Sam Sartor
Git(Hub|Lab)
# Coroutines 2: Electric Boogaloo 2020-6-26

TODO: short introduction, credit other contributes

Addition 1: Yield

At the heart of this proposal is the idea that generators are really not a new type but just an extra feature of closures. Specifically:

The statement yield x; is allowed in closures. It behaves the same as return x; except the next time the closure is called, execution resumes immediately following the statement instead of at the top of the body.

Notice how on every yield, the closure arguments are reassigned:

|x| {
    let y = x;
    yield; // x gets re-assigned
    dbg!(x, y);
}

In the unified coroutines proposal (RFC 2781) which heavily inspired yield closures, this is called "magic mutation" but that is really a misnomer. Parameter bindings can be moved into new bindings if desired. And if not, they are replaced when the closure is resumed.

Addition 2: FnPin

So that closures can set up self-referential state across yields, we also need a new trait FnPin:

pub trait FnPin<Args>: FnOnce<Args> {
    extern "rust-call" fn call_pin(Pin<&mut self>, args: Args) -> Self::Output;
}

Most of the time, closures will implement both FnMut and FnPin. However, if the closure body contains live borrows across yield points, only an FnPin implementation will be provided.

Addition 3: Mut modifier

Closures are allowed to move captures if they can guarantee that they will never be reentered. Normally this is done by only implementing FnOnce. However, "once" yield closures still need be callable multiple times.

To solve this, there will be some way to force a "once" closure to implement FnPin/FnMut and panic if reentered. The syntax is open to bikeshedding but must not be ambiguous. I propose adding a mut modifier in addition to move:

mut || {
    drop(capture_a);
    yield;
    drop(capture_b);
}

Since mut || .. is nothing but an evil footgun when used without yield, a deny(mut_without_yield) lint will also be added.

Uses

Most coroutine proposals for Rust tend to immediately focus on the proposal's usability (or lack thereof) as an implementation detail of async. But async is already a solved problem in Rust. I think that focusing on it now will only clutter the coroutine design space.

Instead consider how yield can seriously up the power of plain old closures in existing situations:

escaped_text.chars().filter_map(|c| {
    if c != '\\' {
        return Some(c);
    }
    yield None;
    Some(match c {
        'x' => {
            yield None;
            let most = c;
            yield None;
            let least = c;
            char::from_digit(most, 16)? * 16 + char::from_digit(least, 16)?
        },
        'n' => '\n',
        'r' => '\r',
        't' => '\t',
        '0' => '\0',
        '\\' => '\\',
        _ => c,
    })
})

Yield closures apply well to all kinds of pushdown parsers in general. For example, here is a really simple lisp repl:

// Will run until all whitespace chars are consumed.
fn whitespace(c: char) -> Poll<()> {
    match c.is_whitespace() {
        true => Pending,
        false => Ready(()),
    }
}

// Will run until all chars in an expression are consumed.
// Returns the value of the expression.
fn expression() -> impl FnMut(char) -> Poll<Result<u32, Invalid>> {
    |c| {
        poll!(whitespace, c);
        if c == '(' {
            yield Pending; // Consume (
            poll!(whitespace, c); // Consume whitespace

            let (mut value, op): (u32, fn(u32, u32) -> u32) = match c {
                '+' => (0, |a, b| a + b),
                '*' => (1, |a, b| a * b),
                '|' => (0, |a, b| a | b),
                '&' => (!0, |a, b| a & b),
                _ => return Ready(Err(Invalid::Operator)),
            };
            yield Pending; // Consume operator

            if !c.is_whitespace() && c != ')' {
                return Ready(Err(Invalid::Spacing));
            }
            poll!(whitespace, c); // Consume whitespace

            let mut args = expression();
            while c != ')' {
                value = op(value, poll!(args, c)?); // Consume expression
                poll!(whitespace, c); // Consume whitespace
            }

            yield Pending; // Consume )
            Ready(Ok(value))
        } else if c.is_digit() {
            let mut value = 0;
            while c.is_digit() {
                value *= 10;
                value += char::from_digit(c).ok_or(Invalid::Number)?;
                yield Pending; // Consume digit
            }
            Ready(Ok(value))
        } else {
            Ready(Err(Invalid::Expression))
        }
    }
}

// REPL
fn main() -> Result<(), Error> {
    let mut eval = expression();
    for line in stdin().lock()?.lines() {
        for c in line?.chars() {
            if let Ready(x) = eval(c) {
                println!("= {}", x?);
            }
        }
    }
    Ok(())
}

TODO: talk about streams, custom futures, pushdown parsers, argument consumption, async desugar, etc

Extensions

There could be a new .await(...) syntax which polls FnPin::call_pin with the given expressions as arguments. It would be internally very similar to .await, which polls Future::poll with the implicit async context. The key difference being, .await(..) can be used anywhere yield is permitted. This is useful for interacting with arbitrary sorts of poll functions & closures:

move mut |ctx: &mut Context| {
    let mut buffer = [0u8; 4096];
    pin_mut!(read);

    loop {
        let n = AsyncRead::poll_read.await(read.as_mut(), ctx, &mut buffer)?;

        if n == 0 {
            return Ready(None);
        }

        for &byte in buffer.iter().take(n) {
            yield Ready(Some(Ok(byte)));
        }
    }
}

It also isn't clear currently how yield would work inside async closures. The obvious solution is to allow yield as an escape hatch inside async blocks so that the existing desugar continues to make sense:

TODO: yield in async

Comparison

This proposal is effectively identical to RFC 2781 except for the following items:

TODO: talk about generator state wrapping, async closures & yield in async blocks