Stream your export: 6.3 seconds to first byte becomes 29 milliseconds

The shape is everywhere, and it is the obvious thing to write:

app.MapGet("/ledger", async (CancellationToken ct) =>
    Results.Ok(await ReadAllRowsAsync(ct)));

Read all the rows, hand the list to the framework, let it serialise. It is one line, it is easy to test, and for a few hundred rows it is completely fine. The trouble starts when the export grows and nobody revisits the shape, because that await means the endpoint will not say a single word until the last row is in memory.

I built two endpoints over one source to see what the difference actually is. Two hundred thousand rows, coming from something that fetches five hundred at a time and takes fifteen milliseconds per page — a stand-in for a database cursor, or a paged upstream API:

                       buffered      streamed
time to first byte     ~6,300 ms       ~29 ms
time to last byte      ~6,430 ms     ~6,320 ms
peak managed heap       ~39.4 MB      ~17.0 MB
response body            21.1 MB       21.1 MB

What the numbers say, and what they do not

The first row is the headline: about two hundred times sooner to the first byte of the response body. That is the difference between a caller that can start parsing at once and a caller that stares at an open socket for six seconds.

The second row is the part I want to be honest about. Streaming did not make the work finish much sooner. The source still takes six seconds to drain, and no serialisation strategy changes that floor. What streaming buys here is small: in every run I did, the streamed endpoint finished a bit before the buffered one, because the buffered endpoint only starts serialising after the drain is done, while the streamed one overlaps the two. On a fast local connection that gap is a rounding error next to the six-second drain. It would not stay that way on a slow link, where the buffered response pays its whole transfer time after the drain instead of overlapping it — but that is not what this table measures.

The fourth row matters for the same reason. Identical bodies, to the byte. Streaming is not compression and it does not save bandwidth. It changes when bytes leave and how many of them the server is holding at once, and that is the whole of it.

Which brings me to the third row, the one that is easy to overstate. Peak managed heap went from 39 MB to 17 MB on this run, a bit over twice as good, not the hundredfold you might hope for. It is also the least stable number in the table: rerun the demo and the streamed side can land anywhere from the mid teens to the mid twenties, because it is mostly measuring garbage that has not been swept yet rather than anything fixed. The buffered side does not move like that, for the same reason it is rooted rather than merely allocated. The rows are allocated either way; the total allocated bytes barely moved, at 39.6 MB against 37.8 MB. The difference is not allocation, it is retention: the buffered endpoint holds all two hundred thousand rows alive at once because the list is still rooted, while the streamed one lets each row become garbage the moment it has been written. What holds across every run is the shape, not the exact ratio: a ceiling that stops rising with the size of the export, which is the property worth having when the export doubles.

Returning the enumerable

In a minimal API the change is that you stop awaiting and stop wrapping:

app.MapGet("/ledger", (CancellationToken ct) => ReadRowsAsync(ct));

The handler now returns IAsyncEnumerable<Row>. ASP.NET Core serialises it as a JSON array and writes elements as they arrive; the converter doing the work shows up by name in stack traces as IAsyncEnumerableOfTConverter, which is a handy thing to recognise. There is no Content-Length, because nobody knows the length yet — on HTTP/1.1 the response goes out chunked.

The source side is an ordinary async iterator:

public static async IAsyncEnumerable<Row> ReadRowsAsync(
    [EnumeratorCancellation] CancellationToken ct = default)
{
    for (var i = 0; i < rows; i++)
    {
        ct.ThrowIfCancellationRequested();
        if (i % PageSize == 0) await FetchNextPageAsync(ct);
        yield return Produce(i);
    }
}

The attribute is load-bearing

[EnumeratorCancellation] is the piece people leave off, and leaving it off compiles.

A consumer cancels an async iterator by handing a token to GetAsyncEnumerator, which is what WithCancellation(token) does for you. For that token to reach your method body, the compiler needs to know which parameter it should be substituted into. The attribute is how you say so. Without it the generated enumerator quietly ignores the token, and your ct parameter keeps whatever was passed at the call site — usually default.

Two iterators identical apart from the attribute, enumerated through the same WithCancellation call with the same token, cancelled after 200 ms:

with [EnumeratorCancellation]   stopped at ~6,000 rows after ~207 ms
without the attribute           ran to completion: 50,000 rows in ~1,550 ms

The compiler is not silent about this, which I had wrong until I built the thing: it raises CS8425, saying the cancellation token parameter will be unconsumed. That is a warning. It compiles, it ships, and it sits in a build log with the other warnings unless your project turns warnings into errors. Worth a grep through your solution, and worth knowing the number.

In an endpoint you often do not need the attribute at all, because the request token flows in as a parameter and you can pass it down yourself. It becomes load-bearing the moment somebody consumes your iterator with WithCancellation, which is exactly the moment you will not be watching.

The failure that arrives as a 200

This is the cost that changed how I think about streaming endpoints.

An exception thrown while producing row 100 is an ordinary 500. I measured it: status 500, Content-Length: 0, empty body. Nothing had been flushed yet, so the framework could still throw away the half-built response and write an error instead.

An exception thrown while producing row 50,000 is a different animal:

status            200 OK
transfer-encoding chunked
content-length    (none)
reading the body  threw HttpRequestException: Error while copying content to a stream

The status line went out with the first chunk, thousands of rows ago. There is no way to take it back. The server logs its exception and drops the connection; the caller has a 200, a partial JSON array, and a broken read. A client that retries on non-2xx will not retry this. A client that catches the read error and moves on will silently process a truncated export.

The boundary is the first flush, not the first row, which is why this is so easy to miss in testing. In this default path, a small response never gets flushed before the handler finishes, so a small test streams and fails exactly like a buffered endpoint, and everything looks fine until production has enough rows to cross the threshold.

If a truncated answer would be dangerous for your callers, the honest options are to give them a framing that can express failure — newline-delimited JSON with a terminator record, or a trailer — or to accept the buffering and keep the clean 500.

The other things that bite

Streaming holds resources for the whole response. With EF Core that means a DbContext and a database connection stay open for as long as the client takes to read, and a slow client now holds a connection from your pool. Buffered endpoints release that early and pay in memory instead. That is a real trade and it goes the other way under connection pressure. And if the query is tracked, the memory ceiling this post is about never shows up at all: the change tracker keeps every entity alive for the life of the DbContext, so a streamed export needs AsNoTracking() to get the retention win, not just the IAsyncEnumerable return type.

Anything between you and the caller can undo it. A reverse proxy configured to buffer responses will collect the whole body before forwarding, and your twenty-nine milliseconds become six seconds again with no change on your side. So will a client that reads to the end before parsing. Even curl buffers unless you pass -N, which is a good five-second check that the thing you built is actually streaming.

And no Content-Length means no progress bar and no early size rejection. Sometimes that matters to the people calling you.

Where I would use it

Exports, reports, log tails, anything where the row count is decided by data rather than by a page size, and anything where the caller can do something useful with the first rows. Those are the cases where a ceiling on memory and an immediate first byte are worth the sharper failure mode.

For a paged list endpoint returning fifty rows, none of this applies. Buffer it, keep the clean error handling, and spend the attention somewhere it changes a number.

The demo is at github.com/gdhami-net/streaming-api-demo. One command prints the table above, another shows the cancellation difference, and a third reproduces both halves of the mid-stream failure — including the one that comes back as a 200.