Backpressure

Suppose a file reader produces 100 items per second while the next stage processes 60. As long as that difference persists, 40 more items accumulate each second. A larger buffer can hold them for a while, but it too will fill if the consumer continues at the same rate.

With backpressure, the sender paces production or transmission according to the receiver's available capacity. Data flows downstream from producer to consumer, while signals indicating how much more can be accepted travel upstream. TCP receive windows, waiting on an asynchronous queue, and stream demand address this problem in different units and execution models.

The receiver advertises how much it can accept

In RFC 793, the 1981 TCP specification, a receiver advertises a window stating how many bytes it can currently accept. The sender takes that limit into account even if it has more data ready. Protocols communicated receiver capacity to senders before asynchronous stream libraries appeared.¹

When the receiving TCP stack accepts data, the application may still have an order to process or a file to convert. If the application keeps reading from the socket into an unbounded queue, work can accumulate internally even as it frees the TCP receive buffer. The producer and consumer beyond the socket need their own connection between downstream capacity and upstream waiting.

Networking also distinguishes flow control from congestion control. A receiver advertises its own capacity; the sender's congestion control governs what the network path can carry. RFC 9293, the current TCP base specification, distinguishes the send window from the congestion window.²

In an asynchronous pipeline, where does waiting occur?

When one thread calls functions in sequence, it waits for a call to finish before making the next one. Move work to a queue or a separate Task, and a producer can create more work without waiting for the consumer to finish. The system then needs a place and a limit for data waiting between them.

If the producer awaits every queue put, a full queue prevents it from adding the next item until a slot opens. If instead it keeps creating Tasks to perform each put, the queue can remain small while Tasks waiting outside it accumulate.

The upper arrows show data flow, and the lower path shows waiting or demand signals traveling upstream. Whether through queue waiting or explicit demand, that feedback governs the producer's next action.

A server uses queues between stages as control points

In their 2001 SEDA paper, Welsh, Culler, and Brewer divided a server into processing stages and made the queues between them explicit. Each stage could inspect its backlog and adjust its thread count or processing batch.³

(Original source)

Events pass through the incoming queue to a handler, which sends events to other stages. The controller adjusts resource use and scheduling in this stage.

In §3.3 the authors explain that a queue may reject new items, allowing an application to wait at a full queue or drop events. Placing a queue between stages does not set bounds automatically; the application uses that control point to slow production or shed load.

Separate stages are easier to tune individually, but a slow stage also needs a way to affect the stages ahead of it. The problem seen in socket flow control reappears inside the server.

Demand passed between libraries

The Reactive Streams participants defined interfaces so asynchronous stream libraries could exchange demand and cancellation under common rules. The commit tagged 1.0.0 dates from April 2015, and the official announcement states their goal of standardizing asynchronous stream processing with non-blocking backpressure.⁴

(Original source)

The opening sentence of the official introduction summarizes the participants' goal, while the specification defines the contract for each signal.⁵

A Publisher sends elements and a Subscriber receives them. The Subscriber calls request(n) on its Subscription to say how many more it can accept. Publisher rule 1.1 prevents cumulative delivery from exceeding cumulative demand, and Subscription rule 3.8 adds each new request to the remaining demand.⁶

Suppose the Subscriber calls request(2) and receives one element. One unit of demand remains; a subsequent request(3) brings the remainder to four. This count permits delivery of elements. It is not a rate of four per second, and a producer with no data, or one that ends the stream, can send fewer than requested.

While remaining demand is zero, the Publisher sends no additional data elements. New requests use a positive number; request(0) is invalid, not a pause command. Completion and error are separate signals from data elements, so they may be delivered even when no demand remains.⁷

Java 9 also provides Publisher, Subscriber, Subscription, and Processor through java.util.concurrent.Flow. Its documentation describes these interfaces as corresponding to Reactive Streams. Library authors must match the contracts for demand, cancellation, and termination as well as the method for sending data if their components are to connect.⁸

Ways to express waiting

In a blocking implementation, a thread may wait. In an asynchronous implementation, the runtime can run other work while a coroutine is suspended. Reactive Streams requires non-blocking exchange of control signals when connecting asynchronous components, so a thread need not be held waiting on that exchange.⁹

Counting demand also requires a choice about when an item is received versus when its work is done. Suppose a Subscriber starts a new database task for every element it receives and immediately replenishes demand. It can obey the element-delivery rules while the number of database tasks in progress keeps growing. A stage that limits resource use must decide when completed work permits another request.

A small example with two queue slots

The following runs in Python 3.11 or later. The producer puts values into a queue with maxsize=2, awaiting each put. When the queue fills, await queue.put() suspends the producer coroutine, and the event loop can run other work such as the consumer.¹⁰

import asyncio

async def produce(queue):
    for item in range(6):
        await queue.put(item)
    await queue.put(None)

async def consume(queue):
    values = []
    while True:
        item = await queue.get()
        if item is None:
            return values
        await asyncio.sleep(0.01)
        values.append(item)

async def main():
    queue = asyncio.Queue(maxsize=2)
    async with asyncio.TaskGroup() as group:
        group.create_task(produce(queue))
        consumer = group.create_task(consume(queue))
    assert consumer.result() == list(range(6))
    print(consumer.result())  # [0, 1, 2, 3, 4, 5]

if __name__ == "__main__":
    asyncio.run(main())

The input consists of six values, 0 through 5, and None marks termination. The single consumer processes the earlier values in order, then receives the marker and exits. The parent waits for both tasks on leaving the TaskGroup, so this flow does not use join and task_done to count completed items separately.

If the consumer fails with an ordinary exception, TaskGroup also cancels the producer. To stop both tasks together, use a path that ends their shared lifetime, such as cancelling the parent task. Cancellation of an individual consumer with CancelledError is handled differently from an ordinary child failure.¹¹

This example passes queue waiting through one process. Reactive Streams passes demand through its separate interfaces and signal rules. Each API has its own termination contract for the same underlying coordination problem.

Compare immediate rejection when the queue is full. This Python 3 example uses put_nowait: instead of waiting to admit the second item, it receives QueueFull and reports that the item was not accepted. The final read returns first, the item already in the queue.¹²

import asyncio

queue = asyncio.Queue(maxsize=1)
queue.put_nowait("first")
try:
    queue.put_nowait("second")
except asyncio.QueueFull:
    print("second: not accepted")

print(queue.get_nowait())  # first

Using await put with the same one-slot capacity waits instead. Run this example on Python 3.11 or later. The put_started event establishes when the second insertion starts; while the first item remains in the queue, the second put has not completed.

import asyncio

async def main():
    queue = asyncio.Queue(maxsize=1)
    await queue.put("first")
    put_started = asyncio.Event()

    async def put_second():
        put_started.set()
        await queue.put("second")

    async with asyncio.TaskGroup() as group:
        pending = group.create_task(put_second())
        await put_started.wait()
        print(pending.done())  # False
        first = await queue.get()
        await pending
        print(first, await queue.get())  # first second

if __name__ == "__main__":
    asyncio.run(main())

The first get frees a slot for the producer's second item. All we have done with the first item is save it in a variable; no separate business processing has started. This shows where waiting for a queue slot differs from waiting for the consumer to finish its work.

What happens when the buffer fills?

The queue's two slots bound the number of waiting items. An item that the consumer has already taken for processing and the next item held by the producer sit outside those slots. The consumer's values list is another store for results, holding only six values here. A bound on item count therefore has a different scope from a bound on total process memory in bytes.

Some sources, such as sensor measurements or mouse movements, cannot slow the rate at which input originates. At the stream boundary, an application can briefly buffer, combine values, or drop some of them. The Reactive Streams specification likewise says buffering or dropping may be needed to respect demand bounds when the original source cannot be slowed.¹³

If input continues to exceed processing capacity, finite storage eventually fills. Adding a place to retain data longer and increasing sustainable throughput address different limits. Preserving every input requires slowing production enough or securing enough processing capacity.

In TCP the receiver advertises a window, in SEDA the application controls stage queues, and in reactive streams a Subscriber requests a count of elements. Following both the forward flow of data and the upstream path of constraints makes clear which producer is waiting and why.