Skip to content

client: stream call_client_stream request body via channel#84

Merged
iainmcgin merged 1 commit intoanthropics:mainfrom
connyay:client-stream-buffering
May 7, 2026
Merged

client: stream call_client_stream request body via channel#84
iainmcgin merged 1 commit intoanthropics:mainfrom
connyay:client-stream-buffering

Conversation

@connyay
Copy link
Copy Markdown
Contributor

@connyay connyay commented May 6, 2026

Previously call_client_stream concatenated every encoded envelope into a single BytesMut and handed it to full_body(), so the transport never saw the first byte until the entire iterator had been drained. Peak memory was O(total) and a slow- or large-yielding iterator buffered everything in process memory before any of it hit the wire.

Switch to the same channel-backed body pattern call_bidi_stream uses: a depth-32 mpsc channel feeds ChannelBody, the iterator drain pushes one envelope per item with natural backpressure on HTTP/2 flow control, and the response future is driven concurrently so frames flush as they're produced. Peak memory drops to roughly channel_depth * envelope_size.

The transport send is dispatched onto a separate task so its body is polled while we drain the iterator (a transport whose send() future contains the actual I/O would otherwise deadlock once the channel filled). On wasm32 there is no tokio runtime; spawn onto wasm-bindgen-futures' single-threaded executor instead. Either way, the response is bridged back via a oneshot so the awaitee is uniform across architectures.

The IntoIterator<Item = Req> public signature is unchanged, so codegen and existing callers are unaffected. A new regression test in tests/streaming installs a recording ClientTransport into the generated EchoServiceClient and asserts the body arrives as N frames for N messages instead of one concatenated frame; this fails on the previous buffered code (1 frame of 68 bytes) and passes on the new code.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 6, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@iainmcgin iainmcgin enabled auto-merge (squash) May 7, 2026 15:18
@iainmcgin iainmcgin disabled auto-merge May 7, 2026 20:42
…s#84)

Previously call_client_stream concatenated every encoded envelope into a
single BytesMut and handed it to full_body(), so the transport never saw
the first byte until the entire iterator had been drained. Peak memory
was O(total) and a slow- or large-yielding iterator buffered everything
in process memory before any of it hit the wire.

Switch to the same channel-backed body pattern call_bidi_stream uses: a
depth-32 mpsc channel feeds ChannelBody, the iterator drain pushes one
envelope per item with natural backpressure on HTTP/2 flow control, and
the response future is driven concurrently so frames flush as they're
produced. Peak memory drops to roughly channel_depth * envelope_size.

The transport send is dispatched onto a separate task so its body is
polled while we drain the iterator (a transport whose send() future
contains the actual I/O would otherwise deadlock once the channel
filled). On wasm32 there is no tokio runtime; spawn onto
wasm-bindgen-futures' single-threaded executor instead. Either way, the
response is bridged back via a oneshot so the awaitee is uniform across
architectures.

The IntoIterator<Item = Req> public signature is unchanged, so codegen
and existing callers are unaffected. A new regression test in
tests/streaming installs a recording ClientTransport into the generated
EchoServiceClient and asserts the body arrives as N frames for N
messages instead of one concatenated frame; this fails on the previous
buffered code (1 frame of 68 bytes) and passes on the new code.
@iainmcgin iainmcgin force-pushed the client-stream-buffering branch from 0c80c70 to 41e39d1 Compare May 7, 2026 20:48
@iainmcgin iainmcgin enabled auto-merge (squash) May 7, 2026 20:48
@iainmcgin iainmcgin merged commit c2aa616 into anthropics:main May 7, 2026
12 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators May 7, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants