A stream is a protocol, not a slow JSON response
Most OpenAI-compatible streaming endpoints use server-sent events over a long-lived HTTP response. The client expects discrete events, correct ordering, a recognizable terminal condition, and timely delivery. Combining chunks, buffering them until the response is large, or changing data boundaries can break an SDK even when the final text looks correct.
Document the exact stream contract your clients consume. Record the content type, event prefix, blank-line delimiters, incremental fields, tool-call deltas, finish event, usage event, and termination marker. The examples in the unified AI API documentation are a starting point; production acceptance tests should use the SDKs and request shapes your users actually run.
Prevent buffering at every hop
A stream can be buffered by the application server, reverse proxy, CDN, compression middleware, or client library. Disable proxy response buffering for the API hostname, flush events promptly, and use timeouts that allow long generations without leaving abandoned connections forever. Test the public hostname through Cloudflare and OpenResty rather than assuming a successful direct-container request proves the edge path.
Do not enable blanket caching on API responses. Static website assets can benefit from a CDN, but authenticated streaming responses must remain dynamic. Compression also needs testing because some middleware waits for a larger buffer before emitting compressed data, increasing time to first token even though total throughput appears normal.
Normalize common events without erasing semantics
A compatibility layer can normalize common text and tool-call events, but it should not invent certainty where providers differ. Reasoning fields, safety metadata, citations, media events, and tool semantics may not have lossless equivalents. Publish those limitations in the model and capability catalog, and expose provider-native endpoints when translation would discard information the application needs.
Tool calls deserve special tests. Some providers stream a tool name once and arguments over many fragments; others interleave multiple calls. The gateway must preserve call identifiers and ordering so the client can assemble valid arguments and return the tool result to the correct conversation turn.
Make cancellation observable and deterministic
When a browser tab closes or a client cancels a request, propagate cancellation upstream when the provider and transport support it. Continuing an expensive generation after the consumer has disappeared wastes capacity and may still create a supplier charge. At the same time, cancellation is not proof that the upstream did no work, so the usage record must remain open until the final outcome is known.
Log the client disconnect time, upstream cancellation attempt, provider response, delivered units, and reported usage. Distinguish client cancellation from gateway timeout and upstream failure; those categories lead to different refund, retry, and reliability decisions.
Use layered timeouts instead of one global number
Separate connection timeout, time to response headers, time to first event, idle time between events, and maximum total duration. A short connection timeout catches an unavailable provider quickly, while a longer idle timeout allows a reasoning model to pause before emitting the next visible event. One global timeout cannot express both requirements safely.
The client timeout should exceed the gateway's useful failure window so the gateway can return a structured error instead of an unexplained socket close. Proxy timeouts must exceed the gateway's stream policy. Monitor first-event and inter-event latency separately; a healthy average total duration can hide a poor user experience at the beginning of every request.
Retry only before the stream becomes externally visible
Once the client has received content, replaying the request through another provider can duplicate text, tool calls, or charges. Safe automatic retry is usually limited to failures before the first event and only for request types that do not have external side effects. After partial delivery, return a clear terminal error and let the application decide whether to start a new user-visible attempt.
This boundary belongs in the broader API gateway migration checklist. Contract tests should verify not only successful event streams but also the exact behavior before and after the first delivered chunk.
Treat usage settlement as part of the stream
Reserve or validate available balance before opening the upstream request, then finalize the charge from reported usage when the stream ends. If usage is missing because of a disconnect, retain enough attempt data to reconcile later. Do not silently treat an interrupted stream as free or charge a full maximum estimate without a documented correction path.
Store a stable request ID across routing, stream delivery, usage, and customer support. The guide to keys, budgets, rate limits, and reconciliation explains how these records support both customer-facing limits and supplier-invoice checks.
Test the stream like a failure-prone distributed system
Build tests for normal completion, slow first event, long pause, malformed event, tool-call fragments, client cancellation, upstream reset, proxy timeout, provider rate limit, and failure after partial output. Run at least one test with the real public proxy chain and one with a production SDK, because command-line output alone may not reproduce browser or library buffering.
For every scenario, assert the user-visible result, the gateway log classification, the usage record, and whether a retry occurred. Streaming is ready for production only when all four views agree. A fast demo that prints tokens is useful, but it is not a complete operational test.