Best practices for streaming response APIs

Background

APIs built on large AI models often use a streaming response to deliver services. A streaming response allows the system to transmit data in real time as it processes a request. Users can begin receiving results while the model is still generating the full output, which reduces wait times and improves interactivity. This approach also enhances the user experience, especially in applications that require real-time feedback. For example, in conversational systems or online customer service, users can see parts of the model's response instantly, making the interaction feel more natural and coherent.

Cloud Marketplace also supports monetizing these APIs, providing subscribers with tools to debug and consume them in a streaming manner.

API Gateway: Publishing streaming response APIs

1. When you publish an api group in API Gateway, refer to the documentation for supporting streaming data transfer (SSE). Then, select the Support Streaming Data Transfer option for the api group that contains your streaming API.

2. When you publish an API in API Gateway, set the API Backend Response Timeout as needed. The default is typically 10 seconds. This prevents timeouts when the backend process is slow to respond.

Debugging streaming responses

The API debugging feature on Cloud Marketplace product pages supports streaming response APIs. For API responses that comply with the Server-Sent Events (SSE) protocol, the API debugging tool displays the streaming results in the Call Result tab. The API debugging tool appends new events from the SSE response to the results until the stream is complete.

For example, after you click Send Request for a GET /sse endpoint, the call result shows the incoming SSE events, each prefixed with data:. The content is displayed incrementally (for example, data: Hel → data: Hello → data: Hello, how can I help).

Best practices

1. Adhere to Server-Sent Events

The API output must comply with the Server-Sent Events (SSE) protocol. The HTTP response must use Content-Type: text/event-stream as the content type. An SSE stream consists of a series of lines that start with data:. Each message is terminated by two newline characters. A message can include multiple data lines and optional fields, such as id (the event ID) and event (the event type). The following is an example of a simple SSE-compliant response:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

id: 1
event: message
data: This is the first message
data: This is a continuation of the same message

id: 2
event: message
data: This is the second message

2. Return cumulative results

In applications using a large AI model, answers are generated progressively as the model produces each token. When sending this output as SSE events, each new message should include the content of all previous messages. This cumulative format helps the client parse messages and reconstruct the full response.