Real-time multimodal interaction flow

This topic describes the real-time multimodal interaction flow between the server and the client.

VAD mode

To enable VAD mode, set the session.turn_detection.type parameter of the client event to "server_vad". In this mode, the server performs Voice Activity Detection (VAD) on incoming audio and responds when it detects speech. This mode is the default and is designed for scenarios where the client continuously sends audio to the server.

• The server sends the input_audio_buffer.speech_started event when it detects the start of speech.
• The client can append audio to the buffer at any time by sending an input_audio_buffer.append event.
• The server sends the input_audio_buffer.speech_stopped event when it detects the end of speech.
• The server commits the input audio buffer by sending an input_audio_buffer.committed event.
• The server sends a conversation.item.created event, which contains the user message item created from the audio buffer.

Tool calling flow

In VAD mode, when a server-generated response requires a tool call, the following interaction flow occurs:

• After detecting the end of speech and generating a response, the server determines that a tool call is required.
• The server sends a response.function_call_arguments.delta event, which contains incremental data for the tool call arguments.
• The server sends a response.function_call_arguments.done event to indicate that all tool call arguments have been sent.
• The client executes the tool call and obtains the result.
• The client sends a conversation.item.create event that contains the tool call result.
• The client sends a response.create event to trigger a response based on the tool call result.

Manual mode

To enable Manual mode, set the session.turn_detection parameter of the client event to null. In this mode, the client requests a server response by explicitly sending input_audio_buffer.commit and response.create events. This mode is suitable for push-to-talk scenarios, such as sending voice messages in chat applications.

• The client can append audio to the buffer by sending an input_audio_buffer.append event.
• The client commits the input audio buffer by sending an input_audio_buffer.commit event. This commit creates a new user message item in the conversation.
• The server responds by sending an input_audio_buffer.committed event.
• The client sends a response.create event to trigger the model to generate the final response.
• The server responds by sending a conversation.item.created event.

Tool calling flow

In Manual mode, when a server-generated response requires a tool call, the following interaction flow occurs:

• After the client sends a response.create event, the server generates a response and detects that a tool call is required.
• The server sends a response.function_call_arguments.delta event, which contains incremental data for the tool call arguments.
• The server sends a response.function_call_arguments.done event to indicate that the transfer of tool call arguments is complete.
• The client executes the tool call and obtains the result.
• The client sends the tool call result using a conversation.item.create event.
• The client sends a response.create event to trigger the model to generate the final response.
• The server generates a response based on the tool call result and sends it to the client using response.audio.delta or response.text.delta events.

Qwen3.8-Omni-Flash-Realtime MCP

Function Calling and MCP can be configured together in session.tools. They cannot be used with web search (enable_search). Model Studio does not charge an additional MCP tool invocation fee. Model inference remains subject to model pricing.

Tool discovery

1. Send session.update with MCP entries in session.tools.
2. Handle mcp_list_tools.in_progress, which may arrive before session.updated. session.updated confirms acceptance of configuration, not tool availability.
3. The service discovers tools. It sends conversation.item.created with a mcp_list_tools item, followed by mcp_list_tools.completed or mcp_list_tools.failed with the same item_id. The item contains the final filtered tools or an error.

Audio input and response.create can continue during discovery, but only Responses created after discovery completes can use the corresponding tools.

MCP calls and continuation

1. The server creates an mcp_call item through response.output_item.added.
2. Concatenate zero or more response.mcp_call_arguments.delta fragments. Use the complete JSON string in response.mcp_call_arguments.done as authoritative.
3. When approval is required, handle conversation.item.created with item.type="mcp_approval_request". Decide whether to allow the call and reply with conversation.item.create, item.type="mcp_approval_response", the exact approval item.id, and boolean approve. Approval is required by default.
4. Approved calls, or calls configured to require no approval, emit response.mcp_call.in_progress. The service executes the MCP call, then emits response.mcp_call.completed or response.mcp_call.failed. Rejection or approval timeout emits failed without execution.
5. Consume response.output_item.done for the final call item. The parent response.done arrives after every MCP item in that Response reaches a terminal state. Its response.output includes final snapshots: successful calls have completed and output; failed calls have failed and error.
6. To generate a model answer from the MCP results, send one response.create after the parent response.done. Do not attach MCP results: they are already in the conversation. Do not use the manual function_call_output return flow for these server-executed MCP calls.

See client configuration fields, approval replies, and server events and items.

Usage limits

The following default MCP constraints apply to each session:

Item	Default value
MCP servers	8
MCP tools	128
Tool discovery timeout	30 seconds
Approval timeout	60 seconds
Tool execution timeout	30 seconds
Single upstream response size	2 MB
Cumulative tool result size	8 MB
Cumulative tool calls	256