Real-time multimodal interaction flow-Alibaba Cloud Model Studio(Model Studio)-阿里云帮助中心

This topic describes the real-time multimodal interaction flow between the server and the client.

VAD mode

To enable VAD mode, set the session.turn_detection parameter of the client event to "server_vad". In this mode, the server performs Voice Activity Detection (VAD) on incoming audio and responds when it detects speech. This mode is the default and is designed for scenarios where the client continuously sends audio to the server.

server_vad

The server sends the input_audio_buffer.speech_started event when it detects the start of speech.
The client can append audio to the buffer at any time by sending an input_audio_buffer.append event.
The server sends the input_audio_buffer.speech_stopped event when it detects the end of speech.
The server commits the input audio buffer by sending an input_audio_buffer.committed event.
The server sends a conversation.item.created event, which contains the user message item created from the audio buffer.

Tool calling flow

In VAD mode, when a server-generated response requires a tool call, the following interaction flow occurs:

After detecting the end of speech and generating a response, the server determines that a tool call is required.
The server sends a response.function_call_arguments.delta event, which contains incremental data for the tool call arguments.
The server sends a response.function_call_arguments.done event to indicate that all tool call arguments have been sent.
The client executes the tool call and obtains the result.
The client sends a conversation.item.create event that contains the tool call result.
The server automatically generates a response based on the tool call result.

Manual mode

To enable Manual mode, set the session.turn_detection parameter of the client event to null. In this mode, the client requests a server response by explicitly sending input_audio_buffer.commit and response.create events. This mode is suitable for push-to-talk scenarios, such as sending voice messages in chat applications.

manual

The client can append audio to the buffer by sending an input_audio_buffer.append event.
The client commits the input audio buffer by sending an input_audio_buffer.commit event. This commit creates a new user message item in the conversation.
The server responds by sending an input_audio_buffer.committed event.
The client sends a response.create event to trigger the model to generate the final response.
The server responds by sending a conversation.item.created event.

Tool calling flow

In Manual mode, when a server-generated response requires a tool call, the following interaction flow occurs:

After the client sends a response.create event, the server generates a response and detects that a tool call is required.
The server sends a response.function_call_arguments.delta event, which contains incremental data for the tool call arguments.
The server sends a response.function_call_arguments.done event to indicate that the transfer of tool call arguments is complete.
The client executes the tool call and obtains the result.
The client sends the tool call result using a conversation.item.create event.
The client sends a response.create event to trigger the model to generate the final response.
The server generates a response based on the tool call result and sends it to the client using response.audio.delta or response.text.delta events.