Python SDK-Alibaba Cloud Model Studio(Model Studio)-阿里云帮助中心

The key interfaces and request parameters for Qwen-Omni real-time using the DashScope Python SDK.

Prerequisites

Your SDK version must be 1.25.17 or later. Before you start, see Real-time multimodal interaction flow.

Getting started

Download the sample code from GitHub. Three calling methods are available:

Audio conversation example: Captures real-time audio input from the microphone, enables VAD mode to automatically detect the start and end of speech, and supports voice interruption.

enable_turn_detection parameter must be set to True.

We recommend that you use headphones for audio playback to prevent echoes from triggering voice interruption.
Audio and video conversation example: Captures real-time audio and video input from the microphone and camera, enables VAD mode to automatically detect the start and end of speech, and supports voice interruption.

enable_turn_detection parameter must be set to True.

We recommend that you use headphones for audio playback to prevent echoes from triggering voice interruption.
Local call: Uses local audio and images as input and enables Manual mode, which lets you manually control the sending pace.

enable_turn_detection parameter must be set to False.

Request parameters

Set the following request parameters in the constructor (__init__) of the OmniRealtimeConversation class.

Parameter	Type	Description
model	str	The Qwen-Omni model to use. See Model list.
callback	`OmniRealtimeCallback`	Callback instance that handles server-side events.
url	str	The call address: Singapore region: wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime. Beijing region: wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/realtime Replace `{WorkspaceId}` with your actual workspace ID.

Important

Alibaba Cloud Model Studio has released workspace-specific domains for the China (Beijing) and Singapore regions. The new dedicated domains deliver superior performance and higher stability for inference requests. We recommend migrating to the new domains:

China (Beijing): from wss://dashscope.aliyuncs.com to wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com
Singapore: from wss://dashscope-intl.aliyuncs.com to wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com

{WorkspaceId} is your workspace ID, which can be found on the Workspace Details page in the Alibaba Cloud Model Studio console. The existing domain remains fully functional.

Configure the following request parameters with the update_session method.

Parameter	Type	Description
output_modalities	list[MultiModality]	The model output modality. Set to [MultiModality.TEXT] for text only, or [MultiModality.TEXT, MultiModality.AUDIO] for both audio and text.
voice	str	The voice for audio output. For supported voices, see Voice list. Default voices: `Qwen3.5-Omni`series model : `"Tina"` `Qwen3-Omni-Flash-Realtime`series model: `"Cherry"` `Qwen-Omni-Turbo-Realtime`series model: `"Chelsie"`
input_audio_format	AudioFormat	The format of the user's input audio. Currently only supports `PCM_16000HZ_MONO_16BIT`, which represents a PCM audio stream at a 16 kHz sample rate.
output_audio_format	AudioFormat	The format of the model's output audio. Currently only supports `PCM_24000HZ_MONO_16BIT`, which represents a PCM audio stream at a 24 kHz sample rate.
smooth_output	bool	This parameter is supported only by the Qwen3-Omni-Flash-Realtime series. True: Get a conversational response. False: Get a more formal, written-style response. However, this may result in poor quality if the content is difficult to read aloud. None: The model automatically selects a conversational or formal response style.
instructions	str	A system message that sets the model's objective or role. For example: You are an AI agent for a five-star hotel. Answer customer questions about room types, facilities, prices, and booking policies. Be accurate and friendly. Always respond with a professional and helpful attitude. Do not provide unverified information or information outside the hotel's scope of services.
enable_input_audio_transcription	bool	Enables speech recognition for input audio.
input_audio_transcription_model	str	Speech recognition model for input audio transcription. The value is always qwen3-asr-flash-realtime. This parameter is not configurable.
turn_detection_type	str	The Voice Activity Detection (VAD) type. Valid values: `server_vad` (default): Detects the end of user speech based on acoustic features. `semantic_vad`: Detects the end of user speech based on semantic validity. This mode can filter out meaningless speech, such as filler words and background noise. This mode is supported only by the Qwen3.5-Omni-Realtime series model.
turn_detection_threshold	float	VAD detection threshold. Increase in noisy environments and decrease in quiet environments. The closer the value is to -1, the more likely noise is to be detected as speech. The closer the value is to 1, the less likely noise is to be detected as speech. Default: 0.2. Valid values: [-1.0, 1.0].
turn_detection_silence_duration_ms	int	Silence duration that indicates end of speech. If exceeded, the model triggers a response. Default: 800. Valid values: [200, 6000].
turn_detection_param	dict	Additional `turn_detection` configuration parameters. Currently supports `idle_timeout_ms` (int): the idle timeout in milliseconds. Applies only to `qwen3.5-omni-plus-realtime` and `qwen3.5-omni-flash-realtime` models in `server_vad` mode. After the server finishes audio playback and the user remains silent beyond this duration (no `speech.started` triggered), the model proactively generates a response to prompt the user to continue the conversation. Valid range: [5000, 30000]. Example: `turn_detection_param={'idle_timeout_ms': 5000}`
enable_search	bool	This parameter takes effect only when you use the Qwen3.5-Omni-Realtime series model. Enables web search. Default: `false`. When enabled, the model can search the web to answer real-time questions. Tool calling (tools) and web search (enable_search) are incompatible. You cannot enable both at the same time.
search_options	object	Web search options. Takes effect only when `enable_search` is enabled. Currently, you can only set `enable_source` (Boolean), which controls whether to return search result sources. Set to `true` to enable. Example: `search_options={'enable_source': True}`.
tools	list[dict]	This parameter takes effect only when you use the Qwen3.5-Omni-Realtime series model. Tool definitions. When provided, the model can call external tools to respond to user questions. If a tool is called, the model does not generate audio and only returns the tool calling parameters. Each tool is a dictionary that contains the following fields: `type` (`string`, required): Must be set to `"function"`. `function` (`dict`, required): The definition of the tool function. It contains the following fields: `name` (`string`, required): The custom name of the tool function. We recommend that you use a name that is the same as the function name, such as `get_current_weather` or `get_current_time`. `description` (`string`, optional): A description of the tool function. The model uses this description to decide whether to call the function. `parameters` (`dict`, optional): Input parameter descriptions. The model uses them to extract parameters. If the tool function does not require input parameters, you do not need to specify this field. It contains the following fields: `type` (`string`, required): Must be set to `"object"`. `properties` (`dict`, optional): Describes the name, data type, and description of each input parameter. The key is the parameter name, and the value is a dictionary that contains the data type (`type`) and description (`description`). `required` (`list`, optional): Specifies which input parameters are required.
temperature	float	Sampling temperature that controls content diversity. Higher values produce more diverse content; lower values produce more deterministic content. Valid values: [0, 2). Because both temperature and top_p control content diversity, we recommend that you set only one of them. Qwen3.5-Omni-Realtime series model:0.7 `qwen3-omni-flash-realtime` series model:0.9 `qwen-omni-turbo-realtime` series model:1.0 `qwen-omni-turbo` models do not support modification.
top_p	float	Probability threshold for nucleus sampling that controls content diversity. Higher values produce more diverse content; lower values produce more deterministic content. Valid values: (0, 1.0]. Because both temperature and top_p control content diversity, we recommend that you set only one of them. Default top_p values: Qwen3.5-Omni-Realtime series model:0.8 `qwen3-omni-flash-realtime` series model:1.0 `qwen-omni-turbo-realtime` series model:0.01 `qwen-omni-turbo` models do not support modification.
top_k	integer	Candidate set size for sampling. For example, 50 means only the 50 highest-scoring tokens form the candidate set. Larger values increase randomness; smaller values increase determinism. Set to None or a value greater than 100 to disable top_k and use only top_p. The value must be greater than or equal to 0. Default top_k values: Qwen3.5-Omni-Realtime series model:20 `qwen3-omni-flash-realtime` series model:50 `qwen-omni-turbo-realtime` series model:20 `qwen-omni-turbo` models do not support modification.
max_tokens	integer	Maximum number of tokens to return. `max_tokens` setting does not affect the generation process of the LLM. If the number of tokens generated by the model exceeds `max_tokens`, the returned content is truncated. Default and maximum values equal the model's maximum output length. See the Model Studio console for details. Use max_tokens to limit output length for generating summaries, keywords, controlling costs, or reducing response time. `qwen-omni-turbo` models do not support modification.
repetition_penalty	float	Controls repetition in generated sequences. Higher values reduce repetition. 1.0 means no penalty. Must be greater than 0. Default repetition_penalty values: Qwen3.5-Omni-Realtime series model:1.0 Other models: 1.05 `qwen-omni-turbo` models do not support modification.
presence_penalty	float	Controls content repetition in model output. Valid values: [-2.0, 2.0]. Positive values reduce repetition; negative values increase it. Default presence_penalty values: Qwen3.5-Omni-Realtime series model:1.5 Other models: 0.0 Scenarios: Higher values suit scenarios requiring diversity or creativity, such as creative writing or brainstorming. Lower values suit scenarios requiring consistency or technical precision, such as technical documents. `qwen-omni-turbo` models do not support modification.
seed	integer	Makes generation more deterministic, ensuring consistent results across runs. Passing the same seed with identical parameters produces the same result as much as possible. Valid values: 0 to 2³¹−1. Default value: -1. `qwen-omni-turbo` models do not support modification.

Key interfaces

OmniRealtimeConversation class

Import with from dashscope.audio.qwen_omni import OmniRealtimeConversation.

Method signature

Server-side response event (delivered via callback)

Description

def connect(self,) -> None

Server-side event

Session created

session.updated

Session configuration updated

Creates a connection with the server.

def update_session(self,
                       output_modalities: list[MultiModality],
                       voice: str,
                       input_audio_format: AudioFormat = AudioFormat.
                       PCM_16000HZ_MONO_16BIT,
                       output_audio_format: AudioFormat = AudioFormat.
                       PCM_24000HZ_MONO_16BIT,
                       enable_input_audio_transcription: bool = True,
                       input_audio_transcription_model: str = None,
                       enable_turn_detection: bool = True,
                       turn_detection_type: str = 'server_vad',
                       prefix_padding_ms: int = 300,
                       turn_detection_threshold: float = 0.2,
                       turn_detection_silence_duration_ms: int = 800,
                       turn_detection_param: dict = None,
                       **kwargs) -> None

session.updated

Session configuration updated

Updates session configuration. For parameter details, see Request parameters.

After connecting, the server returns default session configurations. Call this method immediately after connecting to update settings.

When the server receives the session.update event, it validates the parameters. Invalid parameters return an error; otherwise the server updates session configuration.

def append_audio(self, audio_b64: str) -> None

None

Appends Base64-encoded audio to the cloud input buffer. The buffer is temporary storage that can be written to and committed later.

If "turn_detection" is enabled, the audio buffer is used for voice detection, and the server decides when to commit.
If "turn_detection" is disabled, the client can send up to 15 MiB of audio per event. Streaming smaller blocks makes VAD more responsive.

def append_video(self, video_b64: str) -> None

None

Adds Base64-encoded image data to the cloud video buffer. Accepts local images or real-time video stream captures.

Image input limits:

The image format must be JPG or JPEG. The recommended image resolution is 480p or 720p, with a maximum of 1080p.
A single image after Base64 encoding must not exceed 256 KB. We recommend keeping the raw image size below 190 KB before encoding.
The image data must be Base64-encoded.
We recommend that you send images to the server at a frequency of 1 image per second.

def clear_appended_audio(self, ) -> None

input_audio_buffer.cleared

Deletes the audio received by the server

Deletes the audio from the current cloud buffer.

def commit(self, ) -> None

input_audio_buffer.committed

Server received the committed audio

Commits audio and video from the cloud buffer. Returns an error if the buffer is empty.

If "turn_detection" is enabled, the client does not need to send this event. The server automatically commits the audio buffer.
If "turn_detection" is disabled, the client must commit the audio buffer to create a user message item.

Note:

If audio transcription is configured for the session using input_audio_transcription, the system transcribes the audio.
Committing the input audio buffer does not create a response from the model.

def create_response(self,
        instructions: str = None,
        output_modalities: list[MultiModality] = None) -> None

Server-side event

Server starts generating a response

response.output_item.added

New output content is available in the response

Server-side event

Conversation item created

response.content_part.added

New output content added to the assistant message item

response.audio_transcript.delta

Incrementally generated transcribed text

response.audio.delta

Incrementally generated audio from the model

response.audio_transcript.done

Text transcription completed

response.audio.done

Audio generation completed

response.content_part.done

Streaming of text or audio content for the assistant message is complete

response.output_item.done

Streaming of the entire output item for the assistant message is complete

response.done

Response completed

Instructs the server to create a model response.

When you configure a session in "turn_detection" mode, the server automatically creates a model response.

def cancel_response(self, ) -> None

None

Cancels the in-progress response. If there is no response to cancel, the server responds with an error.

def create_item(self, item: dict) -> None

None

Sends a conversation.item.create event to the server. In tool calling scenarios, use this method to send the tool execution result back to the server.

The item parameter is a dictionary that must contain the following fields:

type: Must be "function_call_output".
call_id: The call_id from the response.function_call_arguments.done event.
output: A string that contains the tool execution result.

def close(self, ) -> None

None

Terminates the task and closes the connection.

def get_session_id(self) -> str

None

Gets the session_id of the current task.

def get_last_response_id(self) -> str

None

Gets the response_id of the last response.

Callback interface (OmniRealtimeCallback)

The server returns response events and data via callbacks. Implement callback methods to process server responses.

Import with from dashscope.audio.qwen_omni import OmniRealtimeCallback.

Method	Parameters	Return value	Description
`def on_open(self) -> None`	None	None	Called after the server connection is established.
`def on_event(self, message: str) -> None`	message: A server-side response event.	None	Contains interface call responses and model-generated text and audio. See Server-side events.
`def on_close(self, close_status_code, close_msg) -> None`	close_status_code: The status code for closing the WebSocket. close_msg: The closing message for the WebSocket.	None	Called after the server closes the connection.

FAQ

Q: How do I align input audio and images?

The Qwen-Omni real-time model uses the audio stream as the input timeline. Images are inserted into the audio stream based on the time they are sent. You can add images at any point in the audio timeline.

In real-time interaction scenarios, you can enable or disable video input at any time.

Q: What is the recommended frequency for inputting images and audio?

In real-time interaction scenarios, we recommend sending images at a frame rate of 1 or 2 fps and sending audio in 100 ms packets.

Q: What are the differences between the two modes of the turn_detection switch?

When turn_detection is enabled, it supports two modes: server_vad and semantic_vad:

Enable "turn_detection":
- Input state: The cloud-based VAD detects the end of a sentence in the input audio and immediately triggers Qwen-Omni inference to return the reply text and speech.
- Response state: In this state, you can continue to send audio and video input without interruption while the model is responding. After the response is complete, the state returns to the input state to await the next speech input.
- Interruption: If the user starts speaking while the model is responding, an interruption is triggered. The service immediately stops the current response and switches to the input state.
Disable "turn_detection":
- You must manually determine the end of a round of audio and video input and trigger Qwen-Omni inference to obtain a response using commit and create_response.
- While the model is responding, you must stop sending audio and video input. You can resume input for the next round only after the model has finished responding.
- You must use the cancel_response method to interrupt the model's response.

Note that when turn_detection is enabled, you can still actively trigger a response using commit and create_response, and actively interrupt it using cancel_response.

Q: Why do I need to select another model for input_audio_transcription?

Qwen-Omni real-time is an end-to-end multimodal model. Its text output is a response to the input, not a direct transcription of the input audio. A separate ASR model is required for transcription. Currently, the model is determined by the built-in configuration and is not configurable.