DocsICP FilingConsole
官方文档
首页

Qwen-Audio Realtime WebSocket API reference

更新时间：
复制 MD 格式
产品详情
我的收藏

The Qwen-Audio Realtime API provides real-time voice conversation capabilities over the WebSocket protocol. Clients interact with the server by sending and receiving JSON events. The API supports audio input, text input, voice activity detection (VAD), and streaming audio and text output.

User guide: Realtime Audio Chat (Qwen-Audio-Realtime). For detailed descriptions of client events and server events, see Client events and Server events.

Important

Alibaba Cloud Model Studio has released a workspace-specific domain for the China (Beijing) region. The new dedicated domain delivers superior performance and higher stability for inference requests. We recommend migrating from dashscope.aliyuncs.com to {WorkspaceId}.cn-beijing.maas.aliyuncs.com.

Replace {WorkspaceId} with your actual Workspace ID. The existing domain remains fully functional.

Service endpoint

The WebSocket URL is fixed as follows. Specify the model name using the model query parameter (replace <model_name> with the actual model name):

China (Beijing)

wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/realtime?model=<model_name>

Replace {WorkspaceId} (including the curly braces) with your actual workspace ID.

Important

Use the wss:// protocol. Set the Authorization in the request header. Pass the model name in the model URL query parameter.

Request headers

Include the following headers in your request:

Parameter

Type

Required

Description

Authorization

string

Yes

Authentication token in the format Bearer <your_api_key>. Replace <your_api_key> with your actual API key.

user-agent

string

No

Client identifier for server-side request tracking.

X-DashScope-WorkSpace

string

No

Alibaba Cloud Model Studio workspace ID.
Important

Authorization is verified during the WebSocket handshake. If the API key is invalid or missing, the handshake fails with an HTTP 401/403 error.

Key concepts

  • Session: A single WebSocket connection corresponds to one session, which maintains configuration and conversation context.

  • Conversation item: An individual message in a conversation, maintained in sequence.

  • Response: The output generated by a single model inference, containing one or more output items . An output item can be an assistant message or a function call.

  • Function calling: An output item the model produces when it needs the client to run a tool function. After the client runs the tool, it sends the result back in function_call_output and triggers the next inference with response.create.

  • Turn detection: Controls when to trigger model inference.

Interaction modes

The Qwen-Audio Realtime API supports three interaction modes, configured through the session.update event's turn_detection.type parameter:

Mode

turn_detection.type

Description

Use cases

server_vad

server_vad

Server-side VAD detects speech start and end, and automatically triggers inference.

Hands-free conversation, voice assistants

smart_turn

smart_turn

Intelligent turn detection that combines acoustic and semantic analysis to determine turn boundaries, not just voice signals. Non-semantic sounds (such as "uh" and "ah") don't trigger a turn or interrupt playback.

Low-latency natural conversation, high-quality interruption

push-to-talk

null

The client manually submits audio and triggers inference.

Push-to-talk, precise control

Interaction flow

For detailed descriptions of client events and server events, see Client events and Server events.

server_vad mode

The server performs voice activity detection on incoming audio and automatically triggers inference after detecting the end of speech.

How to enable: Set session.update event's turn_detection.type to server_vad.

Complete conversation turn

The following diagram illustrates the typical interaction sequence in server_vad mode:

Server Client Server Client Session initialization Voice input loop [User speaking] loop [User speaking] Response generation loop [Streaming output] loop [Multi-turn interaction] connect session.created session.update session.updated input_audio_buffer.append input_audio_buffer.speech_started input_audio_buffer.append conversation.item.input_audio_transcription.delta input_audio_buffer.speech_stopped conversation.item.input_audio_transcription.completed commit audio buffer input_audio_buffer.committed conversation.item.created response.created response.output_item.added conversation.item.created response.content_part.added response.audio_transcript.delta response.audio.delta response.audio_transcript.done response.audio.done response.content_part.done response.output_item.done response.done

The interaction proceeds as follows:

  1. The client establishes a WebSocket connection, and the server returns a session.created event.

  2. The client sends session.update to configure session parameters, and the server returns session.updated.

  3. The client continuously sends input_audio_buffer.append to stream audio data.

  4. The server detects the start of speech and returns input_audio_buffer.speech_started. It also streams ASR transcription deltas via conversation.item.input_audio_transcription.delta.

  5. The server detects the end of speech and returns input_audio_buffer.speech_stopped, input_audio_buffer.committed, and conversation.item.created.

  6. The server automatically generates a response, streaming text and audio deltas (response.audio_transcript.delta, response.audio.delta), and finally returns response.done.

User barge-in

If VAD detects the user starting to speak while the model is playing back a response, the server cancels the current response (returns response.done with status cancelled), then begins a new round of voice input and response. The following diagram illustrates the user barge-in interaction sequence:

Server Client Server Client Server streaming output loop [Streaming] User barge-in New voice input round loop [User speaking] New inference round response.audio.delta input_audio_buffer.append (user starts speaking) response.done(status=cancelled) input_audio_buffer.speech_started input_audio_buffer.append conversation.item.input_audio_transcription.delta input_audio_buffer.speech_stopped conversation.item.input_audio_transcription.completed commit audio buffer input_audio_buffer.committed conversation.item.created response.created

smart_turn mode

smart_turn mode detects end of speech by combining acoustic perception and semantic understanding, filtering out backchannel responses, background noise, and other non-semantic sounds. Non-semantic sounds are passed through as conversation.item.ambient_audio_transcription.delta events without triggering a conversation turn.

How to enable: Set session.update event's turn_detection.type to smart_turn.

Complete conversation turn

The following diagram illustrates the typical interaction sequence in smart_turn mode:

Server Client Server Client Session initialization loop [User speaking] Invalid speech (may occur 0-N times) loop [User speaking] loop [User speaking] Response generation loop [Streaming output] loop [Multi-turn interaction] connect session.created session.update session.updated input_audio_buffer.append conversation.item.ambient_audio_transcription.delta conversation.item.ambient_audio_transcription.completed input_audio_buffer.append input_audio_buffer.speech_started input_audio_buffer.append conversation.item.input_audio_transcription.delta input_audio_buffer.speech_stopped conversation.item.input_audio_transcription.completed commit audio buffer input_audio_buffer.committed conversation.item.created response.created response.output_item.added conversation.item.created response.content_part.added response.audio_transcript.delta response.audio.delta response.audio_transcript.done response.audio.done response.content_part.done response.output_item.done response.done

Key differences from server_vad mode:

  • Non-semantic sounds ("uh", "ah", etc.) don't trigger inference. Instead, they're returned through ambient_audio_transcription events.

  • Previously validated speech may be revoked (input_audio_buffer.speech_stopped returns reason=turn_invalid), in which case inference isn't triggered.

  • While waiting for the user's next input, the client can explicitly send response.create to trigger inference.

User barge-in

Barge-in handling is largely the same as in server_vad mode. The following diagram illustrates the user barge-in interaction sequence:

Server Client Server Client Server streaming output loop [Streaming] User barge-in New voice input round loop [User speaking] New inference round response.audio.delta input_audio_buffer.append (user starts speaking) response.done(status=cancelled) input_audio_buffer.speech_started input_audio_buffer.append conversation.item.input_audio_transcription.delta conversation.item.input_audio_transcription.completed input_audio_buffer.speech_stopped commit audio buffer input_audio_buffer.committed conversation.item.created response.created

Invalid turn

Previously validated speech may be revoked (input_audio_buffer.speech_stopped returns reason=turn_invalid), in which case inference isn't triggered. The client should continue sending audio and wait for the next valid speech. The following diagram illustrates the invalid turn interaction sequence:

Server Client Server Client loop [User speaking] loop [User speaking] Turn invalid Continue sending audio, wait for next turn input_audio_buffer.append input_audio_buffer.speech_started input_audio_buffer.append conversation.item.input_audio_transcription.delta input_audio_buffer.speech_stopped(reason=turn_invalid)

Speaker enhancement configuration flow

In smart_turn mode, when voiceprint_audio_urls is included in the first session.update, the server asynchronously performs voiceprint registration (loading the target speaker's audio features) and notifies the client of the registration progress through events. A failed voiceprint registration does not block the normal conversation flow.

The voiceprint registration interaction sequence is as follows:

  1. The client sends session.update with voiceprint audio URLs in turn_detection.voiceprint_audio_urls. The server returns session.created.

  2. The server immediately starts voiceprint registration asynchronously and pushes voiceprint_audio_list.in_progress before returning session.updated. This event carries the item_id that uniquely identifies the registration task.

  3. The server returns session.updated to confirm that the session configuration has taken effect.

  4. After registration completes, the server pushes a terminal event (the item_id matches the one in step 2):

    • Registration succeeded: voiceprint_audio_list.completed.

    • Registration failed: voiceprint_audio_list.failed, with a reason field describing the failure (for example, the audio URL could not be downloaded).

Note

voiceprint_audio_urls is only configurable in the first session.update event. The field is ignored in subsequent session.update calls.

push-to-talk mode

The client manually controls audio submission and inference triggering. Use this mode when you need precise control over when audio is submitted and inference begins.

How to enable: Set session.update event's turn_detection to null.

Complete conversation turn

The following diagram illustrates the typical interaction sequence in push-to-talk mode:

Server Client Server Client Session initialization Voice input loop [User speaking] Response generation loop [Streaming output] loop [Multi-turn interaction] connect session.created session.update session.updated input_audio_buffer.append conversation.item.input_audio_transcription.delta input_audio_buffer.commit (user releases button) conversation.item.input_audio_transcription.completed input_audio_buffer.committed conversation.item.created response.create (manually trigger inference) response.created response.output_item.added conversation.item.created response.content_part.added response.audio_transcript.delta response.audio.delta response.audio_transcript.done response.audio.done response.content_part.done response.output_item.done response.done

The interaction proceeds as follows:

  1. The client continuously sends input_audio_buffer.append to stream audio data.

  2. After the user finishes speaking, the client sends input_audio_buffer.commit to commit the buffer.

  3. The client sends response.create to manually trigger inference.

  4. The server generates a response, streaming text and audio.

User barge-in

The client sends response.cancel to cancel the current response, and the server returns response.done (with status cancelled and reason client_cancelled). The following diagram illustrates the user barge-in interaction sequence:

Server Client Server Client Server streaming output loop [Streaming] User barge-in New voice input round loop [User speaking] New inference round response.audio_transcript.delta response.audio.delta response.cancel response.done(status=cancelled, reason=client_cancelled) input_audio_buffer.append conversation.item.input_audio_transcription.delta input_audio_buffer.commit conversation.item.input_audio_transcription.completed input_audio_buffer.committed conversation.item.created response.create response.created

Mode operation constraints

Operation

push-to-talk

server_vad

smart_turn

session.update

All parameters can be changed in IDLE state; some are restricted in non-IDLE state

All parameters can be changed in IDLE state; some are restricted in non-IDLE state

All parameters can be changed in IDLE state; some are restricted in non-IDLE state

input_audio_buffer.append

Allowed

Allowed

Allowed

input_audio_buffer.commit

Allowed

Ignored

Ignored

input_audio_buffer.clear

Allowed

Ignored

Ignored

response.create

Allowed. Audio must be committed first via input_audio_buffer.commit. Not allowed while a response is generating.

Allowed when no response is being generated; not allowed while a response is being generated

Allowed while waiting for the user's next input; not allowed during an active turn (from input_audio_buffer.speech_started to response.done)

response.cancel

Allowed (during inference)

Allowed (during inference)

Allowed (during inference)

conversation.item.create/delete/retrieve

Allowed

Allowed

Allowed
Note

turn_detection and input_audio_format can only be changed before the first audio is sent (IDLE state).

Error handling

Type

Behavior

Examples

Client error (invalid_request_error)

Connection stays open; the client receives an error event

Invalid parameters, disallowed state, duplicate item_id

Server error (server_error)

Connection terminated

LLM connection failure, storage failure
Previous：Real-time voice conversationNext：Client events