DocsICP FilingConsole
Documentation
Home Page Alibaba Cloud Model Studio API Reference (Models) Audio Speech translation Real-time audio and video translation (Qwen-Livetranslate-Realtime) Server events

Server events

Updated at:
Copy as MD
Product
My Favorites

Server-side events for the qwen3.5-livetranslate-flash-realtime API.

Reference: Real-time audio and video translation - Qwen

error

Error message returned by the server.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always error.

errorobject

Detailed information about the error.

Properties

typestring

The error type.

codestring

The error code.

messagestring

The error message.

paramstring

The parameter that is related to the error, such as session.modalities.

{
  "event_id": "event_RoUu4T8yExPMI37GKwaOC",
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "code": "invalid_value",
    "message": "Invalid modalities: ['audio']. Supported combinations are: ['text'] and ['audio', 'text'].",
    "param": "session.modalities"
  }
}

session.created

When a client connects, the server returns this event first with the default session configurations.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always session.created.

sessionobject

The session configuration.

Properties

idstring

The unique identifier for the session.

objectstring

The value is always realtime.session.

modelstring

The model in use.

modalitiesarray

The output modality settings for the model.

voicestring

The voice for the audio generated by the model.

input_audio_formatstring

The format of the input audio. Defaults to pcm.

sample_rateinteger

The sample rate of the input audio, in Hz.

output_audio_formatstring

The format of the output audio. Defaults to pcm.

turn_detectionobject

The VAD (Voice Activity Detection) configuration.

translationobject(Optional)

The translation configuration.

Properties

language string (Optional)

The target language for translation.

corpusobject(Optional)

Custom terminology configuration for improving translation accuracy of domain-specific terms.

Properties

corpus.phrasesobject(Optional)

A mapping of source-language terms to target-language translations. For details, see Supported languages.

{
    "event_id": "event_QxBGpjBDmDDQQWDtrqBKB",
    "type": "session.created",
    "session": {
        "id": "sess_OozZ1vtbPt2muDflHODIH",
        "object": "realtime.session",
        "model": "qwen3.5-livetranslate-flash-realtime",
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "Cherry",
        "input_audio_format": "pcm",
        "sample_rate": 16000,
        "output_audio_format": "pcm",
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.2,
            "silence_duration_ms": 1000
        },
        "translation": {
           "language": "en",
           "corpus": {
               "phrases": {
                   "人工智能": "Artificial Intelligence",
                   "机器学习": "Machine Learning"
               }
           }
        }
    }
}

session.updated

Returned after a successful session.update request. If the request fails, an error event is returned instead.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always session.updated.

sessionobject

The session configuration.

Properties

idstring

The unique identifier for the session.

objectstring

The value is always realtime.session.

modelstring

The model in use.

modalitiesarray

The output modality settings for the model.

voicestring

The voice for the audio generated by the model.

sample_rateinteger(Optional)

The sample rate of the input audio, in Hz.

input_audio_formatstring

The format of the input audio. The value is always pcm.

output_audio_formatstring

The format of the output audio. The value is always pcm.

input_audio_transcriptionobject

The input audio transcription configuration. Returned only when the input_audio_transcription.model parameter is configured in the session.

Properties

modelstring

The speech recognition model.

languagestring

The configured speech recognition language.

turn_detectionobject

The VAD (Voice Activity Detection) configuration. This field is not returned in Manual mode (when the client sets this parameter to null in session.update).

Properties

typestring

The VAD type. The value is always server_vad.

thresholdfloat

The VAD detection sensitivity.

prefix_padding_msinteger

The duration of audio (in milliseconds) retained before speech starts, to avoid losing the beginning of speech.

silence_duration_msinteger

The minimum duration of silence (in milliseconds) required after speech ends to determine that speech has concluded.

create_responseboolean

Whether to automatically trigger a translation response after VAD detects that speech has ended.

interrupt_responseboolean

Whether to interrupt the current translation response being generated when VAD detects a new speech turn.

translationobject(Optional)

The translation configuration.

Properties

language string (Optional)

The target language for translation.

corpusobject(Optional)

Custom terminology configuration for improving translation accuracy of domain-specific terms.

Properties

corpus.phrasesobject(Optional)

A mapping of source-language terms to target-language translations.

enable_voice_cloneboolean

Whether to enable voice cloning.

voice_clone_optionsobject

Voice cloning control parameters. Returned only when enable_voice_clone is true.

Properties

frequencystring

The voice cloning frequency.

{
    "event_id": "event_QxBGpjBDmDDQQWDtrqBKB",
    "type": "session.updated",
    "session": {
        "id": "sess_OozZ1vtbPt2muDflHODIH",
        "object": "realtime.session",
        "model": "qwen3.5-livetranslate-flash-realtime",
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "Ethan",
        "sample_rate": 16000,
        "input_audio_format": "pcm",
        "output_audio_format": "pcm",
        "input_audio_transcription": {
            "model": "qwen3-asr-flash-realtime",
            "language": "zh"
        },
        "turn_detection": {
            "type": "server_vad",
            "threshold": 0.2,
            "prefix_padding_ms": 300,
            "silence_duration_ms": 1000,
            "create_response": true,
            "interrupt_response": true
        },
        "translation": {
           "language": "en",
           "corpus": {
               "phrases": {
                   "人工智能": "Artificial Intelligence",
                   "机器学习": "Machine Learning"
               }
           }
        },
        "enable_voice_clone": true,
        "voice_clone_options": {
            "frequency": "once"
        }
    }
}

session.finished

Session is finished and all audio translations are complete.

Sent only after the client sends a session.finish request. The client can then disconnect.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always session.finished.

{
    "event_id": "event_xxx",
    "type": "session.finished"
}

response.created

Returned when the server generates a new model response.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always response.created.

responseobject

The response object.

Properties

id string

The unique identifier for the response.

conversation_id string

The unique identifier for the current session.

object string

The object type. For this event, the value is always realtime.response.

status string

The response status. Valid values:

  • completed
  • failed
  • in_progress
  • incomplete

modalities array

Response modality.

voice string

The voice of the generated audio.

output_audio_format string

The format of the output audio.

output array

This event is currently empty.

{
    "event_id": "event_L8hHVI5jYis6BzAjnPWJh",
    "type": "response.created",
    "response": {
        "id": "resp_P79OOMs8LnrXVpiIHUCKR",
        "object": "realtime.response",
        "conversation_id": "conv_UFClXtYkRkFXrs48y8pmK",
        "status": "in_progress",
        "modalities": [
            "text",
            "audio"
        ],
        "voice": "Cherry",
        "output_audio_format": "pcm16",
        "output": []
    }
}

response.done

Returned after response generation is complete. Contains all output items except raw audio data.

event_id string

The unique identifier for this event.

type string

The event type. The value is always response.done.

response object

The response object.

Properties

id string

The unique identifier for the response.

conversation_id string

The unique identifier for the current session.

object string

The object type. For this event, the value is always realtime.response.

status string

The status of the response.

modalities array

The modality of the response.

voice string

The voice used for the audio generated by the model.

output_audio_format string

The format of the output audio.

output object

The output of the response.

Properties

id string

The unique identifier for the response output.

type string

The type of the output item. The value is currently always message.

object string

The object type of the output item. The value is currently always realtime.item.

status string

The status of the output item.

role string

The role of the output item.

content array

The content of the output item.

Properties

type string

The type of the output content. The value is text for plain text output and audio when the output includes audio.

text string

The text content of the output.

transcript string

The text transcription of the audio content.

usage object

The token consumption information for this response.

{
  "event_id": "event_CNea8oXNipVanSg2VIzkO",
  "type": "response.done",
  "response": {
    "id": "resp_TfhYTqej692vsGA2jNEtH",
    "object": "realtime.response",
    "conversation_id": "conv_ZtyLfKVm8XqLwYRlsuDih",
    "status": "completed",
    "modalities": [
      "text",
      "audio"
    ],
    "voice": "Cherry",
    "output_audio_format": "pcm16",
    "output": [
      {
        "id": "item_MKtkMwN9RtcyE9eJShyWy",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
          {
            "type": "audio",
            "transcript": "Hello? "
          }
        ]
      }
    ],
    "usage": {
      "total_tokens": 56,
      "input_tokens": 47,
      "output_tokens": 9,
      "input_tokens_details": {
        "text_tokens": 20,
        "audio_tokens": 27
      },
      "output_tokens_details": {
        "text_tokens": 2,
        "audio_tokens": 7
      }
    }
  }
}

response.text.text

Returned for text-only output when the model generates text incrementally.

event_idstring

A unique identifier for the event.

typestring

The type of the event. The value is always response.text.text.

textstring

The incremental text that is returned.

response_idstring

The response ID.

item_idstring

A unique identifier for the message item.

output_indexinteger

Currently, the value is always 0.

content_indexinteger

Currently, the value is always 0.

stashstring

Temporary text generated by the model. Concatenate stash with the current text to form an interim result. The system updates text and stash continuously using response.text.text events until it receives a response.text.done event. The final complete text is then available in the text field.

{
    "event_id": "event_B1lIeyOXR7qJMEExbqtTG",
    "type": "response.text.text",
    "response_id": "resp_B1lIdtjF4Noqpn5NOjznj",
    "item_id": "item_B1lIdJsAJlJiFs8ztWpJt",
    "output_index": 0,
    "content_index": 0,
    "text": "How are",
    "stash": " you today?"
}

response.text.done

Returned when text generation finishes for text-only output, or if the response is interrupted, incomplete, or canceled.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always response.text.done.

response_idstring

The unique identifier for the response.

item_idstring

The unique identifier for the message item.

output_indexinteger

The value is currently always 0.

content_indexinteger

The value is currently always 0.

text string

The complete text output from the model.

{
    "event_id": "event_B1lIeE2Nac33zn5V7h2mm",
    "type": "response.text.done",
    "response_id": "resp_B1lIdtjF4Noqpn5NOjznj",
    "item_id": "item_B1lIdJsAJlJiFs8ztWpJt",
    "output_index": 0,
    "content_index": 0,
    "text": "How can I assist you today?"
}

response.audio.delta

Returned when audio output is enabled and the model generates audio incrementally.

event_idstring

A unique identifier for the event.

typestring

The event type. The value is always response.audio.delta.

response_idstring

A unique identifier for the response.

item_idstring

A unique identifier for the message item.

output_index integer

The value is always 0.

content_index integer

The value is always 0.

delta string

The incremental audio data that is output by the model. The data is Base64-encoded.

{
    "event_id": "event_B1osWMZBtrEQbiIwW0qHQ",
    "type": "response.audio.delta",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_OFaPGtzfWCPyGzxnuEX9i",
    "output_index": 0,
    "content_index": 0,
    "delta": "UklGRnoGAABXQVZFZm10IBAAAAAB..."
}

response.audio.done

Returned when audio generation is complete. Also returned if the response is interrupted, incomplete, or canceled. Does not contain complete audio data.

event_idstring

The unique identifier for this event.

typestring

The event type. This is always response.audio.done.

response_idstring

The unique identifier for the response.

item_idstring

The unique identifier for the message item.

output_indexinteger

The value is always 0.

content_indexinteger

The value is always 0.

{
    "event_id": "event_B1osWMWoDRYyITDyNYcBu",
    "type": "response.audio.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_OFaPGtzfWCPyGzxnuEX9i",
    "output_index": 0,
    "content_index": 0
}

input_audio_buffer.speech_started

Returned when the server-side VAD (Voice Activity Detection) detects that the user has started speaking.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always input_audio_buffer.speech_started.

audio_start_msinteger

The time point (in milliseconds) when speech was detected to start, as an offset from the beginning of the audio stream.

item_idstring

The unique identifier for the associated message item.

{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.speech_started",
    "audio_start_ms": 568,
    "item_id": "item_xxx"
}

input_audio_buffer.speech_stopped

Returned when the server-side VAD detects that the user has stopped speaking, indicating the end of the current speech input turn. Because the translation response is generated synchronously with the streaming speech input, the actual translation response may have already started during the speech input process, without needing to wait for this event.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always input_audio_buffer.speech_stopped.

audio_end_msinteger

The time point (in milliseconds) when speech was detected to end, as an offset from the beginning of the audio stream.

item_idstring

The unique identifier for the associated message item.

{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.speech_stopped",
    "audio_end_ms": 3900,
    "item_id": "item_xxx"
}

input_audio_buffer.committed

In Manual mode (turn_detection is null), after the client sends an input_audio_buffer.commit event, the server returns this event as confirmation and automatically starts generating the translation response.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always input_audio_buffer.committed.

{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.committed"
}

input_audio_buffer.cleared

After the client sends an input_audio_buffer.clear event, the server returns this event as confirmation that the uncommitted audio data in the buffer has been cleared.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always input_audio_buffer.cleared.

{
    "event_id": "event_xxx",
    "type": "input_audio_buffer.cleared"
}

conversation.item.created

Returned when a new message item is created in the conversation. This event is triggered in the following scenarios:

  • When the server starts generating a translation response, it creates a corresponding assistant message item (at this point, content is an empty array, and the content is progressively populated with the streaming response).
  • In Manual mode, after the client sends an input_audio_buffer.commit event, the server additionally creates a message item corresponding to the user's input audio (content contains {"type": "input_audio"}).

For the same VAD segment, the server creates separate message items for the speech recognition result and the translation result. The item.id of the speech recognition result matches the previous_item_id in the translation result event. Use these values to associate and display the source text with its translation.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always conversation.item.created.

previous_item_idstring

The unique identifier of the previous message item. For a translation result event, this value matches the item.id of the speech recognition result for the same VAD segment.

itemobject

The message item information.

Properties

idstring

The unique identifier for the message item.

typestring

The value is always message.

objectstring

The value is always realtime.item.

statusstring

The status of the message item.

rolestring

The role of the message. Valid values: assistant or user.

contentarray

The content of the message. When a response is first created, this is an empty array that is progressively populated with the streaming response. For user message items created by commit in Manual mode, it contains {"type": "input_audio"}.

{
    "event_id": "event_xxx",
    "type": "conversation.item.created",
    "previous_item_id": "item_asr_xxx",
    "item": {
        "id": "item_translation_xxx",
        "object": "realtime.item",
        "type": "message",
        "status": "in_progress",
        "role": "assistant",
        "content": []
    }
}

conversation.item.input_audio_transcription.text

When input_audio_transcription.model is configured, the server streams speech recognition results in the original source language.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always conversation.item.input_audio_transcription.text.

item_idstring

The unique identifier for the message item.

content_indexinteger

The value is currently always 0.

textstring

The confirmed recognition text.

stashstring

The recognition text that is pending confirmation. This text may be corrected by subsequent events.

languagestring

The detected source language.

emotionstring

The detected emotion of the recognized audio. Supported emotions are as follows:

  • surprised: Surprised
  • neutral: Neutral
  • happy: Happy
  • sad: Sad
  • disgusted: Disgusted
  • angry: Angry
  • fearful: Fearful
{
    "event_id": "event_xxx",
    "type": "conversation.item.input_audio_transcription.text",
    "item_id": "item_xxx",
    "content_index": 0,
    "text": "",
    "stash": "The weather is really nice today",
    "language": "zh",
    "emotion": "neutral"
}

conversation.item.input_audio_transcription.completed

When input_audio_transcription.model is configured, returns the final recognition result after speech recognition completes.

event_idstring

The unique identifier for this event.

typestring

The event type. This is always conversation.item.input_audio_transcription.completed.

item_idstring

The unique identifier for the message item.

content_indexinteger

This is currently always 0.

transcriptstring

The complete speech recognition result in the original source language.

languagestring

The detected source language.

emotionstring

The detected emotion of the recognized audio. Supported emotions:

  • surprised: surprised
  • neutral: calm/neutral
  • happy: happy
  • sad: sad
  • disgusted: disgusted
  • angry: angry
  • fearful: fearful
{
    "event_id": "event_xxx",
    "type": "conversation.item.input_audio_transcription.completed",
    "item_id": "item_xxx",
    "content_index": 0,
    "transcript": "What a beautiful day! Let's go for a walk in the park together.",
    "language": "en",
    "emotion": ""
}

conversation.item.input_audio_transcription.failed

Returned when audio input was received but speech recognition failed. This event is handled separately from other error events, making it easier for the client to identify the specific associated item.

typestring

The event type. The value is always conversation.item.input_audio_transcription.failed.

item_idstring

The ID of the associated conversation item.

content_indexinteger

The index of the content part that contains the audio.

error.codestring

The error code.

error.messagestring

The error message.

{
    "event_id": "event_xxx",
    "type": "conversation.item.input_audio_transcription.failed",
    "item_id": "item_xxx",
    "content_index": 0,
    "error": {
        "code": "xxx",
        "message": "xxx",
        "param": "xxx"
    }
}

response.audio_transcript.text

Returned for audio output to display real-time translation.

event_idstring

The unique identifier for this event.

typestring

The type of the event. The value is always response.audio_transcript.text.

response_idstring

The unique identifier for the response.

item_idstring

The unique identifier for the message item.

output_index integer

Currently, the value is always 0.

content_index integer

Currently, the value is always 0.

textstring

The confirmed translation text segment.

stashstring

Temporary translation text, concatenated with text to form interim results. The system updates text and stash continuously via response.audio_transcript.text events until a response.audio_transcript.done event is received. Then retrieve the final translation from transcript.

{
  "event_id": "event_xxx",
  "type": "response.audio_transcript.text",
  "response_id": "resp_xxx",
  "item_id": "item_xxx",
  "output_index": 0,
  "content_index": 0,
  "text": "Hello,",
  "stash": " who are you?"
}

response.audio_transcript.done

Returned when audio output is enabled and text generation finishes.

event_idstring

The unique identifier for this event.

typestring

The event type. This is always response.audio_transcript.done.

response_idstring

The unique identifier for the response.

item_idstring

The unique identifier for the message item.

output_index integer

This is currently always 0.

content_index integer

This is currently always 0.

transcriptstring

The complete text.

{
    "event_id": "event_VN4Q4GJugLcc1S23viW8E",
    "type": "response.audio_transcript.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_JvJauNH2CTXb1D9WV6pD4",
    "output_index": 0,
    "content_index": 0,
    "transcript": "How can I assist you today?"
}

response.output_item.added

Returned when a new output item is added during response generation.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always response.output_item.added.

response_idstring

The unique identifier for the response.

output_index integer

The value is currently always 0.

item object

Information about the output item.

Properties

idstring

The unique identifier for the output item.

typestring

The value is always message.

objectstring

The value is always realtime.item.

statusstring

The status of the output item.

rolestring

The role of the message.

contentstring

The content of the message.

{
    "event_id": "event_B4O5yPt3Gjnjy5eYH3plG",
    "type": "response.output_item.added",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "output_index": 0,
    "item": {
        "id": "item_OFaPGtzfWCPyGzxnuEX9i",
        "object": "realtime.item",
        "type": "message",
        "status": "in_progress",
        "role": "assistant",
        "content": []
    }
}

response.output_item.done

Returned when an output item is completed.

event_idstring

The unique identifier for this event.

typestring

The event type. The value is always response.output_item.done.

response_idstring

The unique identifier for the response.

output_indexinteger

The value is currently always 0.

itemobject

Information about the output item.

Properties

idstring

The unique identifier for the output item.

objectstring

The value is always realtime.item.

typestring

The value is always message.

statusstring

The status of the output item.

rolestring

The role of the message sender.

contentstring

The content of the message.

{
    "event_id": "event_XkiwbYTBC9Wcdwy6uYJ2G",
    "type": "response.output_item.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "output_index": 0,
    "item": {
        "id": "item_JvJauNH2CTXb1D9WV6pD4",
        "object": "realtime.item",
        "type": "message",
        "status": "completed",
        "role": "assistant",
        "content": [
            {
                "type": "audio",
                "text": "Hello, I am a large language model developed by Alibaba Cloud. My name is Qwen. How can I help you?"
            }
        ]
    }
}

response.content_part.added

Returned when a new content part is added.

event_id string

The unique ID of the event.

type string

The type of the event. The value is always response.content_part.added.

response_id string

The unique ID of the response.

item_id string

The unique ID of the message item.

output_index integer

The value is always 0.

content_index integer

The value is always 0.

part object

Outputs item information.

Properties

type string

The type of the content part.

text string

The text of the content part.

{
    "event_id": "event_J2UixwYKZsXg7c9YXZetL",
    "type": "response.content_part.added",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_OFaPGtzfWCPyGzxnuEX9i",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "audio",
        "text": ""
    }
}

response.content_part.done

Returned when a content part is completed.

event_idstring

The unique identifier for this event.

typestring

The event type. This is always response.content_part.done.

response_idstring

The unique identifier for the response.

item_idstring

The unique identifier for the message item.

output_index integer

The value is always 0.

content_index integer

The value is always 0.

part object

Information about the content part.

Properties

typestring

The type of the content part.

textstring

The text of the content part.

{
    "event_id": "event_VN4Q4GJugLcc1S23viW8E",
    "type": "response.content_part.done",
    "response_id": "resp_P79OOMs8LnrXVpiIHUCKR",
    "item_id": "item_JvJauNH2CTXb1D9WV6pD4",
    "output_index": 0,
    "content_index": 0,
    "part": {
        "type": "audio",
        "text": "Hello, I am a large language model developed by Alibaba Cloud. My name is Qwen. How can I help you?"
    }
}
Previous：Client eventsNext：Python SDK