Audio and Video File Translation – Qwen

Updated at:

qwen3-livetranslate-flash translates audio and video files across 18 languages. It accepts audio or video input and returns translated text, synthesized audio, or both via a streaming API. For video input, visual context improves translation accuracy (e.g., distinguishing "medical mask" vs. "masquerade mask" based on video frames).

How it works

  1. Set the languages: Refer to Supported languages and set the source language (source_lang) and target language (target_lang) in the translation_options parameter.

    Omitting source_lang enables automatic language detection, but specifying the language improves translation accuracy.

  2. Input the file to translate: The messages array must contain exactly one message with role set to user, and the content field must contain the URL or Base64 data of the audio or video file to translate.

  3. Control the output modality:Use the modalities parameter to control the output modality:

    • ["text"]: text output only;

    • ["text","audio"]: outputs text and Base64-encoded audio.

      If the output includes audio, set the voice (voice) in the audio parameter. Refer to Supported voices.

The following is the core code using the OpenAI Python SDK:

# 导入依赖与创建客户端...
completion = client.chat.completions.create(
    model="qwen3-livetranslate-flash",    # 选择模型
    # messages 仅包含一条 user 消息,content 为待翻译文件
    messages = [
    {
      "role": "user",
      "content": [
        {
          "type": "input_audio",
          "input_audio": {
            "data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
            "format": "wav"
          }
        }
      ]
    }
  ],
    # translation_options 非 OpenAI 标准参数,需通过 extra_body 传入
    extra_body={"translation_options": {"source_lang": "zh", "target_lang": "en"}},
    # 示例:输出文本和音频
    modalities = ["text","audio"],
    audio={"voice": "Cherry", "format": "wav"},
    stream=True
)
Usage restrictions
  • Single-turn translation: The model is designed for translation tasks and does not support multi-turn conversations.
  • No system message: Setting global behavior through the system role is not supported.
  • Invocation method:Only streaming output compatible with the OpenAI protocol is supported.
Response parsing

chunk is the streaming response object:

  • Text: read chunk.choices[0].delta.content.
  • Audio: read chunk.choices[0].delta.audio["data"]. The model outputs audio at a 24 kHz sample rate.

Supported models

Model nameVersionContext lengthMax inputMax output
(tokens)
qwen3-livetranslate-flash

Currently equivalent to qwen3-livetranslate-flash-2025-12-01

Stable version

53,248

49,152

4,096

qwen3-livetranslate-flash-2025-12-01

Snapshot version

How to use

The qwen3-livetranslate-flash model supports audio or video input and outputs text and audio.

Make sure you have obtained and configured an API key and set the API key as an environment variable. If you call through the OpenAI SDK, install the SDK.

The following examples use audio input. To input a video file, uncomment the corresponding code.

import os
from openai import OpenAI

client = OpenAI(
    # 新加坡和北京地域的API Key不同。获取API Key:https://help.aliyun.com/zh/model-studio/get-api-key
    # 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:api_key="sk-xxx",
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    # 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
    base_url="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
)

# ----------------音频输入 ----------------
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "input_audio",
                "input_audio": {
                    "data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
                    "format": "wav",
                },
            }
        ],
    }
]

# ----------------视频输入(需取消注释)----------------
# messages = [
#     {
#         "role": "user",
#         "content": [
#             {
#                 "type": "video_url",
#                 "video_url": {
#                     "url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241115/cqqkru/1.mp4"
#                 },
#             }
#         ],
#     },
# ]

completion = client.chat.completions.create(
    model="qwen3-livetranslate-flash",
    messages=messages,
    modalities=["text", "audio"],
    audio={"voice": "Cherry", "format": "wav"},
    stream=True,
    stream_options={"include_usage": True},
    extra_body={"translation_options": {"source_lang": "zh", "target_lang": "en"}},
)

for chunk in completion:
    print(chunk)
import OpenAI from "openai";

const client = new OpenAI({
    // 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:apiKey: "sk-xxx",
    // 新加坡和北京地域的API Key不同。获取API Key:https://help.aliyun.com/zh/model-studio/get-api-key
    apiKey: process.env.DASHSCOPE_API_KEY,
    // 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
    baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
});

// ---------------- 音频输入 ----------------
const messages = [
    {
        role: "user",
        content: [
            {
                type: "input_audio",
                input_audio: {
                    data: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
                    format: "wav",
                },
            },
        ],
    },
];

// ---------------- 视频输入(需取消注释) ----------------
// const messages = [
//     {
//         role: "user",
//         content: [
//             {
//                 type: "video_url",
//                 video_url: {
//                     url: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241115/cqqkru/1.mp4",
//                 },
//             },
//         ],
//     },
// ];

async function main() {
    const completion = await client.chat.completions.create({
        model: "qwen3-livetranslate-flash",
        messages: messages,
        modalities: ["text", "audio"],
        audio: { voice: "Cherry", format: "wav" },
        stream: true,
        stream_options: { include_usage: true },
        translation_options: { source_lang: "zh", target_lang: "en" },
    });

    for await (const chunk of completion) {
        console.log(JSON.stringify(chunk));
    }
}

main();
# ======= 重要提示 =======
# 新加坡和北京地域的API Key不同。获取API Key:https://help.aliyun.com/zh/model-studio/get-api-key
# 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
# === 执行时请删除该注释 ===
curl -X POST https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1/chat/completions \
-H "Authorization: Bearer $DASHSCOPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
    "model": "qwen3-livetranslate-flash",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
                        "format": "wav"
                    }
                }
            ]
        }
    ],
    "modalities": ["text", "audio"],
    "audio": {
        "voice": "Cherry",
        "format": "wav"
    },
    "stream": true,
    "stream_options": {
        "include_usage": true
    },
    "translation_options": {
        "source_lang": "zh",
        "target_lang": "en"
    }
}'

These examples use public file URLs. To use local files, see the audio and video sections in Input a Base64-encoded local file.

Parse Base64 audio data

The model outputs audio in a streaming, Base64-encoded format. You can process the data in one of the following two ways:

  • Concatenate and decode: concatenate all returned Base64 fragments, decode them all at once after generation completes, and save the result as an audio file.
  • Real-time playback: decode each Base64 fragment in real time and play it directly.
# Installation instructions for pyaudio:
# APPLE Mac OS X
#   brew install portaudio
#   pip install pyaudio
# Debian/Ubuntu
#   sudo apt-get install python-pyaudio python3-pyaudio
#   or
#   pip install pyaudio
# CentOS
#   sudo yum install -y portaudio portaudio-devel && pip install pyaudio
# Microsoft Windows
#   python -m pip install pyaudio

import os
from openai import OpenAI
import base64
import numpy as np
import soundfile as sf

# 初始化OpenAI客户端
client = OpenAI(
    # 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:api_key="sk-xxx",
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    # 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
    base_url="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "input_audio",
                "input_audio": {
                    "data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
                    "format": "wav",
                },
            }
        ],
    }
]
completion = client.chat.completions.create(
    model="qwen3-livetranslate-flash",
    messages=messages,
    modalities=["text", "audio"],
    audio={"voice": "Cherry", "format": "wav"},
    stream=True,
    stream_options={"include_usage": True},
    extra_body={"translation_options": {"source_lang": "zh", "target_lang": "en"}},
)

# 方式1: 待生成结束后再进行解码
audio_string = ""
for chunk in completion:
    if chunk.choices:
        if hasattr(chunk.choices[0].delta, "audio"):
            try:
                audio_string += chunk.choices[0].delta.audio["data"]
            except Exception as e:
                print(chunk.choices[0].delta.audio["transcript"])
    else:
        print(chunk.usage)

wav_bytes = base64.b64decode(audio_string)
audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
sf.write("audio_assistant_py.wav", audio_np, samplerate=24000)

# 方式2: 边生成边解码(使用方式2请将方式1的代码进行注释)
# # 初始化 PyAudio
# import pyaudio
# import time
# p = pyaudio.PyAudio()
# # 创建音频流
# stream = p.open(format=pyaudio.paInt16,
#                 channels=1,
#                 rate=24000,
#                 output=True)

# for chunk in completion:
#     if chunk.choices:
#         if hasattr(chunk.choices[0].delta, "audio"):
#             try:
#                 audio_string = chunk.choices[0].delta.audio["data"]
#                 wav_bytes = base64.b64decode(audio_string)
#                 audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
#                 # 直接播放音频数据
#                 stream.write(audio_np.tobytes())
#             except Exception as e:
#                 print(chunk.choices[0].delta.audio["transcript"])

# time.sleep(0.8)
# # 清理资源
# stream.stop_stream()
# stream.close()
# p.terminate()
// 运行前的准备工作:
// Windows/Mac/Linux 通用:
// 1. 确保已安装 Node.js (建议版本 >= 14)
// 2. 运行以下命令安装必要的依赖:
//    npm install openai wav
//
// 如果要使用实时播放功能 (方式2), 还需要:
// Windows:
//    npm install speaker
// Mac:
//    brew install portaudio
//    npm install speaker
// Linux (Ubuntu/Debian):
//    sudo apt-get install libasound2-dev
//    npm install speaker

import OpenAI from "openai";

const client = new OpenAI({
    // 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:apiKey: "sk-xxx",
    apiKey: process.env.DASHSCOPE_API_KEY,
    // 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
    baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
});

// ---------------- 音频输入 ----------------
const messages = [
    {
        role: "user",
        content: [
            {
                type: "input_audio",
                input_audio: {
                    data: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
                    format: "wav",
                },
            },
        ],
    },
];

const completion = await client.chat.completions.create({
    model: "qwen3-livetranslate-flash",
    messages: messages,
    modalities: ["text", "audio"],
    audio: { voice: "Cherry", format: "wav" },
    stream: true,
    stream_options: { include_usage: true },
    translation_options: { source_lang: "zh", target_lang: "en" },
});

// 方式1: 待生成结束后再进行解码
// 需要安装: npm install wav
import { createWriteStream } from 'node:fs';  // node:fs 是 Node.js 内置模块,无需安装
import { Writer } from 'wav';

async function convertAudio(audioString, audioPath) {
    try {
        // 解码Base64字符串为Buffer
        const wavBuffer = Buffer.from(audioString, 'base64');
        // 创建WAV文件写入流
        const writer = new Writer({
            sampleRate: 24000,  // 采样率
            channels: 1,        // 单声道
            bitDepth: 16        // 16位深度
        });
        // 创建输出文件流并建立管道连接
        const outputStream = createWriteStream(audioPath);
        writer.pipe(outputStream);

        // 写入PCM数据并结束写入
        writer.write(wavBuffer);
        writer.end();

        // 使用Promise等待文件写入完成
        await new Promise((resolve, reject) => {
            outputStream.on('finish', resolve);
            outputStream.on('error', reject);
        });

        // 添加额外等待时间确保音频完整
        await new Promise(resolve => setTimeout(resolve, 800));

        console.log(`音频文件已成功保存为 ${audioPath}`);
    } catch (error) {
        console.error('处理过程中发生错误:', error);
    }
}

let audioString = "";
for await (const chunk of completion) {
    if (Array.isArray(chunk.choices) && chunk.choices.length > 0) {
        if (chunk.choices[0].delta.audio) {
            if (chunk.choices[0].delta.audio["data"]) {
                audioString += chunk.choices[0].delta.audio["data"];
            }
        }
    } else {
        console.log(chunk.usage);
    }
}
// 执行转换
convertAudio(audioString, "audio_assistant_mjs.wav");

// 方式2: 边生成边实时播放
// 需要先按照上方系统对应的说明安装必要组件
// import Speaker from 'speaker'; // 引入音频播放库

// // 创建扬声器实例(配置与 WAV 文件参数一致)
// const speaker = new Speaker({
//     sampleRate: 24000,  // 采样率
//     channels: 1,        // 声道数
//     bitDepth: 16,       // 位深
//     signed: true        // 有符号 PCM
// });
// for await (const chunk of completion) {
//     if (Array.isArray(chunk.choices) && chunk.choices.length > 0) {
//         if (chunk.choices[0].delta.audio) {
//             if (chunk.choices[0].delta.audio["data"]) {
//                 const pcmBuffer = Buffer.from(chunk.choices[0].delta.audio.data, 'base64');
//                 // 直接写入扬声器播放
//                 speaker.write(pcmBuffer);
//             }
//         }
//     } else {
//         console.log(chunk.usage);
//     }
// }
// speaker.on('finish', () => console.log('播放完成'));
// speaker.end(); // 根据实际 API 流结束情况调用

Billing

Audio

Each second of input or output audio consumes 12.5 tokens.

Video

Video file tokens are divided into video_tokens (visual) and audio_tokens (audio).

  • video_tokens

    This script calculates video_tokens. The calculation flow is as follows:

    1. Frame sampling: frames are sampled at 2 FPS, with the frame count clamped to the range [4, 128].
    2. Resizing: height and width are adjusted to multiples of 32 and dynamically scaled according to the frame count to fit the total pixel limit.
    3. Token calculation: ⌈frames/2⌉ × (height/32) × (width/32) + 2.
# 使用前安装:pip install opencv-python
import math
import os
import cv2

# 固定参数
FRAME_FACTOR = 2

IMAGE_FACTOR = 32

# 视频帧的长宽比
MAX_RATIO = 200

# 视频帧的像素下限
VIDEO_MIN_PIXELS = 128 * 32 * 32

# 视频帧的像素上限
VIDEO_MAX_PIXELS = 768 * 32 * 32

FPS = 2
# 最少抽取帧数
FPS_MIN_FRAMES = 4

# 最大抽取帧数
FPS_MAX_FRAMES = 128

# 视频输入的最大像素值,
VIDEO_TOTAL_PIXELS = 16384 * 32 * 32

def round_by_factor(number, factor):
    return round(number / factor) * factor

def ceil_by_factor(number, factor):
    return math.ceil(number / factor) * factor

def floor_by_factor(number, factor):
    return math.floor(number / factor) * factor

def get_video(video_path):
    cap = cv2.VideoCapture(video_path)
    frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    video_fps = cap.get(cv2.CAP_PROP_FPS)
    cap.release()
    return frame_height, frame_width, total_frames, video_fps

def smart_nframes(total_frames, video_fps):
    min_frames = ceil_by_factor(FPS_MIN_FRAMES, FRAME_FACTOR)
    max_frames = floor_by_factor(min(FPS_MAX_FRAMES, total_frames), FRAME_FACTOR)
    duration = total_frames / video_fps if video_fps != 0 else 0
    if duration - int(duration) > (1 / FPS):
        total_frames = math.ceil(duration * video_fps)
    else:
        total_frames = math.ceil(int(duration) * video_fps)
    nframes = total_frames / video_fps * FPS
    nframes = int(min(min(max(nframes, min_frames), max_frames), total_frames))
    if not (FRAME_FACTOR <= nframes <= total_frames):
        raise ValueError(f"nframes should in interval [{FRAME_FACTOR}, {total_frames}], but got {nframes}.")
    return nframes

def smart_resize(height, width, nframes, factor=IMAGE_FACTOR):
    min_pixels = VIDEO_MIN_PIXELS
    total_pixels = VIDEO_TOTAL_PIXELS
    max_pixels = max(min(VIDEO_MAX_PIXELS, total_pixels / nframes * FRAME_FACTOR), int(min_pixels * 1.05))
    if max(height, width) / min(height, width) > MAX_RATIO:
        raise ValueError(f"absolute aspect ratio must be smaller than {MAX_RATIO}, got {max(height, width) / min(height, width)}")
    h_bar = max(factor, round_by_factor(height, factor))
    w_bar = max(factor, round_by_factor(width, factor))
    if h_bar * w_bar > max_pixels:
        beta = math.sqrt((height * width) / max_pixels)
        h_bar = floor_by_factor(height / beta, factor)
        w_bar = floor_by_factor(width / beta, factor)
    elif h_bar * w_bar < min_pixels:
        beta = math.sqrt(min_pixels / (height * width))
        h_bar = ceil_by_factor(height * beta, factor)
        w_bar = ceil_by_factor(width * beta, factor)
    return h_bar, w_bar

def video_token_calculate(video_path):
    height, width, total_frames, video_fps = get_video(video_path)
    nframes = smart_nframes(total_frames, video_fps)
    resized_height, resized_width = smart_resize(height, width, nframes)
    video_token = int(math.ceil(nframes / FPS) * resized_height / 32 * resized_width / 32)
    video_token += 2  # 视觉标记
    return video_token

if __name__ == "__main__":
    video_path = "spring_mountain.mp4"  # 你的视频路径
    video_token = video_token_calculate(video_path)
    print("video_tokens:", video_token)
  • audio_tokens

    Each second of audio consumes 12.5 tokens. Audio shorter than 1 second is billed as 1 second.

For token pricing, see Model list.

API reference

For the input and output parameters of the qwen3-livetranslate-flash model, see Audio and video file translation - Qwen.

Supported languages

The language codes in the following table can be used to specify the source and target languages.

Some target languages support text output only and do not support audio output.

Language code

Language

Supported output modalities

en

English

Audio, text

zh

Chinese

Audio, text

ru

Russian

Audio, text

fr

French

Audio, text

de

German

Audio, text

pt

Portuguese

Audio, text

es

Spanish

Audio, text

it

Italian

Audio, text

id

Indonesian

Text

ko

Korean

Audio, text

ja

Japanese

Audio, text

vi

Vietnamese

Text

th

Thai

Text

ar

Arabic

Text

yue

Cantonese

Audio, text

hi

Hindi

Text

el

Greek

Text

tr

Turkish

Text

Supported voices

Voice name

voice parameter

Voice effectDescriptionSupported languages

Cherry

Cherry

A cheerful, approachable, and naturally friendly young woman.

Chinese, English, French, German, Russian, Italian, Spanish, Portuguese, Japanese, Korean

Nofish

Nofish

A designer who cannot pronounce retroflex consonants.

Chinese, English, French, German, Russian, Italian, Spanish, Portuguese, Japanese, Korean

Shanghai-Jada

Jada

A brisk, energetic Shanghai lady.

Chinese

Beijing-Dylan

Dylan

A young man who grew up in the hutongs of Beijing.

Chinese

Sichuan-Sunny

Sunny

A Sichuan girl sweet to the core.

Chinese

Tianjin-Peter

Peter

Tianjin crosstalk style, a professional supporting performer.

Chinese

Cantonese-Kiki

Kiki

A sweet best friend from Hong Kong.

Cantonese

Sichuan-Eric

Eric

An unconventional, street-smart man from Chengdu, Sichuan.

Chinese

FAQ

Q: What content is translated when I input a video file?

A: The model translates the audio content of the video. Visual information is used to assist context understanding and improve accuracy.

Example:

When the audio content is This is a mask:

  • If the frame shows a protective face mask, it is translated as "This is a medical mask";
  • If the frame shows a masquerade mask, it is translated as "This is a masquerade mask".