Audio and Video File Translation – Qwen
qwen3-livetranslate-flash translates audio and video files across 18 languages. It accepts audio or video input and returns translated text, synthesized audio, or both via a streaming API. For video input, visual context improves translation accuracy (e.g., distinguishing "medical mask" vs. "masquerade mask" based on video frames).
How it works
-
Set the languages: Refer to Supported languages and set the source language (
source_lang) and target language (target_lang) in thetranslation_optionsparameter.Omitting
source_langenables automatic language detection, but specifying the language improves translation accuracy. -
Input the file to translate: The
messagesarray must contain exactly one message withroleset touser, and thecontentfield must contain the URL or Base64 data of the audio or video file to translate. -
Control the output modality:Use the
modalitiesparameter to control the output modality:-
["text"]: text output only; -
["text","audio"]: outputs text and Base64-encoded audio.If the output includes audio, set the voice (
voice) in theaudioparameter. Refer to Supported voices.
-
The following is the core code using the OpenAI Python SDK:
# 导入依赖与创建客户端...
completion = client.chat.completions.create(
model="qwen3-livetranslate-flash", # 选择模型
# messages 仅包含一条 user 消息,content 为待翻译文件
messages = [
{
"role": "user",
"content": [
{
"type": "input_audio",
"input_audio": {
"data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
"format": "wav"
}
}
]
}
],
# translation_options 非 OpenAI 标准参数,需通过 extra_body 传入
extra_body={"translation_options": {"source_lang": "zh", "target_lang": "en"}},
# 示例:输出文本和音频
modalities = ["text","audio"],
audio={"voice": "Cherry", "format": "wav"},
stream=True
)
Usage restrictions
- Single-turn translation: The model is designed for translation tasks and does not support multi-turn conversations.
- No system message: Setting global behavior through the
systemrole is not supported. - Invocation method:Only streaming output compatible with the OpenAI protocol is supported.
chunk is the streaming response object:
- Text: read
chunk.choices[0].delta.content. - Audio: read
chunk.choices[0].delta.audio["data"]. The model outputs audio at a 24 kHz sample rate.
Supported models
| Model name | Version | Context length | Max input | Max output |
|---|---|---|---|---|
| (tokens) | ||||
qwen3-livetranslate-flash
| Stable version | 53,248 | 49,152 | 4,096 |
qwen3-livetranslate-flash-2025-12-01 | Snapshot version | |||
How to use
The qwen3-livetranslate-flash model supports audio or video input and outputs text and audio.
Make sure you have obtained and configured an API key and set the API key as an environment variable. If you call through the OpenAI SDK, install the SDK.
The following examples use audio input. To input a video file, uncomment the corresponding code.
import os
from openai import OpenAI
client = OpenAI(
# 新加坡和北京地域的API Key不同。获取API Key:https://help.aliyun.com/zh/model-studio/get-api-key
# 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:api_key="sk-xxx",
api_key=os.getenv("DASHSCOPE_API_KEY"),
# 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
base_url="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
)
# ----------------音频输入 ----------------
messages = [
{
"role": "user",
"content": [
{
"type": "input_audio",
"input_audio": {
"data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
"format": "wav",
},
}
],
}
]
# ----------------视频输入(需取消注释)----------------
# messages = [
# {
# "role": "user",
# "content": [
# {
# "type": "video_url",
# "video_url": {
# "url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241115/cqqkru/1.mp4"
# },
# }
# ],
# },
# ]
completion = client.chat.completions.create(
model="qwen3-livetranslate-flash",
messages=messages,
modalities=["text", "audio"],
audio={"voice": "Cherry", "format": "wav"},
stream=True,
stream_options={"include_usage": True},
extra_body={"translation_options": {"source_lang": "zh", "target_lang": "en"}},
)
for chunk in completion:
print(chunk)
import OpenAI from "openai";
const client = new OpenAI({
// 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:apiKey: "sk-xxx",
// 新加坡和北京地域的API Key不同。获取API Key:https://help.aliyun.com/zh/model-studio/get-api-key
apiKey: process.env.DASHSCOPE_API_KEY,
// 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
});
// ---------------- 音频输入 ----------------
const messages = [
{
role: "user",
content: [
{
type: "input_audio",
input_audio: {
data: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
format: "wav",
},
},
],
},
];
// ---------------- 视频输入(需取消注释) ----------------
// const messages = [
// {
// role: "user",
// content: [
// {
// type: "video_url",
// video_url: {
// url: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241115/cqqkru/1.mp4",
// },
// },
// ],
// },
// ];
async function main() {
const completion = await client.chat.completions.create({
model: "qwen3-livetranslate-flash",
messages: messages,
modalities: ["text", "audio"],
audio: { voice: "Cherry", format: "wav" },
stream: true,
stream_options: { include_usage: true },
translation_options: { source_lang: "zh", target_lang: "en" },
});
for await (const chunk of completion) {
console.log(JSON.stringify(chunk));
}
}
main();
# ======= 重要提示 =======
# 新加坡和北京地域的API Key不同。获取API Key:https://help.aliyun.com/zh/model-studio/get-api-key
# 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
# === 执行时请删除该注释 ===
curl -X POST https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1/chat/completions \
-H "Authorization: Bearer $DASHSCOPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-livetranslate-flash",
"messages": [
{
"role": "user",
"content": [
{
"type": "input_audio",
"input_audio": {
"data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
"format": "wav"
}
}
]
}
],
"modalities": ["text", "audio"],
"audio": {
"voice": "Cherry",
"format": "wav"
},
"stream": true,
"stream_options": {
"include_usage": true
},
"translation_options": {
"source_lang": "zh",
"target_lang": "en"
}
}'
These examples use public file URLs. To use local files, see the audio and video sections in Input a Base64-encoded local file.
Parse Base64 audio data
The model outputs audio in a streaming, Base64-encoded format. You can process the data in one of the following two ways:
- Concatenate and decode: concatenate all returned Base64 fragments, decode them all at once after generation completes, and save the result as an audio file.
- Real-time playback: decode each Base64 fragment in real time and play it directly.
# Installation instructions for pyaudio:
# APPLE Mac OS X
# brew install portaudio
# pip install pyaudio
# Debian/Ubuntu
# sudo apt-get install python-pyaudio python3-pyaudio
# or
# pip install pyaudio
# CentOS
# sudo yum install -y portaudio portaudio-devel && pip install pyaudio
# Microsoft Windows
# python -m pip install pyaudio
import os
from openai import OpenAI
import base64
import numpy as np
import soundfile as sf
# 初始化OpenAI客户端
client = OpenAI(
# 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:api_key="sk-xxx",
api_key=os.getenv("DASHSCOPE_API_KEY"),
# 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
base_url="https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
)
messages = [
{
"role": "user",
"content": [
{
"type": "input_audio",
"input_audio": {
"data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
"format": "wav",
},
}
],
}
]
completion = client.chat.completions.create(
model="qwen3-livetranslate-flash",
messages=messages,
modalities=["text", "audio"],
audio={"voice": "Cherry", "format": "wav"},
stream=True,
stream_options={"include_usage": True},
extra_body={"translation_options": {"source_lang": "zh", "target_lang": "en"}},
)
# 方式1: 待生成结束后再进行解码
audio_string = ""
for chunk in completion:
if chunk.choices:
if hasattr(chunk.choices[0].delta, "audio"):
try:
audio_string += chunk.choices[0].delta.audio["data"]
except Exception as e:
print(chunk.choices[0].delta.audio["transcript"])
else:
print(chunk.usage)
wav_bytes = base64.b64decode(audio_string)
audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
sf.write("audio_assistant_py.wav", audio_np, samplerate=24000)
# 方式2: 边生成边解码(使用方式2请将方式1的代码进行注释)
# # 初始化 PyAudio
# import pyaudio
# import time
# p = pyaudio.PyAudio()
# # 创建音频流
# stream = p.open(format=pyaudio.paInt16,
# channels=1,
# rate=24000,
# output=True)
# for chunk in completion:
# if chunk.choices:
# if hasattr(chunk.choices[0].delta, "audio"):
# try:
# audio_string = chunk.choices[0].delta.audio["data"]
# wav_bytes = base64.b64decode(audio_string)
# audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
# # 直接播放音频数据
# stream.write(audio_np.tobytes())
# except Exception as e:
# print(chunk.choices[0].delta.audio["transcript"])
# time.sleep(0.8)
# # 清理资源
# stream.stop_stream()
# stream.close()
# p.terminate()
// 运行前的准备工作:
// Windows/Mac/Linux 通用:
// 1. 确保已安装 Node.js (建议版本 >= 14)
// 2. 运行以下命令安装必要的依赖:
// npm install openai wav
//
// 如果要使用实时播放功能 (方式2), 还需要:
// Windows:
// npm install speaker
// Mac:
// brew install portaudio
// npm install speaker
// Linux (Ubuntu/Debian):
// sudo apt-get install libasound2-dev
// npm install speaker
import OpenAI from "openai";
const client = new OpenAI({
// 若没有配置环境变量,请用阿里云百炼API Key将下行替换为:apiKey: "sk-xxx",
apiKey: process.env.DASHSCOPE_API_KEY,
// 以下为华北2(北京)地域的URL。请在示例中填写实际的业务空间 ID,各地域的URL不同。
baseURL: "https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/compatible-mode/v1",
});
// ---------------- 音频输入 ----------------
const messages = [
{
role: "user",
content: [
{
type: "input_audio",
input_audio: {
data: "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
format: "wav",
},
},
],
},
];
const completion = await client.chat.completions.create({
model: "qwen3-livetranslate-flash",
messages: messages,
modalities: ["text", "audio"],
audio: { voice: "Cherry", format: "wav" },
stream: true,
stream_options: { include_usage: true },
translation_options: { source_lang: "zh", target_lang: "en" },
});
// 方式1: 待生成结束后再进行解码
// 需要安装: npm install wav
import { createWriteStream } from 'node:fs'; // node:fs 是 Node.js 内置模块,无需安装
import { Writer } from 'wav';
async function convertAudio(audioString, audioPath) {
try {
// 解码Base64字符串为Buffer
const wavBuffer = Buffer.from(audioString, 'base64');
// 创建WAV文件写入流
const writer = new Writer({
sampleRate: 24000, // 采样率
channels: 1, // 单声道
bitDepth: 16 // 16位深度
});
// 创建输出文件流并建立管道连接
const outputStream = createWriteStream(audioPath);
writer.pipe(outputStream);
// 写入PCM数据并结束写入
writer.write(wavBuffer);
writer.end();
// 使用Promise等待文件写入完成
await new Promise((resolve, reject) => {
outputStream.on('finish', resolve);
outputStream.on('error', reject);
});
// 添加额外等待时间确保音频完整
await new Promise(resolve => setTimeout(resolve, 800));
console.log(`音频文件已成功保存为 ${audioPath}`);
} catch (error) {
console.error('处理过程中发生错误:', error);
}
}
let audioString = "";
for await (const chunk of completion) {
if (Array.isArray(chunk.choices) && chunk.choices.length > 0) {
if (chunk.choices[0].delta.audio) {
if (chunk.choices[0].delta.audio["data"]) {
audioString += chunk.choices[0].delta.audio["data"];
}
}
} else {
console.log(chunk.usage);
}
}
// 执行转换
convertAudio(audioString, "audio_assistant_mjs.wav");
// 方式2: 边生成边实时播放
// 需要先按照上方系统对应的说明安装必要组件
// import Speaker from 'speaker'; // 引入音频播放库
// // 创建扬声器实例(配置与 WAV 文件参数一致)
// const speaker = new Speaker({
// sampleRate: 24000, // 采样率
// channels: 1, // 声道数
// bitDepth: 16, // 位深
// signed: true // 有符号 PCM
// });
// for await (const chunk of completion) {
// if (Array.isArray(chunk.choices) && chunk.choices.length > 0) {
// if (chunk.choices[0].delta.audio) {
// if (chunk.choices[0].delta.audio["data"]) {
// const pcmBuffer = Buffer.from(chunk.choices[0].delta.audio.data, 'base64');
// // 直接写入扬声器播放
// speaker.write(pcmBuffer);
// }
// }
// } else {
// console.log(chunk.usage);
// }
// }
// speaker.on('finish', () => console.log('播放完成'));
// speaker.end(); // 根据实际 API 流结束情况调用
Billing
Audio
Each second of input or output audio consumes 12.5 tokens.
Video
Video file tokens are divided into video_tokens (visual) and audio_tokens (audio).
-
video_tokensThis script calculates
video_tokens. The calculation flow is as follows:- Frame sampling: frames are sampled at 2 FPS, with the frame count clamped to the range [4, 128].
- Resizing: height and width are adjusted to multiples of 32 and dynamically scaled according to the frame count to fit the total pixel limit.
- Token calculation: ⌈frames/2⌉ × (height/32) × (width/32) + 2.
# 使用前安装:pip install opencv-python
import math
import os
import cv2
# 固定参数
FRAME_FACTOR = 2
IMAGE_FACTOR = 32
# 视频帧的长宽比
MAX_RATIO = 200
# 视频帧的像素下限
VIDEO_MIN_PIXELS = 128 * 32 * 32
# 视频帧的像素上限
VIDEO_MAX_PIXELS = 768 * 32 * 32
FPS = 2
# 最少抽取帧数
FPS_MIN_FRAMES = 4
# 最大抽取帧数
FPS_MAX_FRAMES = 128
# 视频输入的最大像素值,
VIDEO_TOTAL_PIXELS = 16384 * 32 * 32
def round_by_factor(number, factor):
return round(number / factor) * factor
def ceil_by_factor(number, factor):
return math.ceil(number / factor) * factor
def floor_by_factor(number, factor):
return math.floor(number / factor) * factor
def get_video(video_path):
cap = cv2.VideoCapture(video_path)
frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
video_fps = cap.get(cv2.CAP_PROP_FPS)
cap.release()
return frame_height, frame_width, total_frames, video_fps
def smart_nframes(total_frames, video_fps):
min_frames = ceil_by_factor(FPS_MIN_FRAMES, FRAME_FACTOR)
max_frames = floor_by_factor(min(FPS_MAX_FRAMES, total_frames), FRAME_FACTOR)
duration = total_frames / video_fps if video_fps != 0 else 0
if duration - int(duration) > (1 / FPS):
total_frames = math.ceil(duration * video_fps)
else:
total_frames = math.ceil(int(duration) * video_fps)
nframes = total_frames / video_fps * FPS
nframes = int(min(min(max(nframes, min_frames), max_frames), total_frames))
if not (FRAME_FACTOR <= nframes <= total_frames):
raise ValueError(f"nframes should in interval [{FRAME_FACTOR}, {total_frames}], but got {nframes}.")
return nframes
def smart_resize(height, width, nframes, factor=IMAGE_FACTOR):
min_pixels = VIDEO_MIN_PIXELS
total_pixels = VIDEO_TOTAL_PIXELS
max_pixels = max(min(VIDEO_MAX_PIXELS, total_pixels / nframes * FRAME_FACTOR), int(min_pixels * 1.05))
if max(height, width) / min(height, width) > MAX_RATIO:
raise ValueError(f"absolute aspect ratio must be smaller than {MAX_RATIO}, got {max(height, width) / min(height, width)}")
h_bar = max(factor, round_by_factor(height, factor))
w_bar = max(factor, round_by_factor(width, factor))
if h_bar * w_bar > max_pixels:
beta = math.sqrt((height * width) / max_pixels)
h_bar = floor_by_factor(height / beta, factor)
w_bar = floor_by_factor(width / beta, factor)
elif h_bar * w_bar < min_pixels:
beta = math.sqrt(min_pixels / (height * width))
h_bar = ceil_by_factor(height * beta, factor)
w_bar = ceil_by_factor(width * beta, factor)
return h_bar, w_bar
def video_token_calculate(video_path):
height, width, total_frames, video_fps = get_video(video_path)
nframes = smart_nframes(total_frames, video_fps)
resized_height, resized_width = smart_resize(height, width, nframes)
video_token = int(math.ceil(nframes / FPS) * resized_height / 32 * resized_width / 32)
video_token += 2 # 视觉标记
return video_token
if __name__ == "__main__":
video_path = "spring_mountain.mp4" # 你的视频路径
video_token = video_token_calculate(video_path)
print("video_tokens:", video_token)
-
audio_tokensEach second of audio consumes 12.5 tokens. Audio shorter than 1 second is billed as 1 second.
For token pricing, see Model list.
API reference
For the input and output parameters of the qwen3-livetranslate-flash model, see Audio and video file translation - Qwen.
Supported languages
The language codes in the following table can be used to specify the source and target languages.
Some target languages support text output only and do not support audio output.
Language code | Language | Supported output modalities |
|---|---|---|
en | English | Audio, text |
zh | Chinese | Audio, text |
ru | Russian | Audio, text |
fr | French | Audio, text |
de | German | Audio, text |
pt | Portuguese | Audio, text |
es | Spanish | Audio, text |
it | Italian | Audio, text |
id | Indonesian | Text |
ko | Korean | Audio, text |
ja | Japanese | Audio, text |
vi | Vietnamese | Text |
th | Thai | Text |
ar | Arabic | Text |
yue | Cantonese | Audio, text |
hi | Hindi | Text |
el | Greek | Text |
tr | Turkish | Text |
Supported voices
| Voice name |
| Voice effect | Description | Supported languages |
|---|---|---|---|---|
Cherry | Cherry | A cheerful, approachable, and naturally friendly young woman. | Chinese, English, French, German, Russian, Italian, Spanish, Portuguese, Japanese, Korean | |
Nofish | Nofish | A designer who cannot pronounce retroflex consonants. | Chinese, English, French, German, Russian, Italian, Spanish, Portuguese, Japanese, Korean | |
Shanghai-Jada | Jada | A brisk, energetic Shanghai lady. | Chinese | |
Beijing-Dylan | Dylan | A young man who grew up in the hutongs of Beijing. | Chinese | |
Sichuan-Sunny | Sunny | A Sichuan girl sweet to the core. | Chinese | |
Tianjin-Peter | Peter | Tianjin crosstalk style, a professional supporting performer. | Chinese | |
Cantonese-Kiki | Kiki | A sweet best friend from Hong Kong. | Cantonese | |
Sichuan-Eric | Eric | An unconventional, street-smart man from Chengdu, Sichuan. | Chinese |
FAQ
Q: What content is translated when I input a video file?
A: The model translates the audio content of the video. Visual information is used to assist context understanding and improve accuracy.
Example:
When the audio content is This is a mask:
- If the frame shows a protective face mask, it is translated as "This is a medical mask";
- If the frame shows a masquerade mask, it is translated as "This is a masquerade mask".