what are the apis for model studio speech model services-Intelligent Speech Interaction(ISI)-阿里云帮助中心

Overview

The Paraformer speech recognition file transcription API transcribes common audio or video files and returns the results.

Common audio and video files are typically recorded at a sample rate of 16 kHz or higher. For these files, you can use the paraformer-v1 model for Chinese and English speech recognition or the paraformer-MTL-v1 model for over 20 languages and Chinese dialects. The paraformer-v1 model generally provides higher accuracy for Chinese and English speech. Phone call recordings are usually recorded at 8 kHz. For these recordings, select the paraformer-8k-v1 model for optimal speech recognition results.

Paraformer speech recognition returns a rich set of results, such as full-text, sentence-level text, words, and timestamps. The model performs punctuation prediction and inverse text regularization by default.

Because audio and video files are often large, file transfer and speech recognition processing can take a significant amount of time. The file transcription API submits tasks using asynchronous invocation. After the file transcription is complete, you can use the query interface to retrieve the speech recognition results. The API supports batch processing, which lets you upload up to 100 file URLs in a single request. After all files are transcribed, you can retrieve all the transcription results at once.

Note

For partnership inquiries or technical consultation, you can submit a ticket.

Model overview

Model name	Model description
paraformer-v1	A Chinese and English speech recognition model in the Paraformer speech model service. It supports speech recognition for audio or video files with a sample rate of 16 kHz or higher.
paraformer-8k-v1	A Chinese speech recognition model in the Paraformer speech model service. It supports 8 kHz speech recognition for phone calls.
paraformer-mtl-v1	A multilingual speech recognition model in the Paraformer speech model service. It supports speech recognition for audio or video files with a sample rate of 16 kHz or higher. Supported languages and dialects include the following: Mandarin Chinese, Chinese dialects (Cantonese, Wu, Minnan, Northeastern, Gansu, Guizhou, Henan, Hubei, Hunan, Ningxia, Shanxi, Shaanxi, Shandong, Sichuan, Tianjin), English, Japanese, Korean, Spanish, Indonesian, French, German, Italian, and Malay.

API reference

Prerequisites

Activate the service and obtain an API key. For more information, see Get and configure an API key and Configure an API key as an environment variable.
Ensure that the SDK is installed. For more information, see Install the SDK.

Audio and video file transcription

Use the model parameter to specify the speech recognition model. Use the file_urls parameter to specify the audio or video files for recognition. The URLs must use the HTTP or HTTPS protocol.

Note

Replace your-dashscope-api-key in the example with your API key to ensure the code runs correctly.

# For prerequisites running the following sample, visit https://help.aliyun.com/document_detail/611472.html

import dashscope
import json

dashscope.api_key='your-dashscope-api-key'

task_response=dashscope.audio.asr.Transcription.async_call(
    model='paraformer-v1',
    file_urls=['https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_female.wav',
    	'https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_male.wav']
    )

transcribe_response=dashscope.audio.asr.Transcription.wait(task=task_response.output.task_id)
print(json.dumps(transcribe_response.output, indent=4, ensure_ascii=False))

The file transcription API uses asynchronous invocation. You must handle the asynchronous logic for submitting tasks, querying statuses, and retrieving results.

Note

The file transcription service processes tasks submitted through the API on a best-effort basis. The specific queue waiting time depends on the length of the concurrent queue and the duration of the files in other tasks. Therefore, an exact waiting time cannot be provided. Typically, the queue waiting time is less than a few minutes. After a task leaves the queue and enters the processing state, the file is transcribed at a speed several hundred times faster than real time.

The API supports major audio and video file formats, such as the following:

.aac, .amr, .avi, .flac, .flv, .m4a, .mkv, .mov, .mp3, .mp4, .mpeg, .ogg, .opus, .wav, .webm, .wma, and .wmv.

Note

Because many audio and video formats and their variants exist, it is not technically feasible to test all of them. Therefore, the API cannot guarantee that all formats will be recognized correctly. We recommend that you test your files to verify that you can obtain the expected speech recognition results.

The API supports transcribing up to 100 file URLs specified by the file_urls parameter. Each file must be 2 GB or smaller.

If a file that you want to process exceeds these limits, you can pre-process the file to reduce its size. For more information about file pre-processing, see Pre-process video files to improve file transcription efficiency.

dashscope.audio.asr.Transcribe.async_call()

Submits a task to the file transcription service using an asynchronous call and returns information about the submitted task.

Parameter settings

Parameter	Type	Default value	Description
model	string	-	Specifies the name of the Paraformer model for audio and video file transcription. Valid values are paraformer-v1, paraformer-8k-v1, and paraformer-MTL-v1. For information about how to choose a model, see Model overview.
file_urls	List[string]	-	A list of URLs for the audio or video files to be transcribed. The URLs must use the HTTP or HTTPS protocol. You can specify up to 100 file URLs.
channel_id (optional)	List[int]	[0]	Specifies the indexes of the audio tracks to be recognized in a multi-track file. Provide the indexes as a list. For example, [0] recognizes the first track, and [0, 1] recognizes the first and second tracks.

Sample response

{
    "status_code": 200,
    "request_id": "8c59f00c-7723-455e-922d-ac3a31838170",
    "code": "",
    "message": "",
    "output": {
        "task_id": "bd725f8f-f699-4962-bcad-38a9fc2bcd7c",
        "task_status": "PENDING"
    },
    "usage": null
}

Response parameter descriptions

Response parameter	Type	Description
code	int	The status code. For more information, see Status code description.
task_id	string	The ID of the submitted asynchronous task. This ID is the globally unique identifier for the task and is used for subsequent operations such as querying the task status and retrieving results.

dashscope.audio.asr.Transcribe.wait()

Waits in a blocking manner for an asynchronous task to finish (reach the SUCCEEDED or FAILED state) and then returns the task status and file transcription results. The possible asynchronous task statuses are PENDING, RUNNING, SUCCEEDED, and FAILED. When the task is in the PENDING or RUNNING state, the wait() call is blocked. When the task is in the SUCCEEDED or FAILED state, the wait() call returns the task status and results.

Parameter settings
Parameter
Type
Default value
Description
task
string
-
The task_id returned when you submitted the task by calling async_call().

Sample response

{
    "status_code": 200,
    "request_id": "548c1e4b-1383-4a48-b747-ff46382ffd9d",
    "code": null,
    "message": "",
    "output": {
        "task_id": "bd725f8f-f699-4962-bcad-38a9fc2bcd7c",
        "task_status": "SUCCEEDED",
        "results": [
            {
                "file_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_male.wav",
                "transcription_url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/0cbf0acf/20230517/14%3A30/4545b696-08f6-4330-825e-c1a0c3f67eaf-1.json?Expires=1684391431&OSSAccessKeyId=LTAI***************4G8qL&Signature=1z8zxHVQqdmUKgxr2ldipyC6lOU%3D",
                "subtask_status": "SUCCEEDED"
            },
            {
                "file_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_female.wav",
                "transcription_url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/0cbf0acf/20230517/14%3A30/9e5c4e54-3612-47c3-a824-5eeeea3059ee-1.json?Expires=1684391431&OSSAccessKeyId=LTAI***************4G8qL&Signature=WlzLbAk0zaCVJ5SkATd6xNcEQQ0%3D",
                "subtask_status": "SUCCEEDED"
            }
        ]
    },
    "usage": {
        "duration": 8
    }
}

Response parameter descriptions

Response parameter	Type	Description
code	int	The audio format of the source file.
task_id	string	The ID of the queried task.
task_status	string	The status of the queried task.
file_url	string	The URL of the file processed in the transcription task.
status	string	The processing status of the file.
transcription_url	string	The URL for the processing result of the file.

dashscope.audio.asr.Transcribe.fetch()

Queries an asynchronous task and returns its current status and transcription results. The possible asynchronous task statuses are PENDING, RUNNING, SUCCEEDED, and FAILED. Unlike the wait() call, the fetch() call is non-blocking and immediately returns the current status and results of the queried task. You must write code to poll the task until it is complete, and then retrieve the transcription results.

Parameter settings
Parameter
Type
Default value
Description
task
string
-
The task_id returned when you submitted the task by calling async_call().

Sample response

PENDING and RUNNING states

    "status_code": 200,
    "request_id": "04ea046b-ccca-4765-882c-9f60bd985150",
    "code": null,
    "message": "",
    "output": {
        "task_id": "bd725f8f-f699-4962-bcad-38a9fc2bcd7c",
        "task_status": "RUNNING"
    },
    "usage": null
}

SUCCEEDED state

{
    "status_code": 200,
    "request_id": "548c1e4b-1383-4a48-b747-ff46382ffd9d",
    "code": null,
    "message": "",
    "output": {
        "task_id": "bd725f8f-f699-4962-bcad-38a9fc2bcd7c",
        "task_status": "SUCCEEDED",
        "results": [
            {
                "file_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_male.wav",
                "transcription_url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/0cbf0acf/20230517/14%3A30/4545b696-08f6-4330-825e-c1a0c3f67eaf-1.json?Expires=1684391431&OSSAccessKeyId=LTAI***************4G8qL&Signature=1z8zxHVQqdmUKgxr2ldipyC6lOU%3D",
                "subtask_status": "SUCCEEDED"
            },
            {
                "file_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_female.wav",
                "transcription_url": "https://dashscope-result-bj.oss-cn-beijing.aliyuncs.com/0cbf0acf/20230517/14%3A30/9e5c4e54-3612-47c3-a824-5eeeea3059ee-1.json?Expires=1684391431&OSSAccessKeyId=LTAI***************4G8qL&Signature=WlzLbAk0zaCVJ5SkATd6xNcEQQ0%3D",
                "subtask_status": "SUCCEEDED"
            }
        ]
    },
    "usage": {
        "duration": 8
    }
}

Response parameter descriptions

Response parameter	Type	Description
code	int	The audio format of the source file.
task_id	string	The ID of the queried task.
task_status	string	The status of the queried task.
file_url	string	The URL of the file processed in the transcription task.
status	string	The processing status of the file.
transcription_url	string	The URL for the processing result of the file.

Audio and video file transcription result file

When a file transcription task succeeds, you can retrieve the transcription result by calling wait() or fetch(). This result is saved as a file, and its link is provided in the transcription_url. Open this URL to retrieve the result.

Note

The URL for an asynchronous task's transcription result file is valid for 24 hours from the time the task is completed.

Sample response

{
    "file_url": "https://dashscope.oss-cn-beijing.aliyuncs.com/samples/audio/paraformer/hello_world_female.wav",
    "properties":
    {
        "audio_format": "pcm_s16le",
        "channels":
        [
            0
        ],
        "original_sampling_rate": 16000,
        "original_duration_in_milliseconds": 4087
    },
    "transcripts":
    [
        {
            "channel_id": 0,
            "content_duration_in_milliseconds": 3780,
            "text": "Hello world from Alibaba DAMO Academy Speech Lab.",
            "sentences":
            [
                {
                    "begin_time": 60,
                    "end_time": 3840,
                    "text": "Hello world from Alibaba DAMO Academy Speech Lab.",
                    "words":
                    [
                        {
                            "begin_time": 60,
                            "end_time": 780,
                            "text": "Hello ",
                            "punctuation": ""
                        },
                        {
                            "begin_time": 780,
                            "end_time": 1320,
                            "text": "world",
                            "punctuation": ""
                        },
                        {
                            "begin_time": 1320,
                            "end_time": 1720,
                            "text": "from",
                            "punctuation": ""
                        },
                        {
                            "begin_time": 1720,
                            "end_time": 2460,
                            "text": "Alibaba",
                            "punctuation": ""
                        },
                        {
                            "begin_time": 2460,
                            "end_time": 3000,
                            "text": "DAMO Academy",
                            "punctuation": ""
                        },
                        {
                            "begin_time": 3000,
                            "end_time": 3840,
                            "text": "Speech Lab",
                            "punctuation": "."
                        }
                    ]
                }
            ]
        }
    ]
}

Response parameter descriptions

Response parameter	Type	Description
audio_format	string	The format of the audio in the source file.
channels	List[int]	The audio track index information of the source file. Returns [0] for a single-track audio file, [0, 1] for a dual-track audio file, and so on.
original_sampling_rate	int	The sample rate of the audio in the source file, in Hz.
original_duration	int	The original duration of the audio in the source file, in ms.
channel_id	int	The index of the audio track for the transcription result. The index starts from 0.
content_duration	int	The duration of the content in the audio track that is identified as speech, in ms. Important The Paraformer speech recognition model service only transcribes the content in the audio track that is identified as speech. Billing is based on this duration. Non-speech content is not measured or billed. Typically, the speech content duration is shorter than the original audio duration. Because the determination of whether speech content exists is made by an AI model, there may be some discrepancy with the actual situation.
transcript	string	The paragraph-level speech transcription result.
sentences	List[]	The sentence-level speech transcription results.
words	List[]	The word-level speech transcription results.
begin_time	int	The start timestamp, in ms.
end_time	int	The end timestamp, in ms.
text	string	The speech transcription result.
punctuation	string	The predicted punctuation mark after the word, if any.

Status code description

For general status codes of Model Studio model services, see Error messages.