Speech transcription

更新时间:
复制 MD 格式

This topic describes the AI capabilities of speech transcription and how to implement this feature.

Speech transcription is a core feature of Tingwu that transcribes speech from audio and video files or real-time audio streams into text. This feature is the first and required node in the Tingwu API service chain and cannot be disabled. It supports languages such as Chinese, English, Cantonese, and Japanese. You can configure speaker diarization using the transcription parameters.

Request parameters

Parameter Name

Type

Required

Description

Transcription

object

No

The object for speech recognition control parameters.

Transcription.DiarizationEnabled

boolean

No

Specifies whether to enable speaker diarization during transcription.

Transcription.Diarization

object

No

The speaker diarization object.

Transcription.Diarization.SpeakerCount

int

No

0: The number of speakers is not specified.

2: The number of speakers is two.

Transcription.PhraseId

string

No

The ID of the hotword vocabulary.

Transcription.Model

string

No

The speech transcription model. Use this parameter to call a domain-specific model to improve recognition accuracy in a specific domain. If this parameter is empty, the default model is used. The available parameters are:

  • "domain-automotive": A speech recognition model for sales conversations in the automotive domain. It can be used for both real-time and offline transcription tasks.

  • "domain-education": A speech recognition model for online courses in the education domain. It can only be used for offline transcription tasks.

  • Speech recognition models for other domains will be available soon.

For descriptions of other request parameters, see the document for your scenario:

Note

The Transcription.Model parameter: Use this parameter to call a domain-specific model to improve recognition accuracy in a specific domain. The available domain-specific models are listed in the following table:

Model name

Parameter value

Supported language

Sample rate

Real-time/Offline

Scenarios

Speech recognition model for sales conversations in the automotive domain

domain-automotive

Chinese

16 kHz

Offline

Suitable for speech recognition in the automotive industry, including scenarios such as in-store reception, test drives, and car model promotions.

Speech recognition model for online courses in the education domain

domain-education

Chinese

16 kHz

Offline

Suitable for speech recognition in the education industry, including scenarios such as online courses.

Example settings

// No settings
{
    "Input":{
        ...
    },
    "Parameters":{

    }
}

// Enable speaker diarization
{
    "Input":{
        ...
    },
    "Parameters":{
        "Transcription":{
            "DiarizationEnabled":true,
            "Diarization":{
                "SpeakerCount":2
            }
        }
    }
}

Code examples

#!/usr/bin/env python
#coding=utf-8

import os
import json
import datetime
from aliyunsdkcore.client import AcsClient
from aliyunsdkcore.request import CommonRequest
from aliyunsdkcore.auth.credentials import AccessKeyCredential

def create_common_request(domain, version, protocolType, method, uri):
    request = CommonRequest()
    request.set_accept_format('json')
    request.set_domain(domain)
    request.set_version(version)
    request.set_protocol_type(protocolType)
    request.set_method(method)
    request.set_uri_pattern(uri)
    request.add_header('Content-Type', 'application/json')
    return request

def init_parameters():
    root = dict()
    root['AppKey'] = 'Enter the AppKey that you created in the Tingwu console'

    # Basic request parameters
    input = dict()
    input['SourceLanguage'] = 'cn'
    input['TaskKey'] = 'task' + datetime.datetime.now().strftime('%Y%m%d%H%M%S')
    input['FileUrl'] = 'Enter the URL of the audio file to test'
    root['Input'] = input

    # AI-related parameters. Set them as needed.
    parameters = dict()

    # Speech recognition control
    transcription = dict()
    # Speaker diarization
    transcription['DiarizationEnabled'] = True
    diarization = dict()
    diarization['SpeakerCount'] = 2
    transcription['Diarization'] = diarization
    parameters['Transcription'] = transcription

    root['Parameters'] = parameters
    return root

body = init_parameters()
print(body)

# TODO: Set your AccessKeyId and AccessKeySecret as environment variables.
credentials = AccessKeyCredential(os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID'], os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET'])
client = AcsClient(region_id='cn-beijing', credential=credentials)

request = create_common_request('tingwu.cn-beijing.aliyuncs.com', '2023-09-30', 'https', 'PUT', '/openapi/tingwu/v2/tasks')
request.add_query_param('type', 'offline')

request.set_content(json.dumps(body).encode('utf-8'))
response = client.do_action_with_exception(request)
print("response: \n" + json.dumps(json.loads(response), indent=4, ensure_ascii=False))
package com.alibaba.tingwu.client.demo.aitest;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.CommonRequest;
import com.aliyuncs.CommonResponse;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.exceptions.ClientException;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import org.junit.Test;

/**
 * @author tingwu2023
 */
public class TranscriptionTest {

    @Test
    public void testFiletrans() throws ClientException {
        CommonRequest request = createCommonRequest("tingwu.cn-beijing.aliyuncs.com", "2023-09-30", ProtocolType.HTTPS, MethodType.PUT, "/openapi/tingwu/v2/tasks");
        request.putQueryParameter("type", "offline");

        JSONObject root = new JSONObject();
        root.put("AppKey", "Enter the AppKey that you created in the Tingwu console");

        JSONObject input = new JSONObject();
        input.fluentPut("FileUrl", "Enter the URL of the audio file to test")
                .fluentPut("SourceLanguage", "cn")
                .fluentPut("TaskKey", "task" + System.currentTimeMillis());
        root.put("Input", input);

        JSONObject parameters = new JSONObject();
        JSONObject transcription = new JSONObject();
        transcription.put("DiarizationEnabled", true);
        JSONObject speakerCount = new JSONObject();
        speakerCount.put("SpeakerCount", 2);
        transcription.put("Diarization", speakerCount);
        parameters.put("Transcription", transcription);
        root.put("Parameters", parameters);
        System.out.println(root.toJSONString());
        request.setHttpContent(root.toJSONString().getBytes(), "utf-8", FormatType.JSON);

        // TODO: Set your AccessKeyId and AccessKeySecret as environment variables.
        DefaultProfile profile = DefaultProfile.getProfile("cn-beijing", System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"));
        IAcsClient client = new DefaultAcsClient(profile);
        CommonResponse response = client.getCommonResponse(request);
        System.out.println(response.getData());
    }
  
    public static CommonRequest createCommonRequest(String domain, String version, ProtocolType protocolType, MethodType method, String uri) {
        // Create an API request and set its parameters.
        CommonRequest request = new CommonRequest();
        request.setSysDomain(domain);
        request.setSysVersion(version);
        request.setSysProtocol(protocolType);
        request.setSysMethod(method);
        request.setSysUriPattern(uri);
        request.setHttpContentType(FormatType.JSON);
        return request;
    }
}

Example output

{
    "Code":"0",
    "Data":{
        "TaskId":"10683ca4ad3f4f06bdf6e9dd********",
        "TaskStatus":"COMPLETED",
        "Result":{
            "Transcription":"http://speech-swap.oss-cn-zhangjiakou.aliyuncs.com/tingwu_data/output/1738248129743478/10683ca4ad3f4f06bdf6e9dc1f3c1584/10683ca4ad3f4f06bdf6e9dc1f3c1584_Transcription_20231031165005.json?Expires=1698828606&LTAI****************&Signature=GMm%2BdN2tUPx*********ehu74%3D"
        }
    },
    "Message":"success",
    "RequestId":"7a63ee43-8a35-4ef2-8d5c-c76*********"
}

The Transcription field provides the HTTP URL that you can use to download the speech transcription result.

Protocol parsing

The speech transcription result, which is available at the specified URL, is a JSON-formatted message. The following example shows the message structure. The Size, SampleRate, and Language fields in AudioInfo are not included in real-time transcription results.

{
    "TaskId":"10683ca4ad3f4f06bdf6e9dc*********",
    "Transcription":{
        "AudioInfo": {
            "Size": 670663,
            "Duration": 10394,
            "SampleRate": 48000,
            "Language": "cn"
        },
        "Paragraphs":[
            {
                "ParagraphId":"16987422100275*******",
                "SpeakerId":"1",
                "Words":[
                    {
                        "Id":10,
                        "SentenceId":1,
                        "Start":4970,
                        "End":5560,
                        "Text":"Hello,"
                    },
                    {
                        "Id":20,
                        "SentenceId":1,
                        "Start":5730,
                        "End":6176,
                        "Text":"I am"
                    }
                ]
            }
        ]
        "AudioSegments": [
            [12130, 16994],
            [17000, 19720],
            [19940, 28649]
        ]
    }
}

The fields are defined as follows.

Parameter name

Type

Description

TaskId

string

The ID of the task. This ID is generated when the task is created.

Transcription

object

The speech transcription result object.

Transcription.Paragraphs

list[]

A collection of speech transcription results organized into paragraphs.

Transcription.Paragraphs[i].ParagraphId

string

The paragraph-level ID.

Transcription.Paragraphs[i].SpeakerId

string

The speaker ID.

Transcription.Paragraphs[i].Words

list[]

The word information in the paragraph.

Transcription.Paragraphs[i].Words[i].Id

int

The ordinal number of the word. You can usually ignore this parameter.

Transcription.Paragraphs[i].Words[i].SentenceId

int

The sentence ID. Words that have the same SentenceId can be combined into a sentence.

Transcription.Paragraphs[i].Words[i].Start

long

The start time of the word relative to the beginning of the audio. This is a relative timestamp in milliseconds.

Transcription.Paragraphs[i].Words[i].End

long

The end time of the word relative to the beginning of the audio. This is a relative timestamp in milliseconds.

Transcription.Paragraphs[i].Words[i].Text

string

The text of the word.

Transcription.AudioInfo

object

The audio information object.

Transcription.AudioInfo.Size

long

The size of the audio file in bytes.

Transcription.AudioInfo.Duration

long

The duration of the audio in milliseconds. For real-time speech transcription, this field does not indicate the actual audio duration.

Transcription.AudioInfo.SampleRate

int

The audio sample rate.

Transcription.AudioInfo.Language

string

The language of the audio.

Transcription.AudioSegments

list[][]

The range of valid audio segments.

Transcription.AudioSegments[i][0]

int

The start time of the valid audio segment in milliseconds.

Transcription.AudioSegments[i][1]

int

The end time of the valid audio segment in milliseconds.

FAQ

Why is the recognition result empty after I upload a file?

Check whether the original audio or video file contains valid human speech or has excessive background noise.