what is the timestamp feature, timestamp limits-Intelligent Speech Interaction(ISI)-阿里云帮助中心

The asynchronous long-text speech synthesis service can generate sentence-level timestamps for the output audio stream. The service splits the input text into sentences at punctuation marks, such as periods, question marks, and exclamation points. It then provides the timing of each sentence within the audio. This timestamp information is useful for scenarios such as creating captions for video dubbing or highlighting text for audiobook narration. This topic outlines the timestamp feature.

Comparison of short-text and long-text speech synthesis

Speech synthesis provides two methods, and the timestamp support differs between them. Check the following table before you decide which method to use and what level of timestamp granularity you can get.

Item	Short-text speech synthesis	Long-text speech synthesis
API type	WebSocket (wss) streaming API. The RESTful API does not support timestamps.	RESTful API (asynchronous)
Character limit	About 300 characters per request	Up to 100,000 characters in a single synthesis task
Timestamp granularity	Word-level (phoneme boundary)	Sentence-level
Parameter	Set `enable_subtitle` to `true` to enable timestamps	Set `enable_subtitle` to `true` to enable timestamps. For example, in a RESTful API call: `tts.put("enable_subtitle", true);`
Response field	Returns `task_id`	Returns `task_id`
`task_id` location	Returned in the response header	Returned in the response body

Short-text speech synthesis uses the WebSocket (wss) streaming API and supports word-level (phoneme boundary) timestamps. The short-text RESTful API does not support timestamps.
Long-text speech synthesis supports sentence-level timestamps only through the long-text RESTful API, consistent with the Limits section below.
Long-text speech synthesis supports a single synthesis task of up to 100,000 characters.
Both APIs use enable_subtitle=true to enable timestamps and both return task_id, but at different locations in the response.

Limits

For long-text speech synthesis, only the long-text RESTful API supports sentence-level timestamps.

Parameter settings

To enable the timestamp feature, set the enable_subtitle request parameter to true. The following example shows how to set this parameter for a RESTful API call.

// The long-text TTS RESTful API supports sentence-level timestamps. The default value is false, which disables the feature.
tts.put("enable_subtitle", true);

Server-side response

The server returns the caption information in the sentences field of the response.

Parameter	Type	Description
sentences	List	Timestamp information

The format of the sentences field is as follows:

Parameter	Type	Description
begin_time	String	The start timestamp of the TTS audio that corresponds to the text. Unit: ms.
end_time	String	The end timestamp of the TTS audio that corresponds to the text. Unit: ms.

Examples

Sample request:

{
    "payload":{
        "tts_request":{
            "voice":"Aifan",
            "sample_rate":16000,
            "format":"mp3",
            "enable_subtitle":true,
            "text":"Behind my house was a large garden, known as the Hundred-Plant Garden. It was long ago sold with the house to the descendants of Zhu Wen Gong; it has been seven or eight years since I last saw it, and it seemed to contain only wild grass, but at that time, it was my paradise."
        },
        "notify_url":"http://123****.com",
        "enable_notify":false
    },
    "context":{
        "device_id":"my_device_id"
    },
    "header":{
        "appkey":"1iMxP16qgjP****",
        "token":"16aea272b48d4bb188664611837f****"
    }
}

Sample response:

{
    "status":200,
    "data":{
        "sentences":[
            {
                "text":"Behind my house was a large garden, known as the Hundred-Plant Garden",
                "begin_time":"0",
                "end_time":"4247"
            },
            {
                "text":"It was long ago sold with the house to the descendants of Zhu Wen Gong; it has been seven or eight years since I last saw it, and it seemed to contain only wild grass, but at that time, it was my paradise",
                "begin_time":"4247",
                "end_time":"16060"
            }
        ],
        "task_id":"9628f978abab4628b1bcfd5a9da3749f",
        "audio_address":"http://nls-cloud-cn-shanghai.oss-cn-shanghai.aliyuncs.com/jupiter-flow/tmp/9628f978abab4628b1bcfd5a9da3749f.mp3?Expires=1621305670&OSSAccessKeyId=LTAI****************&Signature=OYHTJMQXMpiAx*****",
        "notify_custom":""
    },
    "error_code":20000000,
    "error_message":"SUCCESS",
    "request_id":"7e70c414c31a41ae86b4a5f4241a6f3c"
}