API details

Streaming text-to-speech

Streaming text-to-speech (TTS) converts a text stream into binary audio data and returns the result as a real-time audio stream. This allows end users to hear the speech output almost synchronously, which greatly improves the interactive experience and reduces waiting time. For example, you can send a text stream from a large language model (LLM) directly to the streaming TTS service without any processing, such as concatenation or integration. The service then returns a real-time audio stream. Streaming TTS also provides the following features:

• Supports PCM, WAV, and MP3 encoding formats.

• Supports sending text in a stream. In the same session, you can send text in batches and receive the corresponding audio. The synthesized audio can be played in real time with low latency. The text sent in each batch does not need to form a complete word or sentence. The server-side automatically segments and reorganizes the text based on semantics.

• In a single streaming TTS session, a synthesis request cannot exceed 2,000 characters, and the total number of characters cannot exceed 200,000. One Chinese character is counted as two characters. One English letter, punctuation mark, or space is counted as one character.

• Supports UTF-8 encoded text input.

• Supports setting different voices for various scenarios.

Prerequisites

• Obtain the AppKey and token required for authentication. For more information, see Manage projects and Obtain a token using an SDK.

• If you use an SDK to connect to the service, you must download and install the SDK. For more information, see Getting Started.

Endpoints

Service interaction flow

The interaction flow between the client and the server consists of three stages:

• Stage 1: The client establishes a connection with the server and sends the `StartSynthesis` instruction. The server returns a response event to confirm that the connection is successful. The TTS service can then start.

• Stage 2: The client calls the text sending API operation in a loop to send a text stream to the server. The server returns the synthesized audio data and corresponding event responses.

• Stage 3: After all text is sent, the client sends the `StopSynthesis` instruction. The client then waits for the server to return all the synthesized audio data before disconnecting.

API operations

• startStreamInputTts: Establishes a WebSocket connection with the server and performs operations, such as setting callbacks and parameters.

  Android-Java

  /**
   * Initializes and establishes a connection to start the streaming speech synthesis task.
   * @param callback: The event listener callback. For more information, see the callback descriptions later in this topic.
   * @param ticket: The authentication parameters in a JSON string. For more information, see the parameter settings below.
   * @param parameters: The initialization configuration parameters in a JSON string. For more information, see the description below or the API reference: xxxxx
   * @param session_id: The session ID. You can pass a 32-byte UUID or leave it empty to let the SDK generate one automatically.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int startStreamInputTts(INativeTtsCallback callback,
                                              String ticket,
                                              String parameters
                                              String session_id);

  Object C

  /**
   * Establishes a connection with the server and starts the streaming speech synthesis task.
   * @param ticket: The authentication parameters in a JSON string. For more information, see the description below or the API reference: xxxxx
   * @param parameters: The initialization configuration parameters in a JSON string. For more information, see the description below or the API reference: xxxxx
   * @param session_id: The session ID. You can pass a 32-byte UUID or leave it empty to let the SDK generate one automatically.
   * @param level: The log printing level. A smaller value means more logs are printed.
   * @param save_log: Specifies whether to save the log to a file. The storage directory is the value of the debug_path field in the ticket.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  - (int) startStreamInputTts:(const char *)ticket 
                              parameters:(const char *)parameters 
                              sessionId:(const char *)sessionId 
                              logLevel:(int)logLevel,
                              saveLog:(BOOL)saveLog;

  Java

  /**
   * Starts speech transcription: sends a speech transcription request and synchronously receives a confirmation from the server-side.
   * @throws Exception
   */
  public void startSteamInputTTS()

  Python

  """
  Starts speech synthesis: Sends a speech synthesis request and synchronously receives confirmation from the server.
  
  Parameters:
  -----------
  voice: str
      voice for text-to-speech, default is xiaoyun
  aformat: str
      audio binary format, support: 'pcm', 'wav', 'mp3', default is 'pcm'
  sample_rate: int
      audio sample rate, default is 24000, support:8000, 11025, 16000, 22050,
      24000, 32000, 44100, 48000
  volume: int
      audio volume, from 0~100, default is 50
  speech_rate: int
      speech rate from -500~500, default is 0
  pitch_rate: int
      pitch for voice from -500~500, default is 0
  ex: dict
      dict which will merge into 'payload' field in request
  """
  def startStreamInputTts(
      self,
      voice="longxiaochun",
      aformat="pcm",
      sample_rate=24000,
      volume=50,
      speech_rate=0,
      pitch_rate=0,
  )

  JavaScript

  // This is an ArkTS interface for HarmonyOS.
  /**
   * Initializes and establishes a connection to start the streaming speech synthesis task.
   * @param callback: The event listener callback. For more information, see the callback descriptions later in this topic.
   * @param ticket: The authentication parameters in a JSON string. For more information, see the parameter settings below.
   * @param parameters: The initialization configuration parameters in a JSON string. For more information, see the description below or the API reference: xxxxx
   * @param session_id: The session ID. You can pass a 32-byte UUID or leave it empty to let the SDK generate one automatically.
   * @param log_level: The log level.
   * @param save_log: Specifies whether to save the SDK's internal log files.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public startStreamInputTts( callback:INativeStreamInputTtsCallback,
  			ticket:string, 
                          parameters:string,
  			session_id:string, 
                          log_level:number,
  			save_log:boolean):number

  ticket parameter settings

  Parameter	Type	Required	Description
  url	String	No	The endpoint. The Beijing service is used by default.
  app_key	String	Yes	The AppKey of the project created in the console.
  token	String	Yes	Make sure the token is valid and has not expired.
  complete_waiting_ms	Integer	No	The timeout period for waiting for the STREAM_INPUT_TTS_EVENT_SYNTHESIS_COMPLETE event after calling stop. Unit: ms. Default: 10s.

  parameter settings

  Parameter	Type	Required	Description
  voice	String	No	The voice. Default: longxiaochun.
  format/aformat(python sdk)	String	No	The audio encoding format. Default: pcm. Supported formats: pcm, wav, and mp3.
  sample_rate	String	No	The audio sampling rate. Default: 24000.
  volume	String	No	The reading volume. Range: 0 to 100. Default: 50.
  speech_rate	String	No	The speech rate. Range: -500 to 500. Default: 0.
  pitch_rate	String	No	The pitch. Range: -500 to 500. Default: 0. A larger value produces a higher-pitched voice.

• sendStreamInputTts: Sends a text stream to the server.

  Android-Java

  /**
   * Sends text in a stream.
   * @param text: The streaming text generated from the large language model.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int sendStreamInputTts(String text);

  Object C

  /**
   * Sends text in a stream.
   * @param text: The streaming text generated from the large language model.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  - (int) sendStreamInputTts:(const char *)text;

  Java

  /**
   * Sends text in a stream.
   * @throws Exception
   * @param text: The streaming text generated from the large language model.
   */
  public void sendStreamInputTts(String text);

  Python

  """
  Sends text in a stream.
  Parameters:
  -----------
  text: str
      utf-8 text
  """
  def sendStreamInputTts(self, text)

  JavaScript

  // This is an ArkTS interface for HarmonyOS.
  /**
   * Sends text in a stream.
   * @param text: The streaming text generated from the large language model.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public sendStreamInputTts(text:string):number

• stopStreamInputTts: Stops the streaming speech synthesis process in a blocking manner and disconnects the WebSocket connection from the server.

  Android-Java

  /**
   * Ends the synthesis task. It notifies the server that all text data has been sent and blocks until the server finishes processing and returns all synthesized audio. The blocking timeout can be set by complete_waiting_ms in the start interface.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int stopStreamInputTts()

  Object C

  /**
   * Stops the synthesis task and waits to receive the remaining synthesis results before returning.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public synchronized int stopStreamInputTts()

  Java

  /**
   * Ends the synthesis task. It notifies the server that all text data has been sent and blocks until the server finishes processing and returns all synthesized audio. The blocking timeout can be set by complete_waiting_ms in the start interface.
   */
  public void stopStreamInputTts()

  Python

  """
  Stops the synthesis task and waits to receive the remaining synthesis results before returning.
  """
  def stopStreamInputTts(self)

  JavaScript

  // This is an ArkTS interface for HarmonyOS.
  /**
   * Ends the synthesis task. It notifies the server that all text data has been sent and blocks until the server finishes processing and returns all synthesized audio. The blocking timeout can be set by complete_waiting_ms in the start interface.
   * @param flag_async: Specifies whether the stop interface uses an asynchronous wait. If true, the stop interface returns immediately without blocking. If false, the stop interface blocks until the synthesis is complete or an error occurs. We recommend setting this to true.
   * @return: For more information about error codes, see https://help.aliyun.com/document_detail/459864.html.
   */
  public stopStreamInputTts(flag_async:boolean=true):number

• Callback functions

  Event callback function: Handles callback events from the server.

  Android-Java

  /**
   * Event callback.
   * @param event: The callback event. For more information, see the event list below.
   * @param task_id: The ID of the request task. A new ID is generated for each call.
   * @param session_id: The ID of the request session.
   * @param ret_code: The error code. This is valid when a TTS_EVENT_ERROR event occurs. For more information, see https://help.aliyun.com/document_detail/459864.html.
   * @param error_msg: The error message returned when an error code is generated.
   * @param timestamp: The timestamp information.
   * @param all_response: The complete response information in JSON format.
   */
  void onStreamInputTtsEventCallback(StreamInputTtsEvent event,
                                     String task_id, String session_id, 
                                     int ret_code, String error_msg,
                                     String timestamp, String all_response);

  Object C

  /**
   * Event callback.
   * @param event: The callback event. For more information, see the event list below.
   * @param task_id: The ID of the request task. A new ID is generated for each call.
   * @param session_id: The ID of the request session.
   * @param ret_code: The error code. This is valid when a TTS_EVENT_ERROR event occurs.
   *   For more information, see https://help.aliyun.com/document_detail/459864.html.
   * @param error_msg: The error message returned when an error code is generated.
   * @param timestamp: The timestamp information.
   * @param all_response: The complete response information in JSON format.
   */
  - (void)onStreamInputTtsEventCallback:(StreamInputTtsCallbackEvent)event 
                                         taskId:(char*)taskid 
                                         sessionId:(char*)sessionId 
                                         ret_code:(int)ret_code 
                                         error_msg:(char*)error_msg 
                                         timestamp:(char*)timestamp 
                                         all_response:(char*)all_response;

  Java

  /**
   * The server-side detects the beginning of a sentence.
   * @param response
   */
  abstract public void onSentenceBegin(
      StreamInputSpeechSynthesizerResponse response);
  
  /**
   * The server-side detects the end of a sentence and returns the start and end positions and all timestamps for that sentence.
   * @param response
   */
  abstract public void onSentenceEnd(
      StreamInputSpeechSynthesizerResponse response);
  /**
   * Synthesis complete.
   * @param response
   */
  abstract public void onSynthesisComplete(
      StreamInputSpeechSynthesizerResponse response);
  /**
   * Failure handling.
   * @param response
   */
  abstract public void onFail(StreamInputSpeechSynthesizerResponse response);
  /**
   * Incrementally returns timestamps in response=>payload.
   * @param response
   */
  abstract public void onSentenceSynthesis(
      StreamInputSpeechSynthesizerResponse response);

  JavaScript

  // This is an ArkTS interface for HarmonyOS.
  /**
   * Event callback.
   * @param event: The callback event. For more information, see the event list below.
   * @param task_id: The ID of the request task. A new ID is generated for each call.
   * @param session_id: The ID of the request session.
   * @param ret_code: The error code. This is valid when a TTS_EVENT_ERROR event occurs. For more information, see https://help.aliyun.com/document_detail/459864.html.
   * @param error_msg: The error message returned when an error code is generated.
   * @param timestamp: The timestamp information.
   * @param all_response: The complete response information in JSON format.
   */
  onStreamInputTtsEventCallback(event:StreamInputTtsEvent, 
                                  task_id:string, 
                                  session_id:string,
  				ret_code:number, 
                                  error_msg:string, 
                                  timestamp:string,
  				all_response:string):void;

  Data callback function: Returns the synthesized audio data.

  Android-Java

  /**
   * Returns streaming speech synthesis data.
   * @param data: The synthesized audio data to be written to the player.
   */
  void onStreamInputTtsDataCallback(byte[] data);

  Object C

  /**
   * This callback is continuously invoked when speech synthesis starts to provide a stream of audio data.
   * @param data: The synthesized speech data to be written to a player.
   * @param len: The length of the synthesized speech data.
   */
  - (void)onStreamInputTtsDataCallback:(char*)data len:(int)len;

  Java

  /**
  * Receives the speech synthesis audio data stream.
  * @param message The binary audio data.
  */
  abstract public void onAudioData(ByteBuffer message);

  JavaScript

  // This is an ArkTS interface for HarmonyOS.
  /**
   * Returns streaming speech synthesis data.
   * @param data: The synthesized audio data to be written to the player.
   */
  onStreamInputTtsDataCallback(data:ArrayBuffer|null):void;

Mobile SDK callback events

Python callback functions are configured as parameters when you create an object. The following table describes these parameters.

Common SDK error codes

For more information about error codes, see Error code query.