Streaming text-to-speech iOS SDK-Intelligent Speech Interaction(ISI)-阿里云帮助中心

This topic describes how to use the iOS NUI SDK for Alibaba Cloud Intelligent Speech Interaction. It covers how to download and install the SDK, its key interfaces, and code samples.

Prerequisites

Review the API reference. For more information, see API reference.
You must have a project AppKey. For more information, see Create a project.
Obtain an Access Token. For more information, see Get Token overview.

Download and install

Select and download the mobile SDK.
Important
After you download the SDK, you must replace the sample Alibaba Cloud account information, AppKey, and token in the initialization code to run the sample.
For easier integration, the iOS API uses a pure Objective-C interface starting from version 2.5.14. It no longer uses a mixed C++ interface.
Unzip the package and add `nuisdk.framework` to your project. In the project's Build Phases, add `nuisdk.framework` to Link Binary With Libraries. Then, in the General > Frameworks, Libraries, and Embedded Content section of your build configuration, set `nuisdk.framework` to Embed & Sign.
Open the project in Xcode. The project includes reference code and ready-to-use utility classes for tasks such as audio playback, recording, and file operations. You can copy the source code directly into your project. The sample code for speech synthesis is located in the `StreamInputTtsViewController` class.

Key SDK interfaces

startStreamInputTts: Starts a streaming Text-to-Speech (TTS) task.

/**
 * Establishes a connection with the server and starts a streaming speech synthesis task.
 * @param ticket: Initialization parameters in a JSON string. See the description below.
 * @param parameters: Initialization parameters in a JSON string. See the description below.
 * @param session_id: The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a 32-bit unique ID.
 * @param level: The log printing level. A smaller value prints more logs.
 * @param save_log: Specifies whether to save logs to a file. The storage directory is the value of the debug_path field in the ticket. Note: Log files have no size limit. Ensure that continuous storage does not fill the disk.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) startStreamInputTts:(const char *)ticket parameters:(const char *)parameters sessionId:(const char *)sessionId logLevel:(NuiSdkLogLevel)logLevel saveLog:(BOOL)saveLog;

playStreamInputTts: Synchronously synthesizes a single long sentence. Each call performs one speech synthesis task. The input text supports Speech Synthesis Markup Language (SSML).

/**
 * Establishes a connection with the server and starts a one-way streaming speech synthesis (long-form text-to-speech) task. This is a synchronous interface.
 * @param ticket: Authentication parameters in a JSON string. See the description below or the API reference.
 * @param parameters: Initialization configuration parameters in a JSON string. See the description below or the API reference.
 * @param text: The single sentence to be synthesized. SSML is supported. Only UTF-8 encoded text is supported. For a single synthesis request, use fewer than 5,000 characters. The total must not exceed 100,000 characters. One Chinese character, one English letter, one punctuation mark, or one space is counted as one character.
 * @param session_id: The session ID. Pass a 32-byte UUID or leave it empty for the SDK to generate one automatically.
 * @param level: The log printing level. A smaller value prints more logs.
 * @param save_log: Specifies whether to save logs to a file. The storage directory is the value of the debug_path field in the ticket.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) playStreamInputTts:(const char *)ticket parameters:(const char *)parameters text:(const char*)text sessionId:(const char *)sessionId logLevel:(NuiSdkLogLevel)logLevel saveLog:(BOOL)saveLog;

asyncPlayStreamInputTts: Asynchronously synthesizes a single long sentence. Each call performs one speech synthesis task. The input text supports SSML.

/**
 * Establishes a connection with the server and starts a one-way streaming speech synthesis (long-form text-to-speech) task. This is an asynchronous interface.
 * @param ticket: Authentication parameters in a JSON string. See the description below or the API reference.
 * @param parameters: Initialization configuration parameters in a JSON string. See the description below or the API reference.
 * @param text: The single sentence to be synthesized. SSML is supported. Only UTF-8 encoded text is supported. For a single synthesis request, use fewer than 5,000 characters. The total must not exceed 100,000 characters. One Chinese character, one English letter, one punctuation mark, or one space is counted as one character.
 * @param session_id: The session ID. Pass a 32-byte UUID or leave it empty for the SDK to generate one automatically.
 * @param level: The log printing level. A smaller value prints more logs.
 * @param save_log: Specifies whether to save logs to a file. The storage directory is the value of the debug_path field in the ticket.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) asyncPlayStreamInputTts:(const char *)ticket parameters:(const char *)parameters text:(const char*)text sessionId:(const char *)sessionId logLevel:(NuiSdkLogLevel)logLevel saveLog:(BOOL)saveLog;

cancelStreamInputTts: Immediately stops speech synthesis. No callbacks are returned.

/**
 * Immediately stops the synthesis task. No callbacks are returned.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) cancelStreamInputTts;

stopStreamInputTts: Stops speech synthesis.

/**
 * Ends the synthesis task, notifies the server that all text data has been sent, blocks until the server finishes processing, and returns all synthesized audio. The blocking timeout can be set using complete_waiting_ms in the start interface.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) stopStreamInputTts;

asyncStopStreamInputTts: Asynchronously stops speech synthesis. The call returns immediately, and callbacks continue to return the remaining synthesized content.

/**
 * Ends the synthesis task, notifies the server that all text data has been sent, returns immediately without waiting, and allows callbacks to continue returning the remaining content.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) asyncStopStreamInputTts;

sendStreamInputTts: Streams text to be synthesized.

/**
 * Sends text in a streaming manner.
 * @param text: The streaming text generated from a large language model.
 * @return: For error codes, see https://help.aliyun.com/document_detail/459864.html.
 */
- (int) sendStreamInputTts:(const char *)text;

StreamInputTtsDelegate: The event delegate.

onStreamInputTtsEventCallback: The SDK event callback.

/**
 * Event callback.
 * @param event: The callback event. See the event list below.
 * @param taskid: The task ID for the entire real-time speech synthesis session. It must be consistent throughout the request. It is a 32-bit unique ID.
 * @param sessionId: The ID of the current session. If the client passes this ID in the request, it is returned as is. Otherwise, the server automatically generates a 32-bit unique ID.
 * @param ret_code: The error code. This is valid when a STREAM_INPUT_TTS_EVENT_TASK_FAILED event occurs. For more information, see https://help.aliyun.com/document_detail/459864.html.
 * @param error_msg: The detailed error message. This is valid when a STREAM_INPUT_TTS_EVENT_TASK_FAILED event occurs.
 * @param timestamp: Timestamp information in the synthesis result.
 * @param all_response: The complete response message in JSON string format. You can parse the required information from it.
 */
- (void)onStreamInputTtsEventCallback:(StreamInputTtsCallbackEvent)event taskId:(char*)taskid sessionId:(char*)sessionId ret_code:(int)ret_code error_msg:(char*)error_msg timestamp:(char*)timestamp all_response:(char*)all_response;

StreamInputTtsCallbackEvent events:

Name	Description
TTS_EVENT_SYNTHESIS_STARTED	Speech synthesis has started and is ready for playback.
TTS_EVENT_SENTENCE_BEGIN	The server has detected the start of a sentence.
TTS_EVENT_SENTENCE_SYNTHESIS	Returns incremental speech synthesis results, including the latest audio and timestamps. The data for the current sentence is returned in full, while data across sentences is returned incrementally.
TTS_EVENT_SENTENCE_END	The server has detected the end of a sentence and returns the full timestamp for that sentence.
TTS_EVENT_SYNTHESIS_COMPLETE	The server-side returns a full timestamp for a sentence upon detecting its end.
TTS_EVENT_TASK_FAILED	An error occurred during speech synthesis. See ret_code and error_msg for details.

onStreamInputTtsDataCallback: The callback for synthesized data.

/**
 * Synthetic data callback.
 * @param data: The synthesized audio data to be written to the player.
 */
void onStreamInputTtsDataCallback(byte[] data);

Procedure

Initialize the SDK and the playback component.
Set the required parameters.
Call `startStreamInputTts` to start streaming text-to-speech synthesis.
Call `sendStreamInputTts` to stream the text to be synthesized. In the synthetic data callback, write the data to the player for playback. Streaming playback is recommended.
Call `stopStreamInputTts` to signal that text sending is complete and wait for the synthesis to finish.
Handle the callback that indicates speech synthesis is complete.

Code samples

Start speech synthesis

NSMutableDictionary *ticketJsonDict = [NSMutableDictionary dictionary];
[ticketJsonDict setObject:@"wss://nls-gateway-cn-beijing.aliyuncs.com/ws/v1" forKey:@"url"]; // Required
[ticketJsonDict setObject:@"10000" forKey:@"complete_waiting_ms"];
// Obtain the account access credential:
[_utils getTicket:ticketJsonDict Type:get_token_from_server_for_online_features];
if ([ticketJsonDict objectForKey:@"token"] != nil) {
    NSString *tokenValue = [ticketJsonDict objectForKey:@"token"];
    if ([tokenValue length] == 0) {
        TLog(@"The 'token' key exists but the value is empty.");
    }
} else {
    TLog(@"The 'token' key does not exist.");
}
NSError *error;
NSData *ticketJsonData = [NSJSONSerialization dataWithJSONObject:ticketJsonDict options:0 error:&error];
NSString *ticket = [[NSString alloc] initWithData:ticketJsonData encoding:NSUTF8StringEncoding];

NSDictionary *paramsJsonDict = @{
    @"voice": @"zhixiaoxia",
    @"format": @"pcm",
    @"sample_rate": @(16000),
    @"volume": @(50),
    @"speech_rate": @(0),
    @"pitch_rate": @(0),
    @"enable_subtitle": @(YES)
};
NSData *paramsJsonData = [NSJSONSerialization dataWithJSONObject:paramsJsonDict options:0 error:&error];
NSString *parameters = [[NSString alloc] initWithData:paramsJsonData encoding:NSUTF8StringEncoding];

_streamInputTtsSdk = [StreamInputTts get_instance];
_streamInputTtsSdk.delegate = self;

int ret = [_streamInputTtsSdk startStreamInputTts:[ticket UTF8String] parameters:[parameters UTF8String] sessionId:nil logLevel:NUI_LOG_LEVEL_VERBOSE saveLog:NO];

Stream text for synthesis

NSString * sentence = @"How do you make scrambled eggs with tomatoes?";
[_streamInputTtsSdk sendStreamInputTts:[sentence UTF8String]]

Stop speech synthesis

[_streamInputTtsSdk stopStreamInputTts];

Handle callbacks

onStreamInputTtsEventCallback: The event callback for streaming text-to-speech. You can use this callback to control the player based on the speech synthesis status.

- (void)onStreamInputTtsEventCallback:(StreamInputTtsCallbackEvent)event taskId:(char*)taskid sessionId:(char*)sessionId ret_code:(int)ret_code error_msg:(char*)error_msg timestamp:(char*)timestamp all_response:(char*)all_response {
    NSString *log = [NSString stringWithFormat:@"\nEvent callback (%d): %s", event, all_response];
    TLog(@"%@", log);
    
    if (event == TTS_EVENT_SYNTHESIS_STARTED) {
        TLog(@"onStreamInputTtsEventCallback TTS_EVENT_SYNTHESIS_STARTED");
        // Synthesis started. Start the player.
        [_audioController startPlayer];
    } else if (event == TTS_EVENT_SENTENCE_BEGIN) {
        TLog(@"onStreamInputTtsEventCallback TTS_EVENT_SENTENCE_BEGIN");
    } else if (event == TTS_EVENT_SENTENCE_SYNTHESIS) {
        TLog(@"onStreamInputTtsEventCallback TTS_EVENT_SENTENCE_SYNTHESIS");
    } else if (event == TTS_EVENT_SENTENCE_END) {
        TLog(@"onStreamInputTtsEventCallback TTS_EVENT_SENTENCE_END");
    } else if (event == TTS_EVENT_SYNTHESIS_COMPLETE) {
        TLog(@"onStreamInputTtsEventCallback TTS_EVENT_SYNTHESIS_COMPLETE");
        // Note that this event indicates that speech synthesis is complete, not that playback is complete. The voicePlayer object sends a notification when playback is complete.
        [_audioController drain];
    } else if (event == TTS_EVENT_TASK_FAILED) {
        TLog(@"onStreamInputTtsEventCallback TTS_EVENT_TASK_FAILED:%s", error_msg);
        // Note that this event indicates that speech synthesis is complete, not that playback is complete. The voicePlayer object sends a notification when playback is complete.
        [_audioController drain];
    }
}

onStreamInputTtsDataCallback: The data callback for speech synthesis. Use this callback to write the synthesized data to the player for playback.

- (void)onStreamInputTtsDataCallback:(char*)buffer len:(int)len {
    NSString *log = [NSString stringWithFormat:@"\nAudio callback %d bytes", len];
    TLog(log);
    dispatch_async(dispatch_get_main_queue(), ^{
        self.logTextView.text = [self.logTextView.text stringByAppendingString:log];
        NSRange bottom = NSMakeRange(self.logTextView.text.length -1, 1);
        [self.logTextView scrollRangeToVisible:bottom];
    });
    if (buffer != NULL && len > 0) {
        [_audioController write:(char*)buffer Length:(unsigned int)len];
    }
}