This topic describes how to use the C++ software development kit (SDK) provided by Alibaba Cloud Intelligent Speech Interaction. This topic includes instructions on how to install the SDK, sample code, and a list of frequently asked questions (FAQs).
Download the SDK
The latest version is 3.2.1b, which supports the Linux platform. This version was released on December 25, 2024.
Before you use the SDK, read the API reference. For more information, see the API reference.
The application programming interface (API) definitions in C++ SDK v3.1 are different from those in API v2.0, which is no longer supported. This topic uses the current version as an example.
You can obtain the SDK in one of the following ways.
Method 1: Obtain the latest source code from GitHub. For detailed compilation and running instructions, see the following sections or the readme.md file in the source code.
git clone --depth 1 https://github.com/aliyun/alibabacloud-nls-cpp-sdkMethod 2: Download the required SDK package from the table below. The SDK source code package contains the original SDK code. You must compile the code to generate the library files required for integration. The other SDK packages for specific platforms contain the relevant library and header files and do not require compilation.
Latest SDK package
Platform
MD5
SDK source code
7257c0998654e611cf2e8ca9867670ef
Linux x86_64
9a93df607f26f1558bc1043a425af6d1
Linux aarch64
76c34a3ab397d7285963a139b9270ff4
Where:
alibabacloud-nls-cpp-sdk<version>-master_<github commit id>.zip is the SDK source code package.
NlsCppSdk_<Platform>_<Version>_<github commit id>.tar.gz is the SDK package required for development on the corresponding platform. For more information, see the readme.md file in the package.
SDK package files
scripts/build_linux.sh: A sample compilation script for the Linux platform. This file is in the SDK source code.
CMakeLists.txt: The CMakeList file for the sample code project for the Linux platform. This file is in the SDK source code.
demo folder: Contains sample code. The following table describes the files for the Linux platform.
resource folder: Contains sample audio files for the Voice Service. This folder is in the SDK source code. You can use these files for functional testing. The files are described in the following table.
test0.wav
test1.wav
test2.wav
test3.wav
include the following: Contains the SDK header files from the SDK source code, as described in the following table.
File name
Description
nlsClient.h
SDK instance.
nlsEvent.h
Callback event description.
nlsGlobal.h
Global SDK header file.
nlsToken.h
SDK Access Token instance.
iNlsRequest.h
Base header file for NLS requests.
speechRecognizerRequest.h
Short sentence recognition.
speechSynthesizerRequest.h
Speech synthesis and long-text-to-speech synthesis.
speechTranscriberRequest.h
Real-time audio stream recognition.
FileTrans.h
Audio file recognition.
lib: SDK library files.
readme.md: SDK description.
release.log: Version guide.
version: Version number.
File name | Description |
speechRecognizerDemo.cpp | A demo for short sentence recognition. |
speechSynthesizerDemo.cpp | A demo for speech synthesis. |
speechTranscriberDemo.cpp | A demo for real-time speech recognition. |
fileTransferDemo.cpp | A demo for audio file recognition. |
File name | Description |
Test audio files with a 16 kHz audio sampling rate and 16-bit audio bit depth. |
Compile and run
The tool has the following minimum version requirements:
CMake 3.0
Glibc 2.5
Gcc 4.8.5
Run the following commands in the Linux terminal.
Go to the root directory of the SDK source code.
Generate the SDK library files and executable programs: srDemo for short sentence recognition, stDemo for real-time speech recognition, syDemo for speech synthesis, and daDemo for voice conversation.
./scripts/build_linux.shView the usage instructions for the demo.
cd build/demo ./syDemo
Key interfaces
Basic interfaces
NlsClient: The speech processing client. You can use this client to perform short sentence recognition, real-time speech recognition, and speech synthesis tasks. This client is thread-safe. We recommend that you create only one global instance.
Interface name
Version enabled
Feature description
getInstance
2.x
Gets (creates) an NlsClient instance.
setLogConfig
2.x
Sets the log file and storage path.
setDirectHost
3.x
Skips domain name system (DNS) resolution and directly sets the server IPv4 address. If you call this interface, call it before startWorkThread.
setAddrInFamily
3.1.12
Sets the address family for the socket address structure. The default value is AF_INET, which returns only IPv4-related address information. This interface must be called before startWorkThread.
setUseSysGetAddrInfo
3.1.13
If the libevent DNS cannot meet your needs and fails to complete DNS resolution, you can call this interface to switch to the system's interface. This interface must be called before startWorkThread.
calculateUtf8Chars
3.1.14
Counts the number of characters in the text. You must pass UTF-8 encoded text. One Chinese character, one English letter, or one punctuation mark is counted as one character.
setSyncCallTimeout
3.1.17
Sets the timeout period (in ms) for the synchronous call mode. The default value is 0, which means the synchronous mode is disabled.
When you use the synchronous call mode:
start() is blocked until a result is received from the server.
stop() is blocked until the close() callback is triggered.
setPreconnectedPool
3.3.0
Sets a pre-connection pool for each domain name URL.
Function:
Creates a persistent connection pool for a domain name URL. Connections are automatically reused after requests are complete.
Reduces the connection time before each request is initiated.
Significantly reduces the first-packet latency.
This conflicts with the long-lived connection mode and will disable any configured long-lived connection mode.
Disabled scenario: Tingwu scenarios.
Call constraint: You must call this interface before you call
startWorkThread.
startWorkThread
3.x
Starts the worker threads. The default value is 1, which starts one thread. If you set it to -1, it starts a number of threads equal to the number of CPU cores. For high-concurrency scenarios, we recommend setting this to -1. This can be considered the initialization of the NlsClient instance and must be called.
getVersion
3.x
Gets the SDK version number.
releaseInstance
2.x
Destroys the NlsClient object instance.
CreateSynthesizerRequest
2.x
Creates a speech synthesis object. This object is thread-safe and supports high-concurrency requests. When you create this object, you can use input parameters to specify whether to perform long-text-to-speech synthesis or short-text-to-speech synthesis. You can use short-text-to-speech synthesis for text up to 300 characters. For text longer than 300 characters, consider using long-text-to-speech synthesis. You can call the calculateUtf8Chars interface to count characters.
releaseSynthesizerRequest
2.x
Destroys the speech synthesis object. This must be called after the closed event of the current request.
NlsToken: Creates a Token object to request a token ID. When you request a new token, you must first obtain a valid timestamp. If the token expires, you must request a new one. If you request a token multiple times within its validity period, an incorrect token ID is returned that cannot be used.
Interface name
Feature description
setAccessKeyId
Sets the AccessKey ID of your Alibaba Cloud account.
setKeySecret
Sets the AccessKey secret of your Alibaba Cloud account.
setDomain
Sets the domain name. This parameter is optional.
setServerVersion
Sets the API version. This parameter is optional.
setServerResourcePath
Sets the service path. This parameter is optional.
setRegionId
Sets the service region ID. This parameter is optional.
setAction
Sets the feature. This parameter is optional.
applyNlsToken
Requests a token ID.
getToken
Gets the token ID.
getExpireTime
Gets the token's expiration timestamp in seconds.
getErrorMsg
Gets the error message.
NlsEvent: The event object. You can use this object to obtain the request status code, server response, and error message.
Interface name
Feature description
getStatusCode
Gets the status code. A value of 0 or 20000000 indicates success. A failure returns a corresponding error code.
getErrorMessage
In the TaskFailed callback, gets the error message for a failure that occurs during an NlsRequest operation.
getTaskId
Gets the task ID.
getBinaryData
Gets the binary data returned from the cloud.
getAllResponse
Gets the recognition result returned from the cloud.
Detection Interface
SpeechSynthesizerRequest: The speech synthesis request object. This object is used for both short-text and long-text speech synthesis. For more information about the interface, see the speechSynthesizerRequest.h file.
Interface name
Version enabled
Feature description
setOnSynthesisCompleted
2.x
Sets the callback function for when speech synthesis is complete.
setOnChannelClosed
2.x
Sets the callback function for when the channel is closed.
setOnTaskFailed
2.x
Sets the callback function for errors.
setOnBinaryDataReceived
2.x
Sets the callback function for receiving binary audio data from speech synthesis.
setOnMetaInfo
2.x
Sets the callback function for receiving log information corresponding to the text.
setOnMessage
3.1.16
Sets the callback function for server response messages. All callback output is from this callback and must be parsed by the user. This parameter is optional. After setting it, you must call setEnableOnMessage to enable it.
setAppKey
2.x
Sets the AppKey.
setToken
2.x
Token authentication. All requests must be authenticated using the setToken method before they can be used.
setUrl
2.x
Sets the service URL. This parameter is optional.
setText
2.x
Sets the text content to be synthesized. You can use short-text-to-speech synthesis for text up to 300 characters. For text longer than 300 characters, consider using long-text-to-speech synthesis. You can call the calculateUtf8Chars interface to count characters.
NoteTo call the multi-emotion feature of a voice, add the ssml-emotion tag to the text. For more information, see <emotion>.
If you use the <emotion> tag for a voice that does not support multiple emotions, the `Illegal ssml text` error is reported.
setVoice
2.x
Sets the voice.
setVolume
2.x
Sets the volume.
setFormat
2.x
Sets the output audio encoding format. The default value is PCM. Supported formats are PCM, WAV, and MP3.
setSampleRate
2.x
Sets the audio sampling rate.
setSpeechRate
2.x
Sets the speech rate.
setPitchRate
2.x
Sets the pitch rate.
setMethod
2.x
Sets the synthesis method. The default value is 0.
0: Parametric synthesis. This method is based on statistical parameters. It can adapt to a wide range of prosodic features, has a low synthesizer bit rate, consumes few resources, and delivers high performance with moderate audio quality.
1: Concatenative synthesis. This method learns from a high-quality audio library. It consumes more resources but provides better audio quality that is closer to real human speech. However, it is less stable than parametric synthesis.
setEnableSubtitle
2.x
Indicates whether the caption feature is enabled.
setPayloadParam
2.x
Sets parameters. The input parameter is a JSON-formatted string.
setTimeout
2.x
Sets the connection timeout period. The default value is 5000 ms.
setContextParam
2.x
Sets custom user parameters. The input parameter is a JSON-formatted string.
AppendHttpHeaderParam
2.x
Sets custom HTTP header parameters for the WebSocket phase.
setSendTimeout
3.1.14
Sets the sending timeout period. The default value is 5000 ms.
setEnableOnMessage
3.1.16
Enables the callback for messages returned from the server.
getTaskId
3.1.17
Gets the task ID of the current request.
start
2.x
Starts the SpeechSynthesizerRequest.
cancel
2.x
Immediately closes the speech synthesis process without confirming the shutdown with the server.
C++ SDK error codes
Status code | Status message | Cause | Solution |
0 | Success | Success | |
-10 | DefaultError | Default error | Not currently in use. |
-11 | JsonParseFailed | Invalid JSON format | Check whether the input JSON string is in the correct JSON format. |
-12 | JsonObjectError | Invalid JSON object | Retry the operation. |
-13 | MallocFailed | Malloc failed | Check if there is sufficient memory. |
-14 | ReallocFailed | Realloc failed | Check if there is sufficient memory. |
-15 | InvalidInputParam | Invalid input parameter | Not currently in use. |
-50 | InvalidLogLevel | Invalid log level | Check the configured log level. |
-51 | InvalidLogFileSize | Invalid log file size | Check the configured log file size parameter. |
-52 | InvalidLogFileNum | Invalid number of log files | Check the configured number of log files parameter. |
-100 | EncoderExistent | The NLS encoder already exists. | Retry the operation. |
-101 | EncoderInexistent | The NLS encoder does not exist. | We recommend re-initializing. |
-102 | OpusEncoderCreateFailed | Failed to create the Opus encoder. | We recommend that you reinitialize. |
-103 | OggOpusEncoderCreateFailed | Failed to create the OggOpus encoder. | We recommend that you reinitialize. |
-104 | InvalidEncoderType | Invalid encoder type | OPUS may have been disabled at compile-time but is being used, or check the ENCODER_TYPE. |
-150 | EventClientEmpty | The main worker thread is a null pointer and has been released. | Re-initialize the SDK by calling startWorkThread(). |
-151 | SelectThreadFailed | Failed to select a worker thread. Not initialized. | Re-initialize the SDK by calling startWorkThread(). |
-160 | StartCommandFailed | Failed to send the start command. | Retry the operation. |
-161 | InvokeStartFailed | The request state machine is incorrect, causing start to fail. | Check if the current request has not been created or has already been completed. |
-162 | InvokeSendAudioFailed | The request state machine is incorrect, causing sendAudio to fail. | Check if the current request has been started (that is, the started event callback has been received) or has already been completed. |
-163 | InvalidOpusFrameSize | Invalid Opus frame size. The default is 640 bytes. | In OPU encoding mode, sendAudio accepts only one frame of 640 bytes of data. |
-164 | InvokeStopFailed | The request state machine is incorrect, causing stop to fail. | Check if the current request is either not yet started (the 'started' event callback has not been received) or already complete. |
-165 | InvokeCancelFailed | The request state machine is incorrect, causing cancel to fail. | Check whether the 'started' event callback has not been received or the request is already complete. |
-166 | InvokeStControlFailed | The request state machine is incorrect, causing stControl to fail. | Check if the 'started' event callback has not been received, or if the request is already complete. |
-200 | NlsEventEmpty | The NLS event is empty. | For internal SDK use. The NlsEvent frame is lost. |
-201 | NewNlsEventFailed | Failed to create NlsEvent. | For internal SDK use. Failed to create the NlsEvent frame. |
-202 | NlsEventMsgEmpty | The message in the NLS event is empty. | The message string was found to be empty during parsing by parseJsonMsg(). |
-203 | InvalidNlsEventMsgType | Invalid message type in the NLS event. | For internal SDK use. The event type of the NlsEvent frame is invalid. |
-204 | InvalidNlsEventMsgStatusCode | Invalid message status code in the NLS event. | For internal SDK use. The event message status of the NlsEvent frame is invalid. |
-205 | InvalidNlsEventMsgHeader | Invalid message header in the NLS event. | For internal SDK use. The event message header of the NlsEvent frame is invalid. |
-250 | CancelledExitStatus | cancel has been called. | Not currently in use. |
-251 | InvalidWorkStatus | Invalid working status | For internal SDK use. The internal status of the current request is invalid. |
-252 | InvalidNodeQueue | Invalid NodeQueue in workThread. | For internal SDK use. The current request to be run is invalid. Release the current request and retry. |
-300 | InvalidRequestParams | Invalid input parameters for the request. | The data passed to sendAudio is empty. |
-301 | RequestEmpty | The request is a null pointer. | For internal SDK use. The current request has been released. Release the current request and retry. |
-302 | InvalidRequest | Invalid request | For internal SDK use. The current request has been released. Release the current request and retry. |
-303 | SetParamsEmpty | The input parameter is empty. | Check if the input parameter is empty. |
-350 | GetHttpHeaderFailed | Failed to get the HTTP header. | For internal SDK use. Locate the issue based on the feedback in the logs. |
-351 | HttpGotBadStatus | Bad HTTP status | For internal SDK use. Locate the issue based on the feedback in the logs. |
-352 | WsResponsePackageFailed | Failed to parse the WebSocket response package. | For internal SDK use. Locate the issue based on the feedback in the logs. |
-353 | WsResponsePackageEmpty | The parsed WebSocket response package is empty. | For internal SDK use. Locate the issue based on the feedback in the logs. |
-354 | WsRequestPackageEmpty | The WebSocket request package is empty. | For internal SDK use. Locate the issue based on the feedback in the logs. |
-355 | UnknownWsFrameHeadType | Unknown WebSocket frame header type | For internal SDK use. Locate the issue based on the feedback in the logs. |
-356 | InvalidWsFrameHeaderSize | Invalid WebSocket frame header size | For internal SDK use. Locate the issue based on the feedback in the logs. |
-357 | InvalidWsFrameHeaderBody | Invalid WebSocket frame header body | For internal SDK use. Locate the issue based on the feedback in the logs. |
-358 | InvalidWsFrameBody | Invalid WebSocket frame body | For internal SDK use. Locate the issue based on the feedback in the logs. |
-359 | WsFrameBodyEmpty | The frame data is empty. This is often due to receiving dirty data. | For internal SDK use. Locate the issue based on the feedback in the logs. |
-400 | NodeEmpty | The node is a null pointer. | Release the current request and retry. |
-401 | InvaildNodeStatus | The node status is invalid. | For internal SDK use. Release the current request and retry. |
-402 | GetAddrinfoFailed | Detecting addresses using DNS parsing | For internal SDK use. Check if DNS is available in the current environment. |
-403 | ConnectFailed | Network connection failed. | Check if the current network environment is available. |
-404 | InvalidDnsSource | No DNS on the current device. | For internal SDK use. Check if DNS is available in the current environment. |
-405 | ParseUrlFailed | Invalid URL | Check if the configured URL is valid. |
-406 | SslHandshakeFailed | SSL handshake failed. | For internal SDK use. Check if the current network environment is available and retry. |
-407 | SslCtxEmpty | SSL_CTX is empty. | For internal SDK use. Check if the current network environment is available and retry. |
-408 | SslNewFailed | SSL_new failed. | For internal SDK use. Check if the current network environment is available and retry. |
-409 | SslSetFailed | Failed to set SSL parameters. | For internal SDK use. Check if the current network environment is available and retry. |
-410 | SslConnectFailed | SSL_connect failed. | For internal SDK use. Check if the current network environment is available and retry. |
-411 | SslWriteFailed | Failed to send data over SSL. | For internal SDK use. Check if the current network environment is available and retry. |
-412 | SslReadSysError | SYSCALL error received when reading data over SSL. | For internal SDK use. Check if the current network environment is available and retry. |
-413 | SslReadFailed | Failed to read data over SSL. | For internal SDK use. Check if the current network environment is available and retry. |
-414 | SocketFailed | Failed to create a socket. | For internal SDK use. Check if the current network environment is available and retry. |
-415 | SetSocketoptFailed | Failed to set socket parameters. | For internal SDK use. Check if the current network environment is available and retry. |
-416 | SocketConnectFailed | Failed to establish a socket connection. | For internal SDK use. Check if the current network environment is available and retry. |
-417 | SocketWriteFailed | Failed to send data over the socket. | For internal SDK use. Check if the current network environment is available and retry. |
-418 | SocketReadFailed | Failed to read data from the socket. | For internal SDK use. Check if the current network environment is available and retry. |
-430 | NlsReceiveFailed | Failed to receive NLS frame data. | For internal SDK use. Check if the current network environment is available and retry. |
-431 | NlsReceiveEmpty | Received NLS frame data is empty. | For internal SDK use. Check if the current network environment is available and retry. |
-432 | ReadFailed | Failed to receive data. | For internal SDK use. Check if the current network environment is available and retry. |
-433 | NlsSendFailed | Failed to send NLS data. | For internal SDK use. Check if the current network environment is available and retry. |
-434 | NewOutputBufferFailed | Failed to create a buffer. | For internal SDK use. Check if there is sufficient memory. |
-435 | NlsEncodingFailed | Audio encoding failed. | For internal SDK use. Release the current request and retry. |
-436 | EventEmpty | The event is empty. | For internal SDK use. Release the current request and retry. |
-437 | EvbufferTooMuch | Too much data in evbuffer. | For internal SDK use. The send data buffer is full. The maximum buffer size is 320,000 for 16 kHz audio and 160,000 for 8 kHz audio. Check if you are sending audio data too frequently or sending too much data at once. |
-438 | EvutilSocketFailed | Failed to set evutil parameters. | For internal SDK use. Release the current request and retry. |
-439 | InvalidExitStatus | Invalid exit status | Check if you have already canceled the current request. |
-450 | InvalidAkId | Invalid AccessKey ID for the Alibaba Cloud account. | Check if the AccessKey ID for the Alibaba Cloud account is empty. |
-451 | InvalidAkSecret | Invalid AccessKey secret for the Alibaba Cloud account. | Check if the AccessKey secret for the Alibaba Cloud account is empty. |
-452 | InvalidAppKey | Invalid project AppKey. | Check if the project AppKey for Alibaba Cloud is empty. |
-453 | InvalidDomain | Invalid domain | Check if the input domain is empty. |
-454 | InvalidAction | Invalid action | Check if the input action is empty. |
-455 | InvalidServerVersion | Invalid ServerVersion | Check if the input ServerVersion is empty. |
-456 | InvalidServerResource | Invalid ServerResource | Check if the input ServerResource is empty. |
-457 | InvalidRegionId | Invalid RegionId | Check if the input RegionId is empty. |
-500 | InvalidFileLink | Invalid audio file link | The audio file link for audio file transcription is empty. |
-501 | ErrorStatusCode | Error status code | An error was returned for audio file transcription. For more information, see the error code. |
-502 | IconvOpenFailed | Failed to request a conversion descriptor. | Failed to convert between UTF-8 and GBK. |
-503 | IconvFailed | Encoding conversion failed. | Failed to convert between UTF-8 and GBK. |
-504 | ClientRequestFaild | Account client request failed. | Failed to return the result of audio file transcription. |
-999 | NlsMaxErrorCode |
Other status codes | Status message | Cause | Solution |
10000001 | NewSslCtxFailed | SSL: couldn't create a context! | We recommend reinitializing. |
10000002 | DefaultErrorCode | return of SSL_read: error:00000000:lib(0):func(0):reason(0) | Retry the operation. |
return of SSL_read: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init | |||
10000003 | SysErrorCode | System error. | Handle the error based on the system feedback. |
10000004 | EmptyUrl | URL: The url is empty. | The input URL is empty. Enter a valid URL. |
10000005 | InvalidWsUrl | Could not parse WebSocket url: | The input URL format is incorrect. Enter a valid URL. |
10000007 | JsonStringParseFailed | JSON: Json parse failed. | JSON format is abnormal. Check the logs for specific error details. |
10000008 | UnknownWsHeadType | WEBSOCKET: unkown head type. | Network connection failed. Check if the local DNS resolution and URL are valid. |
10000009 | HttpConnectFailed | HTTP: connect failed. | Failed to connect to the cloud. Check your network and retry. |
10000010 | MemNotEnough | Out of memory. | Check if there is sufficient memory. |
10000015 | SysConnectFailed | connect failed. | Network connection failed. Check if the local DNS resolution and URL are valid. |
10000100 | HttpGotBadStatusWith403 | Got bad status host=xxxxx line=HTTP/1.1 403 Forbidden | Connection was rejected. Check your account, especially if the token has expired. |
10000101 | EvSendTimeout | Send timeout. socket error: | libevent timed out when sending an event. Check for time-consuming tasks in the callback, or if high concurrency is preventing events from being processed promptly. |
10000102 | EvRecvTimeout | Recv timeout. socket error: | libevent timed out when receiving an event. Check for time-consuming tasks in the callback, or if high concurrency is preventing events from being processed promptly. |
10000103 | EvUnknownEvent | Unknown event: | Unknown libevent event. Retry the operation. |
10000104 | OpNowInProgress | Operation now in progress | The connection is in progress. Retry the operation. |
10000105 | BrokenPipe | Broken pipe | The pipe cannot handle the request. Retry the operation. |
10000110 | TokenHasExpired | Gateway:ACCESS_DENIED:The token 'xxx' has expired! | Update the token. |
10000111 | TokenIsInvalid | Meta:ACCESS_DENIED:The token 'xxx' is invalid! | Check the validity of the token. |
10000112 | NoPrivilegeToVoice | Gateway:ACCESS_DENIED:No privilege to this voice! (voice: zhinan, privilege: 0) | You do not have permission to use this voice. |
10000113 | MissAuthHeader | Gateway:ACCESS_DENIED:Missing authorization header! | Check if your account has the required permissions or if the number of concurrent requests is within the limit. |
10000120 | Utf8ConvertError | utf8ToGbk failed | UTF-8 transcoding failed. This is often a system issue. Retry the operation. |
20000000 | SuccessStatusCode | Success |
Service status codes
For more information about service status codes, see the API reference.
Sample code
The demo uses the default public URL that is built into the SDK to access the service. If you are using an Alibaba Cloud ECS instance in the China (Shanghai) region and need to use an internal URL, you must set the internal URL when you create the speechSynthesizerRequest object.
request->setUrl("wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1");In the demo, the synthesized audio is saved to a file. For applications that require high real-time playback, we recommend that you use stream playback. This method lets you play the audio data as you receive it, which reduces latency. Otherwise, you must wait for the synthesis to finish before you can process the audio stream.
For the complete sample code, see the speechSynthesizerDemo.cpp file in the demo directory of the SDK package.
Before you call the API, you must configure environment variables to read access credentials. The environment variable names for the AccessKey ID, AccessKey secret, and AppKey of Intelligent Speech Interaction are NLS_AK_ENV, NLS_SK_ENV, and NLS_APPKEY_ENV.
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include <stdlib.h>
#include <ctime>
#include <string>
#include <iostream>
#include <vector>
#include <fstream>
#include <sys/time.h>
#include <errno.h>
#include "nlsClient.h"
#include "nlsEvent.h"
#include "nlsToken.h"
#include "speechSynthesizerRequest.h"
using namespace AlibabaNlsCommon;
using AlibabaNls::NlsClient;
using AlibabaNls::NlsEvent;
using AlibabaNls::LogDebug;
using AlibabaNls::LogInfo;
using AlibabaNls::LogError;
using AlibabaNls::TtsVersion;
using AlibabaNls::SpeechSynthesizerRequest;
// Custom thread parameters.
struct ParamStruct {
std::string text;
std::string token;
std::string appkey;
std::string audioFile;
};
// Custom callback parameters.
struct ParamCallBack {
public:
ParamCallBack() {
pthread_mutex_init(&mtxWord, NULL);
pthread_cond_init(&cvWord, NULL);
};
~ParamCallBack() {
pthread_mutex_destroy(&mtxWord);
pthread_cond_destroy(&cvWord);
};
std::string binAudioFile;
std::ofstream audioFile;
pthread_mutex_t mtxWord;
pthread_cond_t cvWord;
};
/**
* Globally maintain a service authentication token and its expiration timestamp.
* Before each service call, check if the token has expired.
* If it has expired, generate a new token using the AccessKey ID and AccessKey Secret, and update the global token and its expiration timestamp.
*
* For more information about how to obtain a token, see: https://help.aliyun.com/document_detail/450514.html
*
* Note: Do not generate a new token before every service call. Only generate a new one when the current token is about to expire. A single token can be shared by all concurrent service calls.
*/
std::string g_akId = "";
std::string g_akSecret = "";
std::string g_token = "";
long g_expireTime = -1;
/**
* Generate a new token using the AccessKey ID and AccessKey Secret, and get its expiration timestamp.
*/
int generateToken(std::string akId, std::string akSecret,
std::string* token, long* expireTime) {
NlsToken nlsTokenRequest;
nlsTokenRequest.setAccessKeyId(akId);
nlsTokenRequest.setKeySecret(akSecret);
int ret = nlsTokenRequest.applyNlsToken();
if (ret < 0) {
// Failed to obtain the token.
printf("generateToken Failed, error code:%d msg:%s\n",
ret, nlsTokenRequest.getErrorMsg());
return ret;
}
*token = nlsTokenRequest.getToken();
*expireTime = nlsTokenRequest.getExpireTime();
return 0;
}
/**
* @brief When the SDK receives a message from the cloud that synthesis is complete, the SDK's internal thread reports a Completed event.
* @note After the Completed event is reported, the SDK will close the recognition connection channel internally.
* @param cbEvent The callback event structure. For more information, see nlsEvent.h.
* @param cbParam Custom callback parameter. The default is NULL. You can customize parameters as needed.
* @return
*/
void OnSynthesisCompleted(NlsEvent* cbEvent, void* cbParam) {
ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
// Example of how to print or use custom user parameters.
printf("OnSynthesisCompleted: %s\n", tmpParam->binAudioFile.c_str());
// Get the status code of the message. 0 or 20000000 indicates success. A failure returns a corresponding error code.
// The task ID of the current task. We recommend that you print this ID because it is the unique identifier for interaction with the server and helps with troubleshooting.
printf("OnSynthesisCompleted: status code=%d, task id=%s\n", cbEvent->getStatusCode(), cbEvent->getTaskId());
// Get all the information returned by the server.
//printf("OnSynthesisCompleted: all response=%s\n", cbEvent->getAllResponse());
}
/**
* @brief If an exception occurs during synthesis, the SDK's internal thread reports a TaskFailed event.
* @note After the TaskFailed event is reported, the SDK will close the recognition connection channel internally.
* @param cbEvent The callback event structure. For more information, see nlsEvent.h.
* @param cbParam Custom callback parameter. The default is NULL. You can customize parameters as needed.
* @return
*/
void OnSynthesisTaskFailed(NlsEvent* cbEvent, void* cbParam) {
ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
// Example of how to print or use custom user parameters.
printf("OnSynthesisTaskFailed: %s\n", tmpParam->binAudioFile.c_str());
// The task ID of the current task.
printf("OnSynthesisTaskFailed: status code=%d, task id=%s, error message=%s\n",
cbEvent->getStatusCode(), cbEvent->getTaskId(), cbEvent->getErrorMessage());
}
/**
* @brief After the text is sent to the server, the SDK's internal thread reports the received binary audio data to the user through a BinaryDataRecved event.
* @param cbEvent The callback event structure. For more information, see nlsEvent.h.
* @param cbParam Custom callback parameter. The default is NULL. You can customize parameters as needed.
* @return
* @notice Do not perform blocking operations here. Only transfer the audio data. Performing too many operations in this callback will block subsequent data callbacks and the completed event callback.
*/
void OnBinaryDataRecved(NlsEvent* cbEvent, void* cbParam) {
ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
// Example of how to print or use custom user parameters.
printf("OnBinaryDataRecved: %s\n", tmpParam->binAudioFile.c_str());
const std::vector<unsigned char>& data = cbEvent->getBinaryData(); // getBinaryData(): Gets the binary audio data from text synthesis.
printf("OnBinaryDataRecved: status code=%d, task id=%s, data size=%d\n",
cbEvent->getStatusCode(), cbEvent->getTaskId(), data.size());
// Append the binary audio data to the file.
if (data.size() > 0) {
tmpParam->audioFile.write((char*)&data[0], data.size());
}
}
/**
* @brief Returns log information corresponding to the TTS text and incrementally returns the corresponding caption information.
* @param cbEvent The callback event structure. For more information, see nlsEvent.h.
* @param cbParam Custom callback parameter. The default is NULL. You can customize parameters as needed.
* @return
*/
void OnMetaInfo(NlsEvent* cbEvent, void* cbParam) {
ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
// Example of how to print or use custom user parameters.
printf("OnBinaryDataRecved: %s\n", tmpParam->binAudioFile.c_str());
printf("OnMetaInfo: task id=%s, respose=%s\n", cbEvent->getTaskId(), cbEvent->getAllResponse());
}
/**
* @brief When recognition ends or an exception occurs, the connection channel is closed, and the SDK's internal thread reports a ChannelCloseed event.
* @param cbEvent The callback event structure. For more information, see nlsEvent.h.
* @param cbParam Custom callback parameter. The default is NULL. You can customize parameters as needed.
* @return
*/
void OnSynthesisChannelClosed(NlsEvent* cbEvent, void* cbParam) {
ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
// Example of how to print or use custom user parameters.
printf("OnSynthesisChannelClosed: %s\n", tmpParam->binAudioFile.c_str());
printf("OnSynthesisChannelClosed: %s\n", cbEvent->getAllResponse());
// Notify the sending thread that the final recognition result has been returned and stop() can be called.
pthread_mutex_lock(&(tmpParam->mtxWord));
pthread_cond_signal(&(tmpParam->cvWord));
pthread_mutex_unlock(&(tmpParam->mtxWord));
}
/**
* @brief Worker thread in short-connection mode.
* It loops through the following sequence:
* createSynthesizerRequest <----|
* | |
* request->start() |
* | |
* Receive OnSynthesisChannelClosed callback |
* | |
* releaseSynthesizerRequest(request) ----|
*/
void* pthreadFunc(void* arg) {
// 0: Get parameters such as the token and configuration files from the custom thread parameters.
ParamStruct* tst = (ParamStruct*)arg;
if (tst == NULL) {
std::cout << "arg is not valid." << std::endl;
return NULL;
}
// 1: Initialize custom callback parameters.
ParamCallBack* cbParam = new ParamCallBack();
cbParam->binAudioFile = tst->audioFile;
cbParam->audioFile.open(cbParam->binAudioFile.c_str(), std::ios::binary | std::ios::out);
/*
* 1. Create a SpeechSynthesizerRequest object.
*
* By default, this is a real-time short-text-to-speech synthesis request, which supports synthesizing up to 300 characters at a time.
* One Chinese character, one English letter, or one punctuation mark is counted as one character.
* An error will be reported (or the text will be truncated) for content exceeding 300 characters.
* For synthesizing more than 300 characters at once, consider the long-text-to-speech synthesis feature.
*
* For more information about real-time short-text-to-speech synthesis, see: https://help.aliyun.com/document_detail/84435.html
* For more information about long-text-to-speech synthesis, see: https://help.aliyun.com/document_detail/130509.html
*/
TtsVersion tts_version = AlibabaNls::ShortTts;
int chars_cnt = NlsClient::getInstance()->calculateUtf8Chars(tst->text.c_str());
if (chars_cnt > 300) {
tts_version = AlibabaNls::LongTts;
}
// 2: Create a SpeechSynthesizerRequest object.
SpeechSynthesizerRequest* request =
NlsClient::getInstance()->createSynthesizerRequest(tts_version);
if (request == NULL) {
printf("createSynthesizerRequest failed.\n");
cbParam->audioFile.close();
delete cbParam;
return NULL;
}
// Set the callback function for when speech synthesis is complete.
request->setOnSynthesisCompleted(OnSynthesisCompleted, cbParam);
// Set the callback function for when the speech synthesis channel is closed.
request->setOnChannelClosed(OnSynthesisChannelClosed, cbParam);
// Set the callback function for exceptions.
request->setOnTaskFailed(OnSynthesisTaskFailed, cbParam);
// Set the callback function for receiving text audio data.
request->setOnBinaryDataReceived(OnBinaryDataRecved, cbParam);
// Set the caption information.
request->setOnMetaInfo(OnMetaInfo, cbParam);
request->setAppKey(tst->appkey.c_str());
// Set the authentication token. This is a required parameter.
request->setToken(tst->token.c_str());
// Set the text to be synthesized. This is a required parameter. The text must be UTF-8 encoded.
// For synthesizing more than 300 characters at once, consider the long-text-to-speech synthesis feature.
// For more information about long-text-to-speech synthesis, see: https://help.aliyun.com/document_detail/130509.html
request->setText(tst->text.c_str());
// The speaker, including "xiaoyun", "ruoxi", "xiaogang", etc. This is an optional parameter. The default is xiaoyun.
request->setVoice("siqi");
// To access a personalized voice, the Voice must be a custom voice.
//request->setPayloadParam("{\"enable_ptts\":true}");
// The volume, in the range of 0 to 100. This is an optional parameter. The default is 50.
request->setVolume(50);
// The audio encoding format. This is an optional parameter. The default is wav. Supported formats are pcm, wav, and mp3.
request->setFormat("wav");
// The audio sampling rate, including 8000 and 16000. This is an optional parameter. The default is 16000.
request->setSampleRate(16000);
// The speech rate, in the range of -500 to 500. This is an optional parameter. The default is 0.
request->setSpeechRate(0);
// The pitch rate, in the range of -500 to 500. This is an optional parameter. The default is 0.
request->setPitchRate(0);
// Enable captions.
request->setEnableSubtitle(true);
// 3: start() is an asynchronous operation. On success, it starts returning BinaryRecv events. On failure, it returns a TaskFailed event.
int ret = request->start();
if (ret < 0) {
printf("start() failed. may be can not connect server. please check network or firewalld\n");
NlsClient::getInstance()->releaseSynthesizerRequest(request); // start() failed, release the request object.
cbParam->audioFile.close();
delete cbParam;
return NULL;
}
struct timeval now;
struct timespec outtime;
// 4: Notify the cloud that data sending is complete.
// stop() is a meaningless interface. The process will run to completion whether it is called or not. You must wait for the closed event callback in both cases.
// cancel() stops the work immediately and does not return a callback. On failure, it returns a TaskFailed event.
//ret = request->cancel();
ret = request->stop();
if (ret == 0) {
printf("wait closed callback.\n");
// The voice server might not process the current request in time and might not return any callback within 10 seconds.
// It might then return a TaskFailed callback after 10 seconds. Therefore, a timeout mechanism is required.
gettimeofday(&now, NULL);
outtime.tv_sec = now.tv_sec + 30;
outtime.tv_nsec = now.tv_usec * 1000;
// Wait for the closed event before releasing, otherwise a crash may occur.
pthread_mutex_lock(&(cbParam->mtxWord));
if (ETIMEDOUT == pthread_cond_timedwait(&(cbParam->cvWord), &(cbParam->mtxWord), &outtime)) {
printf("synthesis timeout.\n");
}
pthread_mutex_unlock(&(cbParam->mtxWord));
}
NlsClient::getInstance()->releaseSynthesizerRequest(request);
cbParam->audioFile.close();
delete cbParam;
return NULL;
}
// Synthesize a single piece of text data.
int speechSynthesizerFile(const char* appkey) {
// Get the current system timestamp to check if the token has expired.
std::time_t curTime = std::time(0);
if (g_expireTime - curTime < 10) {
printf("the token will be expired, please generate new token by AccessKey-ID and AccessKey-Secret.\n");
if (generateToken(g_akId, g_akSecret, &g_token, &g_expireTime) < 0) {
return -1;
}
}
ParamStruct pa;
pa.token = g_token;
pa.appkey = appkey;
pa.text = "The weather is great today, perfect for an outdoor trip.";
pa.audioFile = "syAudio.wav";
// Start a worker thread for a single recognition task.
pthread_t pthreadId;
pthread_create(&pthreadId, NULL, &pthreadFunc, (void *)&pa);
pthread_join(pthreadId, NULL);
return 0;
}
// Synthesize multiple pieces of text data.
// In the multi-threaded SDK, one thread corresponds to one piece of text data, not multiple threads to one piece of text data.
// The sample code starts two threads to synthesize two files simultaneously.
// Free-tier users cannot have more than two concurrent connections.
#define AUDIO_TEXT_NUMS 2
#define AUDIO_TEXT_LENGTH 64
#define AUDIO_FILE_NAME_LENGTH 32
int speechTranscriberMultFile(const char* appkey) {
// Get the current system timestamp to check if the token has expired.
std::time_t curTime = std::time(0);
if (g_expireTime - curTime < 10) {
printf("the token will be expired, please generate new token by AccessKey-ID and AccessKey-Secret.\n");
if (generateToken(g_akId, g_akSecret, &g_token, &g_expireTime) < 0) {
return -1;
}
}
const char syAudioFiles[AUDIO_TEXT_NUMS][AUDIO_FILE_NAME_LENGTH] =
{
"syAudio0.wav",
"syAudio1.wav"
};
const char texts[AUDIO_TEXT_NUMS][AUDIO_TEXT_LENGTH] =
{
"The weather is really nice today, I want to go play football.",
"There's a heavy storm tomorrow, I'd rather stay home and watch a movie."
};
ParamStruct pa[AUDIO_TEXT_NUMS];
for (int i = 0; i < AUDIO_TEXT_NUMS; i ++) {
pa[i].token = g_token;
pa[i].appkey = appkey;
pa[i].text = texts[i];
pa[i].audioFile = syAudioFiles[i];
}
std::vector<pthread_t> pthreadId(AUDIO_TEXT_NUMS); // Start worker threads to recognize audio files simultaneously.
for (int j = 0; j < AUDIO_TEXT_NUMS; j++) {
pthread_create(&pthreadId[j], NULL, &pthreadFunc, (void *)&(pa[j]));
}
for (int j = 0; j < AUDIO_TEXT_NUMS; j++) {
pthread_join(pthreadId[j], NULL);
}
return 0;
}
int main(int argc, char* argv[]) {
printf("Usage: ./demo <your appkey> <your AccessKey ID> <your AccessKey Secret>\n");
std::string appkey = getenv("NLS_APPKEY_ENV");
g_akId = getenv("NLS_AK_ENV");
g_akSecret = getenv("NLS_SK_ENV");
// Set the SDK output log as needed. This is optional.
// This indicates that the SDK log is output to log-synthesizer.txt.
// LogDebug means output logs of all levels. It supports LogDebug, LogInfo, LogWarning, and LogError.
// 400 means 400 MB per file. 50 means 50 log files are recorded in a loop.
int ret = NlsClient::getInstance()->setLogConfig(
"log-synthesizer", LogDebug, 400, 50);
if (ret < 0) {
printf("set log failed.\n");
return -1;
}
// Set the socket address type required by the runtime environment. The default is AF_INET.
// Must be called before startWorkThread().
//AlibabaNls::NlsClient::getInstance()->setAddrInFamily("AF_INET");
// For private cloud deployments, you can set a direct IP connection.
// Must be called before startWorkThread().
//NlsClient::getInstance()->setDirectHost("106.15.83.44");
// Some devices may still be unable to obtain a usable IP through the SDK's DNS even after setting the DNS.
// You can call this interface to actively enable the system's getaddrinfo to solve this problem.
//NlsClient::getInstance()->setUseSysGetAddrInfo(true);
// Start the worker thread. This function must be called before creating and starting a request. It can be considered the initialization of NlsClient.
// If the input parameter is negative, it starts the number of available cores in the current system.
// For concurrency below 200, an input parameter of 1 is recommended. For higher concurrency, see the readme for recommendations.
NlsClient::getInstance()->startWorkThread(1);
// Synthesize a single piece of text.
speechSynthesizerFile(appkey.c_str());
// Synthesize multiple pieces of text.
//speechSynthesizerMultFile(appkey.c_str());
// After all work is complete, release nlsClient before the process exits.
// Note that releaseInstance() is not thread-safe. Ensure that all requests have stopped before releasing.
NlsClient::releaseInstance();
return 0;
}