C++ SDK

Quick start

Prepare a Linux build environment, obtain the SDK, configure credentials, and run the complete example. For request parameters and response events, see Real-time speech recognition API reference.

The discontinued API 2.0 has different definitions from API 3.1 used in this example.

Prepare the environment

The minimum Linux build tool versions are CMake 3.0, glibc 2.5, and GCC 4.8.5.

The example uses Linux x86_64, GCC 10.2.1, and glibc 2.32.

Obtain and install the SDK

git clone --depth 1 https://github.com/aliyun/alibabacloud-nls-cpp-sdk

./scripts/build_linux.sh

Configure credentials

Before running the example, set the following environment variables for the process. The AppKey and AccessKey must belong to the account used to access the service. Do not hard-code credentials or write them to logs.

export NLS_APPKEY_ENV='<appkey>'
export NLS_AK_ENV='<accesskey-id>'
export NLS_SK_ENV='<accesskey-secret>'

When running the example in an IDE, also set these variables in its run configuration. The example obtains a token through NlsToken, caches the token and its expiration time, and refreshes it shortly before expiration. Concurrent requests can share a valid token.

Sample code

The complete example initializes the client, obtains a token, handles asynchronous events, sends audio, and releases resources. Use test0.wav from the SDK package and place it in the working directory. The concurrent example also uses test1.wav.

The sample audio has a sample rate of 16000 Hz, and the project uses the Universal model in the console. For other audio, select a model that supports the audio's use case. For model configuration, see Manage projects.

The example waits for the started event before sending audio, stops sending after a failure event, and propagates failures as a nonzero process exit code. On success, TranscriptionCompleted is followed by the channel-closed event.

#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include <stdlib.h>
#include <ctime>
#include <string>
#include <iostream>
#include <vector>
#include <fstream>
#include <sys/time.h>
#include <errno.h>
#include <atomic>
#include "nlsClient.h"
#include "nlsEvent.h"
#include "nlsToken.h"
#include "speechTranscriberRequest.h"

#define FRAME_SIZE 3200
#define SAMPLE_RATE 16000

using namespace AlibabaNlsCommon; 
using AlibabaNls::NlsClient; 
using AlibabaNls::NlsEvent; 
using AlibabaNls::LogDebug; 
using AlibabaNls::LogInfo; 
using AlibabaNls::LogError;
using AlibabaNls::SpeechTranscriberRequest;

struct ParamStruct {
  std::string fileName;
  std::string appkey; 
  std::string token;
  int result = -1; 
};

struct ParamCallBack {
 public:
  ParamCallBack() {
    pthread_mutex_init(&mtxWord, NULL);
    pthread_cond_init(&cvWord, NULL);
  };
  ~ParamCallBack() {
    pthread_mutex_destroy(&mtxWord);
    pthread_cond_destroy(&cvWord);
  };

  std::atomic<bool> started{false}, closed{false}, failed{false}, completed{false};
  int userId;
  char userInfo[8];
  pthread_mutex_t mtxWord;
  pthread_cond_t cvWord;
};

std::string g_akId = "";
std::string g_akSecret = "";
std::string g_token = "";
long g_expireTime = -1;
int g_sync_timeout = 0;
struct timeval tv;
struct timeval tv1;

int generateToken(std::string akId, std::string akSecret,
                  std::string* token, long* expireTime) {
  NlsToken nlsTokenRequest;
  nlsTokenRequest.setAccessKeyId(akId); 
  nlsTokenRequest.setKeySecret(akSecret); 

  int ret = nlsTokenRequest.applyNlsToken();
  if (ret < 0) {

    printf("generateToken Failed, error code:%d msg:%s\n",
        ret, nlsTokenRequest.getErrorMsg());
    return ret;
  }
  *token = nlsTokenRequest.getToken();
  *expireTime = nlsTokenRequest.getExpireTime();
  return 0;
}

unsigned int getSendAudioSleepTime(int dataSize,
                                   int sampleRate,
                                   int compressRate) { 

  const int sampleBytes = 16;     
  const int soundChannel = 1;     
  int bytes = (sampleRate * sampleBytes * soundChannel) / 8;     
  int bytesMs = bytes / 1000;     
  int sleepMs = (dataSize * compressRate) / bytesMs;
  return sleepMs; 
}

void onTranscriptionStarted(NlsEvent* cbEvent, void* cbParam) {
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;

  printf("onTranscriptionStarted: %d\n", tmpParam->userId);

  printf("onTranscriptionStarted: status code=%d, task id=%s\n",
      cbEvent->getStatusCode(), cbEvent->getTaskId());

  pthread_mutex_lock(&(tmpParam->mtxWord));
  tmpParam->started = true;
  pthread_cond_signal(&(tmpParam->cvWord));
  pthread_mutex_unlock(&(tmpParam->mtxWord));
} 

void onSentenceBegin(NlsEvent* cbEvent, void* cbParam) {
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;

  printf("onSentenceBegin: %d\n", tmpParam->userId);
  printf("onSentenceBegin: status code=%d, task id=%s, index=%d, time=%d\n",
      cbEvent->getStatusCode(), cbEvent->getTaskId(),
      cbEvent->getSentenceIndex(), 
      cbEvent->getSentenceTime() 
  );

}

void onSentenceEnd(NlsEvent* cbEvent, void* cbParam) {
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;

  printf("onSentenceEnd: %d\n", tmpParam->userId);
  printf("onSentenceEnd: status code=%d, task id=%s, index=%d, time=%d, begin_time=%d, result=%s\n",
      cbEvent->getStatusCode(),
      cbEvent->getTaskId(),
      cbEvent->getSentenceIndex(), 
      cbEvent->getSentenceTime(), 
      cbEvent->getSentenceBeginTime(), 
      cbEvent->getResult()    
  );

}

void onTranscriptionResultChanged(NlsEvent* cbEvent, void* cbParam) {
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;

  printf("onTranscriptionResultChanged: %d\n", tmpParam->userId);
  printf("onTranscriptionResultChanged: status code=%d, task id=%s, index=%d, time=%d, result=%s\n",
      cbEvent->getStatusCode(),
      cbEvent->getTaskId(),
      cbEvent->getSentenceIndex(), 
      cbEvent->getSentenceTime(), 
      cbEvent->getResult()    
  );

}

void onTranscriptionCompleted(NlsEvent* cbEvent, void* cbParam) {
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
  tmpParam->completed = true;

  printf("onTranscriptionCompleted: %d\n", tmpParam->userId);
  printf("onTranscriptionCompleted: status code=%d, task id=%s\n",
      cbEvent->getStatusCode(),
      cbEvent->getTaskId());
}

void onTaskFailed(NlsEvent* cbEvent, void* cbParam) { 
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;
  pthread_mutex_lock(&tmpParam->mtxWord);
  tmpParam->failed = true;
  pthread_cond_broadcast(&tmpParam->cvWord);
  pthread_mutex_unlock(&tmpParam->mtxWord); 

  printf("onTaskFailed: %d\n", tmpParam->userId); 
  printf("onTaskFailed: status code=%d, task id=%s, error message=%s\n", 
      cbEvent->getStatusCode(), 
      cbEvent->getTaskId(), 
      cbEvent->getErrorMessage()
  );     

}

void onChannelClosed(NlsEvent* cbEvent, void* cbParam) {     
  ParamCallBack* tmpParam = (ParamCallBack*)cbParam;     

  printf("onChannelClosed: %d, %s\n", tmpParam->userId, tmpParam->userInfo); 
  printf("onChannelClosed: response=%s\n", cbEvent->getAllResponse());

  pthread_mutex_lock(&(tmpParam->mtxWord));
  tmpParam->closed = true;
  pthread_cond_broadcast(&(tmpParam->cvWord));
  pthread_mutex_unlock(&(tmpParam->mtxWord));
}

void* pthreadFunction(void* arg) {
  int sleepMs = 0;
  int ret = 0;
  ParamCallBack *cbParam = NULL;

  cbParam = new ParamCallBack();
  cbParam->userId = rand() % 100;
  strcpy(cbParam->userInfo, "User.");

  ParamStruct *tst = (ParamStruct *) arg; 
  if (tst == NULL) {
    printf("arg is not valid\n"); 
    delete cbParam;
    return NULL;
  } 

  std::ifstream fs;
  fs.open(tst->fileName.c_str(), std::ios::binary | std::ios::in);
  if (!fs) {
    printf("Cannot open %s\n", tst->fileName.c_str());
    delete cbParam;
    return NULL;
  }
  
  SpeechTranscriberRequest* request =
      NlsClient::getInstance()->createTranscriberRequest();
  if (request == NULL) {
    printf("createTranscriberRequest failed\n");
    delete cbParam;
    return NULL;
  }

  request->setOnTranscriptionStarted(onTranscriptionStarted, cbParam);

  request->setOnTranscriptionResultChanged(onTranscriptionResultChanged, cbParam);

  request->setOnTranscriptionCompleted(onTranscriptionCompleted, cbParam);

  request->setOnSentenceBegin(onSentenceBegin, cbParam);

  request->setOnSentenceEnd(onSentenceEnd, cbParam);

  request->setOnTaskFailed(onTaskFailed, cbParam);

  request->setOnChannelClosed(onChannelClosed, cbParam);

  request->setAppKey(tst->appkey.c_str());

  request->setToken(tst->token.c_str());

  request->setUrl("wss://nls-gateway-cn-shanghai.aliyuncs.com/ws/v1");
  request->setFormat("opus");

  request->setSampleRate(SAMPLE_RATE);

  request->setIntermediateResult(true);

  request->setPunctuationPrediction(true);

  request->setInverseTextNormalization(true);

  struct timespec outtime;
  struct timeval now;
  
  ret = request->start();
  if (ret < 0) {
    printf("start() failed. may be can not connect server. please check network or firewalld\n");
    NlsClient::getInstance()->releaseTranscriberRequest(request); 
    delete cbParam;
    return NULL;
  } else {
    if (g_sync_timeout == 0) {

      printf("wait started callback.\n");

      gettimeofday(&now, NULL);
      outtime.tv_sec = now.tv_sec + 5;
      outtime.tv_nsec = now.tv_usec * 1000;
      pthread_mutex_lock(&(cbParam->mtxWord));
      while (!cbParam->started && !cbParam->failed && !cbParam->closed) {
      if (ETIMEDOUT == pthread_cond_timedwait(&(cbParam->cvWord), &(cbParam->mtxWord), &outtime)) {
        printf("start timeout.\n");
        pthread_mutex_unlock(&(cbParam->mtxWord));
        request->cancel();
        NlsClient::getInstance()->releaseTranscriberRequest(request);
        delete cbParam;
        return NULL;
      }
      }
      pthread_mutex_unlock(&(cbParam->mtxWord));
    } else {
      
    }
  }

  while (cbParam->started && !cbParam->failed && !cbParam->closed && !fs.eof()) {
    uint8_t data[FRAME_SIZE] = {0};
    fs.read((char *) data, sizeof(uint8_t) * FRAME_SIZE); 
    size_t nlen = fs.gcount();
    if (nlen <= 0) {
      continue;
    }

    ret = request->sendAudio(data, nlen, ENCODER_OPUS); 
    if (ret < 0) {

      printf("send data fail.\n");
      cbParam->failed = true; 
      break;
    } 

    sleepMs = getSendAudioSleepTime(nlen, SAMPLE_RATE, 1); 
    
    usleep(sleepMs * 1000);
  } 

  printf("sendAudio done.\n");

  fs.close();

  ret = cbParam->failed || cbParam->closed ? -1 : request->stop();
  if (ret == 0) {
      if (g_sync_timeout == 0) {

        printf("wait closed callback.\n");

        gettimeofday(&now, NULL);
        outtime.tv_sec = now.tv_sec + 5;
        outtime.tv_nsec = now.tv_usec * 1000;

        pthread_mutex_lock(&(cbParam->mtxWord));
        while (!cbParam->closed) {
        if (ETIMEDOUT == pthread_cond_timedwait(&(cbParam->cvWord), &(cbParam->mtxWord), &outtime)) {
          printf("stop timeout\n");
          pthread_mutex_unlock(&(cbParam->mtxWord));
          NlsClient::getInstance()->releaseTranscriberRequest(request);
          delete cbParam;
          return NULL;
        }
        }
        pthread_mutex_unlock(&(cbParam->mtxWord));
      } else {
        
      }
  } else {
    printf("stop ret is %d\n", ret);
    cbParam->failed = true;
    if (!cbParam->closed) request->cancel();
  }

  NlsClient::getInstance()->releaseTranscriberRequest(request);
  tst->result = (!cbParam->failed && cbParam->completed) ? 0 : -1;
  delete cbParam;
  return NULL;
}

int speechTranscriberFile(const char* appkey) {

  std::time_t curTime = std::time(0);
  if (g_expireTime - curTime < 10) {
    printf("the token will be expired, please generate new token by AccessKey-ID and AccessKey-Secret.\n");
    if (generateToken(g_akId, g_akSecret, &g_token, &g_expireTime) < 0) {
      return -1;
    } 
  }

  ParamStruct pa;
  pa.token = g_token;
  pa.appkey = appkey;
  pa.fileName = "test0.wav";

  pthread_t pthreadId;
  if (pthread_create(&pthreadId, NULL, &pthreadFunction, (void *)&pa) != 0) return -1;
  pthread_join(pthreadId, NULL);
  return pa.result;
}

#define AUDIO_FILE_NUMS 2
#define AUDIO_FILE_NAME_LENGTH 32
int speechTranscriberMultFile(const char* appkey) {

  std::time_t curTime = std::time(0);
  if (g_expireTime - curTime < 10) {
    printf("the token will be expired, please generate new token by AccessKey-ID and AccessKey-Secret.\n");
    if (generateToken(g_akId, g_akSecret, &g_token, &g_expireTime) < 0) {
      return -1;
    }
  }

  char audioFileNames[AUDIO_FILE_NUMS][AUDIO_FILE_NAME_LENGTH] = {
      "test0.wav", 
      "test1.wav"
  };
  ParamStruct pa[AUDIO_FILE_NUMS];
  for (int i = 0; i < AUDIO_FILE_NUMS; i ++) {
    pa[i].token = g_token; 
    pa[i].appkey = appkey;
    pa[i].fileName = audioFileNames[i]; 
  } 

  std::vector<pthread_t> pthreadId(AUDIO_FILE_NUMS);
  int created = 0;
  for (; created < AUDIO_FILE_NUMS; ++created) {
    if (pthread_create(&pthreadId[created], NULL, &pthreadFunction, &pa[created]) != 0) break;
  }
  int result = created == AUDIO_FILE_NUMS ? 0 : -1;
  for (int j = 0; j < created; ++j) {
    pthread_join(pthreadId[j], NULL);
    if (pa[j].result != 0) result = -1;
  }
  return result;
}

int main(int argc, char* argv[]) {
  const char* appkeyEnv = getenv("NLS_APPKEY_ENV");
  const char* akEnv = getenv("NLS_AK_ENV");
  const char* skEnv = getenv("NLS_SK_ENV");
  if (!appkeyEnv || !*appkeyEnv || !akEnv || !*akEnv || !skEnv || !*skEnv) {
    std::cerr << "Set NLS_APPKEY_ENV, NLS_AK_ENV, and NLS_SK_ENV." << std::endl;
    return EXIT_FAILURE;
  }
  std::string appkey = appkeyEnv;
  g_akId = akEnv;
  g_akSecret = skEnv;

  int ret = NlsClient::getInstance()->setLogConfig(
      "log-transcriber", LogError, 10, 3);
  if (ret < 0) {
    printf("set log failed.\n");
    NlsClient::releaseInstance();
    return EXIT_FAILURE;
  }

  if (g_sync_timeout > 0) {
    NlsClient::getInstance()->setSyncCallTimeout(g_sync_timeout);
  }

  NlsClient::getInstance()->startWorkThread(1);

  int result = speechTranscriberFile(appkey.c_str());
  // To transcribe two files concurrently, replace the preceding call with:
  // int result = speechTranscriberMultFile(appkey.c_str());

  NlsClient::releaseInstance();

  return result == 0 ? EXIT_SUCCESS : EXIT_FAILURE;
}

Add the SDK include and lib directories to the project's compiler and linker settings. With NlsSdk3.X_LINUX in the current directory, compile the example file demo.cpp with the following command:

g++ -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1 demo.cpp   -I NlsSdk3.X_LINUX/include -L NlsSdk3.X_LINUX/lib   -lalibabacloud-idst-speech -lpthread -o demo
export LD_LIBRARY_PATH="$PWD/NlsSdk3.X_LINUX/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

Usage notes

• The example calculates sending intervals for 16000 Hz, 16-bit, mono audio. For 8 kHz PCM, wait 100 ms after sending 1600 bytes; for 16 kHz PCM, wait 100 ms after sending 3200 bytes. In OPUS/OPU mode, the input to the SDK is still PCM. The recommendation is 640 bytes followed by a 20 ms wait. Send live recordings at the capture rate; rate limiting is needed when simulating a live recording with a file.

• sendAudio: stop sending audio if the return value is negative. After TranscriptionCompleted or TaskFailed, the SDK closes the channel. Do not continue sending audio.

• In asynchronous mode, a return from start() does not indicate successful startup. Wait for a started or failure event. A return from stop() does not indicate that the interaction has ended. Wait for the channel-closed event. Call setSyncCallTimeout to enable synchronous mode.

• SDK multithreading uses one thread per audio source, not multiple threads sending the same source. The example provides single-file and two-file concurrent entry points. Release the global client only after all requests have ended. releaseInstance() is not thread-safe.

Free users cannot exceed two concurrent connections. For fewer than 200 concurrent requests, one worker thread is recommended. See the SDK readme.md for recommendations at higher concurrency.

The sentence silence threshold is the silence duration after an utterance. Its valid range is 200–2000 ms, with a default of 800 ms. Semantic sentence detection overrides silence-based sentence detection and requires intermediate results to be enabled.

Use setPayloadParam to pass custom or advanced parameters as a JSON string, such as {"vad_model":"farfield"}.

request->setUrl("ws://nls-gateway-cn-shanghai-internal.aliyuncs.com/ws/v1");

Key interfaces

SDK package files

• scripts/build_linux.sh: sample Linux build script in the SDK source.

• CMakeLists.txt: CMake project file for the Linux SDK example.

• demo: integration examples in the SDK package. The Linux examples are listed below.

  File	Description
  speechRecognizerDemo.cpp	Short sentence recognition example.
  speechSynthesizerDemo.cpp	Speech synthesis example.
  speechTranscriberDemo.cpp	Real-time speech recognition example.
  fileTransferDemo.cpp	Recording file recognition example.

• resource: sample audio in the SDK source for functional testing.

  File	Description
  test0.wav test1.wav test2.wav test3.wav	Test audio files with a 16 kHz sample rate and 16-bit samples.

• include: SDK header files, as listed below.

  File	Description
  nlsClient.h	SDK client instance.
  nlsEvent.h	Callback events.
  nlsGlobal.h	Global SDK header.
  nlsToken.h	SDK access token instance.
  iNlsRequest.h	Base NLS request header.
  speechRecognizerRequest.h	Short sentence recognition.
  speechSynthesizerRequest.h	Speech synthesis and long-text speech synthesis.
  speechTranscriberRequest.h	Real-time audio stream recognition.
  FileTrans.h	Recording file recognition.

• lib: SDK libraries.

• readme.md: SDK instructions.

• release.log: Release notes.

• version: Version number.

C++ SDK error codes

Server response status codes

For service status codes, see API reference.

FAQ