Stream ingest from an iOS client

Prerequisites

• Create a real-time recording and obtain an ingest URL

• Install the iOS Push SDK

Key SDK interfaces

• nui_initialize: Initializes the SDK.

  /**
   * Initializes the SDK. The SDK is a singleton. Release the SDK before you initialize it again. Do not call this method in the UI thread to prevent blocking.
   * @param parameters: The initialization parameters. For more information, see the following description.
   * @param level: The log level. A smaller value indicates that more logs are printed.
   * @param save_log: Specifies whether to save logs as files. The files are stored in the directory specified by the debug_path field in the parameters. Note: Log files are not size-limited. Continuous storage may cause the disk to become full.
   * @return For more information, see the error codes.
   */
  -(NuiResultCode) nui_initialize:(const char *)parameters
                         logLevel:(NuiSdkLogLevel)level
                          saveLog:(BOOL)save_log;

  • Parameter details:

    Parameter	Type	Required	Description
    workspace	String	Yes	The path of the working directory. The SDK reads configuration files from this directory.
    app_key	String	Yes	Set this parameter to "default".
    token	String	Yes	Set this parameter to "default".
    url	String	Yes	The MeetingJoinUrl that is returned when a Tingwu real-time recording task is created is the URL to which you push the audio stream for real-time recognition.
    service_mode	String	Yes	Set this parameter to "1" to enable the online feature.
    device_id	String	Yes	The device ID. This ID uniquely identifies a device, such as a MAC address, serial number (SN), or a unique pseudo ID.
    debug_path	String	No	The debug directory. If the save_log parameter is set to true during SDK initialization, log files are saved to this directory.
    save_wav	String	No	This parameter takes effect only when the save_log parameter is set to true during SDK initialization. It specifies whether to save debug audio files. The files are saved to the debug directory. Make sure the debug_path is valid and writable. Note: The audio files are not size-limited. Continuous storage may cause the disk to become full.

• nui_set_params: Sets the SDK parameters in JSON format.

  /**
   * Sets parameters in JSON format. Call this interface after nui_initialize and before nui_dialog_start.
   * @param params: For more information about the parameters, see the following description.
   * @return For more information, see the error codes.
   */
  -(NuiResultCode) nui_set_params:(const char *)params;

  • Parameter details:

    Parameter	Type	Required	Description
    service_type	Int	Yes	Set this parameter to "4". This specifies the type of voice service to request. The value for Tingwu real-time stream ingest is "4".
    nls_config	JsonObject	Yes	The parameter settings for accessing the Voice Service. For more information, see the following details.
    nls_config.sr_format	String	Yes	The value must be 'pcm'. For the corresponding CreateTask - Create a Tingwu task operation, you must also set the encoding format of the audio stream to 'pcm'.
    nls_config.sample_rate	Integer	Yes	The audio sampling rate. The default value is 16000 Hz. When you create a Tingwu task by calling the CreateTask - Create a Tingwu task operation, you must also specify the sample rate of the audio stream. The supported values are 8000 and 16000.

• nui_dialog_start: Starts recognition.

  /**
   * Starts recognition.
   * @param vad_mode: Multiple modes are available. For recognition scenarios, use P2T.
   * @param dialog_params: Sets recognition parameters. You can leave this empty by passing an empty JSON string.
   * @return For more information, see the error codes.
   */
  -(NuiResultCode) nui_dialog_start:(NuiVadMode)vad_mode
                        dialogParam:(const char *)dialog_params;

• nui_dialog_cancel: Stops recognition.

  /**
   * Stops recognition. After this interface is called, the server returns the final recognition result and ends the task.
   * @param force: Specifies whether to forcibly end the task and ignore the final result. A value of false indicates that the task is stopped but waits for the complete result to be returned.
   * @return For more information, see the error codes.
   */
  -(NuiResultCode) nui_dialog_cancel:(BOOL)force;

• nui_release: Releases the SDK resources.

  /**
   * Releases SDK resources.
   * @return For more information, see the error codes.
   */
  -(NuiResultCode) nui_release;

• nui_get_version: Retrieves the current SDK version information.

  /**
   * Gets the current SDK version information.
   * @return The SDK version information as a string.
   */
  -(const char*) nui_get_version;

• nui_get_all_response: Retrieves the complete information of the current event callback.

  /**
   * Gets the complete information of the current event callback in the onNuiEventCallback.
   * @return The complete event information as a JSON string.
   */
  -(const char*) nui_get_all_response;

• NeoNuiSdkDelegate

  onNuiEventCallback: The event callback for the SDK.

  /**
   * The main SDK event callback.
   * @param event: The callback event. For more information, see the event list below.
   * @param dialog: The session ID. Not currently in use.
   * @param wuw: Used for voice wake-up. Not currently supported.
   * @param asr_result: The speech recognition and translation results.
   * @param finish: A flag indicating whether the current round of recognition is complete.
   * @param resultCode: For more information, see the error codes. This is valid when an EVENT_ASR_ERROR event occurs.
   */
  -(void) onNuiEventCallback:(NuiCallbackEvent)nuiEvent
                      dialog:(long)dialog
                   kwsResult:(const char *)wuw
                   asrResult:(const char *)asr_result
                    ifFinish:(BOOL)finish
                     retCode:(int)code;

  List of NuiCallbackEvent events:

  Name	Description
  EVENT_VAD_START	Detects the start of human speech.
  EVENT_VAD_END	Detects the end of human speech.
  EVENT_ASR_PARTIAL_RESULT	Intermediate speech recognition result.
  EVENT_ASR_RESULT	Final speech recognition result.
  EVENT_ASR_ERROR	Determines the cause of the error based on the error code information.
  EVENT_MIC_EEROR	Recording error. This indicates that the SDK has not received any audio for 2 consecutive seconds. Check if the recording system is working correctly.
  EVENT_SENTENCE_START	Real-time speech recognition event. Detects the start of a sentence.
  EVENT_SENTENCE_END	Real-time speech recognition event. Detects the end of a sentence and returns the complete result for that sentence.
  EVENT_SENTENCE_SEMANTICS	Not in use.
  EVENT_RESULT_TRANSLATED	Translation result.
  EVENT_TRANSCRIBER_COMPLETE	Reported after speech recognition is stopped.

  onNuiNeedAudioData: Retrieves audio data.

  /**
   * When recognition starts, this callback is called continuously. The app needs to fill in the audio data in the callback.
   * @param audioData:  The storage area for the audio data.
   * @param len: The number of bytes of audio data to fill.
   * @return The actual number of bytes filled.
   */
  -(int) onNuiNeedAudioData:(char *)audioData length:(int)len;

  onNuiAudioStateChanged: Enables or disables the recording feature based on the audio state.

  /**
   * When interfaces such as start, stop, or cancel are called, the SDK notifies the app through this callback to turn the recording on or off.
   * @param state: The required state for recording (open or close).
   */
  -(void) onNuiAudioStateChanged:(NuiAudioState)state;

  onNuiRmsChanged: The audio energy event callback.

  /**
   * The real-time audio energy value of the audio received during SDK runtime.
   * @param rms: The audio energy value, ranging from -160 to 0.
   */
  -(void) onNuiRmsChanged:(float) rms;

Procedure

1. Initialize the SDK and the recording instance.

2. Configure the parameters as required.

3. Call nui_dialog_start to start recognition.

4. Enable the recorder based on the onNuiAudioStateChanged audio state callback.

5. Provide the recording data in the onNuiNeedAudioData callback.

6. Obtain the recognition result from the EVENT_ASR_PARTIAL_RESULT and EVENT_SENTENCE_END event callbacks. Obtain the translation result from the EVENT_RESULT_TRANSLATED event callback.

7. Call nui_dialog_cancel to stop recognition.

8. After the process is complete, call the nui_release interface to release the SDK resources.

Code examples

NUI SDK initialization

BOOL save_log = NO;
NSString * initParam = [self genInitParams];
[_nui nui_initialize:[initParam UTF8String] logLevel:NUI_LOG_LEVEL_VERBOSE saveLog:save_log];

The genInitParams method generates a JSON string that contains the resource directory and user information. The user information includes the following fields.

-(NSString*) genInitParams {
    NSString *strResourcesBundle = [[NSBundle mainBundle] pathForResource:@"Resources" ofType:@"bundle"];
    NSString *bundlePath = [[NSBundle bundleWithPath:strResourcesBundle] resourcePath];
    NSString *debug_path = [_utils createDir];

    // 1. Interfaces and implementation: https://help.aliyun.com/zh/tingwu/interface-and-implementation?spm=a2c4g.11186623.0.0.2b045060jFsVFw
    //    Follow the steps in the document to first create an AccessKey and a project.
    //    Then, you need to call the CreateTask interface on your server to create a real-time recording and obtain the MeetingJoinUrl.
    //    This MeetingJoinUrl is the url below.
    NSMutableDictionary *dictM = [NSMutableDictionary dictionary];

    // Account and project creation
    //  For information about how to obtain an ak_id and ak_secret, see https://help.aliyun.com/document_detail/72138.html
    [dictM setObject:@"default" forKey:@"app_key"]; // Required. Do not change.
    [dictM setObject:@"default" forKey:@"token"]; // Required. Do not change.

    // Set url to the generated MeetingJoinUrl.
    // Because the MeetingJoinUrl generation process involves an AK/SK, account information cannot be stored on the mobile client. Therefore, it must be generated on the server and sent to the mobile client.
    // For more information, see: https://help.aliyun.com/zh/tingwu/api-tingwu-2023-09-30-createtask
    [dictM setObject:@"wss://tingwu-realtime-cn-hangzhou-pre.aliyuncs.com/api/ws/v1?XXXX" forKey:@"url"]; // Required
    // The path of the working directory. The SDK reads configuration files from this directory.
    [dictM setObject:bundlePath forKey:@"workspace"]; // Required

    // This parameter takes effect only when the save_log parameter is set to true during SDK initialization. It specifies whether to save debug audio files. The files are saved to the debug directory. Make sure the debug_path is valid and writable.
    [dictM setObject:save_wav ? @"true" : @"false" forKey:@"save_wav"];
    // The debug directory. If the save_log parameter is set to true during SDK initialization, this directory is used to save intermediate audio files.
    [dictM setObject:debug_path forKey:@"debug_path"];
    
    // FullCloud = 1 // Select this for online real-time speech recognition.
    [dictM setObject:@"1" forKey:@"service_mode"]; // Required. Do not change.

    NSString *id_string = [[[ASIdentifierManager sharedManager] advertisingIdentifier] UUIDString];
    TLog(@"id: %s", [id_string UTF8String]);
    [dictM setObject:id_string forKey:@"device_id"]; // Required. We recommend that you enter a unique ID to help locate issues.

    NSData *data = [NSJSONSerialization dataWithJSONObject:dictM options:NSJSONWritingPrettyPrinted error:nil];
    NSString * jsonStr = [[NSString alloc]initWithData:data encoding:NSUTF8StringEncoding];
    return jsonStr;
}

Parameter settings

Set the parameters as a JSON string.

-(NSString*) genParams {
    NSMutableDictionary *nls_config = [NSMutableDictionary dictionary];
    [nls_config setValue:@16000 forKey:@"sample_rate"];
    [nls_config setValue:@"pcm" forKey:@"sr_format"];
    NSMutableDictionary *dictM = [NSMutableDictionary dictionary];
    [dictM setObject:nls_config forKey:@"nls_config"];
    [dictM setValue:@(SERVICE_TYPE_SPEECH_TRANSCRIBER) forKey:@"service_type"]; // Required
    
    NSData *data = [NSJSONSerialization dataWithJSONObject:dictM options:NSJSONWritingPrettyPrinted error:nil];
    NSString * jsonStr = [[NSString alloc]initWithData:data encoding:NSUTF8StringEncoding];
    return jsonStr;
}

NSString * parameters = [self genParams];
[_nui nui_set_params:[parameters UTF8String]];

Start recognition

Use the nui_dialog_start interface to start the listener.

-(NSString*) genDialogParams {
    NSMutableDictionary *dialog_params = [NSMutableDictionary dictionary];    
    NSData *data = [NSJSONSerialization dataWithJSONObject:dialog_params options:NSJSONWritingPrettyPrinted error:nil];
    NSString * jsonStr = [[NSString alloc]initWithData:data encoding:NSUTF8StringEncoding];
    return jsonStr;
}

NSString * parameters = [self genDialogParams];
[_nui nui_dialog_start:MODE_P2T dialogParam:[parameters UTF8String]];

Callback handling

• onNuiAudioStateChanged: The recording state callback. The SDK maintains the recording state internally. Use this callback to enable or disable the recorder based on the state.

  -(void)onNuiAudioStateChanged:(NuiAudioState)state{
      TLog(@"onNuiAudioStateChanged state=%u", state);
      if (state == STATE_CLOSE || state == STATE_PAUSE) {
          // The recording module provided in the old version of the sample project is for reference only. You can rewrite the recording module based on your business needs.
          // [_voiceRecorder stop:YES];
  
          // The new version of the sample project provides a new recording module for reference only. You can rewrite the recording module based on your business needs.
          [_audioController stopRecorder:NO];
      } else if (state == STATE_OPEN){
          self.recordedVoiceData = [NSMutableData data];
          // The recording module provided in the old version of the sample project is for reference only. You can rewrite the recording module based on your business needs.
          // [_voiceRecorder start];
  
          // The new version of the sample project provides a new recording module for reference only. You can rewrite the recording module based on your business needs.
          [_audioController startRecorder];
      }
  }

• onNuiNeedAudioData: The recording data callback. Provide the recording data in this callback.

  -(int)onNuiNeedAudioData:(char *)audioData length:(int)len {
      static int emptyCount = 0;
      @autoreleasepool {
          @synchronized(_recordedVoiceData){
              if (_recordedVoiceData.length > 0) {
                  int recorder_len = 0;
                  if (_recordedVoiceData.length > len)
                      recorder_len = len;
                  else
                      recorder_len = _recordedVoiceData.length;
                  NSData *tempData = [_recordedVoiceData subdataWithRange:NSMakeRange(0, recorder_len)];
                  [tempData getBytes:audioData length:recorder_len];
                  tempData = nil;
                  NSInteger remainLength = _recordedVoiceData.length - recorder_len;
                  NSRange range = NSMakeRange(recorder_len, remainLength);
                  [_recordedVoiceData setData:[_recordedVoiceData subdataWithRange:range]];
                  emptyCount = 0;
                  return recorder_len;
              } else {
                  if (emptyCount++ >= 50) {
                      TLog(@"_recordedVoiceData length = %lu! empty 50times.", (unsigned long)_recordedVoiceData.length);
                      emptyCount = 0;
                  }
                  return 0;
              }
  
          }
      }
      return 0;
  }

• onNuiEventCallback: The NUI SDK event callback. Do not call other SDK interfaces from within this event callback to prevent deadlocks.

  -(void)onNuiEventCallback:(NuiCallbackEvent)nuiEvent
                     dialog:(long)dialog
                  kwsResult:(const char *)wuw
                  asrResult:(const char *)asr_result
                   ifFinish:(bool)finish
                    retCode:(int)code {
      TLog(@"onNuiEventCallback event %d finish %d", nuiEvent, finish);
      if (nuiEvent == EVENT_ASR_PARTIAL_RESULT || nuiEvent == EVENT_SENTENCE_END) {
          // Here, asr_result contains the task_id. The task_id helps troubleshoot issues. Record and save it.
          TLog(@"ASR RESULT %s finish %d", asr_result, finish);
          NSString *result = [NSString stringWithUTF8String:asr_result];
      } else if (nuiEvent == EVENT_RESULT_TRANSLATED) {
          // Get the translation result.
          NSString *result = [NSString stringWithUTF8String:asr_result];
      } else if (nuiEvent == EVENT_ASR_ERROR) {
          // In EVENT_ASR_ERROR, asr_result is the error message. Using it with the error code and the task_id makes it easier to troubleshoot issues. Record and save them.
          TLog(@"EVENT_ASR_ERROR error[%d], error mesg[%s]", code, asr_result);
  
          // You can get complete information through nui_get_all_response.
          const char *response = [_nui nui_get_all_response];
          if (response != NULL) {
              TLog(@"GET ALL RESPONSE: %s", response);
          }
      } else if (nuiEvent == EVENT_MIC_ERROR) {
          TLog(@"MIC ERROR");
          // The recording module provided in the old version of the sample project is for reference only. You can rewrite the recording module based on your business needs.
          // [_voiceRecorder stop:YES];
          // [_voiceRecorder start];
  
          // The new version of the sample project provides a new recording module for reference only. You can rewrite the recording module based on your business needs.
          [_audioController stopRecorder:NO];
          [_audioController startRecorder];
      }
  
      // If finish is true (which may indicate an error or the completion of recognition), it means a task lifecycle has ended, and you can start a new recognition.
      if (finish) {
      }
      
      return;
  }

Stop recognition

[_nui nui_dialog_cancel:NO];