Linux C++ SDK

This topic describes how to use the real-time multimodal interaction Linux C++ SDK provided by Alibaba Cloud Model Studio, including SDK download and installation, key interfaces, and code examples.

Prerequisites

Enable an Alibaba Cloud Model Studio real-time multimodal interaction application to get your workspace ID, app ID, and API key.

Multimodal real-time interaction service architecture

Download and install

Audio format specifications

Audio format specifications for WebSocket connections:

• Uplink: Supports PCM and Opus audio formats for speech recognition. Only PCM audio (16 kHz sample rate, 16-bit, single channel) can be sent by users. The SDK automatically encodes Opus audio internally.
• Downlink: Supports single-channel PCM, MP3, and Opus audio streams. Opus audio streams are decoded internally by the SDK, and users receive the corresponding PCM data.

Interface reference

CreateConversation

Create an interaction and set callbacks. The corresponding destruction interface is DestroyConversation.

/**
 * @brief Create an interaction and set callbacks. The corresponding destruction interface is DestroyConversation.
 * @param onMessage : Callbacks for listening to events. See specific callbacks below.
 * @param onEventTrackMessage : Sends internal SDK logs (info level or higher) to the upper layer through a callback.
 * @param user_data : User data returned in callbacks.
 * @return Conversation* : Pointer to the Conversation singleton.
 */
static Conversation* CreateConversation(
    const ConversationCallbackMethod onMessage,
    const EventTrackCallbackMethod onEventTrackMessage,
    void* user_data = NULL);

The ConversationCallbackMethod type is defined as follows:

ConversationCallbackMethod

SDK event callback

/**
 * @brief Callback for various SDK events.
 * @param ConvEvent*
 * : The callback event. Call its specific methods to obtain event details and related information. See ConvEvent below.
 * @param void* : User-defined user_data.
 */
typedef void (*ConversationCallbackMethod)(ConvEvent*, void*);

ConvEvent

Method list

ConvEventType

NetworkEventType

The EventTrackCallbackMethod type is defined as follows:

EventTrackCallbackMethod

Obtain key instrumentation log information

/**
 * @brief Internal SDK logs are sent to the user through a callback.
 * @param ConvLogLevel : Log level of the returned log.
 * @param const char* : Specific log string.
 * @param void* : User-defined user_data.
 */
typedef void (*EventTrackCallbackMethod)(ConvLogLevel, const char*, void*);

enum ConvLogLevel {
  kConvLogLevelVerbose,
  kConvLogLevelDebug,
  kConvLogLevelInfo,
  kConvLogLevelWarning,
  kConvLogLevelError,
  kConvLogLevelNone,
};

DestroyConversation

Destroy the interaction. The corresponding creation interface is CreateConversation.

/**
 * @brief Destroy the interaction. If the interaction is ongoing, Disconnect() is called internally.
 * @return ConvRetCode : Status code.
 */
ConvRetCode DestroyConversation();

Connect

Assemble a start request to initiate a connection to the server. A successful return does not mean the connection to the VoiceChat Server succeeded—it only means the start request was built successfully. Check the response to confirm if the connection succeeded.

/**
 * @brief Establish a connection to the server.
 * @param params : Initialization parameters in JSON string format.
 * @return ConvRetCode : Status code.
 */
ConvRetCode Connect(const char* params);

Model Studio Parameter Settings Instructions

{
  "word_list": [
    {
      "activation": true,
      "name": "小白小白",
      "type": "wakeup",
      "threshold1": 0.01
    },
    {
      "activation": false,
      "name": "小云小云",
      "type": "wakeup"
    }
  ]
}

{
  "nls_rule": {
    "header": [
      "Start_custom1"
    ],
    "payload":[
      "Start_task_group",
      "Start_task"
    ]
  }
}

{
	"nls_params": {
		"Start_custom1": {
			"test": 1
		},
		"Start_task_group": "aigc",
		"Start_task": "multimodal-generation",
		"Start_function": "generation",
		"Start_model": "multimodal-dialog"
	}
}

Disconnect

Assemble a stop request to send an instruction telling the server to terminate the connection. A successful return does not mean the disconnection from the VoiceChat Server succeeded—it only means the stop request was built successfully. Check the response to confirm if the disconnection succeeded.

/**
 * @brief Disconnect the connection.
 * @return ConvRetCode : Status code.
 */
ConvRetCode Disconnect();

Interrupt

Interrupt the interaction to put the AI into listening state.

/**
 * @brief Button (Tap) interrupt. Call this interface to request interruption while playing.
 * @return ConvRetCode : Status code.
 */
ConvRetCode Interrupt();

SendAudioData

Push real-time captured audio data.

/**
 * @brief Push audio data.
 * @param data : Audio data in PCM format.
 * @param data_size : Size of audio data in bytes.
 * @param type : Audio data encoding setting. Currently unused and has no effect.
 * @param timestamp : Timestamp of this data packet (ms).
 * @return ConvRetCode : Status code.
 */
ConvRetCode SendAudioData(const uint8_t* data, size_t data_size,
                          EncoderType type = kEncoderNone,
                          uint64_t timestamp = 0);

SendRefData

Push reference audio data when sending audio data to the player.

/**
 * @brief Push reference audio data when sending audio data to the player.
 * @param data : Audio data in PCM format.
 * @param data_size : Size of audio data in bytes.
 * @param timestamp : Timestamp of this data packet (ms).
 * @return ConvRetCode : Status code.
 */
ConvRetCode SendRefData(const uint8_t* data, size_t data_size,
                          uint64_t timestamp = 0);

SendResponseData

In Listening or Idle state, notify the server to interact proactively with the user. You can either convert uploaded text directly into speech for delivery or upload text to call the LLM and deliver the result as speech.

/**
 * @brief
 * Notify the server to interact proactively with the user. You can either convert uploaded text directly into speech for delivery or upload text to call the LLM and deliver the result as speech.
 * @param params : Initialization parameters in JSON string format.
 * @return ConvRetCode : Status code.
 */
ConvRetCode SendResponseData(const char* params);

parameters settings

parameters examples

Switch to video mode

{
	"type": "prompt",
	"parameters": {
		"biz_params": {
			"videos": [{
				"action": "connect",
				"type": "voicechat_video_channel"
			}]
		}
	}
}

Exit video mode

{
	"type": "prompt",
	"parameters": {
		"biz_params": {
			"videos": [{
				"action": "exit",
				"type": "voicechat_video_channel"
			}]
		}
	}
}

Send image for Q&A

# Send image as base64 data (supports images smaller than 180 KB)
{
	"type": "prompt",
	"parameters": {
		"images": [{
			"type": "base64",
			"value": "base64String"
		}]
	}
}

# Send image URL
{
	"type": "prompt",
	"parameters": {
		"images": [{
			"type": "url",
			"value": "https://xxxxxxx"
		}]
	}
}

UpdateMessage

Update parameters, such as P2T parameters, or send parameters like UpdateInfo.

/**
 * @brief Send parameters such as those related to P2T.
 * @param params : Parameters.
 * @return ConvRetCode : Status code.
 */
ConvRetCode UpdateMessage(const char* params);

Update custom wake words (only valid for kws_duplex)

{
	"type": "update_local_modules",
	"work": "update_keywords_list",
	"wuw": {
		"word_list": [{
			"name": "Xiao Yun",
			"type": "main"
		}]
	}
}

Update other parameters

{"type":"update_nls_parameters", "avatar":{}}
{"type":"update_nls_parameters", "client_info":{}}
{"type":"update_nls_parameters", "agent_chat":{}}

// [Added in SDK V1.2.4]
{"type":"update_nls_parameters", "extra_info":{}, "context":{}}

Universal method for updating parameters

For custom server interaction parameters, use "type": "update_custom_nls_parameters" to enable custom parameter functionality.

nls_rule specifies where to add parameters in the protocol. For example, "payload": ["Start_custom2", "SendSpeech_custom3"] means that when you send the "Start" protocol to the server, the parameter "custom2":{"test":2} is added to the payload. Similarly, "header": ["Start_custom1"] means that when you send the "Start" protocol to the server, the parameter "custom1":{"test":1} is added to the header.

{
	"type": "update_custom_nls_parameters",
	"nls_rule": {
		"header": ["Start_custom1"],
		"payload": ["Start_custom2", "SendHumanSpeech_custom3"],
		"context": ["Stop_custom4"]
	},
	"nls_params": {
		"Start_custom1": {
			"test": 1
		},
		"Start_custom2": {
			"test": 2
		},
       "SendHumanSpeech_custom3": {
			"test": 3
		},
		"Stop_custom4": {
			"test": 4
		}
	}
}

Send parameters (UpdateInfo)

Client update events, such as sending image data.

Model Studio protocol parameters

Model Studio protocol example

{
    "type": "update_info",
	"parameters": {
		"biz_params": {},
		"client_info": {
			"status": {
				"bluetooth_announcement": {
					"status": "stopped"
				}
			}
		},
		"images": [{
			"type": "base64",
			"value": "base64String"
		}]
	}
}

Send universal parameters

{
	"type": "update_custom_message",
	"header": {
		"TEST1": "XXX"
	},
	"payload": {
		"TEST2": "YYY"
	}
}
Corresponding response message:
{
	"header": {
		"TEST1": "XXX",
		"dialog_id": "2b9bea3d2b2749a0808bae06a9da493c",
		"namespace": "VoiceChat",
		"request_id": "c48cfe7bbadd485db2e387136ce6608c",
		"task_id": "fffffff707fe4c288869931b0eeeeeee"
	},
	"payload": {
		"TEST2": "YYY"
	}
}

SetAction

Notify the SDK about audio (playback) start or end events.

  /**
   * @brief Trigger an SDK action, such as notifying the SDK about audio (playback) start/end events.
   * @param action : Type of action to trigger.
   * @param data : Data accompanying this SDK action, such as audio data or a JSON string.
   * @param data_size : Size of the accompanying data in bytes.
   * @return ConvRetCode : Status code.
   */
  ConvRetCode SetAction(ConvAction action, const uint8_t* data = NULL,
                        size_t data_size = 0);

ConvAction

{
	"keyword_str": "Xiao Yun",
	"keyword_tag": "other_keyword"
}

GetState

Get current states.

/**
 * @brief Get current state.
 * @param type : State type.
 * @return ConvRetCode : Status code.
 */
int GetState(StateType type);

StateType

DialogState

GetResponse

Feed VoiceChat-related responses obtained via RTC into the SDK. The response is parsed internally by the VideoChat Native SDK and converted into interaction states and related events for the app.

/**
 * @brief
 * Feed VoiceChat-related responses obtained via RTC into the SDK. The response is parsed internally by the VideoChat Native
 * SDK and converted into interaction states and related events for the app.
 * @param params : Initialization parameters in JSON string format.
 * @return ConvRetCode : Status code.
 */
ConvRetCode GetResponse(const char* params);

Additionally, requests such as AvatarStateChanged can be assembled and processed via updateMessage(). You can also use this getResponse(), with the following format:

{"type":"rtc","from_state":"2","seq_id":"1","to_state":"0"}
or
{"from_state":"2","seq_id":"1","to_state":"0"}
or
{"from_state":2,"seq_id":1,"to_state":0}
or
{"header":{"name":"AvatarStateChanged","namespace":"Conversation"},"payload":{"from_state":2,"seq_id":2,"to_state":0}}
or
{"header":{"name":"AvatarStateChanged","namespace":"Conversation"},"payload":{"from_state":"2","seq_id":"2","to_state":"0"}}

Code examples

Voice dialog initialization

This step sets up callback functions to receive all events during the interaction, including dialog state and results.

void onMessage(ConvEvent* event, void* param) {
  ConvEvent::ConvEventType event_type = event->GetMsgType();
  int dialog_state = event->GetDialogStateChanged();

  if (event_type != ConvEvent::kSoundLevel &&
      event_type != ConvEvent::kBinary) {
    std::cout << "trigger onMessage -->>\n"
              << "  session id: " << event->GetSessionId() << "\n"
              << "  messge type: " << event->GetMsgTypeString() << "\n"
              << "  dialog state: " << dialog_state << "\n"
              << "  response: " << event->GetAllResponse() << std::endl;
  }

  switch (event_type) {
    case ConvEvent::kConversationFailed:
      // Dialog error occurred
      break;
    case ConvEvent::kConversationStarted:
      // Dialog connection succeeded
      break;
    case ConvEvent::kConversationCompleted:
      break;
    case ConvEvent::kSentenceBegin:
      break;
    case ConvEvent::kSentenceEnd:
      break;
    case ConvEvent::kDataOutputStarted:
      // Will receive TTS data next. Start the player here.
      // After starting the player, notify the SDK.
      break;
    case ConvEvent::kDataOutputCompleted:
      // Finished receiving TTS data. Notify the player that data sending is complete.
      // Note: This only means TTS data reception is complete, not playback completion. The cache or player still has a lot of data to play.
      // After full playback, you must notify the SDK.
      break;
    case ConvEvent::kBinary:
      break;
    case ConvEvent::kSoundLevel:
      break;
    case ConvEvent::kDialogStateChanged:
      // Perform business logic based on dialog state.
      break;
    case ConvEvent::kInterruptAccepted:
      // Interruption allowed. Close currently playing audio here.
      break;
    case ConvEvent::kInterruptDenied:
      break;
    case ConvEvent::kVoiceInterruptAccepted:
      // Interruption allowed. Close currently playing audio here.
      break;
    case ConvEvent::kVoiceInterruptDenied:
      break;
    case ConvEvent::kHumanSpeakingDetail:
      break;
    case ConvEvent::kRespondingDetail:
      break;
  }
}

void onEtMessage(ConvLogLevel level, const char* log, void* user_data) {
  std::cout << " ==>> [" << level << "] " << log << std::endl;
}

ConversationCallbackMethod on_message_callback = onMessage;
EventTrackCallbackMethod on_et_callback = onEtMessage;
Conversation* conversation = Conversation::CreateConversation(on_message_callback,
                                                              on_et_callback, NULL);

Initiate voice dialog connection

Build connection parameters, including account information, in JSON format. For details, see the interface documentation. In PushToTalk mode, after a successful connection, the dialog state is Idle.

std::string gen_init_params() {
  Json::FastWriter writer;
  Json::Value root;
  Json::Value agent_chat;
  root["mode"] = g_mode;
  root["chain_mode"] = "ws";
  root["ws_version"] = 3;

  root["debug_mode"] = "xxxxxxx";
  root["workspace"] = "xxxxxxx";
  root["save_log"] = true;
  root["save_wav"] = true;
  root["log_level"] = g_log_level;

  root["url"] = "";
  root["apikey"] = "";
  root["app_id"] = "";
  root["workspace_id"] = "";

  Json::Value upstream;
  upstream["type"] = "AudioOnly";
  upstream["audio_format"] = "opus";  // ASR format. Supports pcm, opus.
  root["upstream"] = upstream;

  Json::Value downstream;
  downstream["type"] = "Audio";
  downstream["voice"] = "longxiaoxia_v2";
  downstream["sample_rate"] = 24000;  // Default TTS sample rate is 24000.
  downstream["audio_format"] = "pcm";  // Downlink audio encoding format. Supports opus, pcm.
  downstream["intermediate_text"] = "transcript,dialog";
  downstream["debug"] = true;
  root["downstream"] = downstream;

  Json::Value client_info; /* Additional user information */
  client_info["user_id"] = "bin23439207";
  root["client_info"] = client_info;

  Json::Value dialog_attributes;
  dialog_attributes["prompt"] = "You are a helpful assistant.";
  root["dialog_attributes"] = dialog_attributes;

  Json::Value acoustic_echo_cancelling_attributes;
  acoustic_echo_cancelling_attributes["enable_external_aec_module"] = false;
  root["acoustic_echo_cancelling_attributes"] =
      acoustic_echo_cancelling_attributes;

  Json::Value voice_activity_detection_attributes;
  voice_activity_detection_attributes["enable_external_vad_module"] = false;
  root["voice_activity_detection_attributes"] =
      voice_activity_detection_attributes;

  Json::Value key_words_spotting_attributes;
  key_words_spotting_attributes["enable_external_kws_module"] = false;
  key_words_spotting_attributes["independent_kws_mode"] = false;
  root["key_words_spotting_attributes"] = key_words_spotting_attributes;

  std::string result = writer.write(root);
  std::cout << "init params: " << result.c_str() << std::endl;
  return result;
}

// Initiate connection. Check the return value to determine if the connection succeeded.
// If the connection fails, use the return value and callback information to diagnose.
ConvRetCode ret = conversation->Connect(gen_init_params().c_str());

End voice dialog

ConvRetCode ret = conversation->Disconnect();

Release voice dialog

conversation->DestroyConversation();
conversation = NULL;

Interrupt current voice dialog

Call this interface when the dialog is in Responding state to interrupt and enter Listening state.

ConvRetCode ret = conversation->Interrupt();

User starts speaking in PushToTalk mode

Call this interface when the dialog state is Idle to enter Listening state. If the dialog state is Responding or Thinking, it interrupts the current state and re-enters Listening. After a successful call, the user's speech is continuously sent to the server for interaction until stopped.

ConvRetCode ret = conversation->SetAction(kStartHumanSpeech);

User stops speaking in PushToTalk mode

The user's speech is continuously sent to the server for interaction until this interface is called. The AI then provides feedback based on what the user said.

ConvRetCode ret = conversation->SetAction(kStopHumanSpeech);

Text-to-speech synthesis

The SDK supports directly requesting the server to synthesize audio from text.

You must send the SendResponseData() request when the client is in Listening state.

If the current state is not Listening, first call the Interrupt() interface to interrupt the current playback.

Json::Value root;
root["text"] = "Happiness is a skill—the inner peace you achieve after letting go of unnecessary external desires.";
root["type"] = "transcript";

Json::StreamWriterBuilder writer;
writer["indentation"] = "";

ConvRetCode ret = conversation->SendResponseData(Json::writeString(writer, root).c_str());

VQA interaction: Send image for multimodal interaction

VQA enables multimodal interaction (image + voice) during a conversation by sending images.

The core process involves sending multimodal interaction commands via the SendResponseData() interface using voice or text.

Json::Value root;
root["text"] = "What's in the picture?";
root["type"] = "prompt";

Json::Value parameters;
Json::Value biz_params;

biz_params["user_defined_params"] = Json::Value(Json::objectValue);
Json::Value videos(Json::arrayValue);
{
    Json::Value video1;
    video1["action"] = "connect";// Enter video mode
    video1["type"] = "voicechat_video_channel";
    videos.append(video1);

    Json::Value video2;
    video2["action"] = "exit";// Exit video mode
    video2["type"] = "voicechat_video_channel";
    videos.append(video2);
}
biz_params["videos"] = videos;
parameters["biz_params"] = biz_params;

Json::Value images(Json::arrayValue);
{
    Json::Value image;
    // Example of sending image as base64 data
    // You can use the SDK's helper interface or implement Base64Encode yourself
    ConversationUtils utils;
    image["type"] = "base64";
    image["value"] = utils.Base64EncodeFromFilePath(path/* image path */);

    // Example of sending image URL
    image["type"] = "url";
    image["value"] = "https://xxxx";

    images.append(image);
}
parameters["images"] = images;

root["parameters"] = parameters;

Json::StreamWriterBuilder writer;
writer["indentation"] = "";
ConvRetCode ret = conversation->SendResponseData(Json::writeString(writer, root).c_str());

LiveAI interaction: Send image for multimodal interaction

To use this feature, set upstream.type to AudioAndVideo in the Connect() input parameters. Then, continuously send image data via the UpdateInfo interface to achieve multimodal interaction (image + voice). Supported image sizes are 720p and 480p. In video mode, send once every 500 ms.

// Example of sending image as base64 data
// You can use the SDK's helper interface or implement Base64Encode yourself
ConversationUtils utils;
std::string base64_content = utils.Base64EncodeFromFilePath(path/* image path */);
if (!base64_content.empty()) {
  Json::Value root;
  root["type"] = "update_info";
  Json::Value parameters;
  parameters["biz_params"] = Json::objectValue;

  Json::Value client_info;
  Json::Value status;
  Json::Value bluetooth_announcement;
  bluetooth_announcement["status"] = "stopped";
  status["bluetooth_announcement"] = bluetooth_announcement;
  client_info["status"] = status;
  parameters["client_info"] = client_info;

  Json::Value image;
  image["type"] = "base64";
  image["value"] = base64_content;
  image["width"] = 480;
  image["height"] = 720;
  Json::Value images(Json::arrayValue);
  images.append(image);
  parameters["images"] = images;
  root["parameters"] = parameters;

  Json::StreamWriterBuilder writer;
  writer["indentation"] = "";
  conversation->UpdateMessage(Json::writeString(writer, root).c_str());
}

Custom prompt variables and value passing

• Configure custom variables in the console project under Prompts.

Define a user_name field to represent the user's nickname. Insert the variable user_name as a placeholder ${user_name} into the Prompt.

In the console Prompts editor, click Custom Variables at the top to add variable fields. Reference them in the Prompt body using the ${variable_name} placeholder format.

• Set variables in code.

In the following example, set "user_name" = "Da Mi".

/* The following example adds biz_params to the <Initiate voice dialog connection> example */
std::string gen_init_params() {
  /* Other initialization parameters are omitted. See <Initiate voice dialog connection>. */

  Json::Value biz_params;
  Json::Value user_prompt_params;
  user_prompt_params["user_name"] = "Da Mi";
  biz_params["user_prompt_params"] = user_prompt_params;
  root["biz_params"] = biz_params;

  std::string result = writer.write(root);
  std::cout << "init params: " << result.c_str() << std::endl;
  return result;
}

// Initiate connection. Check the return value to determine if the connection succeeded.
// If the connection fails, use the return value and callback information to diagnose.
ConvRetCode ret = conversation->Connect(gen_init_params().c_str());

Custom dialog variables and value passing

• Configure custom variables in the console project under Dialog Variables.

Add a name variable to represent the person in front of you, and an emotion variable to represent the user's mood.

In the console Prompts editor, under the Dialog Variables panel, click the + Add button in the upper right corner. Enter variable names (such as name and emotion) and their descriptions (such as person in front of you and user's mood).

• Set variables in code.

In the following example, set "name" = "Zhang San" and "emotion" = "sad".

/* The following example adds biz_params to the <Initiate voice dialog connection> example */
std::string gen_init_params() {
  /* Other initialization parameters are omitted. See <Initiate voice dialog connection>. */

  Json::Value biz_params;
  Json::Value user_query_params;
  user_query_params["name"] = "Zhang San";
  user_query_params["emotion"] = "sad";
  biz_params["user_query_params"] = user_query_params;
  root["biz_params"] = biz_params;

  std::string result = writer.write(root);
  std::cout << "init params: " << result.c_str() << std::endl;
  return result;
}

// Initiate connection. Check the return value to determine if the connection succeeded.
// If the connection fails, use the return value and callback information to diagnose.
ConvRetCode ret = conversation->Connect(gen_init_params().c_str());

SDK error codes

Time series chart description

duplex

push2talk

tap2talk