This topic describes how to implement a Real-time Conversational AI solution using the Alibaba Real-Time Communication (ARTC) software development kit (SDK).
Introduction
This solution uses the ARTC SDK to build a Real-Time Communication (RTC) network and implements real-time interaction by calling Real-time Conversational AI API operations. This provides an efficient and flexible integration path. You can select API operations as needed to provide multiple artificial intelligence interaction experiences, such as intelligent conversation, sentiment analysis, matchmaking assistants, and digital human live streaming. You can also perform custom development and extend features for various application scenarios to improve the user experience.
Architecture
The following figure shows the system architecture for ARTC-based calls between users and AI agents.
Agent integration
Prerequisites
You have created an agent. For more information, see Quick Start for Real-time Conversational AI.
You need to integrate the Alibaba Real-Time Communication (ARTC) SDK. For instructions, see Getting Started with ApsaraVideo Real-time Communication.
Start and end a call
To start a call with an AI agent, follow these steps:
A user sends a request from the application to the AppServer to start a call and obtain an RTC token.
After the AppServer receives the request, it calls the StartAIAgentInstance operation to start the agent and generates an RTC token based on the rules in Token-based authentication.
The AppServer returns the call result and the RTC token to the user.
The user uses the ARTC SDK to join the corresponding RTC channel with the RTC token and starts the call with the agent.
To end a call with an AI agent, follow these steps:
A user sends a request to the AppServer to end the call. The AppServer calls the StopAIAgentInstance operation to stop the AI agent.
The user uses the ARTC SDK to leave the channel and end the session.
Development Reference
You can implement features by referring to the AICallKit integration solution. For client sample code, see Android source code and iOS source code. For server-side sample code, see Server source code.
Feature implementation
The following features are implemented in the audio and video call agent integration solution. You can also use standard ARTC SDK API operations.
Walkie-talkie mode
Prerequisites
You have activated an ApsaraVideo Real-time Communication application. For more information, see Getting Started with ApsaraVideo Real-time Communication.
You have enabled the RTC custom message channel. For more information, see Send and receive custom messages.
Interaction flow
Message protocol
Enable or disable walkie-talkie mode
Message type: 1105
The message body is as follows:
{
"type": 1105,
"senderId": "user_1", // Sender ID
"receiverId": "robot_1", // Agent ID
"data": {
"enable": true // true: enables walkie-talkie mode. false: disables walkie-talkie mode.
}
}Transmit walkie-talkie mode status
Message type: 1007
The message body is as follows:
{
"type": 1007,
"seqId": 5,
"senderId": "robot_1", // AI agent UID
"receiverId": "", // The recipient ID does not need to be specified and is usually empty.
"data": {
"enable": true // true: the agent has entered walkie-talkie mode. false: the agent has exited walkie-talkie mode.
}
}The agent service sends this message to notify you of the result, regardless of whether walkie-talkie mode was enabled or disabled through an OpenAPI call or a DataChannel message.
Press to start speaking
Message type: 1106
The message body is as follows:
{
"type": 1106,
"senderId": "user_1", // Sender ID
"receiverId": "robot_1" // Agent ID
}Release to send speech
Message type: 1107
The message body is as follows:
{
"type": 1107,
"senderId": "user_1", // Sender ID
"receiverId": "robot_1" // Agent ID
}Release to cancel speaking
Message type: 1108
The message body is as follows:
{
"type": 1108,
"senderId": "user_1", // Sender ID
"receiverId": "robot_1" // Agent ID
}Code examples
iOS
// Enable or disable walkie-talkie mode.
public func enablePushToTalk(enable: Bool) {
var sendDict: [String: Any] = [
"type": 1105,
]
sendDict.updateValue(myUid, forKey: "senderId")
sendDict.updateValue(robotUid, forKey: "receiverId")
sendDict.updateValue(["enable": enable], forKey: "data")
if let sendData = sendDict.aicall_jsonString.data(using: .utf8) {
let rtcMsg = AliRtcDataChannelMsg()
rtcMsg.type = .custom
rtcMsg.data = sendData
self.rtcEngine.sendDataChannelMessage(rtcMsg)
self.rtcEngine.muteLocalMic(enable, mode: .allAudioMode)
}
}
// Press and hold to speak. Make sure walkie-talkie mode is enabled.
public func startPushToTalk() {
var sendDict: [String: Any] = [
"type": 1106,
]
sendDict.updateValue(myUid, forKey: "senderId")
sendDict.updateValue(robotUid, forKey: "receiverId")
if let sendData = sendDict.aicall_jsonString.data(using: .utf8) {
let rtcMsg = AliRtcDataChannelMsg()
rtcMsg.type = .custom
rtcMsg.data = sendData
self.rtcEngine.sendDataChannelMessage(rtcMsg)
// Unmute the microphone.
self.rtcEngine.muteLocalMic(false, mode: .allAudioMode)
}
}
// Finish speaking. Make sure walkie-talkie mode is enabled.
public func finishPushToTalk() {
var sendDict: [String: Any] = [
"type": 1107,
]
sendDict.updateValue(myUid, forKey: "senderId")
sendDict.updateValue(robotUid, forKey: "receiverId")
if let sendData = sendDict.aicall_jsonString.data(using: .utf8) {
let rtcMsg = AliRtcDataChannelMsg()
rtcMsg.type = .custom
rtcMsg.data = sendData
self.rtcEngine.sendDataChannelMessage(rtcMsg)
// Mute the microphone.
self.rtcEngine.muteLocalMic(true, mode: .allAudioMode)
}
}
// Cancel this transmission. Make sure walkie-talkie mode is enabled.
public func cancelPushToTalk() {
var sendDict: [String: Any] = [
"type": 1108,
]
sendDict.updateValue(myUid, forKey: "senderId")
sendDict.updateValue(robotUid, forKey: "receiverId")
if let sendData = sendDict.aicall_jsonString.data(using: .utf8) {
let rtcMsg = AliRtcDataChannelMsg()
rtcMsg.type = .custom
rtcMsg.data = sendData
self.rtcEngine.sendDataChannelMessage(rtcMsg)
// Mute the microphone.
self.rtcEngine.muteLocalMic(true, mode: .allAudioMode)
}
}
// Process received DataChannel messages.
public func onDataChannelMessage(_ uid: String, controlMsg: AliRtcDataChannelMsg) {
if controlMsg.type != .custom {
return
}
let dataDict = (try? JSONSerialization.jsonObject(with: controlMsg.data, options: .allowFragments)) as? [String : Any]
guard let dataDict = dataDict else {
return
}
debugPrint("onDataChannelMessage:\(dataDict)")
if dataDict["type"] as? Int32 == 1007 {
let data = dataDict["data"] as? [String: Any]
if let data = data {
if let enable = data["enable"] as? Bool {
// The server has enabled or disabled walkie-talkie mode. Refer to the flow.
DispatchQueue.main.async {
self.enablePushToTalk = enable
self.rtcEngine.muteLocalMic(enable, mode: .allAudioMode)
}
}
}
}
// Process other messages.
...
}Android
public void sendCustomMessage(int msgType, JSONObject data) {
if (null != mAliRtcEngine) {
try {
String senderId; // Your user ID
String receiverId; // Agent user ID
JSONObject jsonObject = new JSONObject();
jsonObject.put("type", msgType);
jsonObject.put("senderId", senderId);
jsonObject.put("receiverId", receiverId);
if (null != data) {
jsonObject.put("data", data);
}
AliRtcEngine.AliRtcDataChannelMsg rtcDataChannelMsg = new AliRtcEngine.AliRtcDataChannelMsg();
rtcDataChannelMsg.type = AliEngineDataMsgCustom;
rtcDataChannelMsg.data = jsonObject.toString().getBytes(StandardCharsets.UTF_8);
mAliRtcEngine.sendDataChannelMsg(rtcDataChannelMsg);
} catch (JSONException ex) {
ex.printStackTrace();
}
}
}
// Enable or disable walkie-talkie mode.
public boolean enablePushToTalk(boolean enable) {
JSONObject jsonObject = new JSONObject();
jsonObject.put("enable", enable);
// Send a message to the agent.
sendCustomMessage(1105, jsonObject);
// Mute the microphone.
mAliRtcEngine.muteLocalMic(enable, AliRtcEngine.AliRtcMuteLocalAudioMode.AliRtcMuteAllAudioMode);
}
// Press and hold to speak. Make sure walkie-talkie mode is enabled.
public func startPushToTalk() {
// Send a message to the agent.
sendCustomMessage(1106, null);
// Unmute the microphone.
mAliRtcEngine.muteLocalMic(false, AliRtcEngine.AliRtcMuteLocalAudioMode.AliRtcMuteAllAudioMode);
}
// Finish speaking. Make sure walkie-talkie mode is enabled.
public func finishPushToTalk() {
// Send a message to the agent.
sendCustomMessage(1107, null);
// Mute the microphone.
mAliRtcEngine.muteLocalMic(true, AliRtcEngine.AliRtcMuteLocalAudioMode.AliRtcMuteAllAudioMode);
}
// Cancel this transmission. Make sure walkie-talkie mode is enabled.
public func cancelPushToTalk() {
// Send a message to the agent.
sendCustomMessage(1108, null);
// Mute the microphone.
mAliRtcEngine.muteLocalMic(true, AliRtcEngine.AliRtcMuteLocalAudioMode.AliRtcMuteAllAudioMode);
}
// Process received DataChannel messages.
private AliRtcEngineNotify mRtcEngineRemoteNotify = new AliRtcEngineNotify() {
@Override
public void onDataChannelMessage(String uid, AliRtcEngine.AliRtcDataChannelMsg msg) {
super.onDataChannelMessage(uid, msg);
if (msg.type == AliEngineDataMsgCustom) {
String dataStr = new String(msg.data);
JSONObject jsonObject = new JSONObject(dataStr);
JSONObject dataJson = jsonObject.optJSONObject("data");
int msgType = jsonObject.optInt("type");
if (msgType == 1007) {
boolean enable = dataJson.optBoolean("enable");
// The agent notifies the walkie-talkie mode status. Handle business logic...
}
}
}
}Web
// Enable or disable walkie-talkie mode.
public enablePushToTalk(enable: boolean) { }
// Press and hold to speak. Make sure walkie-talkie mode is enabled.
public startPushToTalk(): boolean { }
// Finish speaking. Make sure walkie-talkie mode is enabled.
public finishPushToTalk(): boolean { }
// Cancel this transmission. Make sure walkie-talkie mode is enabled.
public cancelPushToTalk(): boolean { }
controller.on('AICallPushToTalkChanged', (enable: boolean) => {
// The current walkie-talkie mode has changed to {enable}.
})