Real-time Conversational AI FAQ

Features

• Do I need to deploy an agent on my origin server?

• How do I connect an agent to a large language model deployed on Alibaba Cloud Model Studio?

• Where do I view large language model token consumption?

Integration

• How do I pass custom parameters to a Model Studio application or workflow?

• Does the Web SDK support setVisionPreviewView?

• Can I switch to visual understanding during a call?

• How do I associate images with speech or text when I integrate a custom TTS service?

Audio configuration

• How do I adjust the client audio capture sample rate?

• How do I adjust the agent audio playback sample rate?

Billing

• Why am I charged for STT during Real-time Conversational AI debugging when the other party does not speak?

Troubleshooting

• Errors occur when starting a call

• The client reports an "AgentNotFound" error when starting a messaging conversation

• The client reports an "UnsupportedWorkflowType" error when starting a messaging conversation

• What do I do if the "Application not found or the application is disabled" error occurs during integration?

• How do I troubleshoot call and messaging connection errors?

• What do I do if a custom voice does not take effect?

• How do I troubleshoot missing agent callbacks?

Performance tuning

• How do I tune slow agent responses and premature interruptions?

• Why does the agent answer twice in walkie-talkie mode?

Device and platform support

• How do I integrate Real-time Conversational AI in WeChat H5 and mini programs?

• What input, output, and model capabilities does Real-time Conversational AI support?

Do I need to deploy an agent on my origin server?

No. The agent is a public cloud service provided by Alibaba Cloud, so no deployment on your side is required. You build and call agents by configuring them in the console and calling the OpenAPI.

How do I connect an agent to a large language model deployed on Alibaba Cloud Model Studio?

The agent is integrated with Alibaba Cloud Model Studio by design. Specify the required parameters in the console to call a large language model on Alibaba Cloud Model Studio. For more information, see Large Language Models (LLMs).

Where do I view large language model token consumption?

View large language model token consumption in the Model Studio console. Callback data does not include token consumption.

How do I pass custom parameters to a Model Studio application or workflow?

• Business parameters — When you pass parameters through the SDK, nest the business parameters under the biz_params field. For example, put them in the biz_params object of the Model Studio application parameters. Parameters passed at the top level do not take effect.

• Customer information — To pass customer information for each call, use UserData.

• Avatar, persona, and voice — The persona of a voice call is passed through to the Model Studio prompt. Pass parameters such as the avatar, persona, and voice dynamically through the agent configuration object when you start the call.

Does the Web SDK support setVisionPreviewView?

No. The setVisionPreviewView API is available only in the Android software development kit (SDK) and is not supported by the Real-time Conversational AI Web SDK. Use the public setAgentView API instead. The SDK handles remote view rendering configuration internally.

Can I switch to visual understanding during a call?

No. The agentType is set in callConfig during initialization and cannot be changed dynamically after a call is established. To switch from a voice-only call to visual understanding mode, end the current call and reconnect with agentType set to AICallAgentType.VisionAgent.

How do I associate images with speech or text when I integrate a custom TTS service?

In Real-time Conversational AI, images and speech or text are submitted together to the multi-modal large language model, which uses the conversation context to understand and respond. The Text-to-speech node is downstream of the model and accepts only text. When you integrate a custom TTS service, you do not need to associate images with speech or text at the TTS API layer.

How do I adjust the client audio capture sample rate?

Real-time Conversational AI currently supports the following audio parameters on the capture side:

• 48 kHz sample rate, mono channel

• 16 kHz sample rate, mono channel

The AICallKit SDK supports a 48 kHz sample rate by default. To switch to 16 kHz, set the audio profile to BasicQualityMode as shown in the following sample code:

self.engine.audioConfig = ARTCAICallAudioConfig(audioProfile: .BasicQualityMode, audioScenario: .MusicMode)
// Call other APIs
...  
// Start a call
self.engine.call(...)

ARTCAICallEngine.ARTCAICallConfig artcaiCallConfig = new ARTCAICallEngine.ARTCAICallConfig();
artcaiCallConfig.audioConfig.audioProfile = ARTCAICallAudioBasicQualityMode;
engine.init(artcaiCallConfig);

How do I adjust the agent audio playback sample rate?

Real-time Conversational AI currently supports the following audio playback parameters for the agent:

• 48 kHz sample rate, mono or stereo channel

• 16 kHz sample rate, mono channel

Real-time Conversational AI supports a 48 kHz mono channel sample rate by default. Adjust the setting in the operation that starts the call. The method depends on how you start the call:

Method 1: Start the call through a server-side operation

If you start the call by calling the GenerateAIAgentCall or StartAIAgentInstance operation, add the AgentConfig parameter and construct the ExperimentalConfig field in AIAgentConfig:

// AudioQualityMode: Integer    
// The sample rate mode of the RTC output.
// 0: 48 kHz mono channel 
// 1: 48 kHz stereo channel
// 2: 16 kHz mono channel
// The value of ExperimentalConfig must be a JSON string.
{
    "ExperimentalConfig":"{\"AudioQualityMode\":2}"
}

Method 2: Start the call through the client-side call operation

This method requires AICallKit SDK 2.5.0 or later. If you already use callConfig as the call startup parameter and use the call(xxx) operation to create and start the call, use this method to change the agent playback sample rate.

let callConfig = ...   // Create and initialize ARTCAICallConfig. For more information, see the user guide.
let agentConfig = ARTCAICallAgentConfig()         // Create an ARTCAICallAgentConfig object
agentConfig.experimentalConfig = [
    "AudioQualityMode": 2
]
...                                                  // Set parameters based on your business requirements
callConfig.agentConfig = agentConfig
self.engine.call(config: callConfig)                 // Start the call

ARTCAICallEngine.ARTCAICallConfig artcaiCallConfig = new ARTCAICallEngine.ARTCAICallConfig();
artcaiCallConfig.agentConfig.experimentalConfig = new JSONObject();
try {
    artcaiCallConfig.agentConfig.experimentalConfig.put("AudioQualityMode", 2);
} catch (JSONException e) {
    e.printStackTrace();
}
engine.init(artcaiCallConfig);

Why am I charged for STT during Real-time Conversational AI debugging when the other party does not speak?

You are charged because speech-to-text (STT) remains enabled while the agent is running, even if the agent does not speak during debugging. The STT fee is calculated at the STT unit price for the actual duration that STT is enabled and running. For example, if STT runs for 34 minutes, you are billed for 34 minutes of actual use at the STT unit price. For more information about the pricing and billing rules for this item, see AI Real-time Interaction.

Errors occur when starting a call

The client reports an "AgentNotFound" error when starting a messaging conversation

In the client code, check whether the agentId and the region are set correctly.

In the left-side navigation pane of the Intelligent Cloud Editor (ICE) console, choose Real-time Conversational AI > Agent Management. Verify that the region selected at the top of the page matches the region configured in your code, and get the correct agentId from the Agent ID/Name column of the agent list.

Client code:

String mAgentId = "XXX";          // Agent ID -> the ID of the messaging conversation agent created in the console
String mRegion = "cn-shanghai";   // The region of the agent -> the region of the messaging conversation agent in the console
ARTCAIChatAgentInfo agentInfo = new ARTCAIChatEngine.ARTCAIChatAgentInfo(mAgentId, mRegion)

// Agent ID -> the ID of the messaging conversation agent created in the console
let agentInfo = ARTCAIChatAgentInfo(agentId: "xxxx")
// The region of the agent -> the region of the messaging conversation agent in the console
agentInfo.region = "cn-shanghai"

The client reports an "UnsupportedWorkflowType" error when starting a messaging conversation

If the error details are "The specified workflowType \"VoiceChat\" is not supported by this interface. Please use a compatible workflowType: [\"MessageChat\"]", check the agent ID and whether the workflow type associated with that agent ID is messaging conversation.

On the Agent Management page, find the target agent and check its Agent Workflow ID column to confirm whether the associated workflow type is MessageChat.

What do I do if the "Application not found or the application is disabled" error occurs during integration?

Use the agent ID, not the Model Studio application ID, for integration. On the agent details page, obtain the corresponding agent ID and key. Verify that the agent configuration used by the server to generate the authentication token matches the configuration on the client.

How do I troubleshoot call and messaging connection errors?

• The channel ID is already in use — Generate a new, unique channel ID for each call. Hang up promptly after the call ends, and wait for the resources to be released. Do not reuse the same channel ID in concurrent scenarios.

• Sending a message reports that the client is not connected — Start the agent and listen for the connection status first. Send messages only after you receive the connected event.

• The engine status callback never becomes ready, or starting a session gets no response — Register the callback that gets the authentication token, and let the server issue the token to complete authentication. Use continuous listeners instead of one-time listeners.

What do I do if a custom voice does not take effect?

Add the language identifier that matches the voice to the speech synthesis configuration, and confirm that the voice passed review in the corresponding region. Specify the recognition language through the recognition language parameter in the agent configuration.

How do I troubleshoot missing agent callbacks?

• Callback URL — The callback URL must use HTTPS and return a complete certificate chain issued by a public CA. (Recommended) Use a free Alibaba Cloud SSL Certificate.

• Server response — The server must receive the POST request correctly and return 200. Also confirm that you called the operation that starts the agent instance to get the instance identity.

How do I tune slow agent responses and premature interruptions?

• Slow responses — Check the semantic segmentation wait time, that is, the semantic wait parameter in turnDetectionConfig. Reducing it noticeably improves the response speed.

• Premature interruptions — Adjust the user silence wait parameter in the agent configuration. The current maximum is about 1,200 ms.

Why does the agent answer twice in walkie-talkie mode?

In walkie-talkie mode, voice activity detection and push-to-talk usually capture audio at the same time, which produces two separate inputs. Enable audio capture only after walkie-talkie mode takes effect.

How do I integrate Real-time Conversational AI in WeChat H5 and mini programs?

• WeChat H5 — Supported. Deploy the page under an ICP-filed HTTPS domain name and load it through a web-view. The first microphone call requires user authorization. Check the compatibility of real-time communication capabilities and verify on real devices.

• Mini programs — Using the real-time communication stream pulling protocol directly is not supported. Embed a web page hosted under an HTTPS domain name through a web-view, and configure the business domain name.

• Service domain name — The system automatically assigns and binds the real-time communication service domain name. You do not need to configure a media transmission domain name separately.

What input, output, and model capabilities does Real-time Conversational AI support?

• Input and output — Speech input is supported, and text can be passed in through the text sending operation. The output is always speech.

• Self-trained models — Self-trained models that comply with the OpenAI interface specification are supported. You must provide the model identifier, key, and HTTPS endpoint.

• Workflows — In a workflow, you can configure a published Model Studio agent through the large language model option.