Use RTC as a transport channel

Choose an integration solution

Alibaba Cloud offers two solutions for integrating AI with RTC:

• End-to-end Real-time Conversational AI solution — Choose this solution to integrate AI with RTC without implementing the AI service orchestration yourself. For details, see Quick Start for Real-time Conversational AI.

• RTC as a transport channel — Choose this solution when your application uses ARTC for audio and video transmission and requires highly customized AI processing. You implement the AI service orchestration. The ARTC SDK establishes the audio and video data transmission link between the client and the server, and you add or integrate the AI features that you need.

This topic describes the transport channel solution.

Prerequisites

Audio and video transmission between clients and servers uses the following two SDK packages:

• Client SDK — The ARTC SDK, which supports Android, iOS, Windows, and H5.

• Server SDK — The Linux SDK.

You supply and orchestrate the AI services that your scenario requires, such as automatic speech recognition (ASR), text-to-speech (TTS), large language model (LLM), and digital human services.

Limitations

Review the following constraints before you select an architecture and a client platform:

• Audio 3A processing is disabled by default — Audio 3A processing on the Linux SDK, including denoising and volume gain, does not take effect until you enable it.

• ASR denoising is unavailable in H5 scenarios — Denoising optimization for ASR cannot be enabled in H5 scenarios.

• H.265 encoding is not in the official build — Because this feature is not in high demand, the official version of the Linux SDK does not include the H.265 encoding module.

Architectures

In every transport channel architecture, the ARTC SDK and the Linux SDK join the same RTC channel and exchange media in the following sequence:

1. The Linux SDK receives the stream sent by the ARTC SDK, decodes it, and passes the decoded data to your business layer.

2. Your business layer orchestrates the AI services that the scenario requires.

3. Your business layer sends the processed data, before encoding, to the Linux SDK.

4. The Linux SDK encodes the data and sends it back to the ARTC SDK for playback or rendering in the application.

The following table maps each scenario to an architecture.

Audio-only architecture

Use this architecture for audio-only scenarios.

The Linux SDK receives an audio stream from the ARTC SDK and passes the decoded audio data to your business layer. You can also orchestrate AI services to process the audio with ASR, TTS, and an LLM, and then send the audio data, before encoding, to the Linux SDK.

Digital human architecture

Use this architecture for digital human scenarios in which the digital human service runs on your backend server.

The Linux SDK receives the audio and video streams from the ARTC SDK and passes the decoded data to your business layer. As needed, you can orchestrate AI services for the audio. After processing by ASR, TTS, an LLM, and the digital human, the audio and video data is sent to the Linux SDK before encoding.

Third-party digital human architecture

Use this architecture when your digital human is from a third-party vendor and the service is not deployed on your backend server.

The third-party digital human vendor integrates the Alibaba Cloud Linux SDK and sends the digital human's audio and video data (PCM+YUV) to the SDK. The SDK then encodes this data into Opus+H.264 and sends it to the end user over the Alibaba Cloud Global Real-time Transport Network (GRTN). At this point, the RTC channel has three participants: the end user and two Linux SDK instances.

Broadcast a welcome message

In Real-time Conversational AI, some scenarios involve the first connection between the client application and the AI voice assistant on the server. In this case, the AI voice assistant must broadcast a welcome message to the client user. For example, in an audio call scenario, after the application user answers the call, the AI voice assistant on the server broadcasts the welcome message "Hello, how can I help you?" to the end user. Such scenarios usually require the welcome message from the server to play in full on the client, without loss, truncation, or dropped words.

Interaction flowchart

The following flowchart shows the interaction that delivers the welcome message.

engine.setAudioProfile(AliRtcEngine.AliRtcAudioProfile.AliRtcEngineHighQualityMode, AliRtcEngine.AliRtcAudioScenario.AliRtcSceneChatroomMode);

engine.setAudioProfile(AliRtcAudioProfile.engineHighQualityMode, audio_scene: AliRtcAudioScenario.sceneChatroomMode)

# Python example
def OnAudioPublishStateChanged(self, oldState: AliRTCEngine.AliEnginePublishState, newState: AliRTCEngine.AliEnginePublishState, elapseSinceLastState: int, channel: str) -> None:
    pass

# Python example
def OnAudioSubscribeStateChanged(self, uid: str, oldState: AliRTCEngine.AliEngineSubscribeState, newState: AliRTCEngine.AliEngineSubscribeState, elapseSinceLastState: int, channel: str) -> None:
    pass

# Python example
def OnGreetReady(self, isReady:bool) -> None:
    pass

Linux SDK extension features

The Linux SDK provides the following extension features. Enable only the ones that your scenario requires.

API references

• For the client-side API, see ARTC SDK.

• For the server-side API, see Linux SDK.