Digital Human Real-time Interaction OpenAPI

Version history

Overview

Interaction example

Prerequisites

1. Create an Alibaba Cloud account and use a RAM user to generate an AccessKey pair (AccessKey ID and AccessKey secret).

2. Grant the necessary RAM permissions to the RAM user that generated the AccessKey pair.

3. Log on to the platform using your Alibaba Cloud account and accept the relevant legal agreements.

API details

1. Batch query digital human project information

Request parameters: ListAvatarProject

Response parameters

Object field descriptions

2. Query digital human project details

2.1. Request parameters: QueryAvatarProjectRequest

2.2. Response parameters: QueryAvatarProjectResponse

2.2.1. frame parameter description

2.2.2. layer parameter description

2.2.3. material parameter description

2.2.4. videoScript parameter description

3. Start a session

Request parameters: StartAvatarSessionRequest

Response parameters: StartAvatarSessionResponse

Parsing the channelToken field

{
  "channelId":"123", // Channel ID
  "token":"", // Token
  "expireTime":600, // Deprecated. Time-to-live (TTL) in seconds. Corresponds to the timestamp in the ARTC join information.
  "timestamp": 600, // Expiration timestamp in seconds. Replaces the expireTime field.
  "nonce":"", // Random number. Not required for custom channel tokens.
  "userId":"", // User ID
  "appId":"" // Application ID
}

4. Stop a session

Request parameters: StopAvatarSessionRequest

Response parameters: StopAvatarSessionResponse

5. Effective resource query

Request parameters: QueryAvatarResourceRequest

QueryAvatarResourceResponse

QueryResourceInfo

6. Save a digital human project

6.1. Request parameters: SaveAvatarProjectRequest

6.1.1. frame parameter description

6.1.2. layer parameter description

6.1.3. material parameter description

6.1.4. videoScript parameter description

6.1.5. Coordinate description

• The coordinate parameters specify the offset from the top-left corner of the frame to the top-left corner of the image.

• If a coordinate value contains a decimal, truncate the decimal part. For example, in the following figure, X is 470 and Y is 700.

6.1.6. Frame rate and bitrate reference table

6.2. Response parameters

7. Manage a digital human project

7.1. Request parameters: OperateAvatarProjectRequest

7.2. Response parameters

8. Query project session status

8.1. Request parameters: QuerySessionInfoRequest

8.2. Response parameters

Session information

9. Query digital human avatar models

ListAnchor

9.1. Request parameters

9.2. Response parameters

AnchorResponse

10. Query voice templates

ListVoiceModels

10.1. Request parameters

10.2. Response parameters

VoiceModelResponse

WebSocket real-time interaction

WebSocket connection workflow

1. Start a session to obtain the WebSocket URL and Alibaba Real-Time Communication (ARTC) channel information.

2. Establish a WebSocket connection.

3. Continuously send "channel ready" packets until you receive a "channel ready completed" response.

4. Use the ARTC SDK and channel information to pull the video stream.

5. (Optional) Send an opening remark driver packet to initiate an opening interaction.

6. (Optional) Send a text driver packet to drive the digital human with text.

7. (Optional) Send an audio driver header packet, data packets, and a trailer packet to drive the digital human based on audio recognition results.

8. (Optional) Send an audio driver data packet to drive the digital human with audio.

9. Stop the session.

10. Close the WebSocket connection.

Example of establishing a WebSocket connection:

@ClientEndpoint
public class WebSocketClient {

    private Session userSession = null;

    public WebSocketClient(URI endpointURI) {
        try {
            WebSocketContainer container = ContainerProvider.getWebSocketContainer();
            container.connectToServer(this, endpointURI);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @OnOpen
    public void onOpen(Session userSession) {
        this.userSession = userSession;
        System.out.println("Connected to server");
    }

    @OnClose
    public void onClose(Session userSession, CloseReason reason) {
        this.userSession = null;
        System.out.println("Disconnected from server: " + reason);
    }

    @OnMessage
    public void onMessage(byte[] message) {

        // 1. Get the frame number from the first byte.
        int frame = Byte.toUnsignedInt(message[0]);

        // 2. Get the data type from the second byte.
        int dataType = Byte.toUnsignedInt(message[1]);

        // 3. Get the business type from the third byte.
        int bizType = Byte.toUnsignedInt(message[2]);

        // 4. Get the serial number from bytes 4 to 11.
        byte[] serialNumberBytes = Arrays.copyOfRange(message, 3, 11);
        String serialNumberString = new String(serialNumberBytes, StandardCharsets.UTF_8);

        // 5. Get the remaining content data. The data structure depends on the specific business type.
        byte[] contentJsonBytes = Arrays.copyOfRange(message, 11, message.length);
        String contentJsonString = new String(contentJsonBytes, StandardCharsets.UTF_8);

        // 6. Execute the corresponding business logic...
    }

    public void sendMessage(byte[] message) {
        try {
            if (this.userSession != null) {
                this.userSession.getBasicRemote().sendBinary(ByteBuffer.wrap(message));
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) throws InterruptedException {

        // 1. Establish a WebSocket connection.
        WebSocketClient client = new WebSocketClient(URI.create("ws://127.0.0.1:7005/v1/interaction?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzZXNzaW9uSWQiOiI4ZjM3YmVhOC1lNDYxLTRhNzktODczZS00Yzg2ZTI2OWU4YTAiLCJleHAiOjE3MjU1MTY3NTIsImFsaXl1bk1haW5JZCI6IjE1Mzk3MDQ3MDY0MTMyNzgifQ.PAvCT7gY1VWNGJQc1cmubVLd76INRVZnhHdoBRV9-Rc&sessionId=8f37bea8-e461-4a79-873e-4c86e269e8a0"));

        // 2. Channel preparation.
        //      2.1 Assemble the data packet.
        byte frameId = (byte) 0;
        byte dataType = (byte) 1;
        byte bizType = (byte) 131;
        // For normal business, you can use a randomly generated 8-byte serial number.
        byte[] serialNumber = new byte[8];
        byte[] contentBytes = new byte[0];

        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();

        // 3. Send the data packet.
        client.sendMessage(message);

    }

}

WebSocket message structure

1. Overall structure

The overall packet format is an 11-byte packet header followed by N bytes of packet content.

The 11-byte packet header consists of a 1-byte frame type, a 1-byte data type, a 1-byte business type, and an 8-byte serial number. The serial number is randomly generated by the client and is used for troubleshooting.

public class InteractionWebSocketProtocol {

    // 1-byte frame type
    private InteractionProtocolFrameEnum interactionProtocolFrameEnum;

    // 1-byte data type
    private DataTypeEnum dataType;

    // 1-byte business type
    private BusinessTypeEnum businessType;

    // 8-byte serial number
    private String serialNumber;

    // N-byte message in binary format
    private byte[] messageBytes;

}

2. Frame types

3. Data types

4. Business types

Business types:

Format of input and output parameters:

4.1. Query opening remark

Request:

Returns:

4.2. Interrupt the digital human's speech

Request:

This operation does not return a value.

4.3. Drive the digital human with text

Request:

Returns:

4.4. Audio recognition and driver [Start]

Request:

Returns

4.5. Audio recognition and driver [Data]

Request: Binary data in PCM format.

Response: For more information about the audio recognition result in question mode, see API description. No response is returned in announcement mode.

For the text response from the digital human, see the response described in the "Drive the digital human with text" section.

4.6. Audio recognition and driver [End]

Request:

Return value: None

4.7. Channel ready

Request:

Returns:

4.8. Audio driver

Request:

Response: None

4.9. General business type

Message format:

Error codes:

WebSocket connection examples

1. Opening remark

Client sends:

    public void sendOpenning() {
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 0; // Opening remark business type: 0
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.

        String content = "{\"sessionId\":\"xxx\"}"; // Replace with your sessionId
        byte[] contentBytes = content.getBytes(); 

        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();

        // 3. Send the data packet.
        client.sendMessage(message);
    }

Client receives:

    @OnMessage
    public void onMessage(byte[] message) {

        // 1. Get the frame number from the first byte.
        int frame = Byte.toUnsignedInt(message[0]);

        // 2. Get the data type from the second byte.
        int dataType = Byte.toUnsignedInt(message[1]);

        // 3. Get the business type from the third byte.
        int bizType = Byte.toUnsignedInt(message[2]);

        // 4. Get the serial number from bytes 4 to 11.
        byte[] serialNumberBytes = Arrays.copyOfRange(message, 3, 11);
        String serialNumberString = new String(serialNumberBytes, StandardCharsets.UTF_8);

        // 5. Get the remaining content data. The data structure depends on the specific business type.
        // If it is an opening remark business
        if (bizType == 0) {
            byte[] contentJsonBytes = Arrays.copyOfRange(message, 11, message.length);
            String contentJsonString = new String(contentJsonBytes, StandardCharsets.UTF_8);
            // 6. Execute the corresponding business logic...
        }
        
    }

2. Interrupt

Client sends:

    public void sendOpenning() {
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 2; // Interrupt business type: 2
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.

        String content = "{\"sessionId\":\"xxx\"}"; // Replace with your sessionId
        byte[] contentBytes = content.getBytes(); 

        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();

        // 3. Send the data packet.
        client.sendMessage(message);
    }

Client receives: None

3. Text driver

Client sends:

    public void sendText() {
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 128; // Text driver business type: 128
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.

        String content = "{\n" +
                "  \"sessionId\":\"xxx\",\n" + // Replace with your sessionId
                "  \"text\":\"xxx\",\n" + // Replace with your text
                "  \"askType\":1 \n" + // 1: Ask a question; 2: Make an announcement
                "}"; 
        byte[] contentBytes = content.getBytes(); 

        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();

        // 3. Send the data packet.
        client.sendMessage(message);
    }

Client receives:

    @OnMessage
    public void onMessage(byte[] message) {

        // 1. Get the frame number from the first byte.
        int frame = Byte.toUnsignedInt(message[0]);

        // 2. Get the data type from the second byte.
        int dataType = Byte.toUnsignedInt(message[1]);

        // 3. Get the business type from the third byte.
        int bizType = Byte.toUnsignedInt(message[2]);

        // 4. Get the serial number from bytes 4 to 11.
        byte[] serialNumberBytes = Arrays.copyOfRange(message, 3, 11);
        String serialNumberString = new String(serialNumberBytes, StandardCharsets.UTF_8);

        // 5. Get the remaining content data. The data structure depends on the specific business type.
        // Text driver business
        if (bizType == 128) {
            byte[] contentJsonBytes = Arrays.copyOfRange(message, 11, message.length);
            String contentJsonString = new String(contentJsonBytes, StandardCharsets.UTF_8);
            // 6. Execute the corresponding business logic...
        }
        
    }

4. Audio recognition and driver

Client sends:

    public void sendAudio() {

        // 1. Send the audio driver header packet.
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 130; // Audio driver header/trailer business type: 130
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.
        String content = "{\n" +
                "\t\"sessionId\":\"xxx\",\n" +
                "\t\"type\": \"startAsrAudio\",\n" +
                "\t\"askType\":1 " +
                "}";
        byte[] contentBytes = content.getBytes(); 
        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();
        client.sendMessage(message);

        // 2. Send the binary audio data multiple times.
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 0; // Data type: binary
        byte bizType = (byte) 129; // Audio driver data business type: 129
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.
        byte[] audioData = new Byte[1024]; // Replace with your audio data
        byte[] contentBytes = audioData;
        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();
        client.sendMessage(message);

        // 3. Send the audio driver trailer packet.
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 130; // Audio driver header/trailer business type: 130
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.
        String content = "{\n" +
                "\t\"sessionId\":\"xxx\",\n" +
                "\t\"type\":\"endAsrAudio\",\n" +
                "\t\"askType\": 1 " +
                "}";
        byte[] contentBytes = content.getBytes(); 
        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();
        client.sendMessage(message);
    }

Client receives:

    @OnMessage
    public void onMessage(byte[] message) {

        // 1. Get the frame number from the first byte.
        int frame = Byte.toUnsignedInt(message[0]);

        // 2. Get the data type from the second byte.
        int dataType = Byte.toUnsignedInt(message[1]);

        // 3. Get the business type from the third byte.
        int bizType = Byte.toUnsignedInt(message[2]);

        // 4. Get the serial number from bytes 4 to 11.
        byte[] serialNumberBytes = Arrays.copyOfRange(message, 3, 11);
        String serialNumberString = new String(serialNumberBytes, StandardCharsets.UTF_8);

        // 5. Get the remaining content data. The data structure depends on the specific business type.
        // Get the audio driver response
        if (bizType == 129) {
            byte[] contentJsonBytes = Arrays.copyOfRange(message, 11, message.length);
            String contentJsonString = new String(contentJsonBytes, StandardCharsets.UTF_8);
            // 6. Execute the corresponding business logic...
        }
        
    }

5. Channel ready

Client sends:

    public void channelReady() {
        byte frameId = (byte) 0; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 131; // Channel ready business type: 131
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.

        String content = "{\"sessionId\":\"xxx\"}"; // Replace with your sessionId
        byte[] contentBytes = content.getBytes(); // For normal business, you can use a randomly generated 8-byte serial number.

        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();

        // 3. Send the data packet.
        client.sendMessage(message);
    }

Client receives:

    @OnMessage
    public void onMessage(byte[] message) {

        // 1. Get the frame number from the first byte.
        int frame = Byte.toUnsignedInt(message[0]);

        // 2. Get the data type from the second byte.
        int dataType = Byte.toUnsignedInt(message[1]);

        // 3. Get the business type from the third byte.
        int bizType = Byte.toUnsignedInt(message[2]);

        // 4. Get the serial number from bytes 4 to 11.
        byte[] serialNumberBytes = Arrays.copyOfRange(message, 3, 11);
        String serialNumberString = new String(serialNumberBytes, StandardCharsets.UTF_8);

        // 5. Get the remaining content data. The data structure depends on the specific business type.
        // Channel ready
        if (bizType == 131) {
            byte[] contentJsonBytes = Arrays.copyOfRange(message, 11, message.length);
            String contentJsonString = new String(contentJsonBytes, StandardCharsets.UTF_8);
            // 6. Execute the corresponding business logic...
        }
        
    }

6. Heartbeat

Client sends:

    public void sendPing() {
        byte frameId = (byte) 14; // Frame type: Control frame
        byte dataType = (byte) 1; // Data type: json
        byte bizType = (byte) 255; // General heartbeat business type: 255
        byte[] serialNumber = new byte[8]; // For normal business, you can use a randomly generated 8-byte serial number.

        String content = "{}"; // Not required
        byte[] contentBytes = content.getBytes(); 

        byte[] message = new byte[11 + contentBytes.length];
        ByteBuffer bf = ByteBuffer.wrap(message)
        .put(frameId)
        .put(dataType)
        .put(bizType)
        .put(serialNumber)
        .put(contentBytes);
        message = bf.array();

        // 3. Send the data packet.
        client.sendMessage(message);
    }

Client receives: None

Integration details

PHP

Integration example

require 'vendor/autoload.php';
use AlibabaCloud\SDK\Imarketing\V20220704\Models\GetOssUploadSignatureRequest;
use AlibabaCloud\SDK\IntelligentCreation\V20240313\IntelligentCreation;
use Darabonba\OpenApi\Models\Config as AlibabaConfig;

$config = new AlibabaConfig();
$config->accessKeyId = '****';
$config->accessKeySecret = '****';
$config->endpoint = "intelligentcreation.cn-zhangjiakou.aliyuncs.com";

$intelligentCreationClient = new IntelligentCreation($config);

$request = new QueryAvatarProjectRequest();
$request->projectId = '111';
try {
  $response = $intelligentCreationClient->queryAvatarProject($request);
  var_dump($response->toMap());
} catch (TeaError $e) {
  Log::error($e);
}

2.12.0

composer require alibabacloud/intelligentcreation-20240313 2.12.0

Java

Integration example

package com.aliyun.intelligentcreation20240313;

import com.aliyun.intelligentcreation20240313.models.*;
import com.aliyun.tea.TeaException;
import com.aliyun.teaopenapi.models.Config;
import com.google.gson.Gson;

import java.util.HashMap;
import java.util.Map;

public class TestAvatarTest {

    public TestAvatarTest() throws Exception {
    }

    public static void main(String[] args) throws Exception {
        TestAvatarTest avatarTest = new TestAvatarTest();

        try {
            String projectId = "780931376329506816";
            avatarTest.queryAvatarProjectTest(projectId);
        } catch (TeaException e) {
            Gson gson = new Gson();
            System.out.println(e.getMessage());
            System.out.println(gson.toJson(e.getData()));
        }
        
    }

    String url = "intelligentcreation.cn-zhangjiakou.aliyuncs.com";
    // Initialize the configuration
    String ak = "**";
    String sk = "**";
    Config config = new Config().setAccessKeyId(ak)
            .setAccessKeySecret(sk)
            .setEndpoint(url);
    // Create a client
    Client client = new Client(config);

    void queryAvatarProjectTest(String projectId) throws Exception {
        Gson gson = new Gson();
        Map<String, Object> map = new HashMap<>();
        map.put("projectId",projectId);
        QueryAvatarProjectRequest request = QueryAvatarProjectRequest.build(map);
        // Call the API
        QueryAvatarProjectResponse response = client.queryAvatarProject(request);

        System.out.println(gson.toJson(response));

        if (response.getStatusCode().equals(200)) {
            System.out.println("queryAvatarProjectTest request successful");
        }
    }

    String startAvatarSessionRequest(String projectId) throws Exception {
        Gson gson = new Gson();
        Map<String, Object> map = new HashMap<>();
        map.put("projectId",projectId);
        StartAvatarSessionRequest startAvatarSessionRequest = StartAvatarSessionRequest.build(map);
        StartAvatarSessionResponse response = client.startAvatarSession(startAvatarSessionRequest);

        System.out.println(gson.toJson(response));
        if (response.getStatusCode().equals(200)) {
            System.out.println("startAvatarSessionRequest request successful");
            return response.getBody().getSessionId();
        }
        return null;
    }

    void stopAvatarSessionRequest(String projectId, String sessionId) throws Exception {
        Gson gson = new Gson();
        Map<String, Object> map = new HashMap<>();
        map.put("projectId",projectId);
        map.put("sessionId",sessionId);
        StopAvatarSessionRequest request = StopAvatarSessionRequest.build(map);
        StopAvatarSessionResponse response = client.stopAvatarSession(request);

        System.out.println(gson.toJson(response));
        if (response.getStatusCode().equals(200)) {
            System.out.println("stopAvatarSessionRequest request successful");
        }
    }

    void queryAvatarResourceRequest(String projectId) throws Exception {
        Gson gson = new Gson();
        Map<String, Object> map = new HashMap<>();
        QueryAvatarResourceRequest request = QueryAvatarResourceRequest.build(map);
        QueryAvatarResourceResponse response = client.queryAvatarResource(request);

        System.out.println(gson.toJson(response));
        if (response.getStatusCode().equals(200)) {
            System.out.println("queryAvatarResourceRequest request successful");
        }
    }

}

2.12.0

<dependency>
  <groupId>com.aliyun</groupId>
  <artifactId>intelligentcreation20240313</artifactId>
  <version>2.12.0</version>
</dependency>

Android (Deprecated)

ApsaraVideo Real-time Communication SDK Integration Guide: Android Platform

Note: mLocalSurfaceContainer is the display container for the video stream.

Note: projectId is the project ID.

1. Add the Maven repository.

maven(url = "https://maven.aliyun.com/nexus/content/repositories/releases")

2. Add the Push SDK dependency.

implementation("com.aliyun.aio:AliVCSDK_ARTC:6.8.7")
implementation("com.aliyun:intelligentcreation20240313:2.1.0")
implementation("com.aliyun:tea-openapi:0.3.4")

3. Initialize the stream pulling engine.

private fun createEngine() {
    mAliRtcEngine = AliRtcEngine.getInstance(applicationContext)
    mAliRtcEngine?.setRtcEngineEventListener(object : AliRtcEngineEventListener() {
        /* Notification of the connection status between the SDK and the server. You must handle connection failures. */
        override fun onConnectionStatusChange(
            aliRtcConnectionStatus: AliRtcEngine.AliRtcConnectionStatus,
        aliRtcConnectionStatusChangeReason: AliRtcEngine.AliRtcConnectionStatusChangeReason
        ) {
            super.onConnectionStatusChange(
                aliRtcConnectionStatus,
                aliRtcConnectionStatusChangeReason
            )
            if (aliRtcConnectionStatus == AliRtcEngine.AliRtcConnectionStatus.AliRtcConnectionStatusFailed) {
                /* TODO: You must handle this. We recommend that you prompt the customer. This is reported only when the SDK has tried various recovery policies and can no longer be used. */
            } else {
                /* TODO: Optional. Add business code, which is generally used for data statistics or UI changes. */
            }
        }

        /* The SDK attempts to handle exceptions on the on-premises device. */
        override fun OnLocalDeviceException(
            aliRtcEngineLocalDeviceType: AliRtcEngine.AliRtcEngineLocalDeviceType,
        aliRtcEngineLocalDeviceExceptionType: AliRtcEngine.AliRtcEngineLocalDeviceExceptionType,
        s: String
        ) {
            //TODO: When this exception occurs, the app must check whether the permissions are granted and whether the device hardware is normal.
        }

        override fun onJoinChannelResult(
            result: Int,
            channel: String,
            userId: String,
            elapsed: Int
        ) {
            super.onJoinChannelResult(result, channel, userId, elapsed)
            Log.i(TAG, "onJoinChannelResult result=$result,channel=$channel,userId=$userId,elapsed=$elapsed")
        }

        override fun onLeaveChannelResult(result: Int, stats: AliRtcEngine.AliRtcStats) {
            super.onLeaveChannelResult(result, stats)
            Log.i(TAG, "onLeaveChannelResult result=$result")
        }
    })

    mAliRtcEngine?.setRtcEngineNotify(object : AliRtcEngineNotify() {
        /* This callback is triggered 30 seconds before the authentication expires. You must refresh the authentication time. */
        override fun onAuthInfoWillExpire() {
            super.onAuthInfoWillExpire()
            /* TODO: You must handle this. The business triggers re-acquisition of the authentication information for the current channel and user, and then sets refreshAuthInfo. */

        }

        /* The business may trigger a kick-out action, so you must also handle this. */
        override fun onBye(code: Int) {
            super.onBye(code)
            /* TODO: We recommend that you handle this based on your scenario. */
        }

        override fun onRemoteUserOnLineNotify(uid: String, elapsed: Int) {
            super.onRemoteUserOnLineNotify(uid, elapsed)
            Log.i(TAG, "onRemoteUserOnLineNotify uid=$uid,elapsed=$elapsed")
        }

        override fun onRemoteUserOffLineNotify(
            uid: String,
            aliRtcUserOfflineReason: AliRtcEngine.AliRtcUserOfflineReason
        ) {
            super.onRemoteUserOffLineNotify(uid, aliRtcUserOfflineReason)
            Log.i(TAG, "onRemoteUserOnLineNotify uid=$uid,aliRtcUserOfflineReason=$aliRtcUserOfflineReason")
        }

        override fun onRemoteTrackAvailableNotify(
            uid: String,
            aliRtcAudioTrack: AliRtcEngine.AliRtcAudioTrack,
            aliRtcVideoTrack: AliRtcEngine.AliRtcVideoTrack
        ) {
            super.onRemoteTrackAvailableNotify(uid, aliRtcAudioTrack, aliRtcVideoTrack)
            Log.i(TAG, "onRemoteUserOnLineNotify uid=$uid,aliRtcAudioTrack=$aliRtcAudioTrack,aliRtcVideoTrack=$aliRtcVideoTrack")
            mLocalSurfaceContainer.post{
                if (aliRtcVideoTrack == AliRtcEngine.AliRtcVideoTrack.AliRtcVideoTrackCamera
                    || aliRtcVideoTrack == AliRtcEngine.AliRtcVideoTrack.AliRtcVideoTrackBoth
                ) {
                    val remote_canvas = AliRtcVideoCanvas()
                    val remoteView = mAliRtcEngine?.createRenderSurfaceView(this@PreviewActivity)
                    if (remoteView != null) {
                        remoteView.setZOrderOnTop(true)
                        remoteView.setZOrderMediaOverlay(true)
                    }
                    remote_canvas.view = remoteView
                    mLocalSurfaceContainer.addView(
                        remote_canvas.view, FrameLayout.LayoutParams(
                            mLocalSurfaceContainer.width,
                            mLocalSurfaceContainer.height
                        )
                    )
                    mAliRtcEngine?.setRemoteViewConfig(
                        remote_canvas,
                        uid,
                        AliRtcEngine.AliRtcVideoTrack.AliRtcVideoTrackCamera
                    )
                } else {
                    mAliRtcEngine?.setRemoteViewConfig(
                        null,
                        uid,
                        AliRtcEngine.AliRtcVideoTrack.AliRtcVideoTrackCamera
                    )
                }
            }
        }
    })
}

4. Configure the engine parameters before you join a session.

private fun initEngineBeforeJoin() {
    /* Optional: Parameter settings before joining the session. */
    mAliRtcEngine?.setChannelProfile(AliRtcEngine.AliRTCSdkChannelProfile.AliRTCSdkInteractiveLive)
    mAliRtcEngine?.setClientRole(AliRtcEngine.AliRTCSdkClientRole.AliRTCSdkLive)
    /* Set the audio properties. */
    mAliRtcEngine?.setAudioProfile(
        AliRtcEngine.AliRtcAudioProfile.AliRtcEngineStereoHighQualityMode,
        AliRtcEngine.AliRtcAudioScenario.AliRtcSceneMusicMode
    )

    /* Optional: Camera preview. Stream ingest will proceed even if this is not set. */
    val canvas = AliRtcVideoCanvas()
    canvas.view = mAliRtcEngine?.createRenderSurfaceView(this)
    mLocalSurfaceContainer.removeAllViews()
    if (canvas.view != null) {
        mLocalSurfaceContainer.addView(
            canvas.view,
            FrameLayout.LayoutParams(
                FrameLayout.LayoutParams.MATCH_PARENT,
                FrameLayout.LayoutParams.MATCH_PARENT
            )
        )
    }
    mAliRtcEngine?.setLocalViewConfig(
        canvas,
        AliRtcEngine.AliRtcVideoTrack.AliRtcVideoTrackCamera
    )
}

5. Initialize the POP API SDK.

private fun initPopApiClient(){
    val config = Config()
        .setAccessKeyId(KEY_ID)
        .setAccessKeySecret(KEY_SECRET)
        .setEndpoint(HOST)
    client = Client(config)
}

6. Retrieve project information.

private suspend fun getProjectInfo() = withContext(Dispatchers.IO) {
    val map: Map<String , String> = mutableMapOf<String, String>().apply {
        put(PROJECT_ID , projectId?:"")
    }
    client?.queryAvatarProject(QueryAvatarProjectRequest.build(map))?.apply {
        Log.e(TAG , "statusCode:${statusCode}")
        if (statusCode == 200) {
            Log.e(TAG , "projectName:${body.projectName}")
            tvProjectName.post {
                tvProjectName.text = body.projectName
            }
        }
    }
}

7. Retrieve session information.

private suspend fun getMeetingInfo() = withContext(Dispatchers.IO){
    val map: Map<String , String> = mutableMapOf<String, String>().apply {
        put(PROJECT_ID , projectId?:"")
    }
    val response: StartAvatarSessionResponse? = client?.startAvatarSession(StartAvatarSessionRequest.build(map))

    response?.apply {
        Log.e(TAG , "statusCode:${statusCode}")
        if (statusCode == 200) {
            Log.e(TAG , "sessionId:${body.sessionId},channelToken:${body.channelToken}")
            sessionId = body.sessionId
            val meetingJsonObj = JSONObject(body.channelToken)
            var channelId = ""
            var token = ""
            var nonce = ""
            var userId = ""
            var appId = ""
            var expireTime = 0L
            var gslbList:MutableList<String> = mutableListOf()
            if (meetingJsonObj.has(CHANNEL_ID)){
                channelId = meetingJsonObj.optString(CHANNEL_ID)
            }
            if (meetingJsonObj.has(TOKEN)){
                token = meetingJsonObj.optString(TOKEN)
            }
            if (meetingJsonObj.has(NONCE)){
                nonce = meetingJsonObj.optString(NONCE)
            }
            if (meetingJsonObj.has(USER_ID)){
                userId = meetingJsonObj.optString(USER_ID)
            }
            if (meetingJsonObj.has(APP_ID)){
                appId = meetingJsonObj.optString(APP_ID)
            }
            if (meetingJsonObj.has(EXPIRE_TIME)){
                expireTime = meetingJsonObj.optLong(EXPIRE_TIME)
            }
            if (meetingJsonObj.has(GSLB_LIST)) {
                val gslbListJson = meetingJsonObj.optJSONArray(GSLB_LIST)
                for (i in 0 until gslbListJson.length()) {
                    gslbList.add(gslbListJson.optString(i))
                }
            }
            join(appId , channelId , userId , nonce,expireTime , token , gslbList.toTypedArray())
        }
    }
}

8. Join a meeting

private fun join(appId:String,channelId:String , userId:String ,
                     nonce:String , expireTime:Long,token:String,gslbList:Array<String>){
    val userInfo = AliRtcAuthInfo()
    userInfo.setAppId(appId)
    userInfo.setChannelId(channelId)
    userInfo.setUserId(userId)
    userInfo.setNonce(nonce)
    userInfo.setTimestamp(expireTime)
    userInfo.setGslb(gslbList)
    userInfo.setToken(token)
    mAliRtcEngine?.joinChannel(userInfo, "testUserName")

    resetCountDown()
}

9. Send a question.

private fun send(){
    val question = etInputQuestion.text.toString().trim()
    if (TextUtils.isEmpty(question)) {
        Toast.makeText(this , "question content cannot be null" , Toast.LENGTH_SHORT).show()
        return
    }
    GlobalScope.launch {
        sendText(question)
    }
}

private suspend fun sendText(text:String) = withContext(Dispatchers.IO){

    val map: Map<String , String> = mutableMapOf<String, String>().apply {
        put(PROJECT_ID , projectId?:"")
        put(SESSION_ID , sessionId)
        put(REQUEST_ID , (System.currentTimeMillis()/1000).toString())
        put(TEXT , text)
        put(TYPE , "1")
    }
    client?.sendTextMsg(SendTextMsgRequest.build(map))?.apply {
        onSendResponse(statusCode == 200 && body.status == "SUCCESS" , text)
    }
}

private fun onSendResponse(result:Boolean , text:String){
    tvExpireHint.post {
        if (result) {
            etInputQuestion.setText("")
            questionAdapter?.data?.add(0 , text)
            questionAdapter?.notifyItemInserted(0)
            rvQuestion.scrollToPosition(0)
            resetCountDown()
            Toast.makeText(this@PreviewActivity , "Sent successfully" , Toast.LENGTH_SHORT).show()
        }else {
            Toast.makeText(this@PreviewActivity , "Failed to send" , Toast.LENGTH_SHORT).show()
        }
    }
}

10. Leave the session.

private fun leave(){
    mAliRtcEngine?.leaveChannel()
    findViewById<LinearLayout>(R.id.llEnd).visibility = View.VISIBLE

    val map: Map<String , String> = mutableMapOf<String, String>().apply {
        put(PROJECT_ID , projectId?:"")
        put(SESSION_ID , sessionId)
    }

    GlobalScope.launch {
        async {
            client?.stopAvatarSession(StopAvatarSessionRequest.build(map))?.apply {
                Log.e(TAG , "statusCode = ${statusCode}, status = ${body.status}")
            }
        }
    }
}

11. Release resources.

override fun onDestroy() {
    super.onDestroy()
    mAliRtcEngine?.destroy()
    mAliRtcEngine = null
    handler.removeCallbacksAndMessages(null)
}

12. The session is automatically closed after 10 minutes of inactivity.

private var handler: Handler = Handler(Looper.getMainLooper())

private val mRunnable = Runnable {
    dealCountDown()
}

 private fun dealCountDown(){
    if (messageCount == 0) {
        // Leave the session
        leave()
        return
    }
    // If less than 1 minute, show a prompt
    val timeStr = when {
        60 == messageCount-> "01:00"
        (messageCount in 1..9) -> "00:0${messageCount}"
        else -> "00:${messageCount}"
    }
    llExpireHint.visibility = View.VISIBLE
    tvExpireHint.text = "No activity detected. The preview will automatically close in ${timeStr} to avoid occupying resources."
    messageCount--
    handler.postDelayed(mRunnable , 1000)
}

// Call after joining the session and sending a question
private fun resetCountDown(){
    handler.removeCallbacksAndMessages(null)
    messageCount = 60
    handler.postDelayed(mRunnable , 1000 * 60 * 9)
}