Digital Human Streaming Service Android SDK

SDK 1.2.2 upgrade notes

Prerequisites

1. The SDK does not include operations for server-side instances, such as starting a digital human, stopping a digital human, or broadcasting text. For these features, see the OpenAPI documentation.

2. The Demo requires a backend application to run. Make sure you have a backend application that can call OpenAPI operations.

3. Limits:

   1. System version: Android 6.0 (API level 23) and later.

   2. CPU architecture: armeabi-v7a and arm64-v8a.

Integration method

1. Download the digital human sample project. The SDK AAR package is `libs/AliyunAvatarSDK_1.2.2.aar` in the sample project.

2. Import the SDK integration dependencies.

3. Obtain the required device permissions.

Integrate the SDK

1. To use the SDK in your project, place the downloaded `AliyunAvatarSDK.aar` file into the `libs` folder of your app module. Then, add the import statement to the `app/build.gradle` file.

implementation fileTree(include the following: ['*.aar'], dir: 'libs')

2. In the `build.gradle` file of the root directory, add the Maven repository address:

allprojects {
  repositories {
    google()
    jcenter()
    // Add the following Maven address
    maven {
      url "http://maven.aliyun.com/nexus/content/groups/public/"
    }
  }
}

3. The AliyunAvatarSDK depends on the OkHttp, DingRTC, and fastjson libraries. Manually import these dependencies.

implementation 'com.squareup.okhttp3:okhttp:4.4.1'
implementation 'com.alibaba:fastjson:1.2.83'
implementation 'com.ding.rtc:dingrtc-full:3.7.5'

4. In the `app/src/main/AndroidManifest.xml` file, add the following code to obtain the required device permissions.

<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
<uses-permission android:name="android.permission.CHANGE_WIFI_STATE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/> 

5. Optional: Configure obfuscation rules.

In the `proguard-rules.pro` file, add -keep class configurations to prevent the obfuscation of public class names in the RTC SDK.

-keep class com.serenegiant.**{*;}
-keep class org.webrtc.**{*;}
-keep class com.ding.rtc.**{*;}
-keep class com.alivc.**{*;}
-keep class com.aliyun.avatarsdk.**{*;}

Interactive digital human

Create an instance

AvatarSDK avatarSDK = AvatarSDK.createDialogAvatarInstance(context,
                                            avatarSDKListener, 
                                            avatarInstanceInfo, 
                                            options);

Parameters

AvatarInstanceInfo

The basic information of the digital human application. You can obtain the parameters from the Digital Human Open Platform - Developer Information page.

Channel

The authentication information required by DingRTC. You can obtain all parameters by calling the StartInstance operation.

AvatarOptions

Custom configurations for advanced features

Initialization

The initialization call performs the following operations:

1. Establishes a WebSocket connection and initializes the digital human's interactive capabilities.

2. Joins an RTC channel and pulls the digital human's audio and video streams from the cloud.

AvatarSDK avatarSDK = AvatarSDK.createDialogAvatarInstance(this, avatarSDKListener, avatarInstanceInfo,options);
avatarSDK.init();

Callbacks and listeners

Attach a callback

• Attach during initialization

private AvatarSDKListener avatarSDKListener = new AvatarSDKListener() {}

@Override
protected void onCreate(Bundle savedInstanceState) {
	AvatarSDK avatarSDK = AvatarSDK.createDialogAvatarInstance(this, avatarSDKListener, avatarInstanceInfo, options);
	avatarSDK.init();
}

• In addition to attaching a listener in the constructor, you can also attach a listener to an instance at runtime.

private AvatarSDKListener avatarSDKListener = new AvatarSDKListener() {}

@Override
protected void onCreate(Bundle savedInstanceState) {
	AvatarSDK avatarSDK = AvatarSDK.createDialogAvatarInstance(this, avatarSDKListener, avatarInstanceInfo, options);
	avatarSDK.setAvatarSDKListener(avatarSDKListener);
	avatarSDK.init();
}

Callback method descriptions

• onInitSuccess: Callback for successful initialization.

onInitSuccess(String uid, DingRtcAudioTrack audioTrack, DingRtcVideoTrack videoTrack);

• onRecordingStart: Callback for when recording starts.

public void onRecordingStart();

• onRecordingStop: Callback for when recording ends.

public void onRecordingStop();

• onAsr: Callback for ASR text.

public void onAsr(String asrResult,String sentenceId);

• onAnswer: Callback for the digital human's answer text.

public void onAnswer(String answerResult, String sentenceId);

• onError: Error callback.

public void onError(int errorCode,String source, String errorMsg);

• onUnknown: Extension callback. Data that is sent from the IM message channel and does not belong to any of the preceding types is returned by the `onUnknown` callback.

public void onUnknown(String msg);

Start recording

Starts collecting audio data from the microphone and sends it to the server. This is not available in custom collection mode.

avatarSDK.startRecording();

Stop audio capture

Stops collecting audio data from the microphone and sending the data to the server-side. This function is not available in custom collection mode.

avatarSDK.stopRecording();

Text interaction

Use text to interact with the digital human.

avatarSDK.sendMessage("Hello");

Advanced features

Get the original DingRtcEngine object

Retrieves the original `DingRtcEngine` object. You can use this object to access advanced RTC features.

DingRtcEngine mAliRtcEngine = avatarSDK.getAliRtcEngine();

Set the listener for DingRtcEngine callbacks

• Local user behavior callback

// Note: Use the avatarSDK instance, not the original DingRTCEngine object.
avatarSDK.setRtcEngineEventListener(new AvatarRtcEngineEventListener(context) {
    @Override
    public void onJoinChannelResult(int result, String channel, String userId, int elapsed) {
        // You must call the super method. Otherwise, the reconnection feature of AvatarSDK will not work.
        super.onJoinChannelResult(result, channel, userId, elapsed);
    }

    @Override
    public void onConnectionStatusChange(DingRtcEngine.DingRtcConnectionStatus status, DingRtcEngine.DingRtcConnectionStatusChangeReason reason) {
        // You must call the super method. Otherwise, the reconnection feature of AvatarSDK will not work.
        super.onConnectionStatusChange(status, reason);
    }

    @Override
    public void onOccurError(int error, String message) {
        // You must call the super method. Otherwise, AvatarSDKListener cannot throw the related exception information.
        super.onOccurError(error, message);
    }
    
    @Override
    public void onAudioVolumeIndication(List<DingRtcAudioVolumeInfo> speakers) {
        super.onAudioVolumeIndication(speakers);
    }
});

• Remote user behavior callback

setRtcEngineEventListener(new AvatarRtcEngineEventListener() {
    @Override
    public void onJoinChannelResult(int result, String channel, String userId, int elapsed) {
        super.onJoinChannelResult(result, channel, userId, elapsed);
        Log.d(TAG, "onJoinChannelResult:" + result + ";channel:" + channel);
    }

    @Override
    public void onConnectionStatusChanged(DingRtcEngine.DingRtcConnectionStatus status, DingRtcEngine.DingRtcConnectionStatusChangeReason reason) {
        super.onConnectionStatusChanged(status, reason);
    }

    @Override
    public void onOccurError(int error, String message) {
        super.onOccurError(error, message);
    }

    @Override
    public void onAudioVolumeIndication(List<DingRtcAudioVolumeInfo> speakers) {
        super.onAudioVolumeIndication(speakers);
    }
});

• Get the raw audio data callback. This replaces `DingRtcEngine().registerAudioObserver()`.

avatarSDK.setAvatarRtcAudioObserver(new AvatarRtcAudioObserver() {
    @Override
    public void onCaptureRawData(DingRtcAudioFrame aliRtcAudioSample) {
        
    }

    @Override
    public void onCaptureData(DingRtcAudioFrame aliRtcAudioSample) {

    }

    @Override
    public void onRenderData(DingRtcAudioFrame aliRtcAudioSample) {

    }

    @Override
    public void onPlaybackAudioFrameBeforeMixing(String s, DingRtcAudioFrame aliRtcAudioSample) {

    }
});

• Volume callback. This replaces `DingRtcEngine().registerAudioVolumeObserver()`.

  Important

  After SDK 1.2.2, use `setRtcEngineEventListener` to listen for volume callbacks.

Audio ducking

Automatically detects when the digital human in the RTC video stream is speaking and pauses audio recording. This feature is disabled by default.

• Enable at initialization:

AvatarOptions options = new AvatarOptions.Builder().autoDodge(true).build();
AvatarSDK avatarSDK = AvatarSDK.createDialogAvatarInstance(this, avatarSDKListener, avatarInstanceInfo, options);

• Enable at runtime:

avatarSDK.setAutoDodge(true);

• Shutdown:

avatarSDK.setAutoDodge(false);

Custom collection

Send custom-collected audio data.

avatarSDK.sendAudioBuffer(buffer);

Destroy the instance

Disconnects from the RTC stream pulling, disconnects the IM connection, stops recording, and destroys the client instance.

avatarSDK.destory();

Broadcast digital human

Create an instance

AvatarSDK avatarSDK = AvatarSDK.createBroadcastAvatarInstance(context,
                                                           avatarSDKListener,
                                                           avatarInstanceInfo,
                                            			   options);

Parameters

AvatarInstanceInfo

The basic information of the digital human application. You can obtain the parameters from the Digital Human Open Platform - Developer Information page.

Channel

The authentication information required by DingRTC. You can obtain all parameters by calling the StartInstance operation.

AvatarOptions

Custom configurations for advanced features

Initialization

The initialization call joins an RTC channel and pulls the digital human's audio and video streams from the cloud.

AvatarSDK avatarSDK = AvatarSDK.createBroadcastAvatarInstance(this, avatarSDKListener, avatarInstanceInfo, null);
avatarSDK.init();

Callbacks and listeners

Attach a callback

• Attach during initialization

private AvatarSDKListener avatarSDKListener = new AvatarSDKListener() {}

@Override
protected void onCreate(Bundle savedInstanceState) {
	AvatarSDK avatarSDK = AvatarSDK.createBroadcastAvatarInstance(this, avatarSDKListener, avatarInstanceInfo, options);
	avatarSDK.init();
}

• In addition to attaching a listener in the constructor, you can also attach a listener to an instance at runtime.

private AvatarSDKListener avatarSDKListener = new AvatarSDKListener() {}

@Override
protected void onCreate(Bundle savedInstanceState) {
	AvatarSDK avatarSDK = AvatarSDK.createBroadcastAvatarInstance(this, avatarSDKListener, avatarInstanceInfo, options);
	avatarSDK.setAvatarSDKListener(avatarSDKListener);
	avatarSDK.init();
}

Callback method descriptions

• onInitSuccess: Callback for successful initialization.

public void onInitSuccess(String uid, DingRtcAudioTrack audioTrack, DingRtcVideoTrack videoTrack);

• onError: Error callback.

public void onError(int errorCode,String source, String errorMsg);

• onUnknown: Extension callback. Data that is sent from the IM message channel and does not belong to any of the preceding types is returned by the `onUnknown` callback.

public void onUnknown(String msg);

Advanced features

Get the original DingRtcEngine object

Retrieves the original `DingRtcEngine` object. You can use this object to access advanced RTC features.

DingRtcEngine mAliRtcEngine = avatarSDK.getAliRtcEngine();

Set the listener for DingRtcEngine callbacks

• Local user behavior callback

// Note: Use the avatarSDK instance, not the original DingRTCEngine object.
avatarSDK.setRtcEngineEventListener(new AvatarRtcEngineEventListener() {
    @Override
    public void onJoinChannelResult(int result, String channel, String userId, int elapsed) {
        super.onJoinChannelResult(result, channel, userId, elapsed);
        Log.d(TAG, "onJoinChannelResult:" + result + ";channel:" + channel);
    }

    @Override
    public void onConnectionStatusChanged(DingRtcEngine.DingRtcConnectionStatus status, DingRtcEngine.DingRtcConnectionStatusChangeReason reason) {
        super.onConnectionStatusChanged(status, reason);
    }

    @Override
    public void onOccurError(int error, String message) {
        super.onOccurError(error, message);
    }

    @Override
    public void onAudioVolumeIndication(List<DingRtcAudioVolumeInfo> speakers) {
        super.onAudioVolumeIndication(speakers);
    }
});

• Remote user behavior callback

// Note: Use the avatarSDK instance, not the original DingRTCEngine object.
avatarSDK.setRtcEngineEventListener(new AvatarRtcEngineEventListener() {
    @Override
    public void onJoinChannelResult(int result, String channel, String userId, int elapsed) {
        super.onJoinChannelResult(result, channel, userId, elapsed);
        Log.d(TAG, "onJoinChannelResult:" + result + ";channel:" + channel);
    }

    @Override
    public void onConnectionStatusChanged(DingRtcEngine.DingRtcConnectionStatus status, DingRtcEngine.DingRtcConnectionStatusChangeReason reason) {
        super.onConnectionStatusChanged(status, reason);
    }

    @Override
    public void onOccurError(int error, String message) {
        super.onOccurError(error, message);
    }

    @Override
    public void onAudioVolumeIndication(List<DingRtcAudioVolumeInfo> speakers) {
        super.onAudioVolumeIndication(speakers);
    }
});

Destroy the instance

Disconnects from the RTC stream pulling, disconnects the IM connection, stops recording, and destroys the client instance.

avatarSDK.destory();

Error codes

Audio and video service fault

When the `onError` callback is triggered, if the `source` field contains `Rtc`, it indicates an RTC error. For more information, see DingRTC Error Codes.

Digital Human service errors

When the `onError` callback is triggered, if the `source` field is `Avatar`, it indicates a Digital Human service error. See the following table for details.

FAQ

1. The stream I pulled is full screen. How can I fit it into a custom-sized container?

   You need to adjust the layout properties of the container. For more information, see the settings method in `AvatarActivity` in the Demo. If the transparent background is disabled, you can modify `LayoutParams` in the `updateRemoteDisplay` method. If the transparent background is enabled, you need to modify `LayoutParams` in the `updateRemoteDisplayByGreenBackground` method. Because the aspect ratio of the video stream is fixed, we recommend that you scale it proportionally. Using a container with a different aspect ratio may cause the video to be cropped. The following code provides an example:

   RelativeLayout.LayoutParams layout = new RelativeLayout.LayoutParams(1080, 1920);