Quick integration

1. Initialize the SDK

Procedure:

1. Implement the EngineEventHandlerInterface class, which contains the callbacks required by RTC. When data is received or the status changes, the callback functions in this class are triggered.

2. Create an EngineEventListener instance.

3. Call the AliRTCEngine.CreateAliRTCEngine function to create an AliRTCEngine instance. During this process, the EngineEventListener instance is registered as the callback object for the engine. After that, you only need to call methods of the AliRTCEngine instance to complete the stream ingest and pulling settings.

4. If you need to ingest streams to multiple channels, create multiple AliRTCEngine instances, with each instance corresponding to a virtual user in a channel.

See the following example:

# Implement the EngineEventHandlerInterface class
...

# Initialize the SDK
eventHandler = EngineEventListener()
coreServicePath = '/path/AliRtcCoreService' # Path to AliRtcCoreService
extra_jobj = {
    "user_specified_disable_audio_ranking": "true"
}
extra = json.dumps(extra_jobj)
linuxEngine = AliRTCEngine.CreateAliRTCEngine(eventHandler, 42000, 45000, "/tmp", coreServicePath, True, extra)

After the CreateAliRTCEngine function is called, an AliRtcCoreService process is started. The function parameters are as follows:

• eventHandler:EngineEventHandlerInterface: the callback object that handles callback logic

• lowPort:int: the lower limit of the port number. A port within the range from lowPort to highPort is randomly selected for inter-process communication.

• highPort:int: the upper limit of the port number

• logPath:str: the path where log files are saved during the running of the Linux SDK

• coreServicePath:str: the actual path of AliRtcCoreService

• h5mode:bool: the H5 compatibility mode. If you want to communicate with the web client, make sure to set this parameter to True. For other scenarios, you can generally set it to False.

• extra:str: a JSON-formatted string that is used to configure additional settings for the SDK

2. Join a channel

For information about the Alibaba Cloud RTC authentication process, see Token authentication.

Procedure:

1. Set basic information such as the username and channel ID, and request a single-parameter token from the AppServer.

2. Configure JoinChannelConfig and set parameters such as the stream ingest and pulling mode.

3. Call the JoinChannel method to join a channel.

4. The OnJoinChannelResult function of the EngineEventListener instance is triggered to notify you of the result of joining the channel.

# -------- Obtain the basic information required for joining a channel --------
authInfo = AuthInfo()
authInfo.appid = ''      # The AppID, which is provided by the business party.
authInfo.userid = ''     # The username, which is the unique identifier of each user in the channel. If duplicate usernames exist, users are kicked out of the channel.
authInfo.username = ''   # The nickname of the user, which can be the same as the userid or can be set separately.
authInfo.channel = ''    # The ID of the channel to join.
# ...Request a single-parameter token for joining a channel from the AppServer.
authInfo.token = ''      # The token for joining a channel, which is returned by the AppServer.

# -------- Configure JoinChannelConfig --------
joinConfig = JoinChannelConfig()
joinConfig.channelProfile = ChannelProfile.ChannelProfileInteractiveLive # The interactive mode.
joinConfig.publishAvsyncMode = PublishAvsyncMode.PublishAvysncWithPts      # The audio and video synchronization mode for stream ingest.
joinConfig.subscribeAudioFormat = AudioFormat.AudioFormatPcmBeforMixing  # The audio subscription format.
joinConfig.subscribeVideoFormat = VideoFormat.VideoFormatH264            # The video subscription format.
joinConfig.isAudioOnly = False                                           # The audio-only mode, which is generally set to False.
joinConfig.subscribeMode = SubscribeMode.SubscribeAutomatically          # The subscription mode.
joinConfig.publishMode = PublishMode.PublishAutomatically                # The stream ingest mode.

# -------- Call JoinChannel to join a channel --------
linuxEngine.JoinChannel(authInfo.token, authInfo.channel, authInfo.userid, authInfo.username, joinConfig)

Description of JoinChannelConfig parameters:

1. Stream ingest mode: Both automatic and manual stream ingest modes are supported. In automatic mode, audio and video stream ingest is automatically enabled after you join a channel. In manual mode, you need to manually call an API to start stream ingest.

2. Subscription mode: Both automatic and manual subscription modes are supported. In automatic mode, when a streamer joins a channel, the streamer's audio and video tracks are automatically subscribed to. In manual mode, you need to manually call an API to specify the streamer's user ID to subscribe to. In scenarios where only audio or video is subscribed to, we recommend that you use SubscribeAudioAutoAndOnly or SubscribeCameraAutoAndOnly to reduce CPU usage caused by the decoder.

3. Audio subscription format: AudioFormatMixedPcm indicates stream mixing. After the audio tracks of all remote users in the channel are subscribed to, they are mixed into a single audio data stream for callback. AudioFormatPcmBeforMixing indicates stream separation. The audio data of each remote user is separately called back.

4. Video subscription format: If this parameter is set to VideoFormatH264, the OnRemoteVideoSample callback is triggered when data is received. Otherwise, the callback is not triggered even if data is received.

5. Audio and video synchronization mode: If PublishAvysncWithPts is selected, audio and video synchronization is performed based on the timestamp passed in the API when audio and video data is ingested. Otherwise, the timestamp is not referenced, and data is ingested based on the actual frequency of calls.

Important

Do not call JoinChannel repeatedly before leaving the channel.

The demo shows how to use GenerateToken to simulate the process of generating parameters for joining a channel by the AppServer. This interface is provided specifically to simplify the development and testing process. In a production environment, obtain the token through interaction with the AppServer to prevent the AppKey from being leaked.

3. Manually enable and disable stream ingest

'''
    * @brief Specify whether to ingest a local video (camera) track.
    * @param enabled Specifies whether to enable the ingest of a local video track.
    - true: enables the ingest of a video track.
    - false: disables the ingest of a video track.
    * @return
    - 0: The setting is successful.
    - <0: The setting fails, and an error code is returned.
    * @note By default, the SDK enables the ingest of a video track. You can call this method to modify the default setting before you join a channel. The setting takes effect after you join the channel.
'''
def PublishLocalVideoStream(enabled:bool) -> int

'''
    * @brief Specify whether to ingest a local audio track.
    * @param enabled Specifies whether to enable the ingest of a local audio track.
    - true: enables the ingest of an audio track.
    - false: disables the ingest of an audio track.
    * @return
    - 0: The setting is successful.
    - <0: The setting fails, and an error code is returned.
    * @note By default, the SDK enables the ingest of an audio track. You can call this method to modify the default setting before you join a channel. The setting takes effect after you join the channel.
'''
def PublishLocalAudioStream(enabled:bool) -> int

If data sending is complete but you do not want to leave the channel immediately, we recommend that you call the preceding two methods to stop stream ingest.

4. Push external video data

'''
    * @brief Specify whether to enable an external video source.
    * @param enable Valid values: true and false.
    * @param type The video source.
    * @note After the external video source is enabled, call the PushExternalVideoFrame method to import video data from the external video source.
'''
def SetExternalVideoSource(enable:bool, sourceType:VideoSource, renderMode:RenderMode) -> int

'''
     * @brief Import external video data. Videos with a resolution of 2K or higher are not supported.
     * @param frame The video data.
     * @param type The video source.
     * @param Multiple input video frame types are supported, such as YUV and RGB24.
'''
def PushExternalVideoFrame(frame:VideoDataSample, sourceType:VideoSource) -> int

5. Push external audio data

If you do not call SetExternalAudioPublishVolume, the audio is ingested at a volume of 100 by default.

'''
     * @brief Specify whether to enable an external audio source for stream ingest.
     * @param enable Valid values: true and false.
     * @param sampleRate The sampling rate, which can be 16,000 Hz, 48,000 Hz, and so on.
     * @param channelsPerFrame The number of channels, which can be 1, 2, and so on.
     * @return A value greater than or equal to 0 indicates success, while a value less than 0 indicates failure.
     * @note You can call the SetExternalAudioPublishVolume method to set the volume of external audio for stream ingest.
'''
def SetExternalAudioSource(enable:bool, sampleRate:int, channelsPerFrame:int) -> int

'''
     * @brief Import external audio data for stream ingest.
     * @param audioSamples The audio data.
     * @param sampleLength The length of the audio data.
     * @param timestamp The timestamp.
     * @return A value of less than or equal to 0 indicates failure. If ERR_AUDIO_BUFFER_FULL is returned, try again after the specified data delivery interval elapses.
'''
def PushExternalAudioFrameRawData(audioSamples:bytes, sampleLength:int, timestamp:int) -> int

'''
     * @brief Set the volume of external audio for stream ingest.
     * @param vol The volume. Valid values: 0 to 100.
'''
def SetExternalAudioPublishVolume(volume:int) -> int

'''
     * @brief Query the volume of external audio for stream ingest.
     * @return vol The volume.
'''
def GetExternalAudioPublishVolume() -> int

6. Configure manual subscription

In the mode of automatic subscription, you do not need to call these methods.

'''
     * @brief Stop or resume subscription to the audio track of a remote user. This method is used for in-channel calls. Calling this method before you join a channel is invalid.
     * @param uid The unique user ID that is assigned by the AppServer.
     * @param sub Specifies whether to subscribe to the audio track of the remote user.
     * - true: subscribes to the audio track of the remote user.
     * - false: stops subscribing to the audio track of the remote user.
     * @return
     * - 0: successful
     * - A value other than 0: failure.
'''
def SubscribeRemoteAudioStream(uid:str, sub:bool) -> int:

'''
     * @brief Stop or resume subscription to the video track of a remote user. This method is used for in-channel calls. Calling this method before you join a channel is invalid.
     * @param uid The unique user ID that is assigned by the AppServer.
     * @param track The source of the video track.
     * - AliEngineVideoTrackNo: an invalid value that does not have any effect.
     * - AliEngineVideoTrackCamera: camera.
     * - AliEngineVideoTrackScreen: screen sharing.
     * - AliEngineVideoTrackBoth: camera and screen sharing.
     * @param sub Specifies whether to subscribe to the video track of the remote user.
     * - true: subscribes to the video track of the remote user.
     * - false: stops subscribing to the video track of the remote user.
     * @return
     * - 0: success.
     * - <0: failure.
     * @note
'''
def SubscribeRemoteVideoStream(uid:str, videoTrack:VideoTrack, sub:bool) -> int

7. Data callbacks

'''
     * @brief The callback for local audio data subscription.
     * @details The audio data of all remote users after mixing, which is ready for playback. This corresponds to AliRTCSdk::Linux::AudioFormatMixedPcm.
     * @param frame The audio data. For more information, see {@link AliRTCSdk::Linux::AudioFrame}.
'''
def OnSubscribeMixAudioFrame(self, frame:AudioFrame) -> None

'''
     * @brief The callback for local audio data subscription.
     * @details The audio data of all remote users after mixing, which is ready for playback. This corresponds to AliRTCSdk::Linux::AudioFormatMixedPcm.
     * @param frame The audio data. For more information, see {@link AliRTCSdk::Linux::AudioFrame}.
'''
def OnSubscribeMixAudioFrame(self, frame:AudioFrame) -> None

'''
     * @brief The callback for subscribed remote video data.
     * @param uid The user ID.
     * @param frame The raw video data.
     * @return
'''
def OnRemoteVideoSample(self, uid:str, frame:VideoFrame) -> None

8. Leave a channel

# End stream ingest
linuxEngine.PublishLocalVideoStream(False)
linuxEngine.PublishLocalAudioStream(False)

If you are still in a channel after you stop stream ingest, manually call the following method to leave the channel. This method can also stop stream ingest.

linuxEngine.LeaveChannel()

9. Destroy the SDK

linuxEngine.Release()
linuxEngine = None

1. Data Channel

Call SetParameter to enable the Data Channel: linuxEngine.SetParameter("{\"data\":{\"enablePubDataChannel\":true,\"enableSubDataChannel\":true}}")

Send messages by using SendDataChannelMessage:

dataChannelMsg = AliEngineDataChannelMsg()
message = "This is data channel message".encode('utf-8')
dataChannelMsg.data = message
dataChannelMsg.dataLen = len(message)
dataChannelMsg.networkTime = t0;     #The network time field, which can also be customized without affecting message sending and receiving.
dataChannelMsg.progress = 0;     #Reserved field
dataChannelMsg.type = AliEngineDataMsgType.AliEngineDataChannelCustom

linuxEngine.SendDataChannelMessage(dataChannelMsg)

When a Data Channel message is received, a callback is invoked:

'''
     * @brief Obtain the remote data in the Data Channel.
     * @param msg The remote message.
'''
def OnDataChannelMsg(self, uid:str, msg:AliEngineDataChannelMsg) -> None

2. SEI

/**
 * @brief Send an SEI message. The message can be up to 4 KB in length. It is used for a small amount of data transmission.
 * @param message The content of the message, which can be up to 4 KB in size.
 * @param length The length of the message. Unit: bytes.
 * @param repeatCount The number of retries. This parameter can be used to prevent message loss caused by packet loss.
 * @param delay The delay for sending the message. Unit: milliseconds.
 * @param isKeyFrame Specifies whether to insert SEI into only key frames.
 * @return <0: success. -1: An SDK internal error occurs.
*/
int SendMediaExtensionMsg(const char* message,
                          size_t length,
                          int repeatCount,
                          uint32_t delay,
                          bool isKeyFrame);

Send messages by using SendMediaExtensionMsg:

seiMsg = "This is SEI message".encode('utf-8')
linuxEngine.SendMediaExtensionMsg(seiMsg, len(seiMsg), 3, 0 , False)

The keyframe interval is relatively long. If you need to send and receive SEI messages at a high frequency, we recommend that you set isKeyFrame to false. In addition, we recommend that you set the number of retries to a value greater than 1, which ensures that messages are effectively sent.

When an SEI message is received, a callback is invoked:

'''
     * @brief The callback that is invoked when an SEI message is received.
     * @param uid The ID of the user who sends the message.
     * @param message The content of the message.
     * @param size The length of the message.
     * @note When a user calls the {@link SendMediaExtensionMsg} method to send a message, a callback is triggered, from which other users receive the message.
'''
def OnMediaExtensionMsgReceived(self, userid:str, message:bytes, size:int) -> None:

1. Disable Audio Ranking

Audio Ranking allows you to subscribe to only the few audio tracks with higher volumes in a room where multiple people chat. This way, you can hear the audio more clearly. The feature is enabled by default. If you want to subscribe to all audio tracks, use the following code to disable Audio Ranking:

{"user_specified_disable_audio_ranking" : "true"}

2. Enable AAC transcoding for audio subscription

For scenarios that require stream pulling and archiving, the SDK provides the capability to transcode subscribed remote audio to AAC encoding. You can use the options provided by AudioTranscodingCodec to obtain remote audio in different formats (PCM, AAC, or both). If you enable AAC transcoding, you can also choose whether to perform resampling to reduce CPU and storage usage. When user_specified_audio_observer_resample_rate is set to 0, resampling is not performed.

{"user_specified_audio_observer_codec":
    AudioTranscodingCodec.AudioTranscodingCodecBothPcmAndAac.value}
{"user_specified_audio_observer_codec_format": 0}  # Currently, only the ADTS format is supported.
{"user_specified_audio_observer_resample_rate": 16000}  # The resampling rate.
{"user_specified_audio_observer_codec_bitrate": 64000}  # The target bitrate for transcoding.

3. Directly obtain H.264 video bitstreams

The SDK provides a feature that lets you directly obtain remote H.264 video bitstreams. You can use the options provided by VideoTranscodingCodec to obtain remote videos in different formats (YUV, H.264, or both).

{"user_specified_video_observer_codec":
    VideoTranscodingCodec.VideoTranscodingCodecBothYuvAndH264.value}
{"user_specified_video_observer_codec_format": 0}  # Currently, only the Annex B format is supported.