Use SEI to implement interaction during co-streaming

Background information

After an H.264 or H.265 video is transcoded during relayed live streaming, the encoding information of the current video is added to the supplemental enhancement information (SEI). The SEI includes the layout information of users in a live channel and can be used in various interaction scenarios during live streaming. A viewer can use SEI to obtain the location information of a co-streamer in the video image.

This topic describes how to use SEI to obtain the location information of a co-streamer in the video image by clicking the co-streamer in the video image.

How to listen to SEI callbacks

Contact technical support to enable the SEI callback listener

By default, the SEI callback listener is disabled. If you want to enable the SEI callback listener, contact technical support and provide your AppId.

Configure callbacks for the player

Sample code for Android:

PlayerConfig config = mAliPlayer.getConfig();
config.mEnableSEI = true;
mAliPlayer.setConfig(config);
mAliPlayer.setOnSeiDataListener(new IPlayer.OnSeiDataListener() {
 @Override
 public void onSeiData(int type, byte[] bytes) {
 // Obtain the SEI callback.
 }
});

Sample code for iOS:

AVPConfig *config = [[AVPConfig alloc] init];
config.enableSEI = YES; // Enable the SEI callback listener.
... 
 
_player = [[AliPlayer alloc] init];
[_player setConfig:config];
_player.delegate = self;
... 

// Use a callback event of the player to process the SEI data.
- (void)onSEIData:(AliPlayer *)player type:(int)type data:(NSData *)data {
 // Receive the mixed SEI data if the event type is 5.
 if (type == 5) {
 NSString *str = [NSString stringWithUTF8String:data.bytes];
	 NSData *strData = [str dataUsingEncoding:NSUTF8StringEncoding];
 NSError *jsonError = nil;
	 NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:strData options:0 error:&jsonError];
 NSLog(@"Mixed SEI dictionary data: %@", dict);
 }
}

A viewer in a live channel can use the callback event whose type is 5 to obtain the SEI about the video layout. The following example shows the information format:

{
 "canvas": {
 "bgnd": 0,
 "h": 1280,
 "w": 720
 },
 "stream": [
 {
 "h": 0.22890624,
 "ms": 0,
 "paneid": 0,
 "type": 0,
 "uid": "Harold3",
 "vad": 16,
 "vol": 39,
 "w": 0.40694445,
 "x": 0.090277776,
 "y": 0.1921875,
 "zorder": 1
 },
 {
 "h": 0.22890624,
 "ms": 1,
 "paneid": 1,
 "type": 0,
 "uid": "Joshua1",
 "vad": 0,
 "vol": 0,
 "w": 0.40694445,
 "x": 0.5013889,
 "y": 0.1921875,
 "zorder": 2
 }
 ],
 "ts": 1676894623891,
 "ver": "1.0.0.20220915"
}

canvas specifies the size of the stream canvas, and stream specifies the IDs and the location information of all members including streamers. You can customize the upper-layer business logic based on the location information.