SEI

Usage notes

Common scenarios for SEI:

• Transmitting timestamps to calculate network latency or synchronize data.

• Transmitting descriptive information (up to 4 KB). We recommend using JSON or a plain string.

• SEI is implemented internally using the SEI extension protocol.

Send SEI messages

Checking the environment

Before enabling SEI, verify that the current environment supports sending and receiving SEI messages.

// Static method
const result = AliRtcEngine.isSupported();
if (!result.detail.isSendMediaExtensionMsgSupported) {
  console.log('Your browser does not support sei.');
}

Send an SEI message

Enabling this feature increases client performance overhead. Call setEnableMediaExtensionMsg before joining a channel to enable this feature.

// Prerequisite: Create an engine instance first.
// Enable media extension. This must be called before joining a channel.
aliRtcEngine.setEnableMediaExtensionMsg(true);

// Assume that an SEI message is used to send a DataView object.
const buffer = new ArrayBuffer(4);
const view = new DataView(buffer);

// Use DataView to set the content of ArrayBuffer to 7f ff 7f ff.
view.setInt8(0, 127); // Set the first byte to 127.
view.setUint8(1, 255); // Set the second byte to 255.
view.setInt16(2, 32767); // Set the third and fourth bytes to 32767.

aliRtcEngine
  .sendMediaExtensionMsg(buffer, 2)
  .then(() => {
    console.log(
      'Successfully sent the SEI message. Content: `7fff7fff`. Repetitions: 2. payloadType: 5.',
    );
  })
  .catch((err: Error) => {
    message.error(err.message);
  });

Media extension messages reuse the audio and video data tunnel. Keep the following limits in mind:

• The maximum send rate is profile messages per second. SEI messages are transmitted in H.264/H.265 streams, so extension information can be attached only when video frames are encoded.

• The custom message body is limited to 4 KB to avoid affecting media data quality.

• The repeatCount parameter in sendMediaExtensionMsg specifies message redundancy. A value greater than 1 sends the message multiple times, which helps prevent loss from network packet drops. Other users in the channel receive the same message multiple times and must deduplicate accordingly.

• During a bypass live stream, subscribers in the channel also receive custom messages.

• Only one media extension message can be transmitted at a time. Calling sendMediaExtensionMsg multiple times overwrites previous data with the new data.

Receive SEI messages

Check the environment

The receiving client must also verify SEI support before enabling this feature.

// Static method
const result = AliRtcEngine.isSupported();
if (!result.detail.isSendMediaExtensionMsgSupported) {
  console.log('Your browser does not support sei.');
}

Receive an SEI message

Receiving SEI messages increases client performance overhead. Call setEnableMediaExtensionMsg before joining a channel.

// Prerequisite: Create an engine instance first.
// Listen for the mediaExtensionMsgReceived event.
aliRtcEngine.on('mediaExtensionMsgReceived', (remoteUserId, data, payloadType) => {
  console.log(`User ${remoteUserId} sent an SEI message with payloadType ${payloadType}. Content:`, data);
});