Windows

Limits

Windows SDK 1.16.2 and later are supported.

Scenarios

Inputting external video data to stream

1. You can call the setExternalVideoSource to enable external video input ingest.
   Note Currently, the texture mode is not supported on Windows.
2. The application side uses the interface configLocalCameraPublish or configLocalScreenPublish configuration to specify the track ingest stream, and then calls the interface publish to start the stream ingest.
   Note The SDK allows stream ingest and then enable external video input, that is, the time series swap in steps 1 and 2. However, in this case, when stream ingest is started by default, video data of the original collection source (camera or screen capture) is pushed first until external input is enabled.
3. The application side continuously calls the interface pushExternalVideoFrame to deliver bare video data to the SDK for stream ingest. The parameter frame passes in information about bare data. The parameter sourceType specifies the stream ingest track type, which is the same as the track type that is enabled in Step 1.
   Note

   • The frequency at which video frame data is delivered is controlled by the application, and the delivery is maintained at intervals based on the video source frame rate until the input stops. We recommend that you enable threads on the application side independently for data delivery to ensure the timeliness of data input.
   • Currently, Windows supports input YUV data (format I420). You must specify the format as AliRtcVideoFormatI420 and the bufferType as AliRtcBufferTypeRawData in the bare data information of the parameter frame.
   • When you assign the bare data information, you need to pass in the data pointer, video width and height information, and stride information. The rotation angle rotation is not supported. Keep the default value 0. The timestamp is the current time, in milliseconds.
4. If the data source input ends or the application aborts external video input, call the interface setExternalVideoSource to disable external video input.

Sample code:

AliRtcEngine *pEngine = AliRtcEngine::sharedInstance(this, "");
.....

//1. Enable external video input
pEngine->setExternalVideoSource(true, false, AliRtcVideoSourceCamera); //camera track enables external input
bPushExternalVideo = true;

//2. Configure Start Stream Ingest
pEngine->configLocalCameraPublish(true);
pEngine->publish();

.....

//3. Independent threads push external video data
int frameRate = 30; // 30 fps
do
{
    size_t frameLength = videoWidth * videoHeight * 3 / 2;
    void* cacheBuf = (void*)malloc(frameLength);

    /*
    Copy and push video data from an external data source to cacheBuf
    */

    AliRtcVideoDataSample sample;
    sample.data = (unsigned char*)cacheBuf;
    sample.format = AliRtcVideoFormatI420;
    sample.width = videoWidth;
    sample.height = videoHeight;
    sample.strideY = videoWidth;
    sample.strideU = videoWidth / 2;
    sample.strideV = videoWidth / 2;
    sample.dataLen = frameLength;
    sample.rotation = 0;

    pEngine->pushExternalVideoFrame(&sample, AliRtcVideoSourceCamera);

    // Control the delivery frequency of frame data.
    Sleep(1000 / frameRate);            
} while (bPushExternalVideo);

.....

//4. Stop external video input
pEngine->setExternalVideoSource(false, false, AliRtcVideoSourceCamera);
bPushExternalVideo = false;

Video Bare Data Output

1. The application must first inherit the AliRtcEventListener class to implement onCaptureVideoSample and onRemoteVideoSample callbacks to receive locally collected video naked data and subscribed remote video naked data.
   Note Currently, Windows only supports the output of data in the YUV(I420) format. You can use AliConvertVideoData static interfaces provided by the SDK to convert data to the RGBA format.
2. Call the operation registerVideoSampleObserver to start the video bare data output. After the start, the video data will be continuously called back through the onCaptureVideoSample and onRemoteVideoSample.
   Note To collect video data locally, you must enable the camera (preview is enabled or a video stream is being pushed) before the callback output is generated. The remote video data must also be generated after the video stream of another user is subscribed to.
3. If you need to stop receiving bare video data, you can call the unRegisterVideoSampleObserver operation to disable the output of bare video data.

Sample code:

AliRtcEngine *pEngine = AliRtcEngine::sharedInstance(this, "");
.....

// 1. Register the bare data output of the video.
pEngine->registerVideoSampleObserver();

// Start previewing, streaming, and subscribe to videos of other users.
.....

// 2. Callback for receiving bare data
void onCaptureVideoSample(AliRtcVideoSource videoSource, AliRtcVideoDataSample *videoSample)
{
    // Process the callback for locally collected data.
}

void onRemoteVideoSample(const AliRtc::String &uid, AliRtcVideoSource videoSource, AliRtcVideoDataSample *videoSample)
{
    // Process remote data callback.
}

//3. Stop video bare data output
pEngine->unRegisterVideoSampleObserver();

Inputting external audio data to stream

1. Call the interface setExternalAudioSource to enable external audio input ingest.
   Note Currently, only input audio PCM data is supported. The data is encoded as Signed 16-bit.
2. The application side uses the interface configLocalAudioPublish to configure audio stream ingest, and then calls the publish interface to start audio stream ingest.
   Note The SDK allows stream ingestion to start with external audio input enabled, that is, the timing swap between step 1 and step 2. However, in this case, when stream ingestion starts by default, the microphone is the first to collect audio until external input is enabled.
3. The application side continuously calls the pushExternalAudioFrameRawData interface to deliver audio PCM data to the SDK. The parameter audioSamples brings the audio data address, the parameter sampleLength indicates the audio length, and the parameter timestamp is the current time.
   Note

   • The frequency at which bare audio data is delivered is controlled by the application. The amount of audio data to be delivered each time should not exceed 240ms of audio data. We recommend that you deliver 20ms of audio data each time and keep the delivery loop until the end. When the input data frequency is too fast and the SDK cache is full and the data cannot be consumed temporarily, the interface returns the error code ERR_AUDIO_BUFFER_FULL. In this case, the application side needs to wait for a data length and retry to deliver this data again until it succeeds. Otherwise, the input audio data will be lost.
   • Consistent with video input, it is also recommended that the application side separately open a thread to deliver audio bare data until the input stops.
   • Since the external input audio data may also have a microphone collecting stream at the same time, the application can set whether it is necessary to push the external input audio and the microphone collecting audio together after mixing, or push only the external input audio separately. By calling the interface setMixedWithMic, the mixing with the microphone can be turned on or off. At the same time, the mixing volume of the input audio can be set and obtained through the interface setExternalAudioPublishVolume and getExternalAudioPublishVolume. The volume adjustment range is [0 - 100]. The default value is 50.
   • When the application pushes the microphone to collect audio and external input audio at the same time, the default behavior of the interface muteLocalMic is to stop all audio pushing (including the microphone and external input). If the application only needs to stop the microphone audio (keep the external input audio pushing), you can set the second parameter mode and select the AliRtcMuteOnlyMicAudioMode mode.
4. If the data source input ends or the application aborts external audio input, call the interface setExternalAudioSource to disable external video and audio input.

Sample code:

AliRtcEngine *pEngine = AliRtcEngine::sharedInstance(this, "");
.....

// 1. Enable external audio input streaming
unsigned int mIAudioRate = 44100; //44.1k
unsigned int mIPlayMonoAudio = 1; // Mono
pEngine->setExternalAudioSource(true, mIAudioRate, mIPlayMonoAudio); // Enable audio input ingest.
mPushExternalAudio = true;

//2. Independent threads push external audio data
unsigned int time_interval = 20; //20 ms
size_t byte_per_sample = 16 / 8; // Signed 16-bit
size_t data_size = mIAudioRate * mIPlayMonoAudio * byte_per_sample / (1000 / time_interval);
unsigned int* data = (unsigned int*)malloc(data_size);
if (data == nullptr)
{
fclose(fin);
PostMessage(WM_COMMAND, MAKEWPARAM(IDC_BUTTON_PLAY_AUDIO, BN_CLICKED), 0);
return -1;
}

do
{
size_t read_size = 0;
if ((read_size = fread(data, 1, data_size, fin)) != data_size) {
 fseek(fin, 0, SEEK_SET);
 read_size = fread(data, 1, data_size, fin);
}

if (read_size > 0)
{
 std::chrono::milliseconds now = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch());
 int ret = mpEngine->pushExternalAudioFrameRawData(data, read_size, now.count());

 if (ret < 0)
 {
 PostMessage(WM_COMMAND, MAKEWPARAM(IDC_BUTTON_PLAY_AUDIO, BN_CLICKED), 0);
 break;
 }
// Ensure that the remote playback tone is synchronized with the local playback tone.
 while (ret == ERR_AUDIO_BUFFER_FULL && mPushExternalAudio)
 {
 Sleep(time_interval); //sleep 20 ms
 if (!mPushExternalAudio) break;

 ret = mpEngine->pushExternalAudioFrameRawData(data, read_size, now.count());
 if (ret < 0)
 {
  PostMessage(WM_COMMAND, MAKEWPARAM(IDC_BUTTON_PLAY_AUDIO, BN_CLICKED), 0);
  break;
 }
 }
}
//Sleep(time_interval); //sleep 20 ms
} while (mPushExternalAudio);

.....

//4. Stop external audio input playback
mpEngine->setExternalAudioSource(false, 0, 0);
mPushExternalAudio = false;

External audio input playback

1. Call the interface setExteranlAudioRender to enable external audio input playback, which is enabled by parameter enable settings. The parameter sampleRate and parameter channelsPerFrame are used to specify the sample rate and the number of channels to input audio data.
   Note Currently, only input audio PCM data is supported. The data encoding is Signed 16-bit. The number and sample rate of channels for input and playback audio can be dynamically changed during the push-through process. In the next step 2, the sample rate and number of channels for the current audio data can be specified in the delivery interface pushExternalAudioRenderRawData.
2. The application side continuously calls the pushExternalAudioRenderRawData interface to deliver the audio PCM data to the SDK for playback. The parameter audioSamples is brought into the audio data address, the parameter sampleLength indicates the audio length, the parameter sampleRate and the parameter channelsPerFrame specify the sample rate and the number of channels to input the audio data, and the parameter timestamp is the current timestamp.
   Note Similar to externally input audio data pushing, playback audio also needs to be delivered at a frequency by the application side. We recommend that the application side independently open threads for data delivery to ensure the timeliness of data input.
3. When the data source input ends or the application aborts audio playback, the interface setExteranlAudioRender is called to disable external audio playback.

Sample code:

AliRtcEngine *pEngine = AliRtcEngine::sharedInstance(this, "");
.....

// 1. Enable external audio input playback
unsigned int sampleRate = 44100; //44.1k
unsigned int channelsPerFrame = 1; // Mono
pEngine->setExteranlAudioRender(true, sampleRate, channelsPerFrame); // Enable audio input playback
bRenderExternalAudio = true;

//2. Independent threads push external audio data
unsigned int audioDataInterval = 20; //20 ms
size_t bytePerSample = 16 / 8; // Signed 16-bit
size_t dataSize = sampleRate * channelsPerFrame * bytePerSample / (1000 / audioDataInterval);
unsigned int* data = (unsigned int*)malloc(dataSize);

do
{
    /*
    Copy and push audio data from an external data source to data
    */

    std::chrono::milliseconds now = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch());
    int ret = mpEngine->pushExternalAudioRenderRawData(data, dataSize, sampleRate, channelsPerFrame, now.count());
    if (ret < 0)
    {
        // An error occurs and the playback is interrupted.
        break;
    }

    // The returned result is not 0. You need to make an error judgment and check whether the buffer is full.
    // If the returned result is 0, the delivery is successful. Continue to read and deliver data without an interval.
    while (ret == ERR_AUDIO_BUFFER_FULL && bRenderExternalAudio)
    {
        Sleep(audioDataInterval); // The buffer is full, with an interval of one data length, and then try again until it succeeds.
        if (!bRenderExternalAudio) break;

        ret = mpEngine->pushExternalAudioRenderRawData(data, read_size, sampleRate, channelsPerFrame, now.count());

        if (ret < 0)
        {
            // An error occurs and the push is interrupted.
            break;
        }
    }
} while (bRenderExternalAudio);

.....

//4. Stop external audio input playback
mpEngine->setExteranlAudioRender(false, 0, 0);
bRenderExternalAudio = false;