iOS

The RTC SDK provides basic retouching features in versions 1.17.9 and later. These features include whitening and skin smoothing. You can use the retouching features to improve the user experience in scenarios such as live streaming, video calls, and video conferences.

The RTC SDK supports third-party retouching features in versions 1.14.0 and later. A third-party retouching SDK typically processes callbacks using the following two types of data:

• Raw YUV data for facial recognition.
• OpenGL texture data for final retouching effect processing.

1. In the constructor of the SDK instance, add [extrasDic setValue:@"TRUE" forKey:@"user_specified_video_preprocess"] to the extension dictionary. Then, serialize extraDic and pass it to the extra parameter of the AliRtcEngine sharedInstance method.
2. After you call the startPreview method to enable local preview, call registerLocalVideoTexture to subscribe to OpenGL texture data.
3. To integrate the facial recognition feature using YUV data, call registerVideoSampleObserver after you call startPreview to subscribe to the pre-processed YUV data from the video capture. This is typically done to perform facial recognition on the captured image.
4. Perform the third-party facial recognition operation in the onCaptureVideoSample callback.
5. In onTextureCreate, initialize the third-party algorithm. The context parameter is the OpenGL context.
6. In onTextureUpdate, perform the per-frame retouching process using the third-party algorithm. If retouching is not required or the third-party algorithm fails, return the input textureId from this function. If the retouching process is successful, return the textureId that is processed by the third-party algorithm.
7. In onTextureDestroy, destroy the third-party algorithm.

The RTC SDK provides methods for integrating facial recognition with raw YUV data. To integrate this feature, you must subscribe to the pre-processed buffer data after video capture. To do this, call the registerVideoSampleObserver method after you call startPreview to obtain and process the pre-processed capture data.

///@brief Subscribes to raw data from pre-processed video captures.
- (void)registerVideoSampleObserver;

///@brief Callback for pre-processing captured RTC video data.
- (BOOL)onCaptureVideoSample:(AliRtcVideoSource)videoSource videoSample:(AliRtcVideoDataSample *_Nonnull)videoSample;

The RTC SDK provides OpenGL texture methods. When you integrate retouching, you must subscribe to the video texture data. Therefore, after you call startPreview, call the methods to obtain the OpenGL texture data and the OpenGL thread environment. The methods to call are as follows:

• The registerLocalVideoTexture method

  ///@brief Subscribes to OpenGL texture data.
  - (void)registerLocalVideoTexture;

• The onTextureCreate callback (The FaceUnity SDK does not process this callback)

  ///@brief Indicates that a local video stream texture is created.
  ///@param[in] context A pointer to the OpenGL context (EGLContext).
  - (void)onTextureCreate:(void *_Nullable)context;

• The onTextureUpdate callback

  ///@brief Indicates that each frame of the video stream is processed.
  ///@param[in] width The width of the video stream texture.
  ///@param[in] height The height of the video stream texture.
  ///@param[in] videoSample The video data.
  - (int)onTextureUpdate:(int)textureId width:(int)width height:(int)height videoSample:(AliRtcVideoDataSample *_Nonnull)videoSample;

• The onTextureDestroy callback (The FaceUnity SDK does not process this callback)

  ///@brief Indicates that the local video stream texture is destroyed.
  - (void)onTextureDestory;

Calls to external retouching SDK methods.