Linux integration development guide

Version updates

Version 1.0 supports single-camera RGB detection and recognition workflows.

Main features

The MOGUAN facial recognition software development kit (SDK) provides the following features:

• Facial detection

• RGB liveness detection

• 1:N facial database search, which supports retrieving from a database of up to 100,000 faces

• 1:1 feature comparison

Specifications

Image library size of 100,000

Memory requirements: The following table lists the additional memory that the facial recognition SDK requires for 1:N search operations.

Retrieval speed

Time consumption

Compatibility

• This SDK supports only the Linux x64 NVIDIA GPU version.

• You can use NVIDIA GPUs such as T4 or A10. The driver version must be 465.19.01 or later.

• The SDK does not provide Debug library files. You must select the Release mode for development and testing.

Programming language

The SDK is implemented in C++ and provides external interfaces in C++.

Model package

The SDK includes a model package. The naming convention for the model package is: DeviceName_(Adult/OtherScenario)_(With/WithoutMask)_(With/WithoutInfrared).

For example, gen_models_hie_adult_none_rgb indicates a model for RGB liveness detection of adults without masks on general-purpose PCs. If you require IR liveness detection or the mask detection feature, join the DingTalk group (23109592) for Alibaba Cloud Visual Intelligence API to contact the helpdesk.

SDK configuration

Configuration file

The SDK package contains the sail_face.yaml configuration file in the config/ directory. The SDK requires this file for initialization and for configuring information, such as which model files to use at each stage.

In-code configuration

In addition to using the sail_face.yaml configuration file, the SDK also supports using std::istream as an initialization parameter to input configuration information within your code. For more information, see the code example in example_export_linux.cpp.

Main classes and error descriptions

All SDK methods are in the sail and sail::face namespaces. Each public SDK interface has automatically generated API-level documentation. You can find the corresponding function descriptions in the API documentation.

Interface class

The main features of the SDK are provided in sail_face.h (sail::face::FaceController). These features include submitting facial recognition requests, obtaining results through synchronous or asynchronous interfaces, and performing various operations on the facial feature vector database. Examples of operations include adding a feature, deleting a feature, 1:N search, and 1:1 comparison. For detailed feature descriptions, see the corresponding documentation.

Error codes and error messages

API usage

How to use FaceController

This topic describes how to use the FaceController interface from a design perspective. For the usage and parameter details of each API operation, see the API documentation for each function.

You can use the SDK in two ways: through synchronous interfaces or asynchronous interfaces.

Synchronous interface

You can use the sail::face::FaceController::PutImageSyncV2 interface. When the function returns, it provides the result that corresponds to the created request. The result class is sail::face::DetectionResult. This result contains all detection results and their corresponding properties.

To retrieve results from the detection stage, you can register the sail::face::FaceController::RegisterOnDetectionResultCallback interface before initialization to obtain all detection results. The callback is also invoked if the detection result is empty.

The synchronous interface is the most traditional model. However, because the algorithm pipeline passes through multiple models, the latency of this interface is the sum of all steps. The overall performance is lower than that of the asynchronous interface.

Asynchronous interface

You can use the sail::face::FaceController::PutImageAsync interface to send an image asynchronously. The function returns immediately. The detection and retrieval results can be obtained through a registered callback. The function registered with the sail::face::FaceController::RegisterOnDetectionResultCallback interface is called each time a detection result is available. Your application can implement relevant logic within this callback.

The advantage of the asynchronous method is that you can obtain UI responses at different stages earlier, which reduces the latency between the camera and the UI. For example, you can display a rectangular box around a face on the screen after the detection result is available. Because this callback occurs during the detection stage, the response is faster than with the synchronous API, which makes the experience feel more real-time.

Workflow description

The following figure shows how the SDK works internally and when callbacks are triggered. This information helps you better understand how to use the SDK.

Create a request

You can create requests using sail::face::FaceRequestBuilder to create a sail::face::FaceRequest request class. You must specify the following information in the builder:

1. Image: All requests must include an RGB image using sail::face::RequestBuilder::RGBImage. The current SDK supports only RGB liveness detection. Do not input sail::face::RequestBuilder::InfraredImage.

2. RequestID: The request ID. For asynchronous requests, the RequestID is required to associate the request. This association lets the application link the input image to the callback result. The RequestID is returned to you through sail::face::DetectionResult::request_id. For synchronous requests, you can enter any RequestID.

3. SourceType: (sail::face::ImageSourceType) Specifies whether the request is for camera data (or continuous images, FromCamera) or a single image (FromPicture). If it is a camera image, the tracking algorithm is used. If it is a single image, tracking is not performed.

4. RequestPurpose: (sail::face::RequestPurpose) This variable specifies the purpose of the request. It includes the following two types:

   1. sail::face::RequestPurpose::StreamAnalysis: Indicates that the request needs to go through the complete workflow. This includes various filtering and liveness detection in the processing pipeline, and finally a database search. This is the most common request type for analyzing data collected from a camera.

   2. sail::face::RequestPurpose::FeatureExtract: Indicates that the request is for feature extraction. Therefore, this request is not subject to liveness detection. The extracted feature is returned to the user. Currently, only normal feature points are supported. Masked features are not supported. This is typically used to add or modify entries in the database.

5. DoFeatureSearch: Controls whether to search the database. To retrieve from the database for the input image and check if a corresponding face is registered, set this value to true.

Callbacks

Callbacks are invoked when either synchronous or asynchronous input interfaces are executed. Because asynchronous interfaces do not have return values, callbacks are the only way to implement application logic in asynchronous mode. In synchronous mode, you can also register callbacks to perform certain operations.

Asynchronous callbacks are generally registered only once, and before FaceController::Init.

RegisterOnDetectionResultCallback

This callback is registered to be invoked after the detection module produces a result. When the detection result is empty, the callback is still invoked. This lets you implement logic for scenarios where no face is detected.

RegisterFacePolicy

Registers a policy callback for face selection. You can implement this by inheriting the SailFacePolicy policy class. Two types of policies can be customized:

1. DetectionSelection and DoExtractAndSearch: DetectionSelection selects a set of detection results based on information such as the coordinates of the face bounding box, face quality, angle, and landmark confidence level for all detected faces. DoExtractAndSearch confirms whether to execute subsequent processes for the selected set of detection results. By default, all faces proceed to the next stage. The selected faces then undergo steps such as feature vector extraction and vector query.

2. LivenessSelection: Filters all detections.

RegisterOnFaceCompareResultCallback

This callback is invoked after face selection, liveness detection, feature vector extraction, and vector database search are completed. It provides the final result, which is the same as the return value of the synchronous API.

Facial vector database

The facial vector database is memory-based and is cleared on startup. Each time the process starts, you must re-add the vectors and UIDs from your user database, such as SQLite, to the vector database. The vector database includes a normal face database (full face) and a masked face database (partial face, not supported in this version).

Note that after you add or delete feature vectors, you must call sail::face::FaceController::RebuildFaceDBIndex to make the changes effective for search results. You can call the add or delete operations multiple times and then call sail::face::FaceController::RebuildFaceDBIndex once to perform a batch operation.

The addition and query of vectors use a user-input key as an identifier. You must ensure that this key is unique.

The following API operations are thread-safe:

• AddFeature: Adds a feature vector to the vector database and associates it with a key.

• RemoveFeature: Deletes a key and its corresponding vector from the vector database.

• RebuildFaceDBIndex: Rebuilds the index. After you add or delete vectors, call RebuildFaceDBIndex to apply the changes to the search API.

• SearchFeature: Finds the N most similar vectors and keys to a target vector from the vector database and sorts them by similarity. This operation returns a QueryResult data structure. After you modify the database, you can use SearchFeature to query the updated data.

• CompareFeature: Compares two input vectors. It returns a comparison result using the same struct as the search operation.

Masks

This feature is not supported in the current version. If you have business requirements for this feature, join the DingTalk group (23109592) for Alibaba Cloud Visual Intelligence API to contact the helpdesk.

Usage examples

Initialization

FaceController controller;
controller.Init(g_model_folder, auth_store)

Get an image

There are two ways to obtain an image:

• Obtain the camera image through a hardware API. The image is usually in YUV format. Then, convert the YUV format to the sail::HardwareImage class. This may involve a memory copy. For more information, see the GetSailImageColorRefFromYuv implementation in example_export_linux.cpp.

• For files in formats such as JPEG and JPG, you can use the SDK's built-in DecodeJpegFile function to read and decode them into a HardwareImage. For PNG, JPG, or other formats, see the GetSailImageColorRef implementation in example_export_linux.cpp, which depends on OpenCV.

Input an image request

Build an input image request. The following code provides an example:

FaceRequestBuilder builder;
DetectionResult result;
auto request_ptr = builder.RGBImage(copied_image_ref)
                           .RequestID(request_id++)
                           .SourceType(sail::face::ImageSourceType::FromCamera)
                           .build();

Synchronous request

Call controller.PutImageSyncV2() to send an image synchronously. The analysis and search results for the image are returned when the function finishes. If an exception occurs, an error code is returned. For more information, you can view the logs.

auto result = controller_.PutImageSyncV2(std::move(request_ptr), err_code);

Asynchronous request

Call controller.PutImageASync() to make an asynchronous request. You can handle events in the registered callback.

controller_.PutImageAsync(std::move(request_ptr));

Example scenarios

Initial database creation and population

1. Extract features from an image (sail::face::FaceController::PutImageSyncV2). Your application must handle the storage of the image and the extracted features, for example, in a database such as SQLite. Note that when you add entries to the database, you must use SourceType(sail::face::ImageSource::FromPicture) and RequestPurpose::FeatureExtract. This has the following effects:

   1. The request does not involve tracking and is not affected by the previous request.

   2. The liveness score is not checked.

   Note

   If you have already processed an image, you can store its facial features in a database that you manage. Each time you initialize the facial recognition SDK, you can perform the AddFeature operation in Step 2 on the features in your database. You do not need to call interfaces such as sail::face::FaceController::PutImageSyncV2.

2. Call the sail::face::FaceController::AddFeature interface to add the extracted feature to the vector database. You must also store the ID returned by sail::face::FaceController::AddFeature.

3. Repeat Step 1 and Step 2 until all images and features are added to the database. Then, call sail::face::FaceController::RebuildFaceDBIndex to reindex. You can reindex after each sail::face::FaceController::AddFeature call, but for batch processing, you can call it once at the end.

   Note

   The add step can also be used to add a single feature.

   Assume you need to process a batch of images in JPEG format. user_id is a unique ID that you specify for each image. The following code provides an example:

   ASSERT_EQ(g_controller.Init(g_config_path, g_auth_store), SF_SUCCESS);
   
   for (auto f : file_paths_of_jpeg) {
     auto request_ptr = build_request_from_jpg(f, 1, false); // This is a helper function.
     SFErrorCode err_code;
     auto result = g_controller.PutImageSyncV2(std::move(request_ptr), err_code)
     ASSERT_EQ(err_code, SF_SUCCESS);
     if(result->detections.size() > 0){
       std::shared_ptr<FaceFeature> feature = result->detections.begin()->feature;
       g_controller.AddFeature(controller_.GetNormalFaceDB(), user_id++, *feature); // You must ensure that user_id is unique. This is just an example.
   }
   ASSERT_EQ(g_controller.RebuildFaceDBIndex(controller_.GetNormalFaceDB()), SF_SUCCESS); 
   
   /// --- Helper function reference code.
   std::unique_ptr<FaceRequest> build_request_from_jpg(string img_path, uint64_t id=1, bool do_search=true){
      HardwareImageColorRef bgr_image_ref = DecodeJpegFile(img_path, GetPlatformMemAllocator()); 
       if(bgr_image_ref->GetSize()[0]<64 || bgr_image_ref->GetSize()[1]<64){
           std::cout<<"Img too small(low 64). size: "
                    <<bgr_image_ref->GetSize()[0]
                    <<bgr_image_ref->GetSize()[1]<<std::endl;
           return nullptr;
       }
   
       FaceRequestBuilder builder;
       auto request_ptr = builder.RGBImage(bgr_image_ref)
                                 .SourceType(ImageSourceType::FromPicture)
                                 .RequestPurpose(RequestPurpose::FeatureExtract)
                                 .RequestID(id)
                                 .DoFeatureSearch(do_search)
                                 .build();
       return request_ptr;
   }

4. Facial retrieval is similar to the database population operation, but you need to change RequestPurpose from RequestPurpose::FeatureExtract to RequestPurpose::StreamAnalysis. The following code provides an example:

   // Retrieve face
   auto request_ptr = build_request_from_jpg(test_img);
   request_ptr->purpose_ = RequestPurpose::StreamAnalysis;
   if (request_ptr == nullptr) {
     std::cout << "build_request_from_jpg error";
     return;
   }
   
   result = g_controller.PutImageSyncV2(std::move(request_ptr), err_code);
   if (err_code != SF_SUCCESS) {
     std::cout << "PutImageSyncV2 erro: " << static_cast<int>(err_code)
               << std::endl;
     return;
   }
   std::cout << "result size: " << result->GetFaceDetBoxes().size() << std::endl;
   for (auto& det : result->GetFaceDetBoxes()) {
     if (det->GetSearchResult() && det->GetSearchResult()->items_.size() != 0) {
       auto item = det->GetSearchResult()->items_[0];
       std::cout << "score: " << item.score_ << ", id: " << item.key_
                 << ", is_same_person: " << item.same_person_
                 << ", live: " << det->GetLiveness();
       std::cout << std::endl;
     }
   }

Delete from the feature database

The procedure is as follows:

1. Use your application logic to obtain the unique_id of the entry to delete. This is the ID you provided in AddFeature. Your application must store this ID.

2. Call the sail::face::FaceController::RemoveFeature() interface to delete the entry. Then, call sail::face::FaceController::RebuildFaceDBIndex() to make the deletion effective.

Unpack the SDK package and run the demo code

Deployment package contents

sail-face-linux-sdk-x86-hie-eaee3e5_release.tgz: The SDK software package. It contains the SDK's .so file, C++ header, documentation, examples, configuration file, and the gen_models_hie_adult_none_rgb model folder.

Run the demo

1. Unpack sail-face-linux-sdk-x86-hie-xxxxxx_release.tgz.

2. Modify the SDK configuration:

   1. In the config/sail_face.yaml configuration file, set common.model_dir to the path of the gen_models_hie_adult_none_rgb model folder. You can modify the path as needed. If the model directory is the same as the directory of the running binary file, you can change it to ./gen_models_hie_adult_none_rgb.

   2. In addition to modifying the configuration file, you can also refer to the config_face_xxx_template configurations in example_export_linux.cpp to modify common.model_dir. By default, the model directory is the same as the binary file directory.

3. Request a license key. For more information, see the documentation in the SDK package.

4. Compile the test sample code cd example && mkdir build && cd build && cmake .. && make -j12 to generate example_export_linux.

5. Execute example_export_linux. Prepare two images in JPG format. They can be of the same person or different people for comparison.

   Note

   If you run the command with the sail_face.yaml configuration file, the file must exist in the directory specified by -c (in this case, config). Also, the information in the configuration file must be consistent with the config_face_xxx_template configuration in example_export_linux.cpp.

FAQ

1. What do I do if initialization fails with a "Load config error"?

Check whether the config/sail_face.yaml file from the installation package is in the path specified for the initialization parameter. Also, check whether common.model_dir in the configuration file is correctly configured with the model file directory.

2. What do I do if initialization fails because the algorithm has no model?

If an error similar to Algo SDK Init failed occurs, it is most likely because the model configuration and the model package do not match, which results in missing files.

3. How do I request an authorization license?

For a Linux x86 SDK license, follow the instructions on the web page to request and download the authentication file.

4. Can I run only certain features without using certain nodes?

In general, the pipeline can be trimmed. There is no single way to do this. You can trim it based on the scenario.

5. Can masked features and normal features be compared?

Masked features and normal features are vectors in two different spaces and cannot be compared. The current version supports only normal face features.

6. How to determine if a mask is worn?

The current version supports only normal face features and does not support mask features. If you have business requirements for this feature, join the DingTalk group (23109592) for Alibaba Cloud Visual Intelligence API to contact the helpdesk.

7. Compilation issues

If a compilation error similar to undefined reference to `sail::face::FaceController::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<sail::face::AuthInfoStore>)' occurs, it may be caused by a high GCC version. You can resolve this by adding add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0) to the CMakeLists.txt file.