Quickly add multilayer content moderation to an API
Quickly add content moderation checks to a workflow.
Solution overview
Append nodes to your existing workflow to check text and image content. If a text check fails, the workflow stops and throws an error that specifies the type of sensitive information. If an image check fails, the workflow returns a default image indicating that the content is sensitive.
Prerequisites
If you have already deployed a runnable workflow in the console, you can proceed to the next step. Otherwise, see Quickly create an API to generate comic illustrations to configure the basic environment.
Procedure
After you prepare the environment, you will have a runnable workflow as shown in the following figure:
Image save content moderation (local and remote)
Delete the original Save Image node from the workflow. Double-click a blank area, search for Alibaba Cloud Content Moderation Save Image, and add the node:
Add the node to the end of the workflow. The result is shown in the following figure:
Parameter description (Important):
enable_safe_check: A boolean value that enables or disables image content moderation.
output_ext: The format of the saved image. The default value is png.
error_on_fail: A boolean value that specifies whether to throw an exception if the check fails. If this parameter is set to false, a default image is returned.
check_model_mode: The moderation mode. The default value is `remote`. The `local` value uses a local model for checks.
remote_confidence_threshold: An integer that specifies the confidence level threshold. The default value is 30. If the confidence level of a detected violation is greater than this value, the content is considered a violation. A lower value increases the chance of false positives but improves security. For more information, see Image Moderation Enhanced Edition V2.0 Synchronous Detection API.
remote_content_safe_service: The moderation service type. You can specify one of the following services: `baselineCheck` for a general baseline check, `baselineCheck_pro` for a general baseline check (Professional Edition), `baselineCheck_cb` for a general baseline check (cross-border), `tonalityImprove` for a content administration check, `tonalityImprove_cb` for a content administration check (cross-border), `aigcCheck` for an AIGC image threat check, `aigcCheck_cb` for an AIGC image threat check (cross-border), `profilePhotoCheck` for a profile picture check, `postImageCheck` for a post and comment image check, `advertisingCheck` for a marketing material check, `liveStreamCheck` for a video and live stream screenshot check, or `riskDetection` for a malicious image check.
Enter violence as the positive prompt to test the result:
Image upload content moderation (local and remote)
Double-click a blank area, search for Alibaba Cloud Content Moderation Upload Image, and add the node. The result is shown in the following figure:
Parameter description (Important):
enable_safe_check: A boolean value that enables or disables image content moderation.
check_model_mode: The moderation mode. The default value is `remote`. The `local` value uses a local model for checks.
remote_confidence_threshold: An integer that specifies the confidence level threshold. The default value is 30. If the confidence level of a detected violation is greater than this value, the content is considered a violation. A lower value increases the chance of false positives but improves security. For more information, see Image Moderation Enhanced Edition V2.0 Synchronous Detection API.
remote_content_safe_service: The moderation service type. You can specify one of the following services: `baselineCheck` for a general baseline check, `baselineCheck_pro` for a general baseline check (Professional Edition), `baselineCheck_cb` for a general baseline check (cross-border), `tonalityImprove` for a content administration check, `tonalityImprove_cb` for a content administration check (cross-border), `aigcCheck` for an AIGC image threat check, `aigcCheck_cb` for an AIGC image threat check (cross-border), `profilePhotoCheck` for a profile picture check, `postImageCheck` for a post and comment image check, `advertisingCheck` for a marketing material check, `liveStreamCheck` for a video and live stream screenshot check, or `riskDetection` for a malicious image check.
Test the result. An exception is thrown at runtime:
Text content moderation
Alibaba Cloud Content Moderation for text (remote)
Search for and add the Alibaba Cloud Content Moderation for Text node:
In the CLIP text encoder node, convert the positive prompt text box to an input:
Connect the output of the content moderation node to the input of the text encoder node.
Parameter description (Important):
origin_text: The text to check.
enable_safe_check: A boolean parameter that specifies whether to enable the Content Moderation check for images. By default, this parameter is set to true.
aliyun_content_safe_service: The moderation service type. Valid values include `nickname_detection` for user nickname checks, `chat_detection` for private chat content checks, `comment_detection` for public chat and comment content checks, `ai_art_detection` for AIGC text checks, `ad_compliance_detection` for advertising compliance checks, and `pgc_detection` for PGC educational material checks.
To test the result, enter a prohibited word and click Run:
Alibaba Cloud LLM text content moderation (remote and local)
Search for Alibaba Cloud Text Content Moderation and click
to add it.In the CLIP text encoder node, convert the positive prompt text box to an input:
Connect the output of the content moderation node to the input of the text encoder node.
Parameter description (Important):
origin_text: The text to check.
temperature: The temperature for the local LLM.
local_llm_seed: The seed for the local Large Language Model (LLM).
local_llm_keep_alive: The keep-alive time for the local LLM.
llm_mode: The LLM mode. `auto` automatically decides whether to call the local Ollama or remote Qwen based on the available GPU memory. `remote` calls the remote Qwen. `local` calls the local Ollama.
model: The name of the LLM model. The default local LLM is `qwen2.5:7b`, and the default remote LLM is `qwen-plus`.
local_llm_system_prompt: The system prompt for the local LLM. If you modify this prompt, ensure that the returned value is a boolean. Otherwise, the node's moderation check will not function correctly.
remote_llm_system_prompt: The system prompt for the remote LLM. If you modify this prompt, ensure that the returned value is a boolean. Otherwise, the node's moderation check will not function correctly.
To test the result, enter violence and click Run:
to add it.
Content moderation return results after API publication
Failed image save moderation check
The name of the returned image is default.png. You can monitor for this string to detect a failed check.
{
"status": 10,
"apiInvokeId": "i_660d04736d74470025e691d0",
"data": {
"images": [
"http://sd-fc-shanghai-daily.oss-cn-shanghai.aliyuncs.com/default.png"
],
"taskId": "co6fcikahf3lhj6cl1d0",
"status": "succeeded"
},
"subErrCode": null,
"subErrMessage": null,
"errCode": null,
"errMessage": null,
"startTime": null,
"endTime": null
}Returns the default image URL:
Failed image upload moderation check
If an uploaded image fails the Alibaba Cloud Content Moderation check, the image upload node throws an error. Note that this is different from a failed image save check, which replaces the image with a default one instead of throwing an error. The returned parameters are as follows:
{
"status": 20,
"apiInvokeId": "i_66167aba6d744700257a1c84",
"data": null,
"subErrCode": "P_Request_ContentSafe",
"sub_err_message": "{'labels': 'political_politicalFigure', 'code': 'content_safe_check_failed'}",
"errCode": "B_API_ServiceError",
"errMessage": "Error invoking downstream service for the API",
"startTime": null,
"endTime": null
}Failed text moderation check
A failed text moderation check immediately returns an error in the following format.
{
"status": 20,
"apiInvokeId": "i_66167aba6d744700257a1c84",
"data": null,
"subErrCode": "P_Request_ContentSafe",
"subErrMessage": "Specific error code included",
"errCode": "B_API_ServiceError",
"errMessage": "Error invoking downstream service for the API",
"startTime": null,
"endTime": null
}The following example shows the parameters that are returned when an LLM check fails:
{
"status": 20,
"err_code": "B_API_ServiceError",
"errMessage": "Error invoking downstream service for the API",
"sub_err_code": "P_Request_ContentSafe",
"sub_err_message": "{'labels': 'llm_not_allow_goon', 'reason': 'llm check not allow goon', 'code': 'content_safe_check_failed'}",
"api_invoke_id": "i_66c7f9789b8059002559f670",
"data": null
}The following parameters are returned when an Alibaba Cloud text moderation check fails:
{
"status": 20,
"apiInvokeId": "i_66167aba6d744700257a1c84",
"data": null,
"subErrCode": "P_Request_ContentSafe",
"subErrMessage": "{'labels': 'sexual_content', 'reason': '{\"riskLevel\":\"medium\",\"riskTips\":\"sexual_vulgar\",\"riskWords\":\"sexual\"}', 'code': 'content_safe_check_failed'}",
"errCode": "B_API_ServiceError",
"errMessage": "Error invoking downstream service for the API",
"startTime": null,
"endTime": null
}For more information about specific error codes, see Text Moderation Enhanced Edition General Service API.