Multimodal real-time moderation API integration guide
This document describes how to call the multimodal real-time moderation API of AI Guardrails.
Step 1: Activate the service
Go to the Activate AI Guardrails page to activate AI Guardrails.
Step 2: Grant permissions to a RAM user
Before you integrate the software development kit (SDK) or call the API, you must grant permissions to a Resource Access Management (RAM) user. You can create an AccessKey pair for your Alibaba Cloud account and RAM users. You must use the AccessKey pair to complete identity verification when you call Alibaba Cloud API operations. For more information about how to obtain an AccessKey pair, see Obtain an AccessKey pair.
Grant permissions to a RAM user
Log on to the RAM console using your Alibaba Cloud account.
Create a RAM user. For details, see Create a RAM user.
Grant the
AliyunYundunGreenWebFullAccesssystem policy to the RAM user. This policy grants full access to Content Moderation. For details, see Manage RAM user permissions.The RAM user can now call the Content Moderation API.
Step 3: Install and integrate the SDK
For more information about AI Guardrails SDKs, see Multimodal SDK reference
API description
Usage notes
You can call this operation to create a text Content Moderation task.
Operation:MultiModalGuard
Supported regions and endpoints:
Region | Public endpoint | Internal endpoint |
China (Shanghai) | https://green-cip.cn-shanghai.aliyuncs.com | https://green-cip-vpc.cn-shanghai.aliyuncs.com |
China (Beijing) | https://green-cip.cn-beijing.aliyuncs.com | https://green-cip-vpc.cn-beijing.aliyuncs.com |
China (Hangzhou) | https://green-cip.cn-hangzhou.aliyuncs.com | https://green-cip-vpc.cn-hangzhou.aliyuncs.com |
China (Shenzhen) | https://green-cip.cn-shenzhen.aliyuncs.com | https://green-cip-vpc.cn-shenzhen.aliyuncs.com |
China (Chengdu) | https://green-cip.cn-chengdu.aliyuncs.com | None |
China (Hong Kong) | https://green-cip.cn-hongkong.aliyuncs.com | None |
China East 1 Finance (Hangzhou) | https://green-cip.cn-hangzhou-finance.aliyuncs.com | None |
China East 2 Finance (Shanghai) | https://green-cip.cn-shanghai-finance-1.aliyuncs.com | None |
Shenzhen Finance | https://green-cip.cn-shenzhen-finance-1.aliyuncs.com | None |
Singapore | https://green-cip.ap-southeast-1.aliyuncs.com | https://green-cip-vpc.ap-southeast-1.aliyuncs.com |
Germany (Frankfurt) | https://green-cip.eu-central-1.aliyuncs.com | https://green-cip-vpc.eu-central-1.aliyuncs.com |
US (Virginia) | https://green-cip.us-east-1.aliyuncs.com | https://green-cip-vpc.us-central-1.aliyuncs.com |
Billing: This is a paid API. Only requests with HTTP status code 200 are metered and billed. Requests that return other error codes are not billed. For more information about billing methods, see the Activation and billing section.
Image requirements:
Images support the following formats: PNG, JPG, JPEG, BMP, WEBP, TIFF, SVG, AVIF, HEIF (for HEIF, the longest edge must be shorter than 8,192 px), GIF (the first frame is used), and ICO (the last image is used).
The size of an image cannot exceed 20 MB. The height and width cannot exceed 30,000 px, and the total number of pixels cannot exceed 250 million px. A resolution greater than 200 × 200 px is recommended. If the resolution is too low, the effectiveness of the Content Moderation algorithm is compromised.
Images must be downloaded within 3 seconds. If the download takes more than 3 seconds, a download timeout is returned.
QPS limits: This operation allows up to 50 queries per second (QPS) for a single user (10 QPS by default when files are involved). If the limit is exceeded, API calls are throttled, which may affect your business. We recommend that you call this operation at a reasonable rate.
Request parameters
Name | Type | Required | Example | Description |
Service | String | Yes | query_security_check_pro |
Important query_security_check_pro and response_security_check_pro are officially released. They classify the detected labels at a finer granularity in terms of content compliance protection, and the total number of labels is significantly larger than that of the previous version.
|
ServiceParameters | JSONString | Yes | The set of parameters required by the moderation service. The value is a JSON string. For the description of each parameter, see ServiceParameters. |
Table 1. ServiceParameters
Name | Type | Required | Example | Description |
content | String | At least one content parameter must be passed in | The text to be moderated | The text content to be moderated. Important A maximum of 2,000 characters can be entered in a single request. |
imageUrls | JSONArray | http://xxxx123 | Only one image is supported. Important | |
fileUrls | JSONArray | http://xxxx456 | Only one file is supported. Important The file size cannot exceed 10 MB. | |
chatId | String | No. This parameter is required if streaming moderation is used. | ABC123 | Uniquely identifies an interaction record of a round of "user input + LLM output". |
sessionId | String | No | 14**** | The session ID. This parameter marks the content of the current request as part of the multi-turn conversation in the same session window. |
done | Boolean | No. This parameter is recommended if streaming moderation is used. | true |
|
dataId | String | No | id123****** | Uniquely identifies the content to be moderated. |
accountId | String | No | 13**** | The account ID that uniquely identifies an account. The context moderation feature must be applied for separately. To enable the feature, contact your business manager or submit a ticket. If the feature is not enabled, accountId is used only as an account identifier and does not accumulate the context of cross-request data for the same account. The complete content to be moderated must be sent in a single request. Sending the same content in multiple requests may cause risks to go undetected. |
ip | String | No | 192.168.1.*** | The IP address of the account ID. |
referer | String | No | www.aliyun.com | The Referer request header, used for hotlink protection and other scenarios. The length cannot exceed 256 characters. |
referenceContent | String | No | Context content | The context content used for comparison with the content to be moderated to detect model hallucinations. Important Model hallucination detection depends on this parameter. If you enable the model hallucination detection capability but do not actually pass in this content, the capability does not take effect. In this case, the API does not return an error and returns the moderation result as normal (Code 200), but the returned Detail does not contain a detection item whose Type is modelHallucination. To check whether hallucination detection takes effect, see the "No hallucination detected (control)" example in the Examples section of this topic. |
accountId, sessionId, and chatId all identify the ownership of a request, but they differ in scope. The following table describes the purposes and use scenarios of the three parameters.
Parameter | Purpose | Use scenario |
accountId | Identifies multiple requests initiated by the same account | Account-level identifier. To accumulate context across requests of an account for moderation, you must apply to enable the context moderation feature. |
sessionId | Marks the content of the current request as part of the streaming response of the same LLM output | Required when you need to chain multiple chunks of content for moderation by session |
chatId | Uniquely identifies an interaction record of a round of "user input + LLM output" | Required when streaming moderation is used |
Response parameters
Name | Type | Example | Description |
Code | Integer | 200 | The status code. For more information, see Code description. |
Data | JSONObject | {"Result":[...]} | The moderation result data. For more information, see Data. |
Message | String | OK | The response message to the request. |
RequestId | String | AAAAAA-BBBB-CCCCC-DDDD-EEEEEEEE**** | The request ID. |
Table 2. Data
Name | Type | Example | Description |
Detail | JSONArray | The content compliance risk labels, confidence scores, and other results of the moderated content. For more information, see Detail. | |
Suggestion | String | Pass | The moderation suggestion.
Note
|
Table 3. Detail
Name | Type | Example | Description |
Suggestion | String | pass | The moderation suggestion.
Note
|
Type | String | contentSecurity | The protection dimension.
|
Level | String | high |
Note We recommend that you handle high-risk content directly, manually review medium-risk content, and handle low-risk content only when high recall is required. In routine operations, handle low-risk content in the same way as content with no detected risk. You can configure the risk scores in the AI Guardrails console.
|
Result | JSONArray | The content compliance risk labels, confidence scores, and other results of the moderated content. For more information, see Result. |
Table 4. Result
Name | Type | Example | Description |
Description | String | Suspected political entity | The description of the Label field. Important This field describes the Label field and may be changed. We recommend that you process results based on the Label field instead of this field. |
Confidence | Float | 81.22 | The confidence score, ranging from 0 to 100 and rounded to two decimal places. Some labels do not have confidence scores. |
Label | String | political_xxx | The label returned after text content detection. Multiple labels and scores may be returned. |
Level | String | high |
Note We recommend that you handle high-risk content directly, manually review medium-risk content, and handle low-risk content only when high recall is required. In routine operations, handle low-risk content in the same way as content with no detected risk. You can configure the risk scores in the AI Guardrails console.
|
Ext | JSONObject | Some protection dimensions return the corresponding extension information. For more information, see Ext. |
Table 5. Ext
Name | Type | Example | Description |
Riskwords | String | AA,BB,CC | Applicable protection dimension: contentModeration: content compliance detection
|
CustomizedHit | JSONArray | [{"LibName":"...","Keywords":"..."}] | Applicable protection dimension: contentModeration: content compliance detection
|
SensitiveData | JSONArray | ["6201112223455"] | Applicable protection dimension: sensitiveData: sensitive content detection The detected sensitive data samples. |
Desensitization | String | ...[Mobile phone number] is my contact information... | Applicable protection dimension: sensitiveData: sensitive content detection The masked content. |
FileUrl | String | https://sase-public-server-files.oss-cn-hangzhou.aliyuncs.com/saas-XXX | Applicable protection dimension: waterMark: digital watermarking identifier The download URL of the watermarked file. |
OutFileSize | String | 152357 | Applicable protection dimension: waterMark: digital watermarking identifier The file size. |
FileUrlExp | String | 1754135551 | Applicable protection dimension: waterMark: digital watermarking identifier The time when the download URL of the watermarked file expires. |
Filename | String | B7VKehJ4gZR.png | Applicable protection dimension: waterMark: digital watermarking identifier The file name. |
OutFileHashMd5 | String | 8b96ff73e8d8060016bb41b16d337871 | Applicable protection dimension: waterMark: digital watermarking identifier The MD5 hash of the file. |
Table 6. CustomizedHit
Name | Type | Example | Description |
LibName | String | Custom library 1 | The name of the custom library. |
Keywords | String | Custom word 1,Custom word 2 | The custom words, separated by commas (,). |
Examples
Request example
{
"Service": "XXX",
"ServiceParameters": {
"content": "This thermos cup supports IP67 water resistance and can be used underwater.",
"chatId": "ABC123",
"dataId": "id123******",
"accountId":"abc",
"sessionId":"abc",
"imageUrls": ["http://xxxx"], # Only one image is supported
"fileUrls": ["http://xxxx"], # Only one file is supported
"referer": "http://www.aliyun.com",
"referenceContent": "Product name: Smart thermos cup. Material: 316 stainless steel. Heat preservation duration: 12 hours. Capacity: 500 ml. Packing list: 1 thermos cup and 1 user manual."
}
}Correct usage: send the complete content to be moderated in a single request. If the same content is sent in multiple requests, each request is moderated independently, and risks may not be detected. To accumulate context across requests of an account for moderation, apply to enable the context moderation feature first.
Response examples:
Moderation by query_security_check_pro, with a system policy hit:
{
"Code": 200,
"RequestId": "AAAAAA-BBBB-CCCCC-DDDD-EEEEEEEE****",
"Message": "OK",
"Data": {
"Suggestion": "block",
"DataId": "id123******",
"Detail": [
{
"Suggestion": "block",
"Type": "maliciousUrl",
"Level": "high",
"Result": [
{
"Description": "Information theft",
"Label": "Info Stealer",
"Level": "high"
}
]
},
{
"Suggestion": "mask",
"Type": "sensitiveData",
"Level": "S2",
"Result": [
{
"Ext": {
"Desensitization": "...[Mobile phone number] is my contact information...",
"SensitiveData": [
"136********"
]
},
"Description": "Mobile phone number (Chinese mainland)",
"Label": "1814",
"Level": "S2"
},
{
"Ext": {
"SensitiveData": [
"**City"
]
},
"Description": "City (Chinese mainland)",
"Label": "1739",
"Level": "S0"
}
]
},
{
"Suggestion": "block",
"Type": "promptAttack",
"Level": "high",
"Result": [
{
"Description": "Refusal suppression jailbreak",
"Confidence": 100,
"Label": "Refusal Supression Jailbreak",
"Level": "high"
}
]
},
{
"Suggestion": "block",
"Type": "customLabel",
"Level": "high",
"Result": [
{
"Description": "Custom label hit",
"Label": "*** Business risk expression",
"Level": "high"
}
]
},
{
"Suggestion": "block",
"Type": "contentModeration",
"Level": "high",
"Result": [
{
"Ext": {
"RiskWords": "The name of a political entity",
"Advice": [
{
"answer": "test",
"hitLabel": "political_entity",
"hitLibName": "pornographic_adult-0827"
}
]
},
"Description": "Suspected political entity",
"Confidence": 100,
"Label": "political_entity",
"Level": "high"
},
{
"Ext": {
"CustomizedHit": [
{
"LibName": "To be blacklisted and blocked",
"KeyWords": "word a,word b,word c"
}
]
},
"Description": "Custom library hit",
"Confidence": 100,
"Label": "customized",
"Level": "high"
}
]
},
{
"Result": [
{
"Description": "Intrinsic hallucination",
"Confidence": 95,
"Label": "Intrinsic Hallucination",
"Level": "medium"
}
],
"Type": "modelHallucination",
"Suggestion": "block",
"Level": "medium"
}
]
}
}Moderation by img_response_security_check, with a system policy hit:
{
"Code": 200,
"Data": {
"Detail": [
{
"Level": "none",
"Result": [
{
"Confidence": 0.0,
"Description": "No risk detected",
"Ext": {
"FileUrl": "https://sase-public-server-files.oss-cn-hangzhou.aliyuncs.com/saas-XXX",
"OutFileSize": 527918,
"FileUrlExp": "1754200240",
"Filename": "wJGz6kmZ1Ce.jpg",
"OutFileHashMd5": "02f5129f606027c7a87b84377ec98f8e"
},
"Label": "nonLabel",
"Level": "none"
}
],
"Suggestion": "pass",
"Type": "waterMark"
},
{
"Level": "high",
"Result": [
{
"Confidence": 90,
"Description": "Advertising Law violation - extreme word",
"Label": "ad_Compliance_WordLimit_Tii",
"Level": "high"
}
],
"Suggestion": "block",
"Type": "contentModeration"
}
],
"Suggestion": "block"
},
"Msg": "OK"
}Moderation by text_img_security_check, with a system policy hit:
{
"Code": 200,
"Data": {
"Detail": [
{
"Ext": {},
"Level": "high",
"Result": [
{
"Confidence": 98.34,
"Description": "Female cleavage",
"Label": "sexual_Cleavage",
"Level": "high"
}
],
"Suggestion": "block",
"Type": "contentModeration"
}
],
"Suggestion": "block"
},
"Msg": "OK"
}Moderation by file_security_sync_check, with a system policy hit:
{
"Code": 200,
"Data": {
"Detail": [
{
"Ext": {},
"Level": "high",
"Result": [
{
"Confidence": 100,
"Description": "Website backdoor",
"Label": "WebShell",
"Level": "high"
}
],
"Suggestion": "block",
"Type": "maliciousFile"
},
{
"Ext": {
"PageSum": 1
},
"Level": "none",
"Result": [
{
"Description": "No risk detected",
"Label": "nonLabel",
"Level": "none"
}
],
"Suggestion": "pass",
"Type": "contentModeration"
}
],
"Suggestion": "block"
},
"Msg": "OK"
}Moderation by text_file_sec_sync_check, with a system policy hit:
{
"Code": 200,
"Data": {
"Detail": [
{
"Ext": {
"PageSum": 4
},
"Level": "none",
"Result": [
{
"Description": "No risk detected",
"Label": "nonLabel",
"Level": "none"
}
],
"Suggestion": "pass",
"Type": "contentModeration"
}
],
"Suggestion": "pass"
},
"Msg": "OK"
}Model hallucination detection examples
Model hallucination detection (in public preview) identifies false or inaccurate information in model output by comparing the content generated by the LLM with the reference content passed in referenceContent. The following examples cover three typical hallucination scenarios and one control scenario in which no hallucination is detected. Each example includes the scenario description, the input content (the content to be moderated + referenceContent), the moderation result, and the result explanation. You can copy the example content to verify the integration effect on your own. Prerequisite: in Configure check items, you have enabled the Model Hallucination dimension for the Service, and in the ServiceParameters of the request, you have passed in referenceContent.
Scenario 1: Fabricating information that does not exist in the reference content
Scenario description: The reference content does not mention water resistance, but the model output fabricates "IP67 water resistance". This is fabricating information that does not exist.
Input content:
{
"Service": "query_security_check_pro",
"ServiceParameters": {
"content": "This thermos cup supports IP67 water resistance and can be used underwater.",
"referenceContent": "Product name: Smart thermos cup. Material: 316 stainless steel. Heat preservation duration: 12 hours. Capacity: 500 ml. Packing list: 1 thermos cup and 1 user manual."
}
}Moderation result:
{
"Code": 200,
"Message": "OK",
"RequestId": "AAAAAA-BBBB-CCCCC-DDDD-EEEEEEEE****",
"Data": {
"Suggestion": "block",
"Detail": [
{
"Type": "modelHallucination",
"Suggestion": "block",
"Level": "medium",
"Result": [
{
"Description": "Intrinsic hallucination",
"Confidence": 95,
"Label": "Intrinsic Hallucination",
"Level": "medium"
}
]
}
]
}
}Result explanation: "IP67 water resistance" in the content to be moderated is not supported by the reference content and is determined to be a model hallucination. The returned Detail contains a detection item whose Type is modelHallucination, the Label is Intrinsic Hallucination, the Confidence is the confidence score of the hallucination determination, and the moderation suggestion is block.
Scenario 2: Fabricating information that contradicts the reference content
Scenario description: The reference content explicitly states that a 7-day no-reason return is supported, but the model output claims that "returns are not supported". This is fabricating information that contradicts the reference content.
Input content:
{
"Service": "query_security_check_pro",
"ServiceParameters": {
"content": "This product does not support returns, and no refund can be requested after the order is placed.",
"referenceContent": "After-sales service: This product supports a 7-day no-reason return, and the return shipping fee is borne by the buyer."
}
}Moderation result:
{
"Code": 200,
"Message": "OK",
"RequestId": "AAAAAA-BBBB-CCCCC-DDDD-EEEEEEEE****",
"Data": {
"Suggestion": "block",
"Detail": [
{
"Type": "modelHallucination",
"Suggestion": "block",
"Level": "medium",
"Result": [
{
"Description": "Intrinsic hallucination",
"Confidence": 95,
"Label": "Intrinsic Hallucination",
"Level": "medium"
}
]
}
]
}
}Result explanation: The content to be moderated directly contradicts the return policy in the reference content and is determined to be a model hallucination (Intrinsic Hallucination). The moderation suggestion is block.
No hallucination detected (control)
Scenario description: The model output is consistent with the reference content, and no hallucination is detected. The API returns a normal result (Code 200), but the Detail does not contain a detection item whose Type is modelHallucination.
Input content:
{
"Service": "query_security_check_pro",
"ServiceParameters": {
"content": "The heat preservation duration of this thermos cup is 12 hours, and its capacity is 500 ml.",
"referenceContent": "Product name: Smart thermos cup. Material: 316 stainless steel. Heat preservation duration: 12 hours. Capacity: 500 ml. Packing list: 1 thermos cup and 1 user manual."
}
}Moderation result:
{
"Code": 200,
"Message": "OK",
"RequestId": "AAAAAA-BBBB-CCCCC-DDDD-EEEEEEEE****",
"Data": {
"Suggestion": "pass",
"Detail": []
}
}Result explanation: The content to be moderated is faithful to the reference content and is not determined to be a hallucination, so the Detail does not contain a modelHallucination detection item. This is also the response behavior when the model hallucination dimension is enabled but referenceContent is not passed in: the API does not report an error, but does not return hallucination detection results. First check whether the request passes in referenceContent.
The values of fields such as
Label,Description, andConfidencein the moderation result are subject to the actual response of the API.The model hallucination detection capability is in public preview.
Code description
Code | Status code | Description |
200 | OK | Cause: The request is processed successfully. |
400 | BAD_REQUEST | Cause: The request is invalid, usually because the request parameters are incorrect. |
408 | PERMISSION_DENY | Cause: The request failed authorization. Possible causes include an unauthorized account, an account with overdue payments, an account for which the service is not activated, or a disabled account. |
500 | GENERAL_ERROR | Cause: A temporary error occurred on the server. |
581 | TIMEOUT | Cause: The request timed out. |
588 | EXCEED_QUOTA | Cause: The request frequency exceeds the quota. This operation allows up to 50 queries per second (QPS) for a single user (10 QPS by default when files are involved). |
What do I do if the model hallucination dimension is enabled but no model hallucination detection item is returned?
When the model hallucination (in public preview) dimension does not take effect in the returned result, no error code is returned: the API returns a normal result (Code 200), but the Detail does not contain a detection item whose Type is modelHallucination. In this case, troubleshoot in the following order:
Make sure that the
ServiceParametersof the request contains thereferenceContentparameter. Model hallucination detection relies on this parameter to compare against the content to be moderated. If the parameter is not passed in, no hallucination detection result is returned. For the parameter description and self-check examples, see the "Request parameters" and "Examples" sections of this topic.Make sure that the Service is a text-based service (such as AI input Content Moderation and AI-generated Content Moderation). The model hallucination dimension is supported only by text-based Services; image, file, video, and audio services do not support this dimension.
Log on to the AI Guardrails console, choose Protection Config > Model protection, select the corresponding Service, go to Configuration management > Protection dimensions, and make sure that the Model Hallucination card is enabled.
In the Model Hallucination card, in Configuration management, check Hallucination Degree: this setting adjusts the interception score of model hallucination detection. Content is determined to be a model hallucination only when the score returned by the model exceeds the configured value. Value range: 60 to 100. Default value: 60. If the determination score of the content does not reach the configured interception score, no hallucination detection result is returned.
If all the preceding items are confirmed but the hallucination detection result is still not returned, submit a ticket and provide the RequestId.