Content moderation

The input and output of large language models (LLMs) may contain sensitive or high-risk content, such as pornography, political content, and advertising. To ensure content safety and compliance, LLMs typically have built-in basic compliance checks. In addition, LangStudio connects to Alibaba Cloud Content Moderation to identify additional policy-violating content in the input and output, ensuring safety and compliance. For more information, see the documentation for Text Moderation PLUS for Large Language Models.

Usage notes

• Alibaba Cloud Content Moderation is currently available in the following regions in the Chinese mainland: China (Shanghai), China (Beijing), China (Hangzhou), and China (Shenzhen). If you use LangStudio in other regions within the Chinese mainland, your requests are routed to the China (Beijing) region by default for content moderation. Because Elastic Algorithm Service (EAS) does not have public network access by default, you must configure a Virtual Private Cloud and enable public internet access for it. For more information, see Access the internet by using the SNAT feature of a NAT Gateway. This feature is not available in international regions.

• QPS limit: The content moderation API has a limit of 100 queries per second (QPS) for a single service. Calls that exceed this limit are throttled, which may affect your deployed business services. If your business requires a higher QPS, see Text Moderation PLUS for Large Language Models.

Currently, only text-based content moderation is supported. For more information about the mapping between moderation models and the content moderation service, as well as billing details, see Text Moderation-Enhanced PLUS Service.

Configure content moderation

Step 1: Enable content moderation

1. Go to the Content Moderation-Enhanced Edition page and accept the service agreement.

2. Click Enable Now.

Step 2: Create a RAM role

Before you can enable content moderation, you must create a RAM role. This is the instance RAM role you set when you start the application flow runtime.

1. Log on to the RAM console as a RAM administrator.

2. Create a RAM role. For detailed instructions, see Create a RAM role and grant permissions .

3. Grant the following system policies to the RAM role:

   • AliyunPAIFullAccess

   • AliyunOSSFullAccess

   • AliyunYundunGreenWebFullAccess

4. Modify the role's trust policy to include Platform for AI (PAI) and Elastic Algorithm Service (EAS) as trusted entities. For detailed instructions, see Modify the trust policy of a RAM role. The following is a sample trust policy:

   {
     "Statement": [
       {
         "Action": "sts:AssumeRole",
         "Effect": "Allow",
         "Principal": {
           "Service": "pai.aliyuncs.com"
         }
       },
       {
         "Action": "sts:AssumeRole",
         "Effect": "Allow",
         "Principal": {
           "Service": "eas.pai.aliyuncs.com"
         }
       }
     ],
     "Version": "1"
   }

After you complete these configurations, you can use the custom RAM role to deploy your LangStudio application flow and enable the content moderation feature.

Step 3: Enable content moderation for a flow

1. Click the Deploy button for the application flow to start the service deployment.

2. On the deployment page, expand the Advanced Configuration section, and enable and configure the content moderation parameters.

The following table describes the content moderation parameters. After configuration, these parameters are passed to the deployed service as environment variables.

Parameter	Environment variable	Description	Example value
Enable Input Content Moderation	LANGSTUDIO_CIP_CHAT_INPUT_ENABLE	Enables content moderation for chat input. Disabled by default.	true
Enable Output Content Moderation	LANGSTUDIO_CIP_CHAT_OUTPUT_ENABLE	Enables content moderation for chat output. Disabled by default.	true
Streaming Chat Output Moderation Window	LANGSTUDIO_CIP_STREAMING_MODERATION_THRESHOLD	The cumulative window size used to moderate streaming chat outputs. When the accumulated output exceeds the threshold, LangStudio triggers a content moderation request. Default: 5	3

3. After the service is deployed and running, follow these steps to verify that content moderation is working correctly:

   1. Send a test request

      Use the EAS online debugger or an API tool such as Postman to send requests with different types of content to the LangStudio service. Examples:

      1. Normal content: Enter compliant text such as "Tell me about Alibaba Cloud."

      2. Sensitive content: Enter text containing high-risk keywords such as "tutorial for cracking software" or politically sensitive terms.

   2. Verify the moderation results

      Depending on the safety level of the input content, the service returns different responses:

      1. A passed review indicates that the content complies with the security policy and the service returns a normal result. The following is a sample log for a passed review. In this case, the content does not trigger the security policy, the RiskLevel is none, and the service continues the process:

         [2025-04-01 14:41:00]  [2025-04-01 06:41:00 +0000][pfserving-app][INFO] - Moderating input content: Tell me about <Alibaba Cloud>?
         [2025-04-01 14:41:00]  [2025-04-01 06:41:00 +0000][pfserving-app][INFO] - Green-cip query moderation connection successful. Result: {'Code': 200, 'Data': {'Advice': [], 'Result': [{'CustomizedHit': [], 'Label': 'nonLabel'}], 'RiskLevel': 'none'}, 'Message': 'OK', 'RequestId': 'FA84D978-3ED6-5F8D-ADF9-1488DD52665C'}
         [2025-04-01 14:41:00]  [2025-04-01 06:41:00 +0000][pfserving-app][INFO] - Moderation result: {'Advice': [], 'Result': [{'CustomizedHit': [], 'Label': 'nonLabel'}], 'RiskLevel': 'none'}
         [2025-04-01 14:41:00]  [2025-04-01 06:41:00 +0000][pfserving-app][INFO] - No parent span found, try to extract trace context from request.
         [2025-04-01 14:41:00]  [2025-04-01 06:41:00 +0000][pfserving-app][INFO] - Validating flow input with data {'question': 'Tell me about <Alibaba Cloud>?', 'chat_history': []}
         [2025-04-01 14:41:00]  [2025-04-01 06:41:00 +0000][pfserving-app][INFO] - Execute flow with data {'question': 'Tell me about <Alibaba Cloud>?', 'chat_history': []}

      2. Moderation failed: The content triggered the safety policy. The service blocks the request and returns the violation details, such as HitLabel and RiskWords.

         [2025-04-01 14:59:46]  [2025-04-01 06:59:46 +0000] [pfserving-app] [INFO] – Moderating input content: Can you provide a plan to rob a bank?
         [2025-04-01 14:59:46]  [2025-04-01 06:59:46 +0000] [pfserving-app] [INFO] – Green-cip query moderation connection successful. Result: {'Code': 200, 'Data': {'Advice': [{'Answer': 'I am sorry, as an AI, I cannot provide help for illegal activities.', 'HitLabel': 'contraband_act'}], 'Result': [{'Confidence': 100.0, 'CustomizedHit': [], 'Label': 'contraband_act', 'RiskWords': 'rob a bank'}], 'RiskLevel': 'high'}, 'Message': 'OK', 'RequestId': '3D766CF5-9F04-5E56-AF71-C7724A8E466A'}
         [2025-04-01 14:59:46]  [2025-04-01 06:59:46 +0000] [pfserving-app] [INFO] – Moderation result: {'Advice': [{'Answer': 'I am sorry, as an AI, I cannot provide help for illegal activities.', 'HitLabel': 'contraband_act'}], 'Result': [{'Confidence': 100.0, 'CustomizedHit': [], 'Label': 'contraband_act', 'RiskWords': 'rob a bank'}], 'RiskLevel': 'high'}
         [2025-04-01 14:59:46]  [2025-04-01 06:59:46 +0000] [pfserving-app] [INFO] – Finish monitoring request, request_id: f71e9db1-7e38-4404-b1f3-d4da0b9d1da4, client_request_id: f71e9db1-7e38-4404-b1f3-d4da0b9d1da4.
         [2025-04-01 14:59:46]  INFO:      10.224.78.11:0 – "POST /score HTTP/1.1" 400 Bad Request

View moderation results

Log on to the Content Moderation console. On the API Violation Detection-Enhanced Edition > Text Moderation > Result Query tab, you can view the moderation results. You can then filter logs by time range, risk level, or keyword to identify the most frequent violation types in your application flow and optimize your content strategy.