Ingest Tencent Cloud log data

Enable Kafka protocol consumption in CLS

For detailed instructions, see the official Tencent Cloud documentation: Consume Logs over Kafka.

1. Go to the Tencent Cloud Log TopicTencent Cloud - User listTencent Cloud - Policies page and select the log storage Region in the upper-left corner.

2. Click the target Log Topic to open its details page.

   • Web Application Firewall: Typically found under the plain Logset. For details, see Ship logs to CLS.

   • Cloud Firewall: You can find the log topic information on the CLS log shipping page.

   • Cloud Firewall: You can find the log topic information on the CLS log shipping page.

3. In the left navigation pane, click Consumption over Kafka. On the Basic Information tab, click Edit, then turn on Current Status. Configure the settings: Click OK.

   Setting	Value
   Timestamp range	History + Latest
   Consumer data format	JSON (select Disable Escape) or Raw Content
   Data compression format	No Compression
   Public access	Enabled
   Service log	Enabled

4. After completing the configuration, view the consumer parameters to get the connection details needed for the next steps:

   Parameter	Description
   Public endpoint	Format: kafkaconsumer-${region}.cls.tencentcs.com:9096
   topic	The Kafka topic
   username	Set to ${LogSetID} (the Logset ID)
   password	Set to ${SecretId}#${SecretKey}

Set up an AccessKey pair

Use either a main account key or a sub-account key with minimum required permissions.

• Main account key: Go to Tencent Cloud API Key Management and click Create Key. Save the generated SecretId and SecretKey by clicking Download CSV FileSub-account Access Key Management or copying them to a local file. For more information, see Managing main account access keysManaging sub-account access key pairsManaging main account access key pairs.

  You can use either an API key or a project key.

• Sub-account key:

  1. On the Tencent Cloud Policies page, create a policy with minimum required permissions. For more information, see Authorization for Kafka Protocol Consumption and Create custom policy by policy syntaxCreate custom policy by policy syntax.

     {
         "version": "2.0",
         "statement": [{
             "action": [
                 "cls:PreviewKafkaRecharge",
                 "cls:CreateKafkaRecharge",
                 "cls:ModifyKafkaRecharge"
             ],
             "resource": "*",
             "effect": "allow"
         }]
     }

  2. Go to the Tencent Cloud User List page and select an existing sub-account or create a new one.

     • Attach the policy you created.

     • On the User Details page, go to the API Key tab and click Create Key. Save the generated SecretId and SecretKey. For more information, see .

Grant Security Center access to Kafka

1. Go to Security Center console - System Configuration - Feature Settings. In the upper-left corner, select the region where your assets are located: Chinese Mainland or Outside Chinese Mainland.

2. On the Multi-cloud Configuration Management tab, select Multi-cloud Assets, then click Grant Permission. In the panel, configure:

   Parameter	Value
   Vendor	Apache
   Connection type	Kafka
   Endpoint	Public access address from Tencent Cloud CLS
   Username	Kafka username from Tencent Cloud CLS
   Password	SecretId#SecretKey (concatenated from Set up an AccessKey pair)
   Communication protocol	sasl_plaintext
   SASL authentication mechanism	plain

3. Under Configure synchronization policy, set the AK Service Status Check interval — the frequency at which Security Center verifies the Tencent Cloud AccessKey pair. Select Disable to turn off this check.

Create a data import task

1. Create a data source (skip if you already have one for Tencent Cloud logs).

   1. Go to . Select the region: Chinese Mainland or Outside Chinese Mainland.

   2. On the Data Source tab, create a data source to receive logs from Tencent Cloud. For instructions, see Set up data sources.

      • Source data source type: Select User Log Service or Agentic SOC Dedicated Collection Channel.

      • Add instances: Create a new Logstore to isolate Tencent Cloud log data.

2. On the Data Import tab, click Add Data. Configure:

   Parameter	Value
   Endpoint	Public access address for Kafka protocol consumption from Tencent Cloud
   Topics	Consumer topic from Tencent Cloud
   Value type	json (if Consumer Data Format is JSON) or text (if Raw Content)
   Data source name	The data source created in step 1
   Target Logstore	Auto-fetched from the selected data source

3. Click OK. Security Center begins pulling logs from Tencent Cloud automatically.

Create a CLS-to-COS shipping task

For detailed instructions, see the official Tencent Cloud documentation: Create a Shipping Task to COS.

1. Go to the Tencent Cloud - log topic page and select the log storage Region.

2. Click the target Log Topic to open its details page.

   • Web Application Firewall: Typically found under the waf_post_logset Logset. For details, see Ship logs to CLS.

3. In the left navigation pane, select Shipping to COS, then click Add Shipping Configuration. Configure the shipping task:

   If a confirmation page for log archival appears, click Still Ship To COS to proceed.

   • Basic configuration:

     Setting	Value
     Time range	No end time (required for ongoing data analysis)
     File size	Trigger value for log delivery. When accumulated log size reaches this value, logs are delivered to COS.
     Shipping interval	Time interval for delivery. Logs from each interval are compressed and sent to COS.

     Important

     Important: File size and shipping interval use a logical OR relationship — delivery triggers when either condition is met.

   • Bucket configuration:

     Setting	Value
     COS bucket and Cloud Firewall	Select or create a bucket to store WAF and Cloud Firewall (CFW) logs
     File naming	Delivery time naming (recommended for easy data identification)
     File compression	gzip or No Compression
     COS storage class	Standard. See Storage Class Overview.

     Warning

     Warning: Security Center does not support lzop or snappy compression.

   • Advanced configuration:

     Setting	Value
     Consumer data format	JSON
     JSON	Disable Escape

4. Get the COS bucket endpoint. Go to the Tencent Cloud Bucket List page and open the bucket you configured. Copy the domain name from the Domain Information section.

   Important

   The endpoint must not include the bucket name. The format is cos.${region}.myqcloud.com.

   

Set up an AccessKey pair

Use either a main account key or a sub-account key with minimum required permissions.

• Main account key: Go to Tencent Cloud API Key Management and click Create Key. Save the generated SecretId and SecretKey. For more information, see .

  You can use either an API key or a project key.

• Sub-account key:

  1. On the page, create a policy with minimum required permissions. For more information, see Authorization for Shipping to COS and Create custom policy by policy syntaxCreate custom policy by policy syntax.

     {
         "version": "2.0",
         "statement": [
             {
                 "effect": "allow",
                 "action": [
                     "cls:DescribeTopics",
                     "cls:DescribeLogsets",
                     "cls:DescribeIndex",
                     "cls:CreateShipper"
                 ],
                 "resource": "*"
             },
             {
                 "effect": "allow",
                 "action": [
                     "tag:DescribeResourceTagsByResourceIds",
                     "tag:DescribeTagKeys",
                     "tag:DescribeTagValues",
                     "cls:ModifyShipper",
                     "cls:DescribeShippers",
                     "cls:DeleteShipper",
                     "cls:DescribeShipperTasks",
                     "cls:RetryShipperTask",
                     "cls:DescribeShipperPreview",
                     "cos:GetService",
                     "cam:ListAttachedRolePolicies",
                     "cam:AttachRolePolicy",
                     "cam:CreateRole",
                     "cam:DescribeRoleList"
                 ],
                 "resource": "*"
             }
         ]
     }

  2. Go to the page and select an existing sub-account or create a new one.

     • Attach the policy you created.

     • On the User Details page, go to the API Key tab and click Create Key. Save the generated SecretId and SecretKey. For more information, see Managing sub-account access keys.

Grant Security Center access to COS

1. Go to . Select the region: Chinese Mainland or Outside Chinese Mainland.

2. On the Multi-cloud Configuration Management tab, select Multi-cloud Assets, click Grant Permission, then select IDC from the dropdown. In the panel, configure:

   Parameter	Value
   Vendor	AWS-S3
   Connection type	S3
   Endpoint	COS bucket access domain name (from Create a CLS-to-COS shipping task)
   Access Key ID	SecretId from Set up an AccessKey pair
   Secret Access Key	SecretKey from Set up an AccessKey pair

3. Under Configure synchronization policy, set the AK Service Status Check interval. Select Disable to turn off this check.

Create a data import task

1. Go to Security Center console > Agentic SOC > Integration Center. Select the region: Chinese Mainland or Outside Chinese Mainland.

2. On the Data Import tab, click Add Data. Configure:

   Parameter	Value
   Endpoint	COS bucket access domain name
   Bucket	COS bucket name
   Data source name	A custom data source with normal status (Custom Log Capability or Agentic SOC Dedicated Data Collection Channel). If none exists, see Set up data sources.
   Target Logstore	Auto-fetched from the selected data source

3. Click OK. Security Center begins pulling logs from Tencent Cloud automatically.