Ingest Tencent Cloud log data

Step 1: Configure Kafka protocol consumption and obtain access keys on Tencent Cloud

Create a CLS-to-Kafka protocol consumption task. For detailed instructions, see the official Tencent Cloud documentation: Consume logs over Kafka.

• Enable Kafka protocol consumption

  • Go to the Tencent Cloud - Log Topic page and select the corresponding log storage region in the upper-left corner.

  • Click the target Log Topic name to open its details page.

    • Web Application Firewall: Typically found under the waf_post_logset Logset. For details, see Ship logs to CLS for Web Application Firewall.

    • Cloud Firewall: Find the log topic information on the CLS log shipping page.

  • In the left navigation pane, click Consumption over Kafka. On the Basic Information tab, click Edit and turn on Current Status. Configure the following settings, then click OK.

    • Timestamp range: History + Latest.

    • Consumer data format: JSON (select Disable Escape) or Raw Content.

    • Data compression format: No Compression.

    • Public access: Enabled.

    • Service log: Enabled.

• Get the Kafka service connection details

  After configuration, click View Consumer Parameters to get the required connection details: the CLS public service address (endpoint), username, and consumer topic for use in subsequent steps.

  Parameter	Description
  Public endpoint	Format: kafkaconsumer-${region}.cls.tencentcs.com:9096.
  Consumer topic	The Kafka topic.
  Username	Set to ${LogSetID} (the Logset ID).
  Password	Set to ${SecretId}#${SecretKey}.

Set up access keys

• Main account key: Go to the Tencent Cloud - API Key Management page and click Create Key. Securely store the generated SecretId and SecretKey (click Download CSV File or copy them to a local file). For more information, see Main account access key management.

  You can use either an API key or a project key.

• Sub-account key:

  • On the Tencent Cloud - Policies page, create a minimum-privilege access policy. For more information, see Kafka protocol consumption authorization and Create custom policy by policy syntax.

    {
        "version": "2.0",
        "statement": [{
            "action": [
                "cls:PreviewKafkaRecharge",
                "cls:CreateKafkaRecharge",
                "cls:ModifyKafkaRecharge"
            ],
            "resource": "*",
            "effect": "allow"
        }]
    }

  • Go to the Tencent Cloud - User List page. Select an existing sub-account or create a new one.

  • Attach the policy you created.

  • On the User Details page, go to the API Key tab and click Create Key. Securely store the generated SecretId and SecretKey (click Download CSV File or copy them to a local file). For more information, see Sub-account access key management.

Step 2: Configure Kafka log import in Alibaba Cloud

Grant Security Center access to Kafka

Access the Security Center console - System Settings - Feature Settings. At the top of the left side of the page, select the region where the asset to be protected is located: Chinese Mainland or Outside Chinese Mainland.

• On the Multi-cloud Configuration Management tab, select Multi-cloud Assets, then click Grant Permission. In the panel, configure:

  • Vendor: Select Apache.

  • Connection type: Select Kafka.

  • Endpoint: Enter the Tencent Cloud Kafka protocol consumption public endpoint.

  • Username: Enter the Tencent Cloud Kafka protocol consumption username.

  • Password: Concatenate the account key information (SecretId#SecretKey) from Set up access keys.

  • Communication protocol: sasl_plaintext.

  • SASL authentication mechanism: plain.

• Configure synchronization policy

  AK Service Status Check: Set the interval for Security Center to automatically verify Tencent Cloud account access key validity. Select "Disable" to turn off this check.

Step 1: Prepare COS storage and obtain access keys on Tencent Cloud

Create a CLS-to-COS shipping task. For detailed instructions, see the official Tencent Cloud documentation: Create a COS shipping task.

• Create a COS shipping task:

  • Go to the Tencent Cloud - Log Topic page and select the corresponding log storage region in the upper-left corner.

  • Click the target Log Topic name to open its details page.

    • Web Application Firewall: Typically found under the waf_post_logset Logset. For details, see Ship logs to CLS for Web Application Firewall.

    • Cloud Firewall: Find the log topic information on the CLS log shipping page.

  • In the left navigation pane, select Shipping to COS, then click Add Shipping Configuration. Configure as follows:

    If the log archival confirmation page appears, click Still Ship To COS.

    • Basic configuration:

      • Shipping time range: To support ongoing data analysis, do not set an end time.

      • Shipping file size: Set the trigger value. When accumulated log size reaches this value, logs are delivered to COS.

      • Shipping interval: Set the delivery interval. Logs from each interval are compressed and shipped to COS.

      Shipping file size and Shipping interval use a logical OR relationship — delivery triggers when either condition is met.

    • Bucket configuration:

      • COS bucket: Select or create a bucket to store logs from Tencent Cloud products such as WAF and Cloud Firewall.

      • File naming: We recommend selecting Delivery time naming for easy data identification.

      • COS path: The storage path for logs in the COS bucket. The default format is /year/month/day/hour/, for example, /2022/7/31/14/. The path supports strftime syntax.

      • File compression: Select gzip or No Compression.

        Security Center does not support lzop or snappy compression for log file parsing.

      • COS storage class: Select Standard storage. For more information, see Storage class overview.

    • Advanced configuration:

      • Consumer data format: Select JSON.

      • JSON: Select Disable Escape.

  • Get the COS bucket endpoint.

    Go to the Tencent Cloud - Bucket List page, locate the COS bucket selected in the previous step. In the bucket details page, copy the domain name from the Domain Information section.

    • The domain must not include the bucket name. The format is: cos.${region}.myqcloud.com.

    • The bucket endpoint is used in subsequent steps.

Set up access keys

• Main account key: Go to the Tencent Cloud - API Key Management page and click Create Key. Securely store the generated SecretId and SecretKey (click Download CSV File or copy them to a local file). For more information, see Main account access key management.

  You can use either an API key or a project key.

• Sub-account key:

  • On the Tencent Cloud - Policies page, create a minimum-privilege access policy. For more information, see Shipping to COS authorization and Create custom policy by policy syntax.

    {
        "version": "2.0",
        "statement": [
            {
                "effect": "allow",
                "action": [
                    "cls:DescribeTopics",
                    "cls:DescribeLogsets",
                    "cls:DescribeIndex",
                    "cls:CreateShipper"
                ],
                "resource": "*"
            },
            {
                "effect": "allow",
                "action": [
                    "tag:DescribeResourceTagsByResourceIds",
                    "tag:DescribeTagKeys",
                    "tag:DescribeTagValues",
                    "cls:ModifyShipper",
                    "cls:DescribeShippers",
                    "cls:DeleteShipper",
                    "cls:DescribeShipperTasks",
                    "cls:RetryShipperTask",
                    "cls:DescribeShipperPreview",
                    "cos:GetService",
                    "cam:ListAttachedRolePolicies",
                    "cam:AttachRolePolicy",
                    "cam:CreateRole",
                    "cam:DescribeRoleList"
                ],
                "resource": "*"
            }
        ]
    }

  • Go to the Tencent Cloud - User List page. Select an existing sub-account or create a new one.

  • Attach the policy you created.

  • On the User Details page, go to the API Key tab and click Create Key. Securely store the generated SecretId and SecretKey (click Download CSV File or copy them to a local file). For more information, see Sub-account access key management.

Configure COS log import in Alibaba Cloud

Grant Security Center access to COS

Access the Security Center console - System Settings - Feature Settings. At the top of the left side of the page, select the region where the asset to be protected is located: Chinese Mainland or Outside Chinese Mainland.

• On the Multi-cloud Configuration Management tab, select Multi-cloud Assets, then click Grant Permission and select IDC from the dropdown. In the panel, configure:

  • Vendor: Select AWS-S3.

  • Connection type: Select S3.

  • Endpoint: Enter the COS bucket endpoint.

  • Access Key ID / Secret Access Key: Enter the keys from Set up access keys.

• Configure synchronization policy

  AK Service Status Check: Set the interval for Security Center to automatically verify Tencent Cloud account access key validity. Select "Disable" to turn off this check.

Create a data import task

Access the Security Center console - Agentic SOC - Management - Access Settings. At the top of the left side of the page, select the region where the asset to be protected is located: Chinese Mainland or Outside Chinese Mainland.

• On the Data Import tab, click Add Data. In the panel, configure:

  • Data Source Type: S3.

  • Endpoint: Enter the COS bucket endpoint.

  • Bucket: Enter the COS bucket configured for CLS log shipping.

  • File Path Prefix Filter: Filter S3 files by path prefix to locate the files to import. For example, if the target files are in the csv/ directory, specify the prefix as csv/. For the specific file path, see COS path.

    We strongly recommend setting this parameter. If unset, the system traverses the entire S3 bucket. When the bucket contains a large number of files, full traversal significantly impacts import efficiency.

  • Data Format: The file parsing format.

    • CSV: A separator-delimited text file. You can use the first row as field names or manually specify field names. Each row after the field names is parsed as log field values.

    • JSON: Reads each line of the S3 file as a JSON object. After parsing, each field in the JSON object maps to a log field.

    • Text: Parses each line of the S3 file as one log entry.

    • Multi-line Text: Multi-line mode. Supports specifying a regular expression for the first or last line to parse logs.

    If you select CSV or Multi-line Text, configure the additional parameters described below.

    CSV

    Parameter	Description
    Separator	Set the log separator. The default value is a comma (,).
    Quote	The quote character used for CSV strings.
    Escape Character	Set the log escape character. The default value is a backslash (\).
    Max Line Span	When a single log entry spans multiple lines, specify the maximum number of lines. The default value is 10.
    First Row as Header	Turn on this switch to use the first row of the CSV file as field names. For example, the first row of the following CSV data is extracted as log field names. remote_addr,remote_user,time_local,request_time,request_length xxx5,-,11/Dec/2020:15:31:06,0,000,133,3650,404,GET xxx5,-,11/Dec/2020:15:32:06,0,000,133,3650,404,GET xxx5,-,11/Dec/2020:15:34:10,0,000,133,3650,404,GET
    Skip rows	Specify the number of log rows to skip. For example, set to 1 to start collecting logs from the second row of the CSV file.

    Multi-line Text

    Parameter	Description
    Regex Match Position	Set the position for regular expression matching: Prefix Regex: Use the regular expression to match the beginning of a log entry. Unmatched lines are part of the current log entry until the maximum line count is reached. Suffix Regex: Use the regular expression to match the end of a log entry. Unmatched lines are part of the next log entry until the maximum line count is reached.
    Regex	Set the correct regular expression based on your log content. For information about how to adjust the regular expression, see How do I debug a regular expression?.
    Max Lines	The maximum number of lines per log entry.

  • Compression format: Select the compression format that matches the file compression setting configured in COS.

• Configure the target data source

  • Data source name: Select a custom data source with normal status (Custom Log Capability / Agentic SOC Dedicated Data Collection Channel). If none exists, see Set up data sources to create one.

  • Target Logstore: Automatically fetched from the selected data source.

  • Click OK to save the configuration. After configuration, Security Center automatically pulls logs from Tencent Cloud.