DocsICP FilingConsole
官方文档
首页

Ingest log data from Huawei Cloud

更新时间：
复制 MD 格式
产品详情
我的收藏

In a multi-cloud environment, security logs are often scattered across different cloud platforms, making unified threat detection and incident response difficult. Security Center's Agentic SOC feature lets you centrally import and analyze security logs from Huawei Cloud products — including Web Application Firewall (WAF) and Cloud Firewall (CFW) — so you can manage security across all your cloud environments from one place.

How it works

  1. Log aggregation: WAF and CFW logs are collected into Huawei Cloud's Log Tank Service (LTS).

  2. Data export: LTS exports log data to Distributed Message Service (DMS) for Kafka or Object Storage Service (OBS), which act as relay points for cross-cloud transfer.

  3. Cross-cloud import: Agentic SOC subscribes to and pulls log data from DMS for Kafka (using the Kafka protocol) or OBS (using the S3 protocol), then ingests it into a specified data source.

  4. Ingestion and standardization: Inside Agentic SOC, an ingestion policy applies standardization rules to parse and normalize the raw logs before storing them in the data warehouse.

image

Supported log types

Agentic SOC supports importing the following log types from Huawei Cloud:

  • Web Application Firewall (WAF) alert logs

  • Cloud Firewall (CFW) alert logs

Step 1: Send logs to LTS

Before importing logs, send all security product logs from Huawei Cloud to LTS.

Web Application Firewall

For detailed instructions, see the Huawei Cloud documentation: Using LTS to Record WAF Logs.

  1. Log in to the Web Application Firewall console. In the upper-left corner, select a region or project, then click Events in the left navigation pane.

  2. On the Log Settings tab, click Connect to LTS and configure the following parameters:

    Important

    The configuration takes about 10 minutes to take effect.

    Parameter Value
    Log Types WAF access logs and WAF attack logs
    Log Group Select the log group where you want to store the logs. Click Create Log Group to create a new one.
    WAF Access Log Stream If you selected WAF access logs, select a WAF access log stream. Click Create Log Stream to create a new one.
    WAF Attack Log Stream If you selected WAF attack logs, select a WAF attack log stream. Click Create Log Stream to create a new one.

Cloud Firewall

For detailed instructions, see the Huawei Cloud documentation: Configuring CFW LogsIngesting CFW Logs to LTS.

  1. Create a log group and log stream.

    1. Log in to the Cloud Log Service console. On the Log Management page, click Create Log Group.

    2. On the Create Log Group page, set Log Group Name and Log Retention Period (Days).

      Note

      We recommend adding the suffix -cfw to the log group name (for example, mylog-cfw) for easier identification.

    3. After the log group is created, find it in the list and click Create Log Stream under the image.png icon.

    4. On the Create Log Stream page, set Log Stream Name and Log Storage Duration (Days).

      Note

      Use suffixes like -attack, -access, and -flow for attack event logs, access control logs, and traffic logs, respectively.

      CFW supports three log stream types:

      Log type Description
      Attack logs Records attack alerts, including event type, protection rule, action, 5-tuple, attack payload, and other details.
      Access logs Records traffic that matches ACL policies, including hit time, 5-tuple, response action, access control rule, and other details.
      Traffic logs Records all traffic passing through the Cloud Firewall, including start time, end time, 5-tuple, byte count, packet count, and other details.

  2. Set up LTS synchronization.

    1. Log in to the Cloud Firewall console. In the upper-left corner, select the region and firewall instance, then choose Log Audit > Log Management in the left navigation pane.

    2. On the Log Management page, click Configure LTS Synchronization. Set Log Group and Log Source to the log group and log stream you created.

Step 2: Choose an import method

Two methods are available for importing Huawei Cloud LTS logs into Security Center. Choose based on your real-time requirements, cost constraints, and configuration complexity.

Aspect Kafka (DMS) OBS
Real-time performance Near-real-time (real-time transfer can be configured) Minute-level latency
Configuration complexity Higher. Requires configuring a Kafka instance, Elastic IP Addresses (EIPs), security groups, and more. Lower. Only requires configuring a transfer task.
Cost (Huawei Cloud) Kafka instance, EIP and traffic, Log Service OBS storage, Log Service
Cost (Alibaba Cloud) Agentic SOC log ingestion traffic Agentic SOC log ingestion traffic
Best for Scenarios requiring near-real-time log analysis, such as stream-based security computing or rapid alert response Scenarios where real-time performance is not critical, focusing on cost-effectiveness, log archiving, or batch offline analysis

Step 3: Configure the data import

Follow the instructions for your chosen import method.

Import data using Kafka (DMS)

Prepare the Kafka data channel on Huawei Cloud

Configure a Kafka instance

  1. Create a Kafka instance.

    1. Go to the Buy Kafka Instance pageCreating a Kafka Instance. On the Quick Config tab, complete the basic and network configurations, including instance specifications and a Virtual Private Cloud (VPC).

    2. In the Access Mode area, select Public Network Access and configure the following parameters:

      Parameter Value
      Public Network Access Select Ciphertext Access.
      Public IP Addresses Select an accessible Elastic IP Address (EIP). If you don't have enough EIPs, click Create Elastic IP to go to the EIP purchase page. For more information, see the Huawei Cloud documentation: Apply for an EIP. After purchase, click the image.png icon next to Elastic IP Address and select the newly purchased EIPs from the drop-down list.
      Kafka Security Protocol SASL_SSL: uses SASL for authentication and SSL certificates for data encryption. SASL_PLAINTEXT: uses SASL for authentication and transmits data in plaintext for better performance.
      SASL PLAIN Mechanism If you set Kafka Security Protocol to SASL_PLAINTEXT, select CRAM-SHA-512.
      Username / Password The credentials the client uses to connect to the Kafka instance. The username cannot be changed after encrypted access is enabled.
      Important

      Purchase at least three EIPs. Save the username and password — you’ll need them later to grant Security Center access to Kafka.

    For more information, see the Huawei Cloud documentation: .

  2. Create a topic.

    1. Go to the Huawei Cloud Kafka Management page. In the upper-left corner, select the region where your Kafka instance is located.

    2. In the left navigation pane, click Kafka Instances. Click the name of your target instance to open its details page, then click Topic Management.

    3. Click Create Topic and configure the parameters. The default settings work for most use cases.

    For more information, see the Huawei Cloud documentation: .

  3. Configure security group rules. After enabling public access, configure security group rules to allow connections to Kafka.

    1. On the Kafka instance details page, click Overview in the left navigation pane. In the Network section, click the image icon next to Security Group.

    2. On the policy configuration page, go to the Inbound Rules tab, click Add Rule, and set the following:

      Field Value
      Policy Allow
      Type IPv4
      Protocol Custom TCP
      Port 9095
      Source 0.0.0.0/0

  4. Note the Kafka connection parameters. On the Kafka instance Overview page, record the Address (Public Network, Ciphertext), the enabled Security Protocol, and the SASL PLAIN Mechanism. You'll need these when connecting Security Center to Kafka.

    For example, the enabled Security Protocol is SASL_SSL, and the enabled SASL Authentication Mechanism is SCRAM-SHA-512. You can also click Download next to SSL Certificate to get the certificate file.

Create a transfer task from LTS to Kafka

For detailed instructions, see the Huawei Cloud documentation: Transferring Logs to DMSIngesting WAF Logs to LTS.

  1. Log in to the . In the left navigation pane, click Log Transfer, then click Configure Log Transfer in the upper-right corner.

  2. Set the following transfer parameters:

    Parameter Value
    Transfer Mode Periodic transfer
    Transfer Destination DMS
    Log Group Name / Log Stream Name The log group and stream you configured in Step 1 (for example, WAF attack logs)
    Kafka Instance The Kafka instance you configured
    Topic The topic you created
    Transfer Interval Real-time
    Format Raw Log Format or JSON

Configure the Kafka log import on Alibaba Cloud

Grant Security Center access to Kafka

  1. Go to Security Center consoleSecurity Center console > Agentic SOC > Integration Center. In the upper-left corner, select your asset region: Chinese Mainland or Outside Chinese Mainland.

  2. On the Multi-cloud Configuration Management tab, select Multi-cloud Assets, click Grant Permission, and select IDC from the drop-down list. In the panel that appears, set the following:

    Parameter Value
    Vendor Apache
    Connection Type Kafka
    Endpoint The IPv4 Encrypted Public Endpoint for Kafka you recorded from Huawei Cloud
    Username / Password The Kafka credentials you configured on Huawei Cloud
    Communication Protocol The security protocol you enabled on Huawei Cloud
    SASL Authentication Mechanism The SASL PLAIN Mechanism you configured on Huawei Cloud

  3. Under Configure synchronization policy, set AK Service Status Check to the interval at which Security Center checks the validity of the Huawei Cloud access key. Select Disable to turn off this check.

Create a data import task

  1. Create a data source for the Huawei Cloud log data. Skip this step if you've already created one.

    1. Go to Security Center console > Agentic SOC > Integration Center. In the upper-left corner, select your asset region.

    2. On the Data Source tab, create a data source for the Huawei Cloud logs. For instructions, see Create a data source: Logs are not ingested into Simple Log Service (SLS).

      Parameter Value
      Source Data Source Type Select User Log Service or Agentic SOC Dedicated Collection Channel.
      Add Instances Create a new Logstore to isolate the data.

  2. On the Data Import tab, click Add Data. In the panel that appears, set the following:

    Parameter Value
    Endpoint The IPv4 Encrypted Public Endpoint for Kafka
    Topics The topic you created on Huawei Cloud
    Value Type See the mapping below
    Transfer format Value type
    JSON format json
    Raw Log Format text

  3. Under Configure the destination data source, set the following:

    • Data Source Name: Select the data source you created.

    • Destination Logstore: Logstores under the selected data source are loaded automatically.

  4. Click OK. Security Center begins pulling logs from Huawei Cloud automatically.

Import data using OBS

Prepare OBS data on Huawei Cloud

Configure LTS to transfer logs to OBS

  1. Create a transfer task.

    1. Log in to the Log Service console. In the left navigation pane, click Log Transfer, then click Configure Log Transfer in the upper-right corner.

    2. Set the following transfer parameters:

      Parameter Value
      Transfer Mode Periodic transfer
      Transfer Destination OBS Bucket
      Log Group Name / Log Stream Name The log group and stream you configured in Step 1 (for example, WAF access log stream)
      OBS Bucket Select an existing OBS bucket or create a new one on the Huawei Cloud Bucket List page
      Custom Log Transfer Path Enabled: set a custom path in the format /LogTanks/RegionName/%GroupName/%StreamName/<custom_transfer_path> (default: lts/%Y/%m/%d). Disabled: logs go to the default path LogTanks/RegionName/2019/01/01/<Log_Group>/<Log_Stream>/<log_file_name>.
      Compression Format uncompressed, gzip, or zip
      Note

      LTS can transfer logs to OBS buckets that use the Standard or Restored Archive storage class.

      Warning

      Security Center does not support parsing log files compressed in the snappy format.

    For detailed instructions, see the Huawei Cloud documentation: Transferring Logs to OBS.

  2. Get the OBS bucket endpoint.

    1. Go to the Huawei Cloud - Bucket List page. Locate the OBS bucket you configured for LTS log transfer and open its details page. In the left navigation pane, click Overview.

    2. In the Domain Name area, note the Endpoint. The format is obs.${region}.myhuaweicloud.com.

Create an access key

  1. Go to the Huawei Cloud My Credentials page. In the left navigation pane, click Access Keys.

  2. Click Create Access Key. Either click Download CSV File or copy the Access Key ID and Secret Access Key to a local file for safekeeping. For more information, see Access Keys.

Configure the OBS log import on Alibaba Cloud

Grant Security Center access to Huawei Cloud OBS

  1. Go to Security Center console > Agentic SOC > Integration Center. In the upper-left corner, select your asset region: Chinese Mainland or Outside Chinese Mainland.

  2. On the Multi-cloud Configuration Management tab, select Multi-cloud Assets, click Grant Permission, and select IDC from the drop-down list. In the panel that appears, set the following:

    Parameter Value
    Vendor AWS-S3
    Connection Type S3
    Endpoint The OBS bucket endpoint (format: obs.${region}.myhuaweicloud.com)
    Access Key ID / Secret Access Key The access key you created on Huawei Cloud

  3. Under Configure synchronization policy, set AK Service Status Check to the interval at which Security Center checks the validity of the Huawei Cloud access key. Select Disable to turn off this check.

Create a data import task

  1. Go to Security Center console > Agentic SOC > Integration Center. In the upper-left corner, select your asset region: Chinese Mainland or Outside Chinese Mainland.

  2. On the Data Import tab, click Add Data. In the panel that appears, set the following:

    Parameter Value
    Endpoint The OBS bucket endpoint
    OBS Bucket The OBS bucket where LTS transfers logs
    File Path Prefix Filter Filter S3 files by file path prefix to pinpoint the files to import. For example, if all files to import are in the csv/ directory, set the prefix to csv/. For details on the file path, see the custom transfer path you configured earlier.
    Compression Format Select the compression format that matches your OBS transfer configuration. Auto-detection is also supported.
    Note

    We strongly recommend setting File Path Prefix Filter. If you do not set this parameter, the system traverses the entire S3 bucket. When the bucket contains a large number of files, a full traversal significantly slows down the import.

  3. Under Configure the destination data source, set the following:

    • Data Source Name: Select a custom data source with a normal status (Custom Log Capability or Agentic SOC Dedicated Data Collection Channel). If no suitable data source is available, create one. For instructions, see Data sources.

    • Destination Logstore: Logstores under the selected data source are loaded automatically.

  4. Click OK. Security Center begins pulling logs from Huawei Cloud automatically.

Step 4: Analyze the imported data

After the data is ingested, set up parsing and detection rules.

  1. Create an ingestion policy. Follow the instructions in Connect products to Agentic SOC 2.0 to create an ingestion policy with the following settings:

    Parameter Value
    Data Source Select the destination data source you configured in the data import task.
    Standardized Rule Select from the built-in standardization rules for Huawei Cloud products.
    Standardization Method For alert logs, this is set to Real-time Consumption by default and cannot be changed.

  2. Configure threat detection rules. Enable or create log detection rules in rule management to analyze logs, generate alerts, and create security events. For instructions, see Configure threat detection rules.

Built-in standardization rules include predefined rules such as WAF Alert Log Standardization Rule and Cloud Firewall Alert Log Standardization Rule. You can filter by vendor in the standardization rules list to view and select the applicable rules.

Billing

This solution incurs costs from both cloud platforms. Review the billing documentation for each product before proceeding.

Huawei Cloud costs (data transfer and storage):

Service Billable items Billing documentation
LTS Log storage, read/write operations, and more Huawei Cloud LTS - Billing overview
DMS for Kafka Instance specifications, public network traffic, and more Huawei Cloud Kafka - Billing overview
OBS Storage capacity, number of requests, public network traffic, and more Huawei Cloud OBS - Billing overview

Alibaba Cloud costs (depend on the data storage method you choose):

For Agentic SOC billing, see Billing details and Pay-as-you-go billing for Threat Analysis and Response. For Simple Log Service (SLS) billing, see SLS billing overview.
Data source type Agentic SOC billable items SLS billable items Notes
Agentic SOC Dedicated Collection Channel Log ingestion fee + log storage and write fees (both consume Log Ingestion Traffic) Fees for items other than log storage and writes (such as public network traffic) Agentic SOC creates and manages the SLS resources. Log storage and write fees are billed through Agentic SOC.
User Log Service Log ingestion fee (consumes Log Ingestion Traffic) All log-related fees (storage, writes, public network traffic, and more) All log resources are managed by SLS. All log-related fees are billed through SLS.

FAQ

No log data appears in SLS after creating a data import task

Check in this order:

  1. Huawei Cloud side: Log in to the Huawei Cloud console and confirm that logs are generated and delivered to the configured LTS log stream, Kafka topic, or OBS bucket.

  2. Credentials: In Security Center, go to the Multi-cloud Assets page and confirm the authorization status is normal and the access key is valid.

  3. Network connectivity (Kafka method only): Confirm that public access is enabled for the Kafka service and that the security group rules allow inbound traffic from Security Center's service IP addresses.

  4. Data import task: Go to the Data Import page in Security Center to review task status and error logs, then make corrections.

Why select `Apache` or `AWS-S3` instead of `Huawei Cloud` when granting permission?

The log import feature uses standard, protocol-compatible interfaces rather than vendor-specific APIs.

  • IDC is the drop-down value that represents the protocol vendor. Apache represents the Kafka protocol, and AWS-S3 represents the S3-compatible object storage protocol.

  • Authorizing Huawei Cloud as a vendor enables Agentic SOC to coordinate security event responses with Huawei Cloud — such as blocking an IP address using threat detection rules — but does not enable log import.

Previous：General data importNext：Ingest Tencent Cloud log data