VPC flow logs

Console

API

Before you create a flow log, ensure you have enabled the flow log feature and created a Project and a Logstore in Simple Log Service (SLS):

• Call OpenFlowLogService to enable the flow log feature.

• Call CreateProject to create a Project, and call CreateLogstore to create a Logstore.

After you meet these prerequisites, you can perform the following operations:

• Call CreateFlowLog to create a flow log. You can also call CreateIndex to create an index.

• Call DeactiveFlowLog to stop a flow log.

• Call ActiveFlowLog to start a flow log.

• Call DeleteFlowLog to delete a flow log.

Terraform

Resources: alicloud_log_project, alicloud_log_store, alicloud_vpc_flow_log

# Specify the region where you want to create the flow log.
provider "alicloud" {
  region = "cn-hangzhou"
}

# Specify the description for the Project, and the names for the Logstore and flow log.
variable "name" {
  default = "vpc-flowlog-example"
}

# Generate a random string for the project name.
resource "random_uuid" "example" {
}

# Create a Simple Log Service (SLS) Project.
resource "alicloud_log_project" "example" {
  project_name = substr("tf-example-${replace(random_uuid.example.result, "-", "")}", 0, 16)
  description  = var.name
}

# Create a Simple Log Service (SLS) Logstore.
resource "alicloud_log_store" "example" {
  project_name          = alicloud_log_project.example.project_name
  logstore_name         = var.name
  shard_count           = 3
  auto_split            = true
  max_split_shard_count = 60
  append_meta           = true
}

# Create a VPC flow log.
resource "alicloud_vpc_flow_log" "example" {
  flow_log_name        = var.name
  log_store_name       = alicloud_log_store.example.logstore_name
  description          = var.name
  traffic_path         = ["all"] # Capture traffic from all scenarios.
  project_name         = alicloud_log_project.example.project_name
  resource_type        = "VPC" # The resource type is VPC.
  resource_id          = "vpc-bp1ekmgzch0bo3hxXXXXXX" # The ID of the VPC.
  aggregation_interval = "1" # The capture window is 1 minute.
  traffic_type         = "All" # Capture all traffic, regardless of whether it is allowed or rejected by access control rules.
}

Console

API

Call GetLogsV2 to query and analyze flow logs.

Analyze internet source IPs

Assume you have a web server that listens on port 80 and is accessible from the internet. Access to the server is restricted by a security group based on source IP addresses.

In this case, you can create a flow log to query the source IP addresses that access port 80 and count the requests that are allowed or rejected by the security group.

1. Create a flow log

   • For Resource Instance, select the ENI of the web server.

   • For Data Transfer Type, select All Traffic.

   • For Delivery Configuration, select Deliver to Log Service and Enable Log Analysis Report.

   • Keep the default settings for the other options.

2. Analyze the flow log

   1. Query statement

      Run the following query to filter for source IP addresses that access port 80 of 10.0.0.1. The query also shows the number of times the security group allowed or rejected each IP address:

      dstaddr:10.0.0.1 AND dstport:80 | SELECT -- Filter for logs where the destination IP address is 10.0.0.1 and the destination port is 80.
      srcaddr,
      SUM(CASE WHEN action = 'ACCEPT' THEN 1 ELSE 0 END) AS accept_count, -- Count 1 for each ACCEPT (allowed) action.
      SUM(CASE WHEN action = 'REJECT' THEN 1 ELSE 0 END) AS reject_count -- Count 1 for each REJECT (rejected) action.
      FROM log
      GROUP BY srcaddr -- Group by source IP address.
      ORDER BY accept_count + reject_count DESC -- Sort the results in descending order by the total number of allowed and rejected requests.

   2. Preview

      The srcaddr column displays the source IPs that accessed port 80, and the accept_count and reject_count columns respectively count the number of flow log entries from each source IP that were allowed and rejected by the security group within the query time range:

      • Five source IP addresses accessed port 80: 120.26.XX.XX, 121.43.XX.XX, 154.212.XX.XX, 176.65.XX.XX, and 198.235.XX.XX.

      • All requests from 120.26.XX.XX were allowed, while all requests from the other public IP addresses were rejected.

Analyze ECS traffic

Parameter	Intra-VPC traffic	Inter-VPC traffic
Diagram
Description	Assume you have deployed three ECS instances in one VPC and there is traffic between the instances. You can use flow logs to analyze the rate and trends of traffic between the ECS instances.	Assume you have two VPCs in different regions and each VPC contains multiple ECS instances. The two VPCs are connected using a peering connection. The peering connection uses pay-as-you-go Cloud Data Transfer (CDT) billing. You notice a recent surge in cross-region traffic costs. In this case, use flow logs to identify the ECS instances that generate high volumes of traffic and optimize your traffic costs.
Flow log configuration	Resource Instance: Select the ECS1 ENI For Analysis and Delivery, select Deliver to Log Service and enable Enable Log Analysis Report. Keep the default settings for the other options.	For Resource Instance, select VPC VPC1. For Analysis and Delivery, select Deliver to Log Service and Enable Log Analysis Report. Keep the default settings for the other options.
Query statement	To query the traffic rate trends between ECS1 and other ECS instances, run the following query: (srcaddr:10.0.0.1 AND dstaddr:10.0.0.*) OR (srcaddr:10.0.0.* AND dstaddr:10.0.0.1 ) | select -- Filter for traffic between ECS1 and other ECS instances. date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, -- Convert the UNIX timestamp to a readable time format. concat(srcaddr,'->', dstaddr) as src_to_dst, -- Concatenate the source and destination IP addresses into a session pair with the format 'source_ip->destination_ip'. sum(bytes*8/60) as bandwidth -- Convert bytes to bits and divide by the capture window of 1 minute to get the rate in bit/s. group by time,srcaddr,dstaddr -- Group by time, source IP, and destination IP. order by time asc -- Sort by time in ascending order. limit 100 -- Display the first 100 results.	To query the session traffic rate trends between the two VPCs, run the following query: (srcaddr:10.0.* AND dstaddr:172.16.*) OR (srcaddr:172.16.* AND dstaddr:10.0.*) | select -- Filter for sessions between the two VPCs. date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, -- Convert the UNIX timestamp to a readable time format. concat(srcaddr,'->', dstaddr) as src_to_dst, -- Concatenate the source and destination IP addresses into a session pair with the format 'source_ip->destination_ip'. sum(bytes*8/60) as bandwidth -- Convert bytes to bits and divide by the capture window of 1 minute to get the rate in bit/s. group by time,srcaddr,dstaddr -- Group by time, source IP, and destination IP. order by time asc -- Sort by time in ascending order. limit 100 -- Display the first 100 results.
Preview	On the Query and Analysis Results page, select the Statistical Chart tab and set the chart type to Area Chart. Set the x-axis field to time, the y-axis field to bandwidth, and the aggregation column to src_to_dst. For the format unit, select bps,Kbps,Mbps(bit). The traffic rate from 10.0.0.1 to 10.0.0.2 is the highest, at approximately 1.4 Mbps. The traffic rate from 10.0.0.1 to 10.0.0.3 is the second highest, at approximately 700 Kbps. Other traffic accounts for a small proportion.	The visualization configuration for the query results is the same as in Scenario 2: select the Area Chart type and set the aggregation column to src_to_dst. The results show a traffic spike of approximately 6 Mbps from 10.0.0.1 to 172.16.0.1.

Analyze NAT gateway traffic

Assume you have multiple ECS instances in a region. All instances are deployed in the same vSwitch and access the internet using SNAT on an Internet NAT Gateway.

You recently notice a sudden increase in outbound traffic from the NAT gateway, which causes slow server responses. Use flow logs to identify the ECS instances that generate a high proportion of the traffic.

1. Create a flow log

   • For Resource Instance, select the vSwitch2 where the Internet NAT Gateway is located.

   • For Delivery Configuration, select Deliver to Log Service and Enable Log Analysis Report.

   • Keep the default settings for the other options.

2. Methods for filtering specific traffic paths

   To filter traffic on a specific path in this scenario, you specify different conditions in the query statement:

   Diagram	No.	Filter method
   	①	Filter traffic from an ECS to a NAT Gateway: direction is in, and srcaddr is the private IP address of the ECS.
   	②	To filter traffic from the NAT gateway to the public network: direction is out, and srcaddr is the private IP address of the NAT gateway.
   	③	Filter traffic from the internet to the NAT gateway: direction is in, and dstaddr is the private IP address of the NAT gateway.
   	④	Filter traffic from the NAT gateway to the ECS instance: direction is out, and dstaddr is the private IP address of the ECS instance.

3. Analyze the flow log

   1. Query statement

      On the path from an ECS instance to the NAT gateway, analyze traffic to a specific public IP address:

      direction: 'in' and srcaddr: 10.0.0.* and dstaddr: 120.26.XX.XX | select -- Filter for logs of ECS instances accessing a specific public IP address.
      date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, srcaddr,  -- Convert the UNIX timestamp to a readable time format.
      sum(bytes*8/60) as bandwidth  -- Convert bytes to bits and divide by the capture window of 1 minute to get the rate in bit/s.
      group by time,srcaddr -- Group by time and source IP address.
      order by time asc  -- Sort by time in ascending order.
      limit 100 -- Display the first 100 results.

      Other common queries

      • On the path from the NAT gateway to an ECS instance, filter inbound traffic from a specific public IP address to all ECS instances:

        direction: 'out' and dstaddr: 10.0.0.* and srcaddr: 120.26.XX.XX | select -- Filter for logs of ECS instances accessing a specific public IP address.
        date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time,   -- Convert the UNIX timestamp to a readable time format.
        dstaddr,
        sum(bytes*8/60) as bandwidth  -- Convert bytes to bits and divide by the capture window of 1 minute to get the rate in bit/s.
        group by time,dstaddr -- Group by time and destination IP address.
        order by time asc  -- Sort by time in ascending order.
        limit 100 -- Display the first 100 results.

      • On the path from an ECS instance to the NAT gateway, filter outbound traffic from ECS instances to all public IP addresses:

        direction: 'in' and srcaddr: 10.0.0.*  | select -- Filter for logs of ECS instances accessing all public IP addresses.
        date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time,  -- Convert the UNIX timestamp to a readable time format.
        concat(srcaddr,'->', dstaddr), -- Concatenate the source and destination IP addresses into a session pair with the format 'source_ip->destination_ip'.
        sum(bytes*8/60) as bandwidth  -- Convert bytes to bits and divide by the capture window of 1 minute to get the rate in bit/s.
        group by time,srcaddr,dstaddr -- Group by time, source IP, and destination IP.
        order by time asc  -- Sort by time in ascending order.
        limit 100 -- Display the first 100 results.

   2. Preview

      The way to visualize the query results is similar to that in Scenario 2: select the area chart type, and set the aggregation column to srcaddr.

      On the path to the NAT gateway, the traffic from 10.0.0.1 (ECS1) to the public IP address 120.26.XX.XX is the highest, at approximately 12 Kbps.

Analyze Express Connect traffic ratio

An enterprise uses two VPCs in an Alibaba Cloud region to deploy different services and connects its on-premises data center to Alibaba Cloud using an Express Connect circuit and CEN.

The IT department plans to use flow logs to monitor and analyze how traffic from different business applications in the VPCs uses the express connect circuit. This analysis helps with network resource planning and performance improvement.

1. Create flow logs

   Create two flow logs and deliver the logs to the same Logstore. The key settings for each flow log are as follows:

   • For Resource Instance, select VPC VPC1 and VPC2.

   • For Sampling Path, select Transit Router.

   • Set Delivery Configuration to Deliver to Log Service, select the same Logstore for the two flow logs, and Enable Log Analysis Report.

   • Keep the default settings for the other options.

2. Analyze the flow logs

   1. Query statement

      To analyze the ratio of traffic from different VPCs to the on-premises data center, run the following query:

      action: ACCEPT and srcaddr: 192.168.* and dstaddr:10.1.* | 
      WITH 
          vpc1_traffic AS (
              SELECT 
                  date_trunc('minute',__time__) AS minute,
                  SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc1_traffic
              FROM 
                  log
              WHERE 
                  srcaddr LIKE '192.168.20.%'
              GROUP BY 
                  date_trunc('minute',__time__)
          ),
          vpc2_traffic AS (
              SELECT 
                  date_trunc('minute',__time__) AS minute,
                  SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc2_traffic
              FROM 
                  log
              WHERE 
                  srcaddr LIKE '192.168.10.%'
              GROUP BY 
                  date_trunc('minute',__time__)
          )
      SELECT 
          COALESCE(vpc1_traffic.minute, vpc2_traffic.minute) AS minute,
          (COALESCE(vpc1_traffic.total_vpc1_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc1_percentage, 
          (COALESCE(vpc2_traffic.total_vpc2_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc2_percentage
      FROM vpc1_traffic FULL OUTER JOIN vpc2_traffic ON vpc1_traffic.minute = vpc2_traffic.minute 
      ORDER BY minute

      The following is an explanation of the SQL statement.

      • Filter conditions:

        • srcaddr: 192.168.*, filtering logs whose source address starts with 192.168.*.

        • dstaddr: 10.1.*, filters logs whose destination address starts with 10.1.*.

        • action: ACCEPT. Filters logs where the value of the action field is ACCEPT.

      • Main query

        • Use a FULL OUTER JOIN to join the vpc1_traffic and vpc2_traffic tables on the minute field.

        • Calculates the traffic percentage for each VPC per minute:

          • vpc1_percentage specifies the percentage of total traffic that is routed to VPC1.

          • vpc2_percentage indicates the percentage of VPC2 traffic in the total traffic.

        • The query results are sorted by minute in ascending order.

      • The WITH subquery:

        The SQL statement includes two subqueries, vpc1_traffic and vpc2_traffic. The following description uses the vpc1_traffic subquery as an example:

        • Use the date_trunc function to reduce the precision of the Unix timestamp (the __time__ field) to the minute, and name it minute.

        • Use the SUM function to calculate the total traffic rate in bit/s for a specific minute, and name it total_vpc1_traffic.

        • Filters traffic records with a source address in the 192.168.20.* CIDR block, which belongs to VPC1.

        • Groups the results by minute.

   2. Preview

      The visualization configuration for the query results is similar to Scenario 2. Select the Area Chart type to show the trend of traffic proportions for different VPCs over time.

      The results show that during the period from 14:50 to 15:50, the proportion of traffic from VPC1 to the on-premises data center is higher.