Collect NetFlow logs with Elastic Agent

• Create an Alibaba Cloud Elasticsearch cluster. For more information, see Create an Alibaba Cloud Elasticsearch cluster. This topic uses an Elasticsearch 8.5 cluster as an example.

• Create an ECS instance. The instance must be in the same Virtual Private Cloud (VPC) as your Elasticsearch cluster. For more information, see Create an instance by using the wizard.

  Note

  The ECS instance serves as the source server from which the Elastic Agent collects NetFlow data.

Step 1: Create an agent policy

1. Log on to the Kibana console. For more information, see Log on to the Kibana console.

2. Go to > Management > Fleet.

3. On the Fleet page, click the Agent policies tab.

4. Click Create agent policy and configure the agent policy in the panel that appears.

   1. For Name, enter Name.

   2. Clear the Collect system logs and metrics checkbox.

   3. Click Advanced Options. Under Agent monitoring, clear the Collect agent logs and Collect agent metrics checkboxes.

      

      Note

      In this scenario, you only need to collect NetFlow records, so you can disable the collection of system logs, metrics, and agent monitoring data.

5. When finished, click Create agent policy in the lower-right corner.

Step 2: Create a Fleet Server integration

1. On the Agent policies tab of the Fleet page, click the agent policy netflow-log.

2. On the Integration tab, click Add integration.

3. On the Browse integrations tab, search for and click Fleet Server.

4. Install the Fleet Server integration.

   1. On the Fleet Server page, click the Set tab.

   2. Click Install Fleet Server assets, and then click Install Fleet Server in the dialog box that appears.

      Note

      After the integration is installed, the Settings tab displays its version.

5. In the upper-right corner of the page, click Add Fleet Server.

6. On the Add Fleet Server integration page, enter a name for the integration and select the agent policy netflow-log.

7. In the lower-right corner, click Save and continue. In the dialog box that appears, click Add Elastic Agent later.

Step 3: Create a NetFlow Records integration

1. On the Integration tab of the netflow-log agent policy, click Add integration.

2. On the Browse integrations tab, search for and click NetFlow Records.

3. Install the NetFlow Records integration.

   1. On the NetFlow Records page, click the Set tab.

   2. Click Install NetFlow Records assets, and then click Install NetFlow Records in the dialog box that appears.

      Note

      After the integration is installed, the Settings tab displays its version.

4. In the upper-right corner of the page, click Add NetFlow Records.

5. On the Add NetFlow Records integration page, configure the integration.

   1. Under Integration Configuration, enter netflow-1 for Integration name.

   2. Next to Collect NetFlow logs, click Change defaults. Enter 0.0.0.0 as the UDP listening address and keep the default UDP port of 2055.

      

   3. Under Where to add this integration, on the Existing hosts tab, select the agent policy netflow-log.

6. When finished, click Save and continue in the lower-right corner. In the dialog box that appears, click Add Elastic Agent later.

Step 1: Configure Fleet Server hosts

1. Log on to the Kibana console. For more information, see Log on to the Kibana console.

2. Go to > Management > Fleet.

3. On the Fleet page, click the Set tab.

   1. Under Fleet Server hosts, click Modify Host.

   2. In the Fleet Server hosts panel, enter the private IP address of the source server in the format https://<private_ip>:<port>, for example, https://172.16.*.***:8220.

      Note

      In this topic, the private IP address is the primary private IP address of the ECS instance. For more configuration details, see Fleet Server hosts.

   3. In the Output section, click the edit icon () in the Actions column.

   4. In the Edit output panel, enter the private IP address of your Alibaba Cloud Elasticsearch cluster in the format http://<private_address>:<port>, for example, http://es-cn-uqm3auln80001****.elasticsearch.aliyuncs.com:9200.

   5. Click Save and apply settings, and then click Save and Deploy in the dialog box that appears.

Step 2: Add an Elastic Agent

Add an Elastic Agent to Fleet Server.

1. Go to > Management > Fleet.

2. Click the Agent policies tab.

3. For the netflow-log agent policy, in the Actions column, select > Add agent.

4. In the Add agent panel, under Select a policy for Fleet Server, confirm that the default agent policy is netflow-log.

5. Under Choose a deployment mode for security, keep the default selection Quick start.

6. Under Add your Fleet Server host, click Add Host on the right.

7. Under Generate a service token, click Generate a service token.

8. Under Install Fleet Server to a centralized host, copy the automatically generated code block and run it on your ECS instance.

   

   After the command finishes, a message containing Successfully confirms that the Elastic Agent is installed and running on the ECS instance.

Step 3: Configure the NetFlow service

This example uses softflowd to generate NetFlow logs. Run the following commands on the ECS instance to install and start the service.

1. Download the softflowd source package.

   wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/softflowd/softflowd-0.9.9.tar.gz

2. Install the libpcap-devel dependency.

   yum install libpcap-devel

3. Compile and install softflowd.

   tar -xvf softflowd-0.9.9.tar.gz
   cd softflowd-0.9.9
   ./configure 
   make
   make install

4. Run softflowd.

   nohup softflowd -v 9 -D -i eth0 -t maxlife=1 -n localhost:2055 >/dev/null 2>&1 &