Use Filebeat to collect Apache log data-Elasticsearch(ES)-阿里云帮助中心

This tutorial shows how to use Filebeat to collect Apache log data, filter it with Alibaba Cloud Logstash, and then send it to Alibaba Cloud Elasticsearch for analysis.

Procedure

Step 1: Prerequisites
Step 2: Configure and install the Filebeat shipper
Step 3: Configure the Logstash pipeline
Step 4: View the data collection results

Step 1: Prerequisites

Create an Alibaba Cloud Elasticsearch instance and a Logstash instance. Both instances must be of the same version and in the same Virtual Private Cloud (VPC).
For more information, see Create an Alibaba Cloud Elasticsearch instance and Create an Alibaba Cloud Logstash instance.
Enable automatic index creation for your Alibaba Cloud Elasticsearch instance.
If your shipper outputs to Elasticsearch, you must enable automatic index creation. For more information, see Configure YML parameters.
Create an Alibaba Cloud ECS instance. This ECS instance must be in the same VPC as your Elasticsearch and Logstash instances.
For more information, see Create an instance by using the wizard.
Important
- Beats supports only Alibaba Cloud Linux, Red Hat, and CentOS operating systems.
- Alibaba Cloud Filebeat collects logs only from ECS servers in the same region and VPC as the Alibaba Cloud Elasticsearch or Logstash instance. Log collection from the public internet is not supported.

Set up an httpd service on the ECS instance.

To simplify log analysis and visualization, we recommend defining the Apache log format as JSON in the httpd.conf file. For more information, see Manually build a Magento 2 e-commerce website (Ubuntu). The following configuration is used in the test environment for this topic.

LogFormat "{\"@timestamp\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"client_ip\":\"%{X-Forwa rded-For}i\",\"direct_ip\": \"%a\",\"request_time\":%T,\"status\":%>s,\"url\":\"%U%q\",\"method\":\"%m\",\"http_host\":\"%{Host}i\",\"server_ip\":\"%A\",\"http_referer\":\"%{Referer}i\",\"http_user_agent\":\"%{User-agent}i\",\"body_bytes_sent\":\"%B\",\"total_bytes_sent\":\"%O\"}"  access_log_json
# Comment out the original CustomLog line and replace it with: CustomLog "logs/access_log" access_log_json

Install Cloud Assistant and Docker on the target ECS instance.
For more information, see Install the Cloud Assistant agent and Install and use Docker and Docker Compose.

Step 2: Configure and install the Filebeat shipper

Log on to the Alibaba Cloud Elasticsearch console.
Navigate to the Beats Data Shippers page.
1. In the top navigation bar, select a region.
2. In the left-side navigation pane, click Beats Data Shippers.
3. Optional: If this is the first time you go to the Beats Data Shippers page, view the information displayed in the message that appears and click OK to authorize the system to create a service-linked role for your account.
  Note
  When Beats collects data from various data sources, Beats depends on the service-linked role and the rules specified for the role. Do not delete the service-linked role. Otherwise, the use of Beats is affected. For more information, see Elasticsearch service-linked roles.
In the Create Shipper section, click ECS Logs.
Configure and install the shipper.
For more information, see Collect ECS service logs by using Filebeat and Shipper YML configuration. This topic uses the following configuration.
Note
- The shipper output must be set to the instance ID of your target Alibaba Cloud Logstash instance. You do not need to specify the output again in the YML configuration.
- The Filebeat file directory must be set to your data source directory. You must also enable log data collection and configure the log path in the YML configuration.
Click Next.
In the Install Shipper wizard, select the ECS instance where you want to install the shipper.
Note
Select the ECS instance that you prepared in the prerequisites.
Start the shipper and check its installation status.
1. Click Start.
  After the shipper is started, a Start Shipper dialog box appears.
2. Click Back to Beats Shippers to return to the Beats Data Shippers page. In the Manage Shippers section, view the Filebeat shipper that you started.
3. Wait until the Shipper Status changes to Enabled 1/1. Then, in the Actions column, click View Instances.
4. On the View Instances page, check the Installed Shippers. A status of Normal Heartbeat indicates that the shipper is installed.

Step 3: Configure the Logstash pipeline

In the left-side navigation pane of the Alibaba Cloud Elasticsearch console, click Logstash Clusters.
In the Actions column for your Logstash instance, click Manage Pipeline.
On the Pipelines page, click Create Pipeline.

Configure the pipeline.

Configure the pipeline by using the following example. For more information about how to configure a pipeline, see Manage pipelines by using configuration files.

input {
  beats {
      port => 8000
    }
}
filter {
  json {
        source => "message"
        remove_field => ["@version", "prospector", "beat", "source", "input", "offset", "fields", "host", "message"]
      }

}
output {
  elasticsearch {
    hosts => ["http://es-cn-mp91cbxsm00******.elasticsearch.aliyuncs.com:9200"]
    user => "elastic"
    password => "<YOUR_PASSWORD>"
    index => "<YOUR_INDEX>"
  }
}

Parameter	Description
input	Receives the data that is collected by Beats.
filter	Filters the collected data. This configuration uses the JSON plugin to decode message data and the remove_field option to delete specified fields. Note The filter configuration in this topic is for testing purposes only and may not apply to all business scenarios. You must modify the filter configuration to suit your business needs. For more information about the supported filter plugins and their usage, see Filter plugins.
output	Sends the data to your Alibaba Cloud Elasticsearch instance. The parameters are described as follows: hosts: Replace the value with the endpoint of your Alibaba Cloud Elasticsearch instance. You can obtain the endpoint from the Basic Information page of the instance. For more information, see View the basic information of an instance. <YOUR_PASSWORD>: Replace the value with your Alibaba Cloud Elasticsearch instance password. <YOUR_INDEX>: Replace the value with your desired index name.

Step 4: View the data collection results

Log on to the Kibana console of your Elasticsearch cluster.
For instructions, see Log on to the Kibana console.
Note
Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.
In the left navigation menu, choose Management > Dev Tools.
In the Console, run the following command to view the collected data.
```
GET <YOUR_INDEX>/_search
```
Note
Replace <YOUR_INDEX> with the index name you defined in the Logstash pipeline's output section.
In the left-side navigation pane, click Discover. Select a time range to view the collected data.
Note
Before you run a query, ensure an index pattern for <YOUR_INDEX> exists. If not, in the Kibana console, go to Management and in the Kibana section, click Index Patterns > Create index pattern to create one.