Debug Logstash pipeline configurations

Procedure

1. Go to the Logstash Clusters page.

2. In the top navigation bar, select the region where the desired cluster resides.

3. Click the ID of the target instance in the instance list.

4. In the left-side navigation pane, click Pipelines.

5. Click Create Pipeline.

6. Configure and start the pipeline.

   1. In the Config Settings section, enter a Pipeline ID and your Config Settings.

      Parameter	Description
      Pipeline ID	Enter a custom ID. This ID automatically maps to the path parameter of the file_extend plugin.
      Config Settings	The Config Settings consists of three sections: input: Specifies the data source. Alibaba Cloud Logstash supports all official Logstash input plugins except for the file plugin. filter: Processes data from the data source. Alibaba Cloud Logstash supports a wide range of filter plugins. output: Sends the processed data to a specified destination. In addition to the standard Logstash output plugins, Alibaba Cloud Logstash provides the proprietary file_extend plugin. Configure this plugin to enable debugging and view pipeline output directly in the console.

      The following code provides an example of a pipeline configuration:

      input {
           elasticsearch {
             hosts => "http://es-cn-0pp1jxv000****.elasticsearch.aliyuncs.com:9200"
             user  => "elastic"
             index => "twitter"
             password => "<YOUR_PASSWORD>"
             docinfo => true
           }
      
      }
      filter {
      
      }
      output {
          elasticsearch {
          hosts => ["http://es-cn-000000000i****.elasticsearch.aliyuncs.com:9200"]
          user => "elastic"
          password => "<your_password>"
          index => "%{[@metadata][_index]}"
          document_id => "%{[@metadata][_id]}"
      
        }
         file_extend {
           path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
         }
      
      }

      Important

      • The file_extend configuration in the output section is commented out by default. To use the debugging feature, you must uncomment this section.

      • The path parameter within the file_extend configuration defaults to a system-generated path. Do not modify this path. Alternatively, you can click Start Configuration Debug to automatically generate the correct path.

      • The system automatically replaces the {pipelineid} variable in the path with the Pipeline ID you specified. Do not change this variable, or the debug log will become inaccessible.

      The Elasticsearch input plugin is useful for tasks like bulk-importing test logs by reading documents from an Alibaba Cloud Elasticsearch cluster. By default, this process stops after reading all data. However, Alibaba Cloud Logstash instances run continuously and automatically restart stopped processes. In single-task scenarios, this behavior can cause duplicate data writes. To prevent this, use the schedule parameter with a cron expression to run the task only once. The pipeline stops after this single execution, preventing repetition. For more information, see the official Logstash documentation on Scheduling.

      For example, to schedule the task to run at 13:20 on March 5:

      schedule => "20 13 5 3 *"

   2. Click Next step to configure pipeline parameters.

      For more information about pipeline parameters, see Manage pipelines by using configuration files.

   3. Save and deploy the pipeline.

      • Save: Saves the pipeline configuration and returns you to the Pipelines page, where the changes are not yet deployed. From the Pipelines list, click Deploy Now in the Actions column to apply the configuration.

      • Save and Deploy: Saves the configuration and immediately restarts the instance to apply the changes.

   4. In the success message, click OK.

7. View the debug log.

   1. After the instance restarts, return to the Pipelines list, find the target pipeline, and click View Debug Logs in the Actions column.

   2. On the Logs page, open the Debug Log tab to view the pipeline's processed output.

      To find logs for a specific pipeline, enter pipelineId: <your_pipeline_id> in the search bar to filter and view the log output data processed by the corresponding pipeline.