Logstash FAQ

Answers common questions about Alibaba Cloud Logstash.

Can I use DRDS as a Logstash data source?
How can I import data from or export data to Logstash over the Internet?
What should I do if an error occurs when using a self-managed Kafka cluster as an input or output for Logstash?
Does the JDBC feature of Alibaba Cloud Logstash support MySQL databases?
Can I monitor Logstash nodes?
Can I upload script files to Logstash?
Can I configure the HTTP input protocol for Logstash?
How do I use Logstash to synchronize data from Simple Log Service to Elasticsearch?
Can Logstash synchronize data in real time?
Why do I receive the "[routing] is missing for join field [joinField]" error when I use Logstash to synchronize data?
After I create a pipeline, the process is stuck and the cluster update does not progress. What should I do?
Can I ping the private IP address of a Logstash cluster?
How do I ensure mapping consistency when I use Logstash to synchronize data between two Elasticsearch clusters?
Does Alibaba Cloud Logstash support the rubydebug mode?

DRDS as a Logstash data source

Yes. Follow the same process as an RDS MySQL data migration. Use Logstash to synchronize data from an RDS MySQL database to an Elasticsearch cluster.

Internet data transfer

Logstash clusters run in a VPC and require a NAT gateway for Internet access. Configure a NAT gateway for data transmission over the Internet.

Errors with a self-managed Kafka cluster

Common errors:

No entry found for connection

Cause: The Logstash node cannot resolve the Kafka hostname.
Solution: Add the following to the server.properties file. Example for a Kafka service at 10.10.10.10:9092:
```
listeners=PLAINTEXT://10.10.10.10:9092
advertised.listeners=PLAINTEXT://10.10.10.10:9092
```
Important
- Replace 10.10.10.10:9092 with your Kafka cluster's address and port.
- Use the Alibaba Cloud Kafka service and add your Logstash node IP addresses to the Kafka cluster whitelist.
could not be established. Broker may not be available

Cause: The Kafka service is unreachable or does not exist.

Solution: Verify that the Kafka service is running and the bootstrap_servers value in the pipeline configuration is correct.

JDBC support for MySQL

Yes. Upload the mysql-connector-java library first. Configure third-party libraries.

Node monitoring

Yes. Associate your Logstash cluster with an Elasticsearch cluster via X-Pack, then monitor nodes in Kibana. Configure X-Pack monitoring.

Uploading script files

No. Logstash transfers data through pipeline configurations only. Manage pipelines by using configuration files.

Configuring HTTP input

Yes. Logstash accepts single-line and multi-line events over HTTP or HTTPS. Http input plugin.

Note

Logstash clusters are not Internet-accessible by default. To collect HTTP requests from the Internet, configure a NAT gateway. Configure a NAT gateway for data transmission over the Internet.

Sync data from Simple Log Service to Elasticsearch

Use the logstash-input-sls plugin. Use the logstash-input-sls plugin.

Real-time data synchronization

Logstash synchronizes data in near-real time. It writes continuously to the destination while the pipeline is active and the source contains data.

Handling the "routing is missing" error

Child documents require a routing value because parent and child documents must reside on the same shard. Set the destination index primary shard count to 1 and add routing =>"0" to your pipeline configuration.

Troubleshooting stuck pipeline updates

Check the cluster's main logs to identify the cause (Query logs). Common causes and solutions:

Cause	Solution
Incorrect pipeline configuration.	Pause the update. After the cluster is paused, modify the pipeline configuration and restart the cluster. View the progress details of a cluster task.
High disk usage.	Upgrade the cluster. Upgrade a cluster. After the upgrade, refresh the cluster and check the update progress.
Auto Indexing is not enabled on the destination Elasticsearch cluster (output: elasticsearch).	Enable Auto Indexing on the Elasticsearch cluster. Configure YML parameters.After you complete the operation, refresh the cluster and monitor the progress of the change.
Beats input port is outside the 8000-9000 range.	Pause the update. After the cluster is paused, modify the port in the pipeline configuration to a value within the 8000 to 9000 range, and restart the cluster.
Source or destination uses a public IP address.	Use one of the following solutions: Pause the update. After the cluster is paused, change the public IP address to a private IP address in the pipeline configuration. Configure a NAT gateway for Internet access. Configure a NAT gateway for data transmission over the Internet. After applying the configuration, refresh the cluster and check the update progress.
The pipeline configuration includes `file_extend`, but the logstash-output-file_extend plugin is not installed.	Use one of the following solutions: Install the logstash-output-file_extend plugin. Use the pipeline configuration debugging feature. After the installation, refresh the cluster and check the update progress. Pause the update. After the cluster is paused, remove the `file_extend` configuration from the pipeline, and restart the cluster.

Pinging the private IP address

By default, you cannot ping the private IP address of a Logstash cluster. Use telnet to test network connectivity instead.

Mapping consistency for Elasticsearch data sync

Alibaba Cloud Elasticsearch disables Auto Indexing by default for security. Logstash creates indexes by submitting data directly, not through the Create index API. Before transferring data, either enable Auto Indexing on the destination cluster or pre-create the index with mappings.

Support for rubydebug mode

No. Use the logstash-output-file_extend plugin to print debug logs instead. Rubydebug and similar modes require stdout output, which Alibaba Cloud Logstash does not support.