This topic describes two methods for synchronizing data from PolarDB-X 1.0 to Alibaba Cloud Elasticsearch, so you can choose the one that fits your use case.
If your business data is stored in PolarDB-X 1.0 and you want to run full-text searches or semantic analytics, you need to synchronize your data to Alibaba Cloud Elasticsearch — a distributed search and analytics engine that lets you store, query, and analyze large datasets in near real time.
Which method should I use?
| If you need... | Use this method |
|---|---|
| Low-latency sync of full data with a delay of seconds | logstash-input-jdbc plugin |
| Offline batch sync of large datasets at a minimum 5-minute interval | DataWorks |
| Custom SQL queries or joint queries across multiple tables | DataWorks |
| To synchronize all data in a PolarDB-X 1.0 database | DataWorks |
Method comparison
| Method | Best for | Minimum latency | Key prerequisite |
|---|---|---|---|
| logstash-input-jdbc plugin | Near-real-time full data sync | A few seconds | JDBC driver uploaded to Logstash cluster |
| DataWorks | Offline batch sync, custom SQL, multi-table joins | 5 minutes | DataWorks service activated |
logstash-input-jdbc plugin
The logstash-input-jdbc plugin keeps your Elasticsearch index in sync with PolarDB-X 1.0 by running scheduled polling queries. On each polling cycle, the plugin identifies the most recently inserted or updated data records in the PolarDB-X 1.0 database and queries all identified data records at a time, then synchronizes them to your Elasticsearch cluster. This method is well-suited when you need to synchronize full data and can tolerate a delay of a few seconds.
When to use this method
You want to synchronize full data and can accept a latency of a few seconds.
You want to query a specific subset of data at a time and synchronize only those records.
Prerequisites
Before you can use this method, complete the following setup steps:
Upload a Java Database Connectivity (JDBC) driver to your Logstash cluster. The driver version must be compatible with the version of your PolarDB-X 1.0 database.
Add the IP addresses of all nodes in your Logstash cluster to the IP address allowlist of your PolarDB-X 1.0 instance.
Confirm that your Logstash cluster and PolarDB-X 1.0 instance are in the same time zone. Mismatched time zones cause inconsistent timestamps during synchronization.
Ongoing data requirements
The
_idfield in Elasticsearch must match theidfield in PolarDB-X 1.0 for each synchronized record.Every record you insert or update must include a field that records the insertion or update time. The plugin uses this field to identify changed data on each polling cycle.
DataWorks
DataWorks is a comprehensive data development service that provides various modules such as Data Integration, DataStudio, and Data Quality. Together, these modules let you import and store structured data, convert and develop data, and synchronize processed data to Elasticsearch or other data systems.
When to use this method
You want to synchronize offline big data. DataWorks can collect offline data at a minimum interval of 5 minutes.
You want to specify custom query statements or perform joint queries across multiple tables before synchronizing data.
You want to synchronize all data in a PolarDB-X 1.0 database.
Prerequisites
Before you can use this method, complete the following setup steps:
Activate the DataWorks service.
If high transmission speed is required or your network environment is complex, customize resource groups for DataWorks.
Add the IP address of the resource group to the IP address allowlist of your PolarDB-X 1.0 instance.
Next steps
After selecting a method, follow the corresponding guide to complete the sync setup: