Stream ETL

Why Choose ETL

• You can configure ETL tasks in directed acyclic graph (DAG) mode or Flink SQL mode.

  • DAG mode

    • Visual editing: The ETL task interface provides three components: Input/Dimension Table, Transform, and Output. You can drag and drop these components to quickly build stream processing tasks.

    • Rich development components:

      • The Input/Dimension Table component (source database) supports self-managed MySQL, RDS for MySQL, PolarDB for MySQL, PolarDB-X 1.0 (formerly DRDS), self-managed Oracle, self-managed PostgreSQL, RDS for PostgreSQL, Db2 for LUW, Db2 for i, and PolarDB for PostgreSQL.

      • The Transform component supports table joins, more than 90 function computations, and field filtering.

      • The Output component (destination database) supports self-managed MySQL, RDS for MySQL, PolarDB for MySQL, AnalyticDB for MySQL 3.0, self-managed Oracle, self-managed PostgreSQL, RDS for PostgreSQL, Db2 for LUW, Db2 for i, and PolarDB for PostgreSQL.

  • Flink SQL mode

    ETL supports Flink SQL, a development language that complies with standard SQL semantics, to configure tasks.

• Industry-leading computing performance: ETL uses the powerful stream data ingestion capabilities of DTS to ensure data accuracy and provide industry-leading real-time computing performance.

• Flexible task monitoring and management: ETL provides a task list page where you can monitor and manage tasks. You can start, stop, and view the details of created tasks.

Scenarios

• Real-time centralization of multi-region or heterogeneous data: You can store heterogeneous data or data from different regions in the same database in real time. This facilitates centralized, efficient management and decision support.

• Real-time data integration: The powerful stream data transformation capabilities of ETL greatly improve data integration efficiency. The low-code development method also reduces the difficulty and cost of data integration, which allows your business to focus on realizing data value.

• Real-time data warehouse: The industry-leading stream data processing capabilities help you quickly build a real-time data warehouse.

• Offline data warehouse acceleration: You can use stream processing to pre-process data and load it into a data warehouse for subsequent in-depth analysis. This process does not affect your business systems and can meet the service requirements of the offline data warehouse.

• Report acceleration: You can build a real-time reporting system to significantly improve report generation efficiency. This system also supports more real-time analysis scenarios and meets the high-efficiency requirements for report generation during digital transformation.

• Real-time computing: You can clean and process streaming data from your business systems in real time to generate feature values and tags. These can be used to support online business computing models, such as for profiling, risk control, and recommendations, or other stream computing scenarios such as real-time dashboards.

Promotions

Documentation

• Configuration

  • Configure an ETL task in DAG mode

    • Process for configuring an ETL task in DAG mode

    • Configure source database information

    • Configure transform components

    • Configure destination database information

    • Create a connection template

  • Configure an ETL task in Flink SQL mode

  • Configure ETL in a DTS data migration or synchronization task

• Best practices

  • Use ETL to analyze real-time orders

  • Typical applications for configuring ETL in a DTS task