Create a data flow

Prerequisites

The following database types are supported:

• MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, MyBase for MySQL, PolarDB-X, AnalyticDB for MySQL V3.0, and MySQL databases from other sources

• SQL Server: ApsaraDB RDS for SQL Server, MyBase for SQL Server, and SQL Server databases from other sources

• PostgreSQL: ApsaraDB RDS for PostgreSQL, PolarDB for PostgreSQL, MyBase for PostgreSQL, AnalyticDB for PostgreSQL, and PostgreSQL databases from other sources

• Oracle

• Db2

• MaxCompute

• Hologres

  Note

  Hologres supports only Data Import task nodes.

• OSS

  Note

  OSS supports only Data Output task nodes.

Procedure

1. Log in to DMS 5.0.

2. In the top navigation bar, choose Data+AI > Data Integration > Batch Integration.

   Note

   If you use the DMS console in simple mode, move the pointer over the icon in the upper-left corner and choose All Features > Data+AI > Data Integration > Batch Integration.

3. Click the target data flow to open its details page.

4. Configure a Data Import node.

   Note

   The first node in a data flow must be a Data Import node, which provides the data source for the flow.

   1. From the Task Type list on the left, drag a Data Import node to the canvas.

   2. Click the Data Import node. On the Data source tab at the bottom of the page, configure the data source.

      Parameter	Actions
      Database Type	Select the type of the source database.
      Database	Enter a keyword to search for and select the source database. (Optional) If the database is not in secure collaboration mode, enter the Database Account and Database password in the Login Instance dialog box.
      Table	Select the source table.

5. Configure a data processing node. This example uses a Data Filtering node to filter data from the source table.

   Note

   Any task type, except for Data Import and Data Output, can be a data processing node.

   1. From the Task Type list on the left, drag a Data Filtering node to the canvas.

   2. Hover over the Data Import node, click the hollow circle that appears on the right, and drag the connector to the Data Filtering node.

   3. Click the Data Filtering node. On the Data Filtering tab at the bottom of the page, configure the Filter Expressions.

      For example, enter name='John' in the Filter Expressions text box.

      Note

      You can enter the filter condition directly in the input box or double-click a function from the function list to help you build the expression.

6. Configure a Data Output node.

   Note

   The last node in a data flow must be a Data Output node, which specifies the destination for the processed data.

   1. From the Task Type list on the left, drag a Data Output node to the canvas.

   2. Click the Data Output node. On the Data Output tab at the bottom of the page, configure the Data Output node.

      • Database:

        Parameter	Description
        Database Type	Select the type of the destination database.
        Database	Search for and select the destination database. Note If you are not logged on to the database, enter the Database Account and Database password in the Login Instance dialog box to log on.
        Table name	Select or enter the destination table.
        SQL Statements Executed Before Writing	Enter the SQL statements to run before writing data.
        SQL Statements Executed After Writing	Enter the SQL statements to run after writing data.
        Automatic Table Creation	Set the Automatic Table Creation switch. Off: If the destination table does not exist, the data flow fails when executed. On: If the destination table does not exist, the data flow automatically creates it upon execution.

      • OSS:

        Parameter	Description
        Database Type	Select OSS.
        OSS Bucket	Search for and select the OSS Bucket.
        OSS Directory	Enter the OSS Directory where the data will be written.
        Table name	Enter the destination table name.
        Overwrite Destination Table	Set the Overwrite Destination Table switch. Off: Appends data to the destination. On: If the table exists, clears all data from the table or partition before writing the new data.
        File Format	Select the storage format for the file. Supported formats are Parquet, ORC, Avro, and CSV.
        Compress	Select a compression method for the file.
        Partition	Select the partition key. You can use partition key values to quickly locate specific data. Note You can select a partition key only after you have configured the Data Import node and connected all nodes on the canvas.

   3. Hover over the Data Filtering node, click the hollow circle that appears on the right, and drag the connector to the Data Output node.

      The icon on the node disappears, indicating that the node dependencies are correctly configured.