DocsICP FilingConsole
官方文档
首页

Configure the Amazon RDS for PostgreSQL Output Component

更新时间：
复制 MD 格式
产品详情
我的收藏

The Amazon RDS for PostgreSQL output component writes data to an Amazon RDS for PostgreSQL data source. When you sync data from other data sources to an Amazon RDS for PostgreSQL data source, configure the target data source for this component after configuring the source data.

Prerequisites

Procedure

  1. In the top menu bar on the Dataphin homepage, click Develop > Data Integration.

  2. In the top menu bar on the Integration page, select a Project. In Dev-Prod mode, also select an environment.

  3. In the left navigation pane, click Offline Integration, then click the offline pipeline you want to develop in the Offline Integration list to open its configuration page.

  4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

  5. In the navigation pane on the left of the Component Library panel, click Output. In the output component list on the right, find the Amazon RDS for PostgreSQL component and drag it onto the canvas.

  6. Click and drag the image icon of the input component to connect it to the Amazon RDS for PostgreSQL output component.

  7. Click the image icon in the Amazon RDS for PostgreSQL output component card to open the Amazon RDS for PostgreSQL Output Configuration dialog box.

    image

  8. In the Amazon RDS for PostgreSQL Output Configuration dialog box, configure the following parameters.

    Parameter

    Description

    Basic Settings

    Step Name

    The name of the Amazon RDS for PostgreSQL output component. Dataphin generates a step name automatically. You can change it based on your business scenario. Naming rules:

    • Use only Chinese characters, letters, underscores (_), and digits.

    • Keep the name no longer than 64 characters.

    Datasource

    The drop-down list shows all Amazon RDS for PostgreSQL data sources. This includes data sources for which you have write-through permission and those for which you do not. Click the image icon to copy the current data source name.

    Schema (optional)

    Select the schema where the table resides. Cross-schema table selection is supported. If you do not specify a schema, the default schema configured for the data source is used.

    Table

    Select the target table for output data. Enter a keyword to search for a table name, or enter the exact table name and click Exact Search. After you select a table, the system automatically checks its status. Click the image icon to copy the selected table name.

    If the target table does not exist in the Amazon RDS for PostgreSQL data source, use the one-click table creation feature to generate it quickly. To do so:

    1. Click One-Click Table Creation. Dataphin automatically matches code to create the target table. This includes the table name (defaulted to the source table name) and field types (preliminarily converted from Dataphin fields).

    2. Modify the SQL script for table creation as needed, then click Create. After the table is created, Dataphin uses it as the target table for output data.

      Note

      • If a table with the same name exists in the development environment, clicking Create returns an error that the table already exists.

      • If no matching table is found, you can still integrate using a manually entered table name.

    Production Table Missing Policy

    Choose how to handle cases when the production table does not exist. Options are No Action or Automatic Creation. Default is Automatic Creation. If you choose No Action, the task publishes without creating the production table. If you choose Automatic Creation, the task creates a table with the same name in the target environment during publishing.

    • No Action: If the target table does not exist, the system warns you during submission but lets you publish anyway. You must create the target table in the production environment before running the task.

    • Automatic Creation: You must Edit The Table-creation Statement, which defaults to the CREATE TABLE statement for the selected table. You can adjust the statement. The table name in the statement uses the placeholder ${table_name}. You can only use this placeholder, and the system replaces it with the actual table name when executing the statement.

      If the target table does not exist, the system first creates it using the statement. If creation fails, publishing fails. Fix the statement based on the error message and republish. If the target table already exists, no action is taken.

    Note

    This setting is available only for projects in Dev-Prod mode.

    Loading Policy

    Select how to write data to the target table. Loading Policy options include the following:

    • Append Data (INSERT INTO): Returns a dirty data error if a primary key or constraint conflict occurs.

    • Update on Primary Key Conflict (ON CONFLICT DO UPDATE SET): Updates mapped fields in existing records when a primary key or constraint conflict occurs.

    Write-Through

    Primary key updates are not atomic operations. Enable write-through if input data contains duplicate primary keys. Otherwise, use parallel writes. Write-through has lower performance than parallel writes.

    Note

    This setting is available only when the loading policy is set to Update on Primary Key Conflict.

    Batch Write Size (optional)

    The size of data written in one batch. You can also set Batch Write Count. The system writes data when either limit is reached. Default is 32 MB.

    Batch Write Count (optional)

    Default is 2,048 rows. Data synchronization uses a batch-write strategy. Parameters include Batch Write Count and Batch Write Size.

    • When the accumulated data reaches either limit (size or count), the system treats it as a full batch and writes it to the target immediately.

    • We recommend setting the batch write size to 32 MB. Adjust the batch write count based on the average record size. Use a larger value to maximize batch benefits. For example, if each record is about 1 KB, set the batch write size to 16 MB. Then set the batch write count to more than 16,384 (16 MB ÷ 1 KB). Here, we use 20,000 rows. With this setup, the system triggers batch writes when the accumulated data reaches 16 MB.

    Pre-SQL Statement (optional)

    An SQL script to run on the database before data import.

    For example, to maintain service availability, run these steps: First, create the target table Target_A. Next, write data to Target_A. After writing completes, rename the live service table Service_B to Temp_C. Then rename Target_A to Service_B. Finally, delete Temp_C.

    Post-SQL Statement (optional)

    An SQL script to run on the database after data import.

    Field Mapping

    Input Fields

    Lists input fields from upstream components.

    Output Fields

    Lists output fields. Supported actions:

    • Field Management: Click Field Management to select output fields.

      image

      • Click the gaagag icon to move Selected Input Fields to Unselected Input Fields.

      • Click the agfag icon to move Unselected Input Fields to Selected Input Fields.

    • Batch Add: Click Batch Add to configure fields in JSON, TEXT, or DDL format.

      • JSON format example:

        // Example:
[{
  "name": "user_id",
  "type": "String"
 },
 {
  "name": "user_name",
  "type": "String"
 }]
        Note

        The name field is the field name. The type field is the data type. For example, "name":"user_id","type":"String" imports the field named user_id and sets its type to String.

      • TEXT format example:

        // Example:
user_id,String
user_name,String

        • The row delimiter separates field entries. Default is line feed (\n). Other options: semicolon (;) and period (.).

        • The column delimiter separates field names from field types. Default is comma (,).

      • DDL format example:

        CREATE TABLE tablename (
    id INT PRIMARY KEY,
    name VARCHAR(50),
    age INT
);

    • Create Output Field: Click + Create Output Field. Enter the Column name and select the Type. Click the image icon to save the row.

    Mapping

    Manually map fields between upstream inputs and target table columns. Quick Mapping options include Row Mapping and Name Mapping.

    • Name Mapping: Maps fields with identical names.

    • Row Mapping: Maps fields by position when source and target column names differ.

  9. Click Confirm to complete the configuration of the Amazon RDS for PostgreSQL output component.

Previous：Configure the Amazon Redshift output componentNext：Configure the Amazon RDS for MySQL Output Component