Configure MongoDB output widget

更新时间:
复制 MD 格式

The MongoDB output widget writes data from external databases or big data platform storage systems into MongoDB for integration and reprocessing.

Prerequisites

Procedure

  1. On the Dataphin home page, choose Development > Data Integration.

  2. In the top menu bar, select a Project (or Environment in Dev-Prod mode).

  3. In the left navigation pane, click Batch Pipeline. Select the target offline pipeline from the Batch Pipeline list to open its configuration page.

  4. Click Component Library in the upper-right corner to open the Component Library panel.

  5. In the Component Library panel, select Output, then drag the MongoDB widget onto the canvas.

  6. Drag the image icon to connect an upstream widget to the MongoDB output widget.

  7. On the MongoDB output widget, click the image icon to open the MongoDB Output Configuration dialog box.image

  8. In the Mongodb Output Configuration dialog box, configure the following parameters.

    Parameter

    Description

    Basic Settings

    Step Name

    Name of the MongoDB output widget. Naming rules:

    • Supports Chinese characters, letters, numbers, and underscores (_).

    • Maximum length: 64 characters.

    Datasource

    Lists all MongoDB data sources, with or without write-through permission. Click the image icon to copy the data source name.

    Table

    Select the target table for output data. Search by keyword or enter the exact name and click Accurate Search. The system checks table status after selection. Click the image icon to copy the table name.

    Update Information (optional)

    Specifies the update configuration. Example: {"isUpsert":"true","upsertkey"""unique_id"}.

    Field Separator (optional)

    Separator between fields. Default: comma (,).

    Field Mapping

    Input Field

    Displays the output fields of the upstream widget.

    Output Field

    Output fields of this widget. Add fields using Batch Add or Create New Output Field:

    • Batch Add: Click Batch Add. JSON and TEXT formats are supported for batch configuration.

      • JSON format example:

        // Example:
        [{"name": "user_id","type": "String"},
        {"name": "user_name","type": "String"}]
        Note

        name: the field to import. type: the data type after import. For example, "name":"user_id","type":"String" imports the field named user_id and sets its data type to String.

      • TEXT format example:

        // Example:
        user_id,String
        user_name,String
        • Row delimiter separating field entries. Default: line feed (\n). Supported: line feed (\n), semicolon (;), period (.).

        • Column delimiter separating field name and type. Default: comma (,).

    • Create New Output Field.

      Click + Create New Output Field, enter a Column name, select a Type, and click the image icon to save.

    • Copy Ancestor Table Field.

      Click Copy Ancestor Table Field to auto-generate output fields from the ancestor table.

    • Manage Output Fields.

      Additional operations on added fields:

      • Click the agag icon in the Actions column to edit a field.

      • Click the agfag icon in the Actions column to delete a field.

    Mapping

    Maps source input fields to target output fields for data synchronization. Two modes are available: Same Name Mapping and Same Row Mapping:

    • Same Name Mapping: Maps fields with the same field name.

    • Same Row Mapping: Maps fields by row position when source and target field names differ.

  9. Click Confirm to save the MongoDB output widget configuration.