Field editing and value assignment

Step 1: Configure a Data Integration task

1. Create a data source. For more information, see Data source management.

2. Create a Data Integration task. For more information, see Configure a real-time synchronization task for a single table.

   Note

   For a Data Integration real-time synchronization task for a single table, you can add a data processing component between the source and destination components. For more information, see Supported data sources and synchronization solutions.

Step 2: Add field editing and value assignment

1. In the configuration of the real-time synchronization task for a single table, turn on the Data Processing switch, click +Add Node, and add the Edit Field and Assign Value component.

2. Configure the field editing and value assignment component.

   In this example, the upstream Kafka source outputs the following fields: _key_ (STRING), _value_ (STRING), _partition_ (LONG), _offset_ (LONG), _timestamp_ (LONG), and _headers_ (STRING). All of these fields are selected for downstream output. This example adds a custom field named add_col of type STRING, sets the value assignment method to constant, and sets the value to 1.

   • Ancestor fields

     • Field Name: The name of an output field from the ancestor component.

     • Type: The data type of an output field from the ancestor component.

     • Value: For an existing ancestor field, the value is its output. For a new field, you can choose one of the following value assignment methods: constant, variable, or function.

     • Actions: For an ancestor field, you can choose whether to pass its output to the downstream component.

   • Custom Field: Click Add Field in the lower-left corner of the page to add a new field.

     • Constant: Converts a manually entered constant string to the specified data type and assigns it to the new field.

     • Variable: Assigns the value of a built-in variable to the new field when the task runs. The following table lists the supported built-in variables.

       Note

       Support for these variables depends on the specific source and destination systems.

       Supported variables

       Parameter		Description
       Source	PROCESS_TIME	The time the synchronization task processes a data record. The value is a 13-digit timestamp in milliseconds.
       	DATASOURCE_NAME_SRC	The name of the source data source.
       	DB_NAME_SRC	The name of the source database.
       	SCHEMA_NAME_SRC	The name of the source schema.
       	TABLE_NAME_SRC	The name of the source table.
       	TOPIC_NAME_SRC	The name of the source topic.
       Destination	TABLE_NAME_DEST	The name of the destination table.
       	TOPIC_NAME_DEST	The name of the destination topic.
       	SCHEMA_NAME_DEST	The name of the destination schema.
       	DB_NAME_DEST	The name of the destination database.
       	DATASOURCE_NAME_DEST	The name of the destination data source.

     • Function: This option uses a function to perform calculations at runtime and assigns the results to the new field. If an exception occurs, the system treats the record as dirty data, includes it in the task's dirty data statistics, and then decides whether to terminate the task based on the configured dirty data tolerance.

       Supported functions

       Function name	Parameters	Description
       DATE_FORMAT	Parameter 1: An ancestor field of a numeric type, such as a 10-digit UNIX timestamp or a 13-digit timestamp in milliseconds. Parameter 2: The timestamp type. Use unix for a 10-digit UNIX timestamp or millis for a 13-digit timestamp in milliseconds. Parameter 3: The time zone used for the conversion. Parameter 4: The format of the date string to be returned, for example yyyy-MM-dd HH:mm:ss:SSS ZZ. yyyy represents the year, MM represents the month, dd represents the day, HH represents the hour in 24-hour format, mm represents the minute, ss represents the second, SSS represents the millisecond, and ZZ represents the time zone.	Converts a timestamp to a time string in a specific format.
       TO_TIMESTAMP	Parameter 1: An ancestor field of a text type that contains a time string in a specific format. Parameter 2: The target timestamp type. Use unix for a 10-digit UNIX timestamp or millis for a 13-digit timestamp in milliseconds. Parameter 3: The time zone used for the conversion. Parameter 4: The time string format, for example, yyyy-MM-dd HH:mm:ss:SSS ZZ, where yyyy is the year, MM is the month, dd is the day, HH is the hour in 24-hour format, mm is the minute, ss is the second, SSS is the millisecond, and ZZ is the time zone.	Converts a time string into a 10-digit or 13-digit timestamp.
       SUBSTRING	Parameter 1: An ancestor field of a text type. Parameter 2: The starting position of the substring. The substring includes the character at the starting position. The minimum starting position is 0. Parameter 3: The length of the substring. If this value is less than or equal to 0, the function returns a substring from the starting position to the end of the string.	Extracts a substring from a string.

Next steps

After configuring the source and field editing and value assignment components, click Output Preview to preview the node's output data and verify its correctness. To write the new fields to the destination table, you must first configure their mappings in the field mapping settings.