Offline integration: Incremental append

Scenario

How do you incrementally append data to a new partition in MaxCompute(or Hive) when the approval status of a record changes in an upstream Office Automation (OA) system's MySQL database? How do you later query the complete status flow details for that approval process?

Solution and features

1. In Dataphin, use a filter in the input component to retrieve recently changed data.

   Go to Develop > Data Integration. In the input component of the target integration node, set the filter condition to gmt_modified=${bizdate}. This retrieves data where the modification time equals the data timestamp, which is the day before the current date. When you configure a daily schedule, the system retrieves changed data every day.

2. Configure the MaxCompute output component. Set Loading Policy to Append Data and specify a partition. The changed data is then appended to the latest partition of the MaxCompute table.

   Note

   The append data loading policy is available only for standard tables.

3. To query the details of the status flow for OA approvals, specify ds≤${bizdate} in the `WHERE` clause.