Offline Integration

Section

Description

①Offline Integration Directory

Manage created offline pipeline tasks in the task directory. The task directory supports the following management operations:

• Directory toolbar: Supports Filter, Upload (upload pipeline development scripts), Refresh, Create folder, and Locate.

• Create integration task: Click the button. Create single pipeline tasks and script pipeline tasks.

  • Single pipeline task: Allows you to sync single or multiple tables from one or more source data sources to one or more destination data sources using a visual drag-and-drop method. For more information, see Create an integration task using a single pipeline.

  • Script Pipeline Task: Lets you sync single or multiple tables from one or more source data sources to one or more target data sources using a script-based approach. For detailed steps, see Create Integration Tasks via Script Mode.

• Offline pipeline task: Supports Copy, Acquire lock, Edit, Clone, Rename, Shift, Delete, Unpublish, Unpublish and delete, and Modify project ownership.

  Important

  In Basic mode, a task is updated to the production environment after successful deletion. Proceed with caution. In Dev-Prod mode, after a task is successfully deleted, go to the pending release list to publish the object to the production environment.

• Offline Integration Task List: Click this to manage the offline integration tasks that you created. For more information, see View the offline integration task list.

• Shift tasks: Shift offline script tasks, offline pipeline tasks, and regular folders within the Offline Integration Directory. Offline script tasks and offline pipeline tasks can also shift to offline full database migration folders. Regular folders cannot shift into or out of offline full database migration folders.

  If a regular folder contains child nodes, you cannot delete it. When you delete an offline full database migration folder, the system deletes all tasks within it, including offline script tasks and offline pipeline tasks.

②Toolbar

The toolbar supports Save, Run, Preview, Scheduling Configuration, Channel Configuration, Quality Monitoring, Submit, Go to Publish, Go to O&M, Refresh, Acquire Lock, and Locate.

• Save: Click the icon in the toolbar to save the offline pipeline you are developing.

• Run: Click the icon in the toolbar to run the offline pipeline you are developing. If your current account lacks row-level permissions, click the Request Permission icon. This redirects you to the Management Hub > Permission Management > My Permissions > Row-Level Permissions page. Request permissions there, then run the pipeline again.

• Preview: Click the icon in the toolbar to preview the integration results after the offline pipeline runs.

• Scheduling Configuration: Click the icon in the toolbar to open the Scheduling Configuration tab in the Properties panel. Scheduling Configuration defines the scheduling rules for periodically scheduled offline pipeline tasks. For more information, see Configure Offline Pipeline Scheduling Configuration.

• Channel Configuration: Click the icon in the toolbar to open the Channel Configuration tab in the Properties panel. Using offline integration pipeline channel configuration, you can define fault tolerance, concurrency, JVM resources, database configuration, and traffic monitoring for integration pipelines at runtime for offline integration tasks. For more information, see Offline Integration Pipeline Channel Configuration.

• Quality Monitoring: Click the icon in the toolbar to open the Quality Monitoring panel. You can configure quality monitoring rules for data tables in integration pipelines. For more information, see Configure Quality Monitoring for Integration Pipelines.

• Submit: Click the icon in the toolbar to submit the offline integration pipeline to the Release Center or Management Center. For more information, see Release Center Overview.

• Go to Publish: Click the icon in the toolbar to publish the latest pending content. For offline integration pipelines already submitted to the Release Center, go to the Release Center to publish them.

• Go to Operation Center. Click the icon in the toolbar to manage published instances. For offline integration pipelines that have been submitted to the Operation Center, you can go to the Operation Center to perform O&M operations. For more information, see Operation Center.

• Switch to Script Mode: Click the icon, then click Switch to Script Mode in the drop-down options. Switch the current offline pipeline to an offline script. This is useful when you need to automatically retrieve source table columns and generate complete JSON Configurations, which improves the efficiency of script mode configuration. After switching to script mode, you cannot switch back to offline pipeline mode.

• Refresh: Click the icon in the toolbar to refresh the current offline pipeline task.

• Acquire Lock: Click the icon in the toolbar to acquire the edit lock for the offline pipeline task.

• Locate: Click the icon in the toolbar to locate the current offline integration pipeline task in the task directory.

• Unpublish: Click the icon, select Unpublish, and unpublish published integration tasks.

• Delete: Click the icon, select Delete, and delete the current integration task.

  Important

  In Basic mode, a task is updated to the production environment after successful deletion. Proceed with caution. In Dev-Prod mode, after a task is successfully deleted, go to the pending release list to publish the object to the production environment.

③Development Canvas

• Visual development: On the development canvas, you can drag and drop components from the component library drawer onto the canvas. For more information about the development components supported by Dataphin and their usage instructions, see Integrated Component Library Development Guide.

• Code Script Development: For script tasks, use the code editor to develop offline pipeline tasks.

④Configuration and Management

Configuration and Management supports property configuration, quality monitoring, related objects, and history for integration tasks.

• Property Configuration: Configure properties for the offline pipeline task, such as Basic Information, Scheduling Configuration, Scheduling Dependencies, and Runtime Configuration. For more information, see Configure Offline Pipeline Task Properties.

• Quality monitoring: By configuring quality monitoring for offline integration pipelines, Dataphin automatically monitors data table quality. If a data table has quality risks, the system automatically sends alerts and notifies alert recipients to help you understand the quality status of your data tables promptly. For specific instructions, see Configure integration pipeline quality monitoring.

• Related Objects: Related objects for an offline pipeline refer to tables referenced and read by the current offline pipeline task, tables that reference and read the current task, and tasks that use the current task as an upstream dependency.

• History: View and manage task historical versions.

⑤Console

When you run or preview a task, the console displays structured runtime result information. This includes preview results, runtime information, channel information, step measures, log information, exception information, performance diagnostics, and runtime code.