Dataphin automatically parses the data lineage of tables and fields when you submit an offline computing task. It also identifies changes to task objects and performs pre-checks to ensure your task meets submission criteria.
Data lineage parsing
During task submission, the system parses data lineage in the development environment. During release, it parses data lineage in the production environment. A maximum of 100,000 lineage relationships are recorded. Relationships beyond this limit are not recorded and cannot be displayed in the asset catalog.
Submission details
The Submission Details dialog box displays the submission content, pre-check results, and submission remarks for an offline computing task.
-
Submission content
You can view the object name, object type, change type, and change content of the submitted task object. The change content includes Basic information, Compute code, Runtime parameters, Scheduling configuration, Scheduling dependencies, Runtime configuration.
-
Pre-check
The system performs the following pre-checks during submission. All checks must pass before you can submit the task.
Check item
Description
Scheduling dependencies
Dataphin uses scheduling dependencies to run nodes in a business process in the correct order, ensuring that business data is generated correctly and on time. For more information, see Configure Scheduling Dependencies for an Offline Task.
Runtime parameters
A runtime parameter assigns a value to a variable in the task code. During scheduling, the system automatically replaces the variable with its assigned value. For more information, see Configure Runtime Parameters for an Offline Task.
Cross-node parameters
Variable parameters passed to the immediate downstream nodes. For more information, see Configure Runtime Parameters for an Offline Task.
Dataset lineage configuration
For Python and Shell tasks, if a referenced dataset is inconsistent with its data lineage, the submission fails. For more information, see and Custom Lineage Configuration.
-
Submission remarks
You can enter submission remarks, with a limit of 128 characters.
Check items
After all pre-checks pass, click Confirm and Submit to view the check items and results in the Submission Details dialog box.
|
Check item |
Description |
|
Configuration check |
Checks required attributes, code length, number of referenced resources, and the offline code template version.
|
|
Parameter configuration |
The system parses all variables in the code and lists the parameter name, type, and current value. It also verifies that all local variables have assigned values. |
|
Syntax check |
For SQL tasks only. Validates the SQL syntax and blocks submission if errors are found. |
|
Object check |
For SQL tasks only. Verifies that all referenced objects have been submitted and published to the production environment, and lists each object with its name, type, and check result.
|
|
Permission check |
Verifies that you have the required permissions for all objects in the code. Results are listed by object name, object type, check result, row-level security status, and an option to apply for permissions.
|
|
Specification check |
For SQL tasks only. Scans SQL code against built-in specifications and displays the result for each rule. For more information, see Coding Specifications. |
|
Dependency check |
The system checks the dependency configuration and output configuration.
|
|
Code review |
The system checks whether the task has a configuration that triggers a code review, such as referencing a global variable with code review enabled. For more information, see View and Configure Built-in Approval Templates. Important
|
|
Action execution |
When the checks reach this item, the system executes the submission. You cannot cancel the submission during this process. |