Offline computing task submission

更新时间:
复制 MD 格式

Dataphin automatically parses the data lineage of tables and fields when you submit an offline computing task. It also identifies changes to task objects and performs pre-checks to ensure your task meets submission criteria.

Data lineage parsing

During task submission, the system parses data lineage in the development environment. During release, it parses data lineage in the production environment. A maximum of 100,000 lineage relationships are recorded. Relationships beyond this limit are not recorded and cannot be displayed in the asset catalog.

Submission details

The Submission Details dialog box displays the submission content, pre-check results, and submission remarks for an offline computing task.

  • Submission content

    You can view the object name, object type, change type, and change content of the submitted task object. The change content includes Basic information, Compute code, Runtime parameters, Scheduling configuration, Scheduling dependencies, Runtime configuration.

  • Pre-check

    The system performs the following pre-checks during submission. All checks must pass before you can submit the task.

    Check item

    Description

    Scheduling dependencies

    Dataphin uses scheduling dependencies to run nodes in a business process in the correct order, ensuring that business data is generated correctly and on time. For more information, see Configure Scheduling Dependencies for an Offline Task.

    Runtime parameters

    A runtime parameter assigns a value to a variable in the task code. During scheduling, the system automatically replaces the variable with its assigned value. For more information, see Configure Runtime Parameters for an Offline Task.

    Cross-node parameters

    Variable parameters passed to the immediate downstream nodes. For more information, see Configure Runtime Parameters for an Offline Task.

    Dataset lineage configuration

    For Python and Shell tasks, if a referenced dataset is inconsistent with its data lineage, the submission fails. For more information, see and Custom Lineage Configuration.

  • Submission remarks

    You can enter submission remarks, with a limit of 128 characters.

Check items

After all pre-checks pass, click Confirm and Submit to view the check items and results in the Submission Details dialog box.

Check item

Description

Configuration check

Checks required attributes, code length, number of referenced resources, and the offline code template version.

  • Pre-check: Verifies that you have filled in all required attributes.

  • Code length: The system checks if the code length is within 100,000 characters. You cannot submit tasks that exceed this limit.

  • Number of referenced resources: The system checks if the number of referenced resources exceeds 100.

  • Offline code template version: The system checks for any changes in the offline code template version.

Parameter configuration

The system parses all variables in the code and lists the parameter name, type, and current value. It also verifies that all local variables have assigned values.

Syntax check

For SQL tasks only. Validates the SQL syntax and blocks submission if errors are found.

Object check

For SQL tasks only. Verifies that all referenced objects have been submitted and published to the production environment, and lists each object with its name, type, and check result.

  • Object Name: The format is [catalog].[object_name].

    • Project name variable: If the catalog in the code is the project name variable ${catalog}, the catalog resolves to the name of the production project.

    • Hard-coded: If you hard-code the names of development or production spaces in your code, the original names are used. If you omit the catalog, the system defaults to using ${catalog}.

    • Omitted catalog: If you omit the catalog, the system defaults to using the ${catalog} variable.

  • Object Type: Checked object types include physical table, field, and logical table.

  • Check Result: The result can be Succeeded or Failed.

    • Succeeded: All checked objects have been submitted and published to the production environment.

      Note

      If the catalog is hard-coded, the system only checks whether the object exists in the corresponding environment.

    • Failed: One or more checked objects have not been submitted or published to the production environment.

      Note

      The check also fails if any specified field does not exist in a checked object.

Permission check

Verifies that you have the required permissions for all objects in the code. Results are listed by object name, object type, check result, row-level security status, and an option to apply for permissions.

  • Object Name: The format is [catalog].[object_name].

    • Project name variable: If the catalog in the code is the project name variable ${catalog}, the catalog resolves to the name of the production project.

    • Hard-coded: If a development or production project name is hard-coded, the original name is used. If the catalog is omitted, the system defaults to using ${catalog}.

    • Omitted catalog: If you omit the catalog, the system defaults to using the ${catalog} project name variable.

  • Object Type: Checked object types include physical table, global variable, logical table, and function.

    Note

    If a referenced function belongs to a project where cross-project reference authentication is enabled, both the user's personal account and the production account must be authenticated during pre-editing, running, and submission.

  • Check Result: Verifies that both the current user account and the production account have read and write permissions for the object. The result is Succeeded or Failed.

    Note
    • If the object is a table, read and write permissions are required for all referenced fields.

    • If the object is a function in a Basic project, the production account needs the corresponding read and write permissions. In a Dev-Prod project, the user's personal account needs the corresponding read and write permissions.

  • Row-level Permission Status: Requires a paid add-on for row-level security. If a table has row-level security enabled and an account lacks the necessary permissions, the system adds the Row-level Permission Status column. The status can be Succeeded, Failed, or Succeeded with Risks.

    • Failed: If the user's personal account does not have row-level security for the corresponding object, the status is Failed.

    • Succeeded with Risks: In a Dev-Prod project, if the user's personal account has row-level security but the production account does not, the status is Succeeded with Risks.

  • Apply for permissions: If an object check fails, you can click the image.png icon at the bottom to apply for permissions. For specific steps, see My Permissions.

Specification check

For SQL tasks only. Scans SQL code against built-in specifications and displays the result for each rule. For more information, see Coding Specifications.

Dependency check

The system checks the dependency configuration and output configuration.

  • Dependency Configuration: The system checks whether the dependency node objects configured for the task exist in the development environment.

  • Output Configuration: The system checks whether the output name of the task is already in use within the current tenant.

Code review

The system checks whether the task has a configuration that triggers a code review, such as referencing a global variable with code review enabled. For more information, see View and Configure Built-in Approval Templates.

Important
  • If code review is enabled for the project where the task resides, a code review is triggered upon submission, regardless of whether the task references a global variable that requires a review.

  • If code review is enabled for the project and the task references a global variable that also requires a code review, you must pass every approval task within the code review for the submission to proceed.

Action execution

When the checks reach this item, the system executes the submission. You cannot cancel the submission during this process.