Configure custom lineage

Limitations

• You can configure custom lineage only for non-SQL compute tasks.

• You can configure a maximum of 20 lineage groups. Each lineage group can have up to 5 inputs and 5 outputs.

• You can add dataset lineage to Python or Shell tasks only after the unstructured data feature is enabled.

Lineage configuration

• Lineage group and lineage: Each task can have multiple lineage groups. Within a lineage group, each input is mapped to each output to create lineage. Lineage groups are independent of each other. Each input and output configuration generates both table-level lineage and field-level lineage.

  For example, if you select field a from table A for Input 1, field b from table B for Input 2, and fields c and d from table C for the output, this generates the following lineage:

  • Table-level lineage: table A-->table C; table B-->table C.

  • Field-level lineage: table A.a-->table C.c; table B.b-->table C.c; table A.a-->table C.d; table B.b-->table C.d.

  You can configure lineage between tables and datasets.

  • Lineage between a table (full table) and a file dataset or a hybrid dataset.

  • Lineage between a table (specified fields) and a hybrid dataset (specified fields).

  • Lineage between a table (full table or specified fields) and another table (full table or specified fields).

• You can specify the environment for the input and output tables. If you set the environment to Auto, Dataphin automatically replaces the environment variables during submission and publishing to generate lineage for the development and production environments, respectively.

• Dataphin processes your custom lineage configuration when you submit or publish a task. If you resubmit or republish the task, the lineage is updated.

Configure lineage

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a project from the top menu bar. If you are using the Dev-Prod mode, you also need to select an environment.

3. In the left-side navigation pane, choose Data Processing > Compute Task.

4. In the compute task list, click the target compute task to open its tab.

5. In the right-side pane, click Lineage to open the Lineage Configuration panel.

6. In the Lineage Configuration panel, click + Add Lineage Group to configure input and output objects.

   • Configure input objects

     1. In the lineage group area, click Configure Input Object. In the Configure Input Object dialog box, set the following parameters. The available parameters vary based on the object type.

        Table

        Parameter	Description
        Environment	The environment of the input and output tables. Valid values are Auto, Development, and Production. If you select Auto, you can choose tables from the development environment. This is equivalent to using a project variable to reference the table. When the task is submitted, the system replaces this variable with the corresponding development environment table and parses the lineage. After the task is published, the system automatically replaces the variable with the corresponding production environment table and parses the lineage. If a corresponding production table does not exist, submission and publishing are not blocked, but the production lineage cannot be parsed. If you select Development or Production, no variable replacement occurs during submission or publishing. The configuration is used as is.
        Object type	Select Table.
        Input table	The supported data table types include physical tables, physical views, dimension logical tables, fact logical tables, summary logical tables, tag logical tables, and logical views.
        Selected scope	Entire table: The system generates lineage based on all fields of the table at the time of submission and publishing. Note Lineage is updated only when the task is successfully submitted and published. Subsequent changes to the input or output table schemas do not automatically trigger a lineage update. Specified fields: Select specific fields from the table based on your business requirements.

        Dataset

        Parameter	Description
        Environment	The environment of the input and output datasets. Valid values are Auto, Development, and Production. If you select Auto, you can choose datasets from the development environment. This is equivalent to using a project variable to reference the dataset. When the task is submitted, the system replaces this variable with the corresponding development environment dataset and parses the lineage. After the task is published, the system automatically replaces the variable with the corresponding production environment dataset and parses the lineage. If a corresponding production dataset does not exist, submission and publishing are not blocked, but the production lineage cannot be parsed. If you select Development or Production, no variable replacement occurs during submission or publishing. The configuration is used as is.
        Object type	Select Dataset.
        Input dataset	You can select only a dataset that has been added to the properties of the current compute task. You can search for the dataset by entering keywords from its name.
        Selected scope	Entire dataset: For a file dataset, only dataset-level lineage is generated. For a hybrid dataset, the system generates lineage based on all fields in the dataset's metadata table at the time of submission and publishing. Specify Fields: You can select all fields in the selected dataset. Only when the selected dataset is a hybrid dataset is there support for field-level lineage. Note When a file dataset is an input or output, the system creates only the overall lineage between the file dataset and the associated object, regardless of whether specific fields are selected in the associated object. The system does not create lineage between a file dataset and the fields of another dataset.

     2. Click OK to save the input object configuration.

   • Configure output objects

     1. In the lineage group area, click Configure Output Object. In the Configure Output Object dialog box, set the required parameters. The output object parameters are the same as the input object parameters. For more information, see the tables above.

     2. Click OK to save the output object configuration.

7. After you configure the input and output objects for all lineage groups, click OK in the Lineage Configuration panel to save the lineage configuration.

Submit and publish the lineage

1. After you configure the lineage, click Submit above the code editor for the current compute task.

2. In the Submission Details dialog box, click Confirm and Submit.

3. In the Submission Details dialog box, the Object Check step validates whether the input and output objects and fields referenced in the lineage configuration exist.

   Note

   • The system performs object checks for the lineage configuration only during submission. No additional checks are performed during publication.

   • When you submit a task, the system parses the table-level and field-level lineage for the development environment. When you publish the task, it parses the lineage for the production environment. The system can parse a maximum of 100,000 lineage relationships per submission or publication. If this limit is exceeded, the system does not record the lineage, and it will not appear in the asset catalog.

View the lineage

After the task is submitted and published, you can view the custom table-level and field-level lineage on the Physical Table Details page. For more information, see Physical Table Details.