Batch register custom entities and lineage relationships via OpenAPI

DataWorks edition

Creating entity definitions, entity objects, custom attributes, and lineage relationships requires DataWorks Professional Edition or higher. Specifically:

• CreateMetaEntityDef creates a custom entity definition or an extended table entity definition. The operation requires Professional Edition or higher.

• BatchCreateMetaEntities batch-creates custom entity objects, extended Database objects, and extended Table objects. The operation requires Professional Edition or higher.

• CreateLineageRelationship registers lineage relationships. The operation requires Professional Edition or higher.

If you only call query operations, refer to the edition requirements stated on each operation's reference page.

Account and role permissions

The sample code uses an AccessKey pair to call OpenAPI. The Alibaba Cloud account, RAM user, or STS identity that owns the AccessKey pair must belong to the target DataWorks tenant and have permission to call DataWorks OpenAPI.

To create custom entity definitions, custom attributes, pure custom entities, and extended table entities, use an account with any of the following permissions:

• Tenant owner.

• Tenant administrator.

• Tenant data-governance administrator.

• An account that has the DataWorks FullAccess permission.

To update the business attributes of MaxCompute tables or columns that DataWorks has collected, the caller must also be the MaxCompute table owner, or the project owner or project administrator of the corresponding DataWorks project. The Demo in this topic only reads the IDs of collected tables and columns to register lineage and does not modify these collected tables or columns.

Data to prepare before running the Demo

To run the Java Demo, replace the constant UPSTREAM_TABLE_ID in the class with a DataWorks-collected table ID that actually exists under your account. Obtain the ID as follows:

• Table ID: call ListTables or GetTable and use the Id field in the response.

• Column ID: call ListColumns or GetColumn and use the Id field in the response.

For tables, columns, datasets, and dataset versions that DataWorks has collected, do not concatenate entity IDs manually. Use the IDs returned by the query operations to avoid lineage registration failures caused by ID-format changes or by the object not being collected yet. Extended tables and extended columns registered through BatchCreateMetaEntities can also be queried later using ListTables, GetTable, ListColumns, or GetColumn to obtain the Id. If you later use these extended tables or columns as lineage endpoints, also prefer IDs returned by the query operations.

If you simply onboard your own object list and lineage list by following the "Batch registration engineering pattern" section, you do not have to use the upstream table example in this Demo. You can pass any upstream and downstream entity IDs that actually exist in your list.

Example entity definition list

Example entity object list

Example lineage relationship list

Step 1: Compile the object and relationship lists

First, compile the object list and relationship list of your enterprise system. The object list typically includes:

• Object name, such as Order Query API, Sales Report, or External Order Table.

• Object type, such as API, report, metric, or database table.

• Business attributes, such as owner, business domain, and source system.

• For tables or columns, the database name, table name, column name, and column type.

The relationship list typically includes:

• Upstream object.

• Downstream object.

• Relationship type, such as table-produces-report, column-maps-column, or API-writes-table.

• Task or process that produces the relationship, such as a sync task, an ETL task, or an API call chain.

For batch onboarding, first organize your entity and lineage lists according to the Data preparation templates section, and then proceed to the API call flow.

Step 2: Determine whether the object has been onboarded by DataWorks

If an object is already collected or managed by DataWorks, do not register it again. Use the entity ID returned by the corresponding query operation. Common ways to obtain the ID:

• Table: call ListTables or GetTable.

• Column: call ListColumns or GetColumn.

• Dataset: call ListDatasets or GetDataset.

• Dataset version: call ListDatasetVersions or GetDatasetVersion.

If the object has not been onboarded by DataWorks yet, choose a registration approach based on the object type:

• For business objects such as reports, APIs, metrics, or systems: register them as custom entities.

• For external data assets that have a database/table/column structure: register them as extended table entities.

Step 3: Create entity definitions

Use CreateMetaEntityDef to create the entity type definition.

Custom entities are suitable for business concepts such as APIs, reports, or metrics. Example:

{
  "Name": "customer_api",
  "DisplayName": "Business API",
  "Description": "API object from an enterprise system",
  "AttributeDefs": [
    {
      "Name": "apiCode",
      "DisplayName": "API code",
      "Type": "STRING",
      "IsOptional": false
    },
    {
      "Name": "serviceName",
      "DisplayName": "Service name",
      "Type": "STRING",
      "IsOptional": true
    }
  ]
}

Extended table entities are suitable for describing external database, table, and column objects that DataWorks does not collect automatically. When you create an extended table, we recommend that you also create the corresponding extended Database object and link the table to it through parentMetaEntityId. This builds a Database-to-Table hierarchy that lets you browse external assets in Data Map with the full database/table layout. Example:

{
  "Name": "bi-table",
  "DisplayName": "BI report table",
  "Description": "Database tables and columns from an external BI system",
  "Extend": "TABLE"
}

For detailed field semantics and naming constraints, see the CreateMetaEntityDef reference page.

Step 4: Create reusable custom attributes

To display or filter entities in Data Map by business attributes, use CreateCustomAttribute to create custom attribute definitions. For example:

• Source system.

• Business domain.

• Owner.

• Data tier.

• Security level.

Example:

{
  "Id": "custom-attribute:source_system",
  "DisplayName": "Source system",
  "Type": "ENUM",
  "ValueEnums": ["CRM", "ERP", "BI", "CUSTOM_APP"],
  "EntityTypes": ["custom_entity-customer_api"],
  "SearchFilterEnabled": true,
  "DisplayEnabled": true
}

After a custom attribute is created, you can use it when you write custom entities, extended table entities, or update business information on collected tables and columns. In EntityTypes, specify the exact entity types to which the attribute applies. To attach the attribute to extended tables or extended columns, specify the concrete types generated by the corresponding extended entity definitions.

Step 5: Create entity objects

Use BatchCreateMetaEntities to write entity objects.

Example of a custom entity object:

{
  "Entities": [
    {
      "EntityType": "custom_entity-customer_api",
      "Name": "api_001",
      "Comment": "Business order-query API",
      "Attributes": {
        "apiCode": "api_001",
        "serviceName": "order-service"
      },
      "CustomAttributes": {
        "source_system": ["CRM"]
      }
    }
  ]
}

Example of an extended table entity object

Before writing an extended table, first write the database entity of the same extension type, then write the table entity with its parentMetaEntityId pointing to that database. This produces a complete Database -> Table -> Column hierarchy that is easier to browse and manage in Data Map. Because BatchCreateMetaEntities requires all entities in a single batch to share the same EntityType, send the database and the table as two separate requests.

First, write the external database:

{
  "Entities": [
    {
      "EntityType": "custom_bi-database",
      "Name": "report_db",
      "Comment": "External BI report database",
      "Attributes": {
        "parentMetaEntityId": "custom_bi:custom_bi_src",
        "technicalMetadata.location": "external://bi/report_db"
      }
    }
  ]
}

Next, write the external table together with its columns through the columns attribute:

{
  "Entities": [
    {
      "EntityType": "custom_bi-table",
      "Name": "sales_report",
      "Comment": "Sales report result table",
      "Attributes": {
        "parentMetaEntityId": "custom_bi-database:custom_bi_src::report_db",
        "columns": "[{\"name\":\"order_id\",\"type\":\"STRING\"},{\"name\":\"amount\",\"type\":\"DECIMAL\"}]"
      }
    }
  ]
}

You do not need to call BatchCreateMetaEntities separately to create the columns. After the table is written successfully, the external columns can participate in column-level lineage using their extended column IDs. You can obtain extended table and column IDs by calling ListTables, GetTable, ListColumns, or GetColumn. If you know the ID rule for an extended entity, you can also construct the ID, for example custom_bi-column:custom_bi_src::report_db::sales_report:order_id.

When the write operation succeeds, the response returns the entity ID. Use this ID for subsequent lineage registration, detail queries, and entity updates.

Step 6: Register lineage relationships

Use CreateLineageRelationship to register the upstream-to-downstream relationship between entities. The direction of a lineage edge is always upstream SrcEntity -> downstream DstEntity.

Example 1: From a DataWorks-collected table to a custom API.

{
  "SrcEntity": {
    "Id": "mysql-table:rm-xxx::sales_db::orders",
    "Name": "orders"
  },
  "DstEntity": {
    "Id": "custom_entity-customer_api:api_001",
    "Name": "api_001"
  },
  "Task": {
    "Id": "table_to_api_001",
    "Type": "custom-lineage-task",
    "Attributes": {
      "scene": "table_to_api"
    }
  }
}

Example 2: From a DataWorks-collected table to an external extended table.

{
  "SrcEntity": {
    "Id": "maxcompute-table:::prod_project:default:dwd_order",
    "Name": "dwd_order"
  },
  "DstEntity": {
    "Id": "custom_bi-table:custom_bi_src::report_db::sales_report",
    "Name": "sales_report"
  },
  "Task": {
    "Id": "dwd_order_to_sales_report",
    "Type": "custom-table-lineage",
    "Attributes": {
      "scene": "external_bi_table"
    }
  }
}

Example 3: From a DataWorks-collected column to an external extended column.

{
  "SrcEntity": {
    "Id": "maxcompute-column:::prod_project:default:dwd_order:order_id",
    "Name": "order_id"
  },
  "DstEntity": {
    "Id": "custom_bi-column:custom_bi_src::report_db::sales_report:order_id",
    "Name": "order_id"
  },
  "Task": {
    "Id": "dwd_order_id_to_report_order_id",
    "Type": "custom-column-lineage",
    "Attributes": {
      "scene": "external_bi_column"
    }
  }
}

For column-level lineage, SrcEntity and DstEntity use column entity IDs. Column IDs can come from ListColumns or GetColumn and cover both DataWorks-collected columns and registered extended-table columns. For detailed request parameters and response fields, see the CreateLineageRelationship reference page.

Step 7: Verify the registration result and consume the lineage

After registration, we recommend that you perform two types of verification.

Verify that entities were written successfully:

• Custom entity: call GetMetaEntity.

• Extended table: call GetTable.

• Extended column: call GetColumn.

• DataWorks-collected objects: call the corresponding Get operation.

Verify that lineage was written successfully:

• Query downstream: call ListLineages with SrcEntityId.

• Query upstream: call ListLineages with DstEntityId.

• To return relationship details, set NeedAttachRelationship=true.

After basic verification, you can use lineage data for daily analysis:

• Upstream lineage tracing: starting from a target entity, call ListLineages recursively against upstream nodes to answer "where does the data behind this report or table come from?"

• Downstream impact analysis: starting from a changed entity, call ListLineages to query downstream and aggregate by downstream entity type and count to answer "which reports, APIs, or external tables are affected by a change to this table or column?"

• Broken-link detection: compare your batch-registered entity list with the lineage endpoint list to find orphan entities that have no upstream or downstream, as candidates for additional registration or cleanup.

• Lineage health checks: regularly check that entities still exist, that lineage relationships are not duplicated, that lineage endpoints are queryable, and that batch writes have no failed items, and produce a health-check report.

The multi-hop impact analysis can be implemented as follows:

queue = [changedEntityId]
visited = set()
while queue is not empty:
    current = queue.pop()
    downstream = ListLineages(SrcEntityId=current)
    for entity in downstream:
        if entity.id not in visited:
            visited.add(entity.id)
            queue.push(entity.id)

Emit entity types, entity names, and impact paths from visited.

In your batch onboarding job, persist the entity IDs and lineage relationship IDs that were registered successfully in each run. For subsequent impact analysis, broken-link detection, or incremental updates, you can then diff against this result table using the query operations, without re-parsing the original business-system data.

Recommended write order

For batch onboarding, we recommend the following execution order:

1. Create or confirm entity definitions by calling CreateMetaEntityDef.

2. Create or confirm custom attributes by calling CreateCustomAttribute.

3. Write extended Database objects by calling BatchCreateMetaEntities.

4. Write extended Table objects together with their Columns.

5. Write pure custom entities, such as reports, APIs, and metrics.

6. Register lineage relationships by calling CreateLineageRelationship.

7. Run query-based verification and generate an onboarding result report.

Batching, retry, and resumable runs

BatchCreateMetaEntities is the batch write operation. Each request can write at most 5 entities, and all entities in a single batch must share the same EntityType. In production, first group the rows by EntityType and then split each group into batches of 5.

CreateLineageRelationship writes one lineage edge per call. For large lineage volumes, control the concurrency and apply backoff-and-retry for throttling or transient service errors.

We recommend the following error-handling pattern:

for batch in splitByEntityType(rows, size=5):
    resp = BatchCreateMetaEntities(batch)
    if resp.Success is not true:
        write_batch_failed(batch, resp.RequestId, resp.Message)
        continue

    for result in resp.Results:
        if result.Success is true:
            mark_done(result.Id)
        else:
            write_row_failed(result.Id, result.ErrorMessage, resp.RequestId)

Recommendations for resumable runs:

• Use stable business IDs for entities and lineage relationships so that the same identifiers are produced across runs.

• After each batch write, record the successful items, failed items, RequestId, failure reasons, and original row numbers.

• On a rerun, skip items that already succeeded and retry only failed or changed items.

• Before a run, use query operations such as GetMetaEntity, GetCustomAttribute, or ListLineageRelationships to confirm whether the target objects already exist.

• 4xx data errors usually require fixing the list before retrying. Throttling, network jitter, and transient service errors can use a bounded backoff-and-retry policy.

For daily client-side sync jobs, keep a "desired state table" and a "last-successful write result table". For incremental sync, process only new and changed objects. For periodic full reconciliations, diff the desired state table against the DataWorks query results to detect missing, duplicated, or deprecated relationships and then re-register or clean them up.

Custom entities become searchable and filterable

After custom entity definitions and entity objects are registered, you can search for the corresponding entities in Data Map. If the entity definition enables search-filter capability on its attributes, the left pane shows the corresponding filters (for example, filtering report entities by business domain, source system, or owner).

Entity detail page shows attribute information

On the detail page of a custom entity, you can view the basic information, regular attributes, and custom attributes. For example, the detail page of a BI report entity shows fields such as report code, business domain, owner, and source system.

Custom attributes are managed in one place

Custom attributes created through CreateCustomAttribute appear on the custom attribute management page of Data Map, where you can review their names, types, and applicable entity types for easy maintenance and reuse.

Extended table entities join the table asset search

After extended Database, Table, and Column objects are registered, external tables join the table asset view of Data Map and can be filtered by data source type and database, presented together with DataWorks-collected tables.

Table-level and column-level lineage in the lineage graph

After "table to custom entity", "table to table", and "column to column" lineage is registered, you can view the upstream and downstream relationships on the lineage page. For example, a sales summary table can simultaneously link to a custom report entity and to an extended table entity, with column-level lineage drilling down.

1. Prepare the project

Project layout:

custom-metadata-lineage-demo
├── pom.xml
└── src
    └── main
        └── java
            └── CustomMetadataLineageDemo.java

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>example</groupId>
  <artifactId>custom-metadata-lineage-demo</artifactId>
  <version>1.0.0</version>
  <properties>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <dependencies>
    <dependency>
      <groupId>com.aliyun</groupId>
      <artifactId>dataworks_public20240518</artifactId>
      <version>${dataworks.sdk.version}</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.1.0</version>
        <configuration>
          <mainClass>CustomMetadataLineageDemo</mainClass>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

2. Write the Demo

Save the following code as src/main/java/CustomMetadataLineageDemo.java.

import com.aliyun.dataworks_public20240518.Client;
import com.aliyun.dataworks_public20240518.models.BatchCreateMetaEntitiesRequest;
import com.aliyun.dataworks_public20240518.models.BatchCreateMetaEntitiesResponse;
import com.aliyun.dataworks_public20240518.models.CreateCustomAttributeRequest;
import com.aliyun.dataworks_public20240518.models.CreateLineageRelationshipRequest;
import com.aliyun.dataworks_public20240518.models.CreateMetaEntityDefRequest;
import com.aliyun.dataworks_public20240518.models.GetCustomAttributeRequest;
import com.aliyun.dataworks_public20240518.models.GetCustomAttributeResponse;
import com.aliyun.dataworks_public20240518.models.GetMetaEntityDefRequest;
import com.aliyun.dataworks_public20240518.models.GetMetaEntityDefResponse;
import com.aliyun.dataworks_public20240518.models.LineageEntity;
import com.aliyun.dataworks_public20240518.models.LineageTask;
import com.aliyun.dataworks_public20240518.models.MetaEntityAttributeDef;
import com.aliyun.dataworks_public20240518.models.MetaEntityWriteResult;
import com.aliyun.teaopenapi.models.Config;

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class CustomMetadataLineageDemo {
    private static final String REGION_ID = "cn-shanghai";

    // DataWorks-collected table ID. Replace with the result of ListTables/GetTable.
    private static final String UPSTREAM_TABLE_ID = "maxcompute-table:::prod_project:default:dwd_sales_order";
    private static final String UPSTREAM_TABLE_NAME = "dwd_sales_order";

    // BI report example.
    private static final String REPORT_ENTITY_TYPE = "custom_entity-bi_report";
    private static final String REPORT_ENTITY_NAME = "daily_sales_report";

    // External extended table example.
    private static final String EXT_PREFIX = "warehouse";
    private static final String EXT_SOURCE = "custom_external_bi_src";
    private static final String EXT_DATABASE = "analytics_db";
    private static final String EXT_TABLE = "ads_sales_summary";
    private static final String EXT_TABLE_ID =
            "custom_" + EXT_PREFIX + "-table:" + EXT_SOURCE + "::" + EXT_DATABASE + "::" + EXT_TABLE;
    private static final String EXT_ORDER_ID_COLUMN_ID =
            "custom_" + EXT_PREFIX + "-column:" + EXT_SOURCE + "::" + EXT_DATABASE + "::" + EXT_TABLE + ":order_id";
    private static final String EXT_AMOUNT_COLUMN_ID =
            "custom_" + EXT_PREFIX + "-column:" + EXT_SOURCE + "::" + EXT_DATABASE + "::" + EXT_TABLE + ":amount";

    public static void main(String[] args) throws Exception {
        Client client = new Client(new Config()
                .setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
                .setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"))
                .setRegionId(REGION_ID));

        if (System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID") == null
                || System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET") == null) {
            throw new IllegalArgumentException(
                    "Please set ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET first");
        }

        String reportId = registerBiReport(client);
        registerExternalTable(client);
        createLineage(client, UPSTREAM_TABLE_ID, UPSTREAM_TABLE_NAME,
                reportId, REPORT_ENTITY_NAME, "dw_table_to_bi_report", "custom-table_to_report");
        createLineage(client, UPSTREAM_TABLE_ID, UPSTREAM_TABLE_NAME,
                EXT_TABLE_ID, EXT_TABLE, "dw_table_to_external_table", "custom-table_to_table");
        createLineage(client, EXT_ORDER_ID_COLUMN_ID, "order_id",
                EXT_AMOUNT_COLUMN_ID, "amount", "ext_col_to_ext_col", "custom-column_to_column");

        System.out.println("Custom entities, extended table, and lineage registered.");
    }

    private static String registerBiReport(Client client) throws Exception {
        ensureMetaEntityDef(client, REPORT_ENTITY_TYPE, new CreateMetaEntityDefRequest()
                .setName("bi_report")
                .setDisplayName("BI report")
                .setDescription("Enterprise BI report asset")
                .setExtend("NONE")
                .setAttributeDefs(Arrays.asList(
                        new MetaEntityAttributeDef()
                                .setName("reportCode").setDisplayName("Report code").setType("STRING").setIsOptional(false),
                        new MetaEntityAttributeDef()
                                .setName("reportUrl").setDisplayName("Report URL").setType("STRING").setIsOptional(true)
                )));

        ensureCustomAttribute(client, "custom-attribute:sourceSystem", new CreateCustomAttributeRequest()
                .setId("custom-attribute:sourceSystem")
                .setDisplayName("Source system")
                .setType("ENUM")
                .setValueEnums(Arrays.asList("BI", "CRM", "ERP"))
                .setEntityTypes(Collections.singletonList(REPORT_ENTITY_TYPE))
                .setDisplayEnabled(true)
                .setSearchFilterEnabled(true));

        Map<String, String> attrs = new HashMap<String, String>();
        attrs.put("reportCode", REPORT_ENTITY_NAME);
        attrs.put("reportUrl", "https://bi.example.com/reports/daily_sales");

        Map<String, List<String>> customAttrs = new HashMap<String, List<String>>();
        customAttrs.put("sourceSystem", Collections.singletonList("BI"));

        String reportId = createOneEntity(client,
                new BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities()
                        .setEntityType(REPORT_ENTITY_TYPE)
                        .setName(REPORT_ENTITY_NAME)
                        .setComment("Daily sales report")
                        .setAttributes(attrs)
                        .setCustomAttributes(customAttrs));
        Thread.sleep(1000L); // After successful creation, the query side may have about 1s of visibility lag.
        return reportId;
    }

    private static void registerExternalTable(Client client) throws Exception {
        ensureMetaEntityDef(client, "custom_" + EXT_PREFIX + "-table", new CreateMetaEntityDefRequest()
                .setName(EXT_PREFIX + "-table")
                .setDisplayName("External warehouse table")
                .setDescription("External tables not collected by DataWorks automatically")
                .setExtend("TABLE"));

        Map<String, String> dbAttrs = new HashMap<String, String>();
        dbAttrs.put("parentMetaEntityId", "custom_" + EXT_PREFIX + ":" + EXT_SOURCE);
        dbAttrs.put("technicalMetadata.location", "external://" + EXT_DATABASE);
        createOneEntity(client, new BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities()
                .setEntityType("custom_" + EXT_PREFIX + "-database")
                .setName(EXT_DATABASE)
                .setComment("External analytics database")
                .setAttributes(dbAttrs));

        Map<String, String> tableAttrs = new HashMap<String, String>();
        tableAttrs.put("parentMetaEntityId", "custom_" + EXT_PREFIX + "-database:" + EXT_SOURCE + "::" + EXT_DATABASE);
        tableAttrs.put("tableType", "TABLE");
        tableAttrs.put("columns", "["
                + "{\"name\":\"order_id\",\"type\":\"STRING\",\"comment\":\"Order ID\",\"position\":1},"
                + "{\"name\":\"amount\",\"type\":\"DECIMAL\",\"comment\":\"Sales amount\",\"position\":2}"
                + "]");
        createOneEntity(client, new BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities()
                .setEntityType("custom_" + EXT_PREFIX + "-table")
                .setName(EXT_TABLE)
                .setComment("External sales summary table")
                .setAttributes(tableAttrs));
        Thread.sleep(1000L);
    }

    private static String createOneEntity(
            Client client,
            BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities entity) throws Exception {
        BatchCreateMetaEntitiesResponse response = client.batchCreateMetaEntities(
                new BatchCreateMetaEntitiesRequest().setEntities(Collections.singletonList(entity)));
        if (response.getBody() == null || !Boolean.TRUE.equals(response.getBody().getSuccess())) {
            throw new RuntimeException("BatchCreateMetaEntities request failed");
        }
        MetaEntityWriteResult result = response.getBody().getResults().get(0);
        if (!Boolean.TRUE.equals(result.getSuccess())) {
            if (isAlreadyExists(result.getErrorMessage())) {
                String existingId = buildExpectedEntityId(entity);
                System.out.println("Entity already exists, reusing: " + existingId);
                return existingId;
            }
            throw new RuntimeException("Entity creation failed: " + result.getErrorMessage());
        }
        System.out.println("Entity created: " + result.getId());
        return result.getId();
    }

    private static String buildExpectedEntityId(
            BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities entity) {
        String entityType = entity.getEntityType();
        String name = entity.getName();
        if (entityType.startsWith("custom_entity-")) {
            return entityType + ":" + name;
        }

        Map<String, String> attributes = entity.getAttributes();
        String parentMetaEntityId = attributes == null ? null : attributes.get("parentMetaEntityId");
        if (entityType.endsWith("-database") && parentMetaEntityId != null) {
            String metaSource = parentMetaEntityId.substring(parentMetaEntityId.indexOf(':') + 1);
            return entityType + ":" + metaSource + "::" + name;
        }
        if (entityType.endsWith("-table") && parentMetaEntityId != null) {
            String parent = parentMetaEntityId.substring(parentMetaEntityId.indexOf(':') + 1);
            return entityType + ":" + parent + "::" + name;
        }

        throw new IllegalArgumentException("Cannot derive existing entity ID from type and attributes: " + entityType + "/" + name);
    }

    private static void createLineage(
            Client client,
            String srcId,
            String srcName,
            String dstId,
            String dstName,
            String taskId,
            String taskType) throws Exception {
        client.createLineageRelationship(new CreateLineageRelationshipRequest()
                .setSrcEntity(new LineageEntity().setId(srcId).setName(srcName))
                .setDstEntity(new LineageEntity().setId(dstId).setName(dstName))
                .setTask(new LineageTask().setId(taskId).setType(taskType)));
        System.out.println("Lineage created: " + srcId + " -> " + dstId);
    }

    private static void ensureMetaEntityDef(
            Client client,
            String entityType,
            CreateMetaEntityDefRequest request) throws Exception {
        if (metaEntityDefExists(client, entityType)) {
            System.out.println("Entity definition already exists, skipped: " + entityType);
            return;
        }
        client.createMetaEntityDef(request);
    }

    private static boolean metaEntityDefExists(Client client, String entityType) throws Exception {
        try {
            GetMetaEntityDefResponse response = client.getMetaEntityDef(
                    new GetMetaEntityDefRequest().setEntityType(entityType));
            return response.getBody() != null
                    && Boolean.TRUE.equals(response.getBody().getSuccess())
                    && response.getBody().getMetaEntityDef() != null;
        } catch (Exception e) {
            if (isNotFound(e)) {
                return false;
            }
            throw e;
        }
    }

    private static void ensureCustomAttribute(
            Client client,
            String id,
            CreateCustomAttributeRequest request) throws Exception {
        if (customAttributeExists(client, id)) {
            System.out.println("Custom attribute already exists, skipped: " + id);
            return;
        }
        client.createCustomAttribute(request);
    }

    private static boolean customAttributeExists(Client client, String id) throws Exception {
        try {
            GetCustomAttributeResponse response = client.getCustomAttribute(
                    new GetCustomAttributeRequest().setId(id));
            return response.getBody() != null
                    && Boolean.TRUE.equals(response.getBody().getSuccess())
                    && response.getBody().getCustomAttribute() != null;
        } catch (Exception e) {
            if (isNotFound(e)) {
                return false;
            }
            throw e;
        }
    }

    private static boolean isNotFound(Exception e) {
        String message = e.getMessage();
        if (message == null) {
            return false;
        }
        String lowerMessage = message.toLowerCase();
        return lowerMessage.contains("not found")
                || lowerMessage.contains("not exist")
                || lowerMessage.contains("not_exists");
    }

    private static boolean isAlreadyExists(String message) {
        if (message == null) {
            return false;
        }
        String lowerMessage = message.toLowerCase();
        return lowerMessage.contains("already exists") || lowerMessage.contains("already exist");
    }
}

3. Run the Demo

Configure the access credentials before running. The sample code reads the AccessKey pair from environment variables to avoid hardcoding the secret.

export ALIBABA_CLOUD_ACCESS_KEY_ID="your-access-key-id"
export ALIBABA_CLOUD_ACCESS_KEY_SECRET="your-access-key-secret"

Execute:

export DATAWORKS_SDK_VERSION="latest-version"
export JAVA_VERSION="8"
mvn -Ddataworks.sdk.version="${DATAWORKS_SDK_VERSION}" \
    -Djava.version="${JAVA_VERSION}" \
    compile exec:java

Notes:

• The Demo follows the creation flow. Before you run it, change the entity types, entity names, and external database/table names at the top of the class to identifiers that are not yet used under your account.

• BatchCreateMetaEntities requires checking both the outer Success and each Results[].Success. The createOneEntity helper in the sample already covers these checks.

• After an entity is created successfully, the query side may have about 1 second of visibility lag. The sample adds a brief wait after creation. In production, replace it with a bounded retry.

4. Core code snippets for batch onboarding

The Demo above runs through the end-to-end flow. For production batch onboarding, prepare your entity object list and lineage relationship list as described in the Data preparation templates section, and convert them into List<EntityRow> and List<LineageRow> as input to the batch methods.

Here, EntityRow maps to the entity object list and LineageRow maps to the lineage relationship list. If your list comes from CSV, Excel, or a database, first parse attributes_json, custom_attributes_json, and task_attributes_json into Map objects, then populate the row objects. The snippets below are core fragments that you can embed in the Demo project. They are not a complete CSV/Excel reader; implement file reading and JSON parsing according to your internal engineering standards.

When you add the following code to the CustomMetadataLineageDemo class, also add these imports:

import java.util.ArrayList;
import java.util.LinkedHashMap;

Batch-create entities:

private static BatchWriteResult batchCreateEntities(Client client, List<EntityRow> entityRows) throws Exception {
    BatchWriteResult report = new BatchWriteResult();

    // BatchCreateMetaEntities requires each batch to share the same EntityType, so group by EntityType first.
    Map<String, List<EntityRow>> grouped = new LinkedHashMap<String, List<EntityRow>>();
    for (EntityRow row : entityRows) {
        if (!grouped.containsKey(row.entityType)) {
            grouped.put(row.entityType, new ArrayList<EntityRow>());
        }
        grouped.get(row.entityType).add(row);
    }

    for (Map.Entry<String, List<EntityRow>> entry : grouped.entrySet()) {
        List<EntityRow> sameTypeRows = entry.getValue();
        for (int from = 0; from < sameTypeRows.size(); from += 5) {
            int to = Math.min(from + 5, sameTypeRows.size());
            List<EntityRow> rowBatch = sameTypeRows.subList(from, to);

            List<BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities> requestEntities =
                    new ArrayList<BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities>();
            for (EntityRow row : rowBatch) {
                requestEntities.add(row.toRequestEntity());
            }

            BatchCreateMetaEntitiesResponse response = client.batchCreateMetaEntities(
                    new BatchCreateMetaEntitiesRequest().setEntities(requestEntities));
            if (response.getBody() == null || !Boolean.TRUE.equals(response.getBody().getSuccess())) {
                for (EntityRow row : rowBatch) {
                    report.failed.add("entity=" + row.name + ", reason=batch request failed");
                }
                continue;
            }

            List<MetaEntityWriteResult> results = response.getBody().getResults();
            if (results == null || results.size() != rowBatch.size()) {
                for (EntityRow row : rowBatch) {
                    report.failed.add("entity=" + row.name + ", reason=unexpected batch result");
                }
                continue;
            }
            for (int i = 0; i < results.size(); i++) {
                MetaEntityWriteResult result = results.get(i);
                if (Boolean.TRUE.equals(result.getSuccess())) {
                    report.successIds.add(result.getId());
                } else {
                    EntityRow row = rowBatch.get(i);
                    report.failed.add("entity=" + row.name + ", reason=" + result.getErrorMessage());
                }
            }
        }
    }

    return report;
}

Batch-register lineage relationships:

private static BatchWriteResult batchCreateLineages(Client client, List<LineageRow> lineages) {
    BatchWriteResult report = new BatchWriteResult();
    for (LineageRow row : lineages) {
        boolean success = false;
        String lastError = null;

        for (int attempt = 1; attempt <= 3; attempt++) {
            try {
                LineageTask task = new LineageTask().setId(row.taskId).setType(row.taskType);
                if (row.taskAttributes != null && !row.taskAttributes.isEmpty()) {
                    task.setAttributes(row.taskAttributes);
                }

                client.createLineageRelationship(new CreateLineageRelationshipRequest()
                        .setSrcEntity(new LineageEntity().setId(row.srcId).setName(row.srcName))
                        .setDstEntity(new LineageEntity().setId(row.dstId).setName(row.dstName))
                        .setTask(task));
                report.successIds.add(row.srcId + " -> " + row.dstId);
                success = true;
                break;
            } catch (Exception e) {
                lastError = e.getMessage();
                try {
                    Thread.sleep(1000L * attempt);
                } catch (InterruptedException interrupted) {
                    Thread.currentThread().interrupt();
                    lastError = interrupted.getMessage();
                    break;
                }
            }
        }

        if (!success) {
            report.failed.add("lineage=" + row.srcId + " -> " + row.dstId + ", reason=" + lastError);
        }
    }
    return report;
}

Helper classes:

private static class EntityRow {
    // Maps to entity_type, name, comment in the entity object list.
    private String entityType;
    private String name;
    private String comment;

    // Maps to attributes_json and custom_attributes_json.
    private Map<String, String> attributes = new HashMap<String, String>();
    private Map<String, List<String>> customAttributes = new HashMap<String, List<String>>();

    // Maps to parent_entity_id and columns_json, mainly used for extended Database/Table.
    private String parentEntityId;
    private String columnsJson;

    private BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities toRequestEntity() {
        Map<String, String> mergedAttributes = new HashMap<String, String>();
        if (attributes != null) {
            mergedAttributes.putAll(attributes);
        }
        if (parentEntityId != null && parentEntityId.length() > 0) {
            mergedAttributes.put("parentMetaEntityId", parentEntityId);
        }
        if (columnsJson != null && columnsJson.length() > 0) {
            mergedAttributes.put("columns", columnsJson);
        }

        return new BatchCreateMetaEntitiesRequest.BatchCreateMetaEntitiesRequestEntities()
                .setEntityType(entityType)
                .setName(name)
                .setComment(comment)
                .setAttributes(mergedAttributes)
                .setCustomAttributes(customAttributes);
    }
}

private static class BatchWriteResult {
    private final List<String> successIds = new ArrayList<String>();
    private final List<String> failed = new ArrayList<String>();
}

private static class LineageRow {
    private String srcId;
    private String srcName;
    private String dstId;
    private String dstName;
    private String taskId;
    private String taskType;
    private Map<String, String> taskAttributes = new HashMap<String, String>();
}

In production, persist BatchWriteResult.failed to a file, database, or message queue as input for resumable runs and manual fix-ups. Do not retry every failure indiscriminately. Issues such as parameter errors, missing entity types, or missing lineage endpoints require fixing the data list first, then rerunning.

Prefer registering objects as DataWorks custom entity objects

Non-onboarded custom-{Type}:{Identifier} objects can participate in lineage registration directly, but they are harder to manage afterwards. For new onboarding, we recommend that you:

1. Create an entity definition with CreateMetaEntityDef.

2. Create entity objects with BatchCreateMetaEntities.

3. Use the returned entity ID for lineage registration.

This lets you view entity details, maintain attributes, reuse custom attributes, and support subsequent governance in Data Map.

Distinguish custom entities from extended table entities

If an object is a business concept such as a report, API, metric, system, or data product, register it as a custom entity. If an object has a well-defined database/table/column structure and DataWorks does not collect it automatically, register it as an extended table entity.

When you register an extended table entity, also create the corresponding extended Database object first, then create the Table and inline Columns. This preserves the original database/table/column hierarchy of the source data and produces an asset view in Data Map that is closer to the real data structure.

Confirm parameters on the API reference page

The examples above illustrate API composition and the onboarding flow. For field mandatoriness, naming rules, enum values, error codes, and SDK code, see the corresponding API reference pages.

Pattern 1: Onboard BI reports to Data Map

Suitable when you want to view which tables a report depends on, or which reports are affected when a table changes. Recommended steps:

1. Use CreateMetaEntityDef to create a report entity definition.

2. Use CreateCustomAttribute to create attributes such as business domain, owner, and source system.

3. Use BatchCreateMetaEntities to batch-write the report entities.

4. Use ListTables and ListColumns to obtain the DataWorks-collected table and column IDs.

5. Use CreateLineageRelationship to register table-to-report and column-to-column lineage.

6. Use ListLineages to verify the upstream and downstream chains.

Pattern 2: Onboard external database tables to Data Map

Suitable when your enterprise has third-party databases or external system tables that DataWorks does not collect automatically, but you want to include them in a unified asset view. Recommended steps:

1. Use CreateMetaEntityDef to create the extended table definition.

2. Use BatchCreateMetaEntities to write the external database first, then write the external table together with its inline columns.

3. Use CreateCustomAttribute to add business attributes such as source system and data domain.

4. Use CreateLineageRelationship to establish lineage between DataWorks tables and external tables.

5. Use GetTable, GetColumn, and ListLineages to verify the result.

Pattern 3: Add lineage between DataWorks-collected assets

Suitable when your enterprise has external pipelines or legacy pipelines that you want to register in Data Map. Recommended steps:

1. Use APIs such as ListTables, ListColumns, and ListDatasets to obtain entity IDs.

2. Call CreateLineageRelationship according to the actual upstream-to-downstream direction.

3. Use ListLineages to verify.

In this pattern, you do not need to create custom entity definitions or write entity objects.