How to create a collector to gather metadata from various data sources into DataWorks-DataWorks(DataWorks)-阿里云帮助中心

DataWorks Data Map provides the Metadata Collection feature, which allows you to centrally consolidate and manage metadata from various DataWorks data sources. In Data Map, you can view the metadata aggregated from all your sources. This topic describes how to create a crawler that collects and consolidates metadata from various data sources into DataWorks.

Overview

Metadata collection is a core feature for building an enterprise-grade Data Map and achieving unified data asset management. A crawler automatically extracts technical metadata (such as databases, tables, and columns), data lineage, and partition information from DataWorks data sources (such as MaxCompute, Hologres, MySQL, and CDH Hive) that are scattered across different workspaces within the same region, and consolidates this information into DataWorks Data Map to create a unified data view.

Metadata collection allows you to:

Build a unified data view: Break down data silos by centrally managing metadata from multiple, heterogeneous sources.
Enable data discovery and search: Allow data consumers to quickly and accurately find the data they need.
Achieve end-to-end data lineage analysis: Clearly trace the origin and flow of data to facilitate impact analysis and troubleshooting.
Empower data governance: Implement Data Classification, access control, quality monitoring, and lifecycle management based on comprehensive metadata.

Billing

By default, each collection task consumes 0.25 CU multiplied by the task run time, incurring compute unit fees. Each successful collection generates a scheduling instance, incurring scheduling fees.

Limitations

If a data source uses an allowlist for access control, you must configure it beforehand. For more information, see Metadata collection allowlist.
The DataWorks deployment and the data source must be in the same region. If you must collect metadata across regions, use a public endpoint when you create the data source. For more information, see Create a data source.
Metadata collection is not supported for AnalyticDB for MySQL data sources that have SSL enabled.

Entry point

Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.
In the left-side navigation pane, click to go to the metadata collection page.

Built-in crawlers

Built-in crawlers are preconfigured and automatically executed (near real-time) by the DataWorks platform. They are used to collect core metadata that is deeply integrated with DataWorks. You do not need to create them. You only need to manage the collection scope.

Important

If you cannot find the target table in Data Map, go to My Data > My Tools > Refresh Table Metadata and manually synchronize the relevant tables.

MaxCompute default crawler

This crawler collects metadata from MaxCompute projects under your account. You can go to the details page, click Modify Data Scope to select the projects to collect, and click Permission Configurations to configure the visibility of metadata within the tenant.

In the Built-in section on the metadata collection page, find the MaxCompute Default Crawler card and click Details.
The MaxCompute Default Crawler details page contains two tabs: Basic Information and Data Scope.
- Basic Information: Displays the basic properties of the crawler, such as the collection type and method. This information is read-only.
- Data Scope: Manages which MaxCompute projects this crawler collects metadata from.
Modify the collection scope:
1. Switch to the Data Scope tab and click the Modify Data Scope button.
2. In the dialog that appears, select or clear the MaxCompute projects that you want to collect metadata from.
  Important
  By default, the scope includes all MaxCompute projects that are associated with workspaces in the current region under your tenant. After you modify the data scope, the metadata objects collected in Data Map are consistent with the current data scope. Metadata of cleared projects is not visible.
3. Click OK to save the changes.
Configure metadata visibility:
- In the Data Scope list, find the target project and click Permission Configurations in the Actions column.
- Select a visibility policy based on your data governance requirements:
  - Public Within Tenant: All members within the tenant can search for and view the metadata of this project.
  - Only members in the associated workspace can search and view.: Only members of specific workspaces can access the metadata of this project, ensuring data isolation.

DLF default crawler

Important

To enable real-time collection of DLF metadata, you must grant the Data Reader permission to the service-linked role AliyunServiceRoleForDataworksOnEmr in the DLF console.

The DLF Default Crawler collects metadata from Data Lake Formation (DLF) under your account.

In the Built-in section on the metadata collection page, find the DLF Default Crawler card and click Details to view basic information.
Switch to the Data Scope tab to view the list of DLF Catalogs that are included in the collection scope and the number of tables they contain.
By default, all accessible catalogs (including DLF and DLF-Legacy) are collected.

Custom crawlers

Custom crawlers allow you to centrally manage metadata across environments and engines.

For regular data sources
You can create custom crawlers for traditional structured or semi-structured data sources such as Hologres, StarRocks, MySQL, Oracle, and CDH Hive. By configuring collection tasks, the system performs deep parsing of physical database and table structures at the source to achieve automated extraction and synchronization of metadata such as column attributes, indexes, and partitions.
For metadata-type data sources (catalog)
For non-DLF-managed, self-declared native lake-format metadata such as Paimon Catalog and other metadata-type data sources, you can also create crawlers for direct collection.

Create Custom Crawler

In the custom crawler list section on the metadata collection page, click Create Metadata Crawler.
Select a collection type: On the type selection page, select the target data source type to collect, such as Hologres or StarRocks.
Configure basic settings and resource group:
- Basic Configurations:
  - Select a workspace: Select the workspace that contains the data source.
  - Select Data Source: Select a previously created target data source from the drop-down list. After you make a selection, the system automatically displays the details of the data source.
  - Name: Name the crawler for easy identification later. By default, the name is the same as the data source name.
- Resource Group Configuration:
  - Resource Group: Select a resource group to run the collection task.
  - Test Network Connectivity: This step is critical. Click Test Network Connectivity to make sure that the resource group can access the data source.
    Important
    Check whether the data source has allowlist restrictions enabled. If you need to collect metadata from a data source with allowlist-based access control, see Add resource group IP addresses to the allowlist and Configure an allowlist for a data source to configure allowlist permissions.
    If the data source does not have allowlist restrictions enabled, see Configure network connectivity for a data source to establish network connectivity for the data source.
    If the connectivity test returns the error Backend service call failed: test connectivity failed.not support data type, contact technical support to upgrade the resource group.
Configure metadata collection:
- Collection Scope: Define the databases (database/schema) to collect. If the data source is at the database granularity, the database corresponding to the data source is selected by default. You can select additional databases beyond the data source.
  Important
  - Each database can be configured in only one crawler. If a database is grayed out, it is already being collected by another crawler.
  - After you reduce the collection scope, metadata outside the scope is no longer searchable in Data Map.
Configure intelligent enhancement and collection plan:
- Intelligent enhancement (Beta):
  - AI-generated descriptions: When enabled, the system uses large language model capabilities to automatically generate business descriptions for your tables and columns after metadata is collected, greatly improving metadata readability and usability. After collection is complete, you can view the AI-generated information (such as table descriptions and column descriptions) on the details page of a table object in Data Map.
- Collection Plan:
  - Trigger Mode: Select manual or periodic.
    - Manual: The crawler runs only when you manually trigger it. This is suitable for one-time or on-demand collection.
    - Periodic: Configure a scheduled task (such as monthly, daily, weekly, or hourly), and the system automatically updates metadata on a periodic basis.
      To configure a minute-level scheduled task, select the hourly collection cycle and select all minute-level granularities to implement a scheduled task that runs every 5 minutes.
      Important
      Only data sources in the production environment support periodic collection.
Save the configuration: Click Save or Save and Run to complete the crawler creation.

Manage custom crawlers

After a crawler is created, it appears in the custom crawler list. You can perform the following management operations:

List operations: In the list, you can directly Run, Stop, or Delete a crawler. Use the Filter and Search features at the top to quickly locate the target crawler.
Important
After you delete a metadata crawler, the metadata objects collected by this crawler in Data Map become invalid. You can no longer search for or view objects and their details from this crawler. Proceed with caution.
Batch operations: Select multiple crawlers in the crawler list and use Batch Run or Batch Stop at the bottom of the list to trigger or terminate multiple collection tasks at a time, improving management efficiency.
Crawler status: The crawler list displays the current status of each crawler. Common statuses include Not Run, Waiting, Running, Successful, Failed, and Manually Terminated.
Note
If the data source associated with a crawler has been disassociated or becomes invalid, the crawler enters the frozen state. A frozen crawler cannot be run and can only be deleted.
View details and logs: Click the name of the target crawler to go to its details page.
- Basic Information: View all configuration items of the crawler.
- Data Scope: View or Modify Data Scope.
  If no collection has been performed, the table count and last update time are empty.
  The following data sources do not support scope modification: EMR Hive, CDH Hive, Lindorm, ElasticSearch, OTS, MongoDB, and AnalyticDB for Spark in AnalyticDB MySQL.
- Run Logs: Track the execution history of each collection task. You can view the start time, duration, status, and volume of data collected for each task. When a task fails, clicking View Logs is the key entry point for locating and resolving issues.
Manually trigger collection: In the upper-right corner of the details page, click the Collect Metadata button to immediately trigger a collection task. This is useful when you want to see a newly created table in Data Map right away.

Next steps

After metadata is successfully collected, you can take full advantage of various Data Map capabilities:

Search for your collected tables in Data Map and view their details, column information, partitions, and data preview. For more information, see View table details.
Analyze the upstream and downstream lineage of a table to understand the end-to-end data processing flow. For more information, see Data lineage.
Add assets to a Data Collection to organize and manage your data from a business perspective. For more information, see Create a Data Collection.

FAQ

Q: Collection of MySQL or other database-type sources times out or fails?
A: Check whether you have added the resource group's vSwitch CIDR block to the allowlist. Verify the vSwitch CIDR Block of the resource group.

Appendix: Collection scope and timeliness

Data source

Data Source Type	Collection Mode	Collection granularity	Metadata update timeliness
Data Source Type	Collection Mode	Collection granularity	Table/Column	Partition	Lineage
MaxCompute	Automatically collected by the system by default	Instance	Regular projects: Real-time External projects: T+1	China mainland regions: Real-time International regions: T+1	Real-time
Data Lake Formation (DLF)	Automatically collected by the system by default	Instance	Real-time Important To enable real-time collection, you must grant the Data Reader permission to the service-linked role `AliyunServiceRoleForDataworksOnEmr` in the DLF console. For more information, see the DLF default crawler section above.	Real-time	Lineage is supported for DLF metadata of Serverless Spark, Serverless StarRocks, Serverless Flink, and EMR Impala engines. Other engines are not supported. Important For EMR clusters, you must enable EMR_HOOK. To display lineage for EMR Impala tasks, you must enable lineage logging in the Impala configuration of the EMR cluster. This is supported only for EMR DataLake clusters and is currently in the gray release phase. Contact Alibaba Cloud technical support to enable this feature before use. For configuration details, see Data lineage.
Hologres	Create a crawler manually	Database	Depends on the collection cycle	Not supported	Real-time
EMR Hive		Instance	Depends on the collection cycle	Depends on the collection cycle	Real-time Important You must enable EMR_HOOK for the cluster. To display lineage for EMR Impala tasks, you must enable lineage logging in the Impala configuration of the EMR cluster. This is supported only for EMR DataLake clusters and is currently in the gray release phase. Contact Alibaba Cloud technical support to enable this feature before use. For configuration details, see Data lineage.
CDH Hive		Instance	Depends on the collection cycle	Real-time	Real-time
StarRocks		Database	Instance mode: Real-time. Connection string mode: Depends on the collection cycle.	Not supported	Real-time Important Only instance mode supports lineage collection. Connection string mode does not support lineage collection.
AnalyticDB for MySQL		Database	Depends on the collection cycle	Not supported	Real-time Note You must submit a ticket to enable the data lineage feature for the AnalyticDB for MySQL instance.
AnalyticDB for Spark		Instance	Real-time	Not supported	Real-time
AnalyticDB for PostgreSQL		Database	Depends on the collection cycle	Not supported	Real-time
Lindorm		Instance	Depends on the collection cycle	Not supported	Real-time
OTS		Instance	Depends on the collection cycle	Not supported	Not supported
MongoDB		Instance	Depends on the collection cycle	Not supported	Not supported
ElasticSearch		Instance	Depends on the collection cycle	Not supported	T+1 update
Paimon Catalog		Catalog	Depends on the collection cycle	Depends on the collection cycle	Not supported
Other data source types (MySQL, PostgreSQL, SQL Server, Oracle, ClickHouse, SelectDB, OceanBase, etc.)		Database	Depends on the collection cycle	Not supported	Not supported

Note

"Depends on the collection cycle" in the table means that metadata is periodically updated based on the collection plan you configured for the crawler (such as monthly, daily, weekly, or hourly).
AnalyticDB for Spark and AnalyticDB for MySQL share the same metadata collection entry point.

Task code

Data Map supports task code search and quick navigation. The following describes the scope of searchable code.

Code source	Collection scope	Collection trigger method
Data Studio	Data Studio - Create a node and edit code	Automatic collection
Legacy Data Studio	Legacy Data Studio - Create a node and edit code
Data Analysis	Data Analysis - Create an SQL query and edit code
Data Service	Data Service - Create an API Data Push service

API assets

Data Map supports viewing metadata of Data Service APIs, as described below:

API Type	Collection scope	Collection trigger method
API generation (wizard mode)	Data Service - Create an API in wizard mode	Automatic collection
API generation (script mode)	Data Service - Create an API in script mode
Register API	Data Service - Register an API
API orchestration	Data Service - Create an API orchestration

AI assets

Data Map supports viewing and managing AI assets, and provides AI asset lineage to track the origin, usage, and evolution of data and models. The following describes the support for each AI asset type.

Asset type	Collection scope	Collection trigger method
Dataset	PAI - Create a dataset/Register a dataset DataWorks - Create a dataset	Automatic collection
AI model	PAI - Model training task/Register a model/Deploy a model service
Algorithm task	PAI - Training task/Workflow task/Distributed training task
Model service	PAI - Deploy a model service (EAS deployment)

Workspace

Data Map supports viewing workspace metadata, as described below:

Project	Collection Mode	Collection trigger method
Workspace	DataWorks - Create a workspace	Automatic collection