Apache Gravitino metadata collection

Prerequisites

• An Apache Gravitino data source is created and passes the connectivity test. For more information, see Apache Gravitino data source.

  Note

  Apache Gravitino serves only as a metadata collection channel in this scenario. After collection, the metadata presented in DataWorks is still from the underlying catalogs, such as Iceberg Catalog, Paimon Catalog, and Hive Catalog.

• A serverless resource group is created and associated with the workspace.

Limits

• Only Relational catalogs are supported for collection, such as Hive, Iceberg, Paimon, and JDBC catalogs. Model, File Set, and Messaging catalog types are not supported.

• Partition information is dynamically retrieved in real time and is not statically stored. Each time you view partitions, the data is fetched on demand.

Create a metadata collection task

1. Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.

2. On the Data Map page, click Metadata Collection in the top navigation bar to go to the metadata collection page.

3. On the metadata collection page, click Create Crawler to go to the Create Metadata Collection page.

4. In the Basic Configuration section, configure the following parameters:

   Parameter	Description
   Collection Type	Select Apache Gravitino.
   Configuration Source	The default value is Data Source.
   Select Workspace	Select the workspace to which the data source belongs, or select All Workspaces to view all available data sources.
   Select Data Source	Select the Apache Gravitino data source that you created. After the selection, the data source type, name, workspace, environment type, and connection information are displayed below.
   Name	The custom name of the metadata collection task, such as gravitino_test.

5. In the Resource Group Configuration section, configure the following parameters:

   Parameter	Description
   Resource Group	Select the serverless resource group for running the collection task. After the selection, the name, identifier, type, and status of the resource group are displayed.
   Test Connectivity	Click Test Connectivity to verify the network connectivity between the resource group and the data source. The connectivity test must pass before you can save the collection configuration.

6. (Optional) In the AI Enhancement Configuration section, select whether to enable AI Description Enhancement.

   After this feature is enabled, the system uses the intelligent analysis capabilities of the DataWorks data agent large language model to automatically generate description recommendations for tables and columns after metadata collection is complete. Asset owners can view the AI-generated descriptions on the table details page and choose to confirm or manually modify the descriptions to improve the accuracy and standardization of metadata management.

7. In the Collection Plan section, configure the following parameters:

   Parameter	Description
   Trigger Method	Select the trigger method for collection: Manual: The collection can only be triggered manually. Periodic: The collection task is automatically run based on the configured schedule.
   Collection Frequency	If you set Trigger Method to Periodic, specify the collection frequency. Valid values: Monthly, Weekly, Daily, and Hourly. If you select Hourly, you can further select specific minutes from the Minutes drop-down list. Multiple values can be selected, with a minimum granularity of 5 minutes (for example, 0, 5, 10, and 15).

8. After the configuration is complete, click Save and Run at the bottom of the page to save the configuration and immediately trigger a collection task. You can also click Save to save the configuration only and manually trigger the collection later.

View database details

After the collection task runs successfully, you can view the databases and tables covered by the collection on the Data Scope tab of the crawler details page.

1. In the Metadata Collection list, click the name of the target crawler to go to the crawler details page.

2. Click the Data Scope tab to view the list of collected databases.

The Data Scope page displays the following information:

After you click View Details, the database details page is displayed. The left side shows the basic information of the database, and the right side shows the table list.

The Basic Information section includes:

• Total Tables: The total number of tables collected in the database.

• Database: The database name.

• Entity ID: The unique identifier of the database.

• Technical Type: The technical type of the database, such as Apache Hive, Iceberg, or Paimon.

• Region: The region where the data source is located.

The Table List displays the following information:

View table details

In the table list on the database details page, click a table name to go to the table details page. The left side displays the basic information, technical information, and business information of the table. The right side displays column details and usage notes.