DocsICP FilingConsole
Documentation
Home Page Dataphin Intelligent Data construction and governance Dataphin-sharing mode (fully managed version) User Guide Asset governance Metadata Center Metadata collection Metadata Collection Overview

Metadata Collection Overview

Updated at:
Copy as MD
Product
My Favorites

Dataphin supports metadata collection from a wide range of sources. These include traditional databases like MySQL and Oracle, big data storage like Hive and Hologres, and application systems. You can view your collection tasks at a glance, including the number of tasks and object types for each data source or application system.

Prerequisites

  • To use an application system as a collection source, you must first create it in Admin Center > Data Source Management > Application System.

  • By default, Dataphin supports metadata collection from relational databases. To collect metadata from other types of data sources, you must purchase the necessary add-ons.

  • You must purchase the , Metadata Collection - Big Data Storage or add-on to collect metadata from the DataWorks application system.

Metadata collection workflow

If a data source is in a network isolated from the Dataphin cluster, you must use a registered scheduling cluster. The collection task writes the collected metadata to object storage, such as OSS, for staging. The process then writes the data to Dataphin. This process incurs additional storage costs.

Procedure

  1. On the Dataphin homepage, choose Governance > Metadata from the top navigation bar.

  2. In the left-side navigation pane, choose Metadata Collection > Collection Overview.

  3. On the Welcome to Metadata Collection and Management page, Dataphin displays cards with information about each data source or application system, such as the number of configured collection tasks and supported object types.

    • Data source: Dataphin supports a wide range of data source types, such as relational databases and big data databases. For more information, see Data sources supported by Dataphin.

      The supported versions for MySQL, Oracle, and Hive (MySQL metadatabase) are as follows:

      • MySQL: MySQL 5.1.43, MySQL 5.6/5.7, MySQL 8, and RDS MySQL.

      • Oracle: Oracle 11g, Oracle 12c, Oracle 18c, Oracle 19c, Oracle 21c, and Oracle 23c.

      • Hive (MySQL metadatabase): CDH 5.x Hive 1.1.0, EMR 3.x Hive 2.3.5, EMR 5.x 3.1.x, CDH 6.x Hive 2.1.1, FusionInsight 8.x Hive 3.1.0, CDP 7.x Hive 3.1.3, and AsiaInfo DP 5.x Hive 3.1.0.

    • Application system: Supported systems include Quick BI and DataWorks.

  4. You can create a collection task for a data source or application system.

    Create task: Hover over a card to create a collection task. For more information, see Create and manage a metadata collection task.

    Note

    You can configure only one collection task for each data source. However, you can configure separate collection tasks for a data source's development and production environments.

Previous：Metadata collectionNext：Create and Manage Metadata Collection Tasks