数据湖元数据管理

前提条件

• 已在数据湖构建（Data Lake Formation）控制台开通数据湖构建。

• 进入RAM访问控制台给AliyunDDIAccessingOSSRole角色添加一个AliyunDDIAccessingDLFRolePolicy自定义策略，策略详情如下：

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "dlf:BatchCreatePartitions",
                "dlf:BatchCreateTables",
                "dlf:BatchDeletePartitions",
                "dlf:BatchDeleteTables",
                "dlf:BatchGetPartitions",
                "dlf:BatchGetTables",
                "dlf:BatchUpdatePartitions",
                "dlf:BatchUpdateTables",
                "dlf:CreateDatabase",
                "dlf:CreateFunction",
                "dlf:CreatePartition",
                "dlf:CreateTable",
                "dlf:DeleteDatabase",
                "dlf:DeleteFunction",
                "dlf:DeletePartition",
                "dlf:DeleteTable",
                "dlf:GetDatabase",
                "dlf:GetFunction",
                "dlf:GetPartition",
                "dlf:GetTable",
                "dlf:ListCatalogs",
                "dlf:ListDatabases",
                "dlf:ListFunctionNames",
                "dlf:ListFunctions",
                "dlf:ListPartitionNames",
                "dlf:ListPartitions",
                "dlf:ListPartitionsByExpr",
                "dlf:ListPartitionsByFilter",
                "dlf:ListTableNames",
                "dlf:ListTables",
                "dlf:RenamePartition",
                "dlf:RenameTable",
                "dlf:UpdateDatabase",
                "dlf:UpdateFunction",
                "dlf:UpdateTable",
                "dlf:UpdateTableColumnStatistics",
                "dlf:GetTableColumnStatistics",
                "dlf:DeleteTableColumnStatistics",
                "dlf:UpdatePartitionColumnStatistics",
                "dlf:GetPartitionColumnStatistics",
                "dlf:DeletePartitionColumnStatistics",
                "dlf:BatchGetPartitionColumnStatistics"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

背景信息

数据湖元数据已适配Databricks 数据洞察的Spark SQL。

适用场景

数据湖元数据具有高可用和易维护的特点，因此适合在如下场景下使用数据湖元数据：

• Databricks 数据洞察集群的生产环境，您无需维护独立的元数据库。

• 横向使用多种大数据计算引擎，例如Databricks 数据洞察、MaxCompute、EMR等，元数据可以集中管理。

• 多个Databricks 数据洞察集群，可以统一管理元数据。

创建集群

创建Databricks 数据洞察集群时，如图元数据选择为数据湖元数据方式，创建详情请参见创建集群。