Alibaba Cloud DLF data source
This guide explains how to connect ApsaraDB for SelectDB to an Alibaba Cloud Data Lake Formation data source to perform federated analytics.
Overview
Alibaba Cloud Data Lake Formation (DLF) is a unified metadata management service on Alibaba Cloud that is compatible with the Hive Metastore protocol. You can use ApsaraDB for SelectDB to connect to and access DLF as you would a Hive Metastore.
Connect to DLF
Create a basic DLF catalog
CREATE CATALOG dlf PROPERTIES (
"type"="hms",
"hive.metastore.type" = "dlf",
"dlf.proxy.mode" = "DLF_ONLY",
"dlf.endpoint" = "dlf-vpc.cn-beijing.aliyuncs.com",
"dlf.region" = "cn-beijing",
"dlf.uid" = "uid",
"dlf.access_key" = "ak",
"dlf.secret_key" = "sk"
);The parameters "type"="hms", "hive.metastore.type" = "dlf", and "dlf.proxy.mode" = "DLF_ONLY" are fixed and cannot be modified.
The following table describes the other key parameters.
|
Parameter |
Description |
|
dlf.endpoint |
The endpoint of the DLF service. For more information, see Data Lake Formation: Supported regions and endpoints. |
|
dlf.region |
The region where the DLF service is deployed. For more information, see Data Lake Formation: Supported regions and endpoints. |
|
dlf.uid |
The ID of your Alibaba Cloud account. You can find the Account ID in the user profile section in the upper-right corner of the Alibaba Cloud console. |
|
dlf.access_key |
The AccessKey ID. For more information, see Create an AccessKey pair. |
|
dlf.secret_key |
The AccessKey secret. For more information, see Create an AccessKey pair. |
|
"dlf.access.public"="true" |
Enables public internet access to data in Object Storage Service (OSS). |
After the catalog is configured, you can access metadata in DLF as you would with a Hive Metastore.
Ensure that the Alibaba Cloud account used to create the catalog has permissions to access DLF. For more information, see Permission description.
Data type mapping
The data type mapping between ApsaraDB for SelectDB and DLF is the same as the mapping for a Hive data source.