DocsICP FilingConsole
Documentation
Home Page MaxCompute User Guide Data Lake Analytics (Lakehouse) Data lakehouse solution 1.0 (Deprecated) Lakehouse 1.0 Appendix Manage external data sources

Manage external data sources

Updated at:
Copy as MD
Product
My Favorites

External data sources connect your MaxCompute projects to existing Hadoop clusters—both E-MapReduce (EMR) and self-managed—without migrating data. Query Hadoop data directly from MaxCompute to build a lakehouse architecture.

With external data sources, you can:

  • Map Hadoop clusters to external MaxCompute projects

  • Query data from a single Hadoop cluster or join data across multiple Hadoop clusters

  • Access Hive-managed tables through the Hive Metastore Service (HMS)

  • Use Kerberos-secured Hadoop clusters with account-level identity mapping

Prerequisites

Before you begin, ensure you have:

  • A MaxCompute project in a supported region

  • A network connection from MaxCompute to the virtual private cloud (VPC) where the Hadoop cluster is deployed. For setup details, see VPC connection scheme

  • The NameNode IP addresses, port numbers, and HMS addresses for your Hadoop cluster

  • (For Kerberos clusters) The krb5.conf file and a keytab file for each mapped account

Usage notes

  • Supported regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Zhangjiakou), and Singapore (Singapore).

  • One-to-one binding: Each external data source can be bound to only one external MaxCompute project.

  • No updates after creation: You cannot update an external data source after it is created. Delete and recreate it if you need to change the configuration.

Create an external data source

  1. Log on to the MaxCompute控制台MaxCompute控制台 and select a region.

  2. On the Manage External Data Sources tab, click Create External Data Source.

  3. In the Create External Data Source dialog box, configure the parameters described in the following table, and then click OK. No Authentication Select No Authentication if Kerberos is disabled for the Hadoop cluster. No additional configuration is required. Kerberos Authentication Select Kerberos Authentication if Kerberos is enabled for the Hadoop cluster, then configure the following fields:

    Parameter Description
    Select MaxCompute Project The MaxCompute project to associate with this external data source. Select from the drop-down list. To find the project name, go to the Project management tab.
    External Data Source Name A custom name for the external data source. The name can contain only lowercase letters, digits, and underscores (_), and must be fewer than 128 characters.
    Network Connection Object The network connection from MaxCompute to the VPC where the Hadoop cluster is deployed. For supported connection types, see VPC connection scheme.
    NameNode Address The IP addresses and port numbers of the active and standby NameNode processes. The default port is 8020. Contact your Hadoop cluster administrator for the exact values.
    HMS Service Address The IP addresses and port numbers of the active and standby HMS processes. The default port is 9083. Contact your Hadoop cluster administrator for the exact values.
    Cluster Name The name of the Hadoop cluster. For a high availability (HA) cluster, this matches the NameNode process name. For a self-managed cluster, get the value from the dfs.nameservices parameter in hdfs-site.xml.
    Authentication Type The authentication mechanism for the Hadoop cluster. MaxCompute uses account mappings to access metadata and data from the cluster. Select No Authentication or Kerberos Authentication to match your cluster configuration.
    Field Description
    Configuration File Click Upload KRB5.conf File to upload the krb5.conf file for the Hadoop cluster. On Linux, this file is located in the /etc directory on the master node (HDFS NameNode processes).
    hmsPrincipals The Kerberos principals of the HMS service. Run the list_principals command on the Kerberos terminal of the Hadoop cluster to get these values. Enter as a comma-delimited string with one principal per HMS IP address. Example: hive/emr-header-1.cluster-20**@EMR.20.COM,hive/emr-header-2.cluster-20@EMR.20**.COM
    Account The Alibaba Cloud account that accesses the Hadoop cluster through MaxCompute. Click Add Account Mapping to add one or more mappings.
    Kerberos Account The Kerberos-authenticated Hadoop user account allowed to access the HMS service.
    Upload File Click Upload Keytab File to upload the keytab file for the Kerberos account. For instructions on creating a keytab file, see Create a keytab configuration file.

View or delete an external data source

  1. Log on to the and select a region.

  2. On the Manage External Data Sources tab, find the external data source you want to manage, then click Details or Delete in the Actions column.

    Before deleting an external data source that is bound to an external project, you must first delete the external project or unbind the external data source from it.

What's next

  • To query data from the connected Hadoop cluster, create an external MaxCompute project and bind it to this external data source.

  • To add a new Kerberos-authenticated Hadoop account, see Create a keytab configuration file.

Previous：AppendixNext：Authorize MaxCompute to access DLF