DocsICP FilingConsole
Documentation
Home Page MaxCompute Developer Reference Data Lake Analytics (Lakehouse) Data lakehouse solution 1.0 (Deprecated) Lakehouse 1.0 Build a data lakehouse by using MaxCompute and Hadoop Build and manage a Hadoop data lakehouse

Build and manage a Hadoop data lakehouse

Updated at:
Copy as MD
Product
My Favorites

Connecting MaxCompute to a Hadoop cluster creates a unified platform to manage, store, and analyze large volumes of structured and semi-structured data with high-concurrency analytics. This guide walks you through building and managing the data lakehouse.

Prerequisites

  • Activate MaxCompute and create a MaxCompute project. For more information, see Activate MaxCompute and Create a MaxCompute project.

    Note

    • If you are creating a new project, select the Hive-compatible data type.

    • To avoid cross-region network fees, make sure that the VPC hosting your Hadoop cluster is in the same region as your MaxCompute project.

  • Your Hadoop cluster must have high availability (HA) enabled. For more information, contact your Hadoop cluster administrator.

Limitations

  • The data lakehouse feature is available only in the following regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Hong Kong), Singapore, and Germany (Frankfurt).

  • The VPC of your Hadoop cluster must be in the same region as your MaxCompute project.

Procedure

Step 1: Grant permissions to MaxCompute

To build the data lakehouse, grant MaxCompute permission to create an elastic network interface (ENI) in your VPC. This establishes a network connection from MaxCompute to the VPC. Log on to the Alibaba Cloud console with the Alibaba Cloud account that owns the VPC and click Authorize.

Step 2: Build a data lakehouse

  1. Log in to the DataWorks console and select a region in the upper-left corner.

    For a list of supported regions, see Limitations.

  2. In the navigation pane on the left, choose Other Items > Lake and Warehouse Integration (Data Lakehouse).

  3. On the Lake and Warehouse Integration (Data Lakehouse) page, click Start.

  4. On the New Data Lake Integration page, configure the following parameters.

    Table 2. Create an external data lake connection

    Parameter

    Description

    Heterogeneous data platform type

    • Alibaba Cloud E-MapReduce/Hadoop cluster: Build a data lakehouse by using MaxCompute and Hadoop.

    • Alibaba Cloud DLF+OSS Data Lake Connection: Build a data lakehouse by using MaxCompute, DLF, and OSS.

    Alibaba Cloud E-MapReduce/Hadoop cluster

    Network connection

    Select or create a network connection from MaxCompute to the VPC of your E-MapReduce or Hadoop cluster. For parameter details, see the Create a network connection between MaxCompute and the destination VPC step in Network connection process.

    Note

    • This feature is free of charge during the public preview.

    • For more information about the basic concepts of network connections, see Network connections.

    Select external data source

    An external data source stores the URL, port, user authentication information, and other settings required to create an external data lake connection. Select or create an E-MapReduce or Hadoop external data source. For more information, see Table 3. External data source parameters.

    Table 3. External data source parameters

    Parameter

    Description

    Select MaxCompute project

    Select the destination MaxCompute project. You can view the project name on the Project Management tab.

    External Data Source Name

    Enter a custom name for the external data source. The name must meet the following requirements:

    • The name can contain only lowercase letters, digits, and underscores (_).

    • The name must be less than 128 characters in length.

    Network Connection Object

    The network connection from MaxCompute to the VPC where your E-MapReduce or Hadoop cluster is deployed. For more information, see Network connection process.

    NameNode Address

    The service addresses and port numbers of the active and standby NameNodes of the destination Hadoop cluster. The port is typically 8020. Contact your Hadoop cluster administrator for details.

    HMS Service Address

    The service addresses and port numbers of the active and standby Hive Metastore Service (HMS) instances in the destination Hadoop cluster. The port is typically 9083. Contact your Hadoop cluster administrator for details.

    Cluster Name

    The name that refers to the NameNode in a high-availability (HA) Hadoop cluster. For a self-built Hadoop cluster, you can find the cluster name in the dfs.nameservices configuration parameter in the hdfs-site.xml file.

    Authentication Type

    MaxCompute uses account mapping to access metadata and data from the Hadoop cluster. The mapped Hadoop accounts are typically protected by an authentication mechanism such as Kerberos. Select the authentication type that matches your cluster configuration. For more information, contact your Hadoop cluster administrator.

    • No Authentication Method: Select this option if Kerberos authentication is not enabled on the Hadoop cluster.

    • Kerberos Account Authentication: Select this option if Kerberos authentication is enabled on the Hadoop cluster.

      • Configuration File: Upload the krb5.conf file from the Hadoop cluster.

        Note

        On a Linux-based Hadoop cluster, the krb5.conf file is typically in the/etc directory on the master NameNode.

      • hmsPrincipals: The HMS service identity. You can run thelist_principals command in the Kerberos terminal of the Hadoop cluster to obtain the HMS principals. Example:

        hive/emr-header-1.cluster-20****@EMR.20****.COM,hive/emr-header-2.cluster-20****@EMR.20****.COM
        Note

        Separate principals for different nodes with a comma. Each principal must correspond to an HMS service address.

      • Add engine permission mapping.

        • Alibaba Cloud account: The Alibaba Cloud account used to access the Hadoop cluster through MaxCompute.

        • Kerberos Account: The Hadoop user account that is authorized by Kerberos and has permissions to access Hive.

        • Upload file: Upload the keytab configuration file for the Kerberos account. For information about how to generate this file, see Generate a keytab configuration file.

    Table 4. Create target data mapping

    Parameter

    Description

    Select external data source object

    By default, this parameter is set to the value of Select external data source.

    Map target database

    The destination database in the Hadoop cluster.

Step 3: Manage the data lakehouse

View metadata of external projects in DataWorks

  1. Log in to the DataWorks console and select a region in the upper-left corner.

  2. In the navigation pane on the left, choose Workspace.

  3. On the Workspaces page, in the Actions column of the target workspace, choose Shortcuts > DataStudio.

  4. On the Data Studio page, click the image icon in the navigation pane on the left. In the Data Catalog, click MaxCompute. On the MaxCompute - Projects tab on the right, add an existing external project or create a new one.

  5. Click the name of the target table to view the table schema below.

    Note

    • The table metadata is updated on a T+1 basis. This means that if you modify the table schema at the mapping source, such as in Hive, the changes appear in DataWorks Data Map the next day. Metadata on the MaxCompute engine side, however, is updated in real time.

In the Public Tables pane, enter keywords in the search box to quickly filter tables from external projects. When you select a target table, the Column Information, Partition Information, and Data Preview tabs appear below, showing the detailed table schema. In the SQL editor on the right, write and run queries against the external table. The query results are displayed in the Result tab below.

Related information

For data lakehouse solutions that use the Delta Lake or Apache Hudi storage format with a Hadoop cluster, see Data lakehouse based on a Hadoop cluster that supports Delta Lake or Apache Hudi.

Previous：Build a data lakehouse by using MaxCompute and HadoopNext：Support for Delta Lake or Hudi on Hadoop clusters