Configure a Hive data source

Step 1: Prepare the Hive data source

Prepare the user account and permissions required to read data from your Hive source.

• If the meta-type is set to Glue, you must have an AWS account with permissions to read metadata from AWS Glue Data Catalog.

• If the storage-type is set to S3, you must have an AWS account with permissions to read data from Amazon S3.

• If the storage-type is set to OSS, you must have an Alibaba Cloud account with permissions to read data from OSS.

• If you use Hive for data validation, you must have a Hive account that can execute SQL through Hive JDBC.

• If you use AWS Athena for data validation, the AWS account must have permissions to execute queries in Athena.

Step 2: Add a data source

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Data Transfer > Migration Service.

3. On the Data Source tab, click Add Data Source.

   To create the service-linked role, click OK in the MaxCompute Service-linked Role dialog box. If this dialog box does not appear, the role already exists.

   On the Add Data Source page, configure the data source and click Add.

   • Data Source Basic Information

     Parameter	Required	Description
     Data Source Name	Yes	A custom name for the data source. The name can contain letters, digits, and Chinese characters, but no other special characters.
     Data Source Type	Yes	Select Hive.
     Network Connection	Yes	Select the network connection to use. Network connections are created on the Manage Configurations > Network Connection page of the MaxCompute console. These connections allow MMS to communicate with a VPC to connect to the data source.
     Metadata Service Type	Yes	HMS: The default metadata management service, Hive Metastore (hive metastore). For this type, configure the following parameter: Metastore URLs AWS-GLUE: The unified metadata management service, AWS Glue Data Catalog (Glue). For this type, configure the following parameters: AWS Region AWS Access Key Id AWS Access Key Secret
     Storage Service Type	Yes	You can select multiple options: When the metadata service type is AWS-GLUE, ALIYUN-OSS is not supported. HDFS ALIYUN-OSS: For this type, configure the following parameters: OSS Endpoint OSS Access Id OSS Access Key AWS-S3: Enter the S3 Endpoint to connect to AWS S3.
     Hive Version	Yes	Select the version based on your environment.
     JDBC URL	No	The Hive JDBC connection information. The format is jdbc:hive2://localhost:10000/default.
     JDBC username	No	The username for the Hive JDBC connection.
     JDBC password	No	The password for the Hive JDBC connection.
     SQL hints for MaxCompute migration job	No	Specifies SQL hints. For more information, see List of flag parameters.
     Default destination MaxCompute project	Yes	The default destination project for the data migration. This setting cannot be changed after creation.
     Destination MaxCompute project list	No	If data from a single data source needs to be migrated to multiple destination projects, you must configure this parameter.
     MaxCompute project for executing migration jobs	Yes	The project where migration jobs, such as Spark and SQL jobs, run on MaxCompute. The jobs use the default compute quota associated with this project.

   • Kerberos information

     Parameter	Description
     Kerberos Authentication	Enables Kerberos authentication for the Hive metastore. If enabled, you must also set the Kerberos principal parameter.
     Kerberos principal	The Kerberos principal for authentication. The value must be consistent with the kdc_realm value in your krb5.conf file.

   • Other information

     The following parameters are optional. Configure them as needed.

     Parameter	Description
     Hive job settings	Used for execution engines such as MapReduce, Spark, and Tez.
     Metastore scheduled update	Enables or disables periodic metadata synchronization from the data source. Valid values: Enable: Periodically synchronizes metadata. Set the Update Cycle to daily or hourly, and configure the Update Started At. Daily: The update runs once per day at the specified time (accurate to the minute). Hourly: The update runs once per hour at the specified minute. Disable: Metadata is not synchronized periodically.
     Metastore access concurrency	The number of concurrent requests to the metastore. Default value: 3.
     Schema Whitelist	The schemas to migrate. To specify multiple schemas, separate them with commas (,).
     Schema Blacklist	The schemas to exclude from the migration. To specify multiple schemas, separate them with commas (,).
     Table Blacklist	The Hive tables to exclude from the migration. The format for a single table is schema.table or table, and multiple tables are separated by a comma (,).
     Table Whitelist	The Hive tables to migrate. The format for a single table is schema.table or table, and multiple tables are separated by commas (,).
     Maximum concurrent migration tasks	The maximum number of concurrent MMS migration tasks. A high concurrency level may overload the compute resources of the source system. Default value: 20.
     Table name character mapping	Maps special characters in source table names to MaxCompute-supported characters. Specify the mappings in JSON format, such as {"%": "_"}.
     Maximum partitions per task	The maximum number of partitions to process in a single migration task. Processing partitions in batches reduces the number of submitted SQL queries and saves time. In most cases, you do not need to change the default value of 50.
     Maximum data size per task (GB)	Default value: 5 GB. In most cases, you do not need to change this value.
     Partition value character mapping	Maps special characters in source partition values to MaxCompute-supported characters. Specify the mappings in JSON format, such as {"/": "_"}.
     Hadoop username for Spark	In simple authentication mode, Spark accesses Hadoop using the default username admin. You can use this parameter to specify a different username.
     Default invalid partition name	The name that Hive uses for default or invalid partitions.

Step 3: Synchronize metadata

When you start the data source, MMS creates a job instance that connects to the data source and synchronizes its metadata. This lets you configure subsequent migration jobs.

1. On the Data Source tab, for the target data source, click Update metadata in the Actions column.

2. The Data Source tab shows the Status of the target data source.

3. If the source metadata changes and you have enabled the Metastore scheduled update feature, the system updates the metadata automatically based on your schedule. Manual synchronization is not required.