E-MapReduce data migration
This topic describes how to migrate data from the distributed file system of E-MapReduce to the Apsara File Storage for HDFS file system.
Background information
E-MapReduce (EMR) is an open source big data platform that runs on Alibaba Cloud. EMR on ECS is the deployment mode in which EMR runs on Elastic Compute Service (ECS) instances.
You can use EMR on ECS with the Apsara File Storage for HDFS. This lets you leverage the high throughput and low latency of the Apsara File Storage for HDFS to achieve higher performance for big data computing and meet evolving business demands.
Prerequisites
-
You have created an E-MapReduce cluster that meets the following requirements. For more information, see Create a cluster.
-
The EMR cluster is of the EMR-3.XX.X series.
-
The cluster has Java Development Kit (JDK) 1.8 or later installed.
-
The cluster has Hadoop 2.7.2 or later installed.
-
-
You have enabled the Apsara File Storage for HDFS, and created a file system, a mount target, a permission group, and a permission group rule. For more information, see HDFS feature for File Storage NAS Quick Start.
When you configure the mount target, ensure that the selected VPC and vSwitch are the same as those used by the E-MapReduce cluster. Follow these steps to find the VPC and vSwitch information:
-
Log on to the Alibaba Cloud E-MapReduce console.
-
In the top navigation bar, select a region.
-
On the Overview page, find the target E-MapReduce cluster on which to mount the Apsara File Storage for HDFS, and click Node Management.
-
Click the
icon next to the target node group to expand the list of nodes. -
Click the ID of the target node to go to its Instance Details page. In the Configuration Information section, find the VPC and vSwitch information.
-
This topic uses an EMR cluster of version EMR-3.51.0 and the data lake use case as an example.
Data migration
-
Log on to the Alibaba Cloud E-MapReduce console.
-
In the top navigation bar, select a region.
-
On the Overview page, find the target E-MapReduce cluster where you want to mount Apsara File Storage for HDFS, and click Cluster Services.
-
Stop the services.
To prevent data loss during migration, you must stop data processing services such as YARN, Hive, and Spark. This section uses the Spark3 service as an example.
-
On the Cluster Services tab, hover over the Spark3 card, and then choose
> Stop. -
In the dialog box that appears, enter a reason for the action and click OK.
A service is stopped when its health status icon turns black. Stop all Spark components before you stop other services.
The Spark3 service includes the SparkHistoryServer, SparkRuntime, and SparkThriftServer components. You can check the Health Status column to confirm that each one has stopped.
-
Repeat the preceding steps to stop the remaining services.
ImportantIf your EMR cluster uses HDFS as its distributed file system, do not stop the HDFS service. Otherwise, the data migration will fail.
-
-
Configure the Hadoop-Common service.
-
On the Cluster Services tab, hover over the Hadoop-Common card and click Configure.
-
Select the core-site.xml tab and click Add Configuration Item.
-
Add the following parameters and click OK.
-
Set the fs.dfs.impl parameter to com.alibaba.dfs.DistributedFileSystem.
-
Set the fs.AbstractFileSystem.dfs.impl parameter to com.alibaba.dfs.DFS.
-
-
In the Save dialog box, confirm the new parameters, enter a reason, and then click Save.
Turn on the Save and Apply switch.
-
Click Deploy Client Configuration, enter a reason, and click OK.
-
-
Deploy the Java SDK for the HDFS feature for File Storage NAS.
-
Download the latest Java SDK for the HDFS feature for File Storage NAS.
-
Copy the Java SDK for the HDFS feature for File Storage NAS to the HDFS dependency directory of the Hadoop-Common service on each node in the EMR cluster.
# Replace aliyun-sdk-dfs-x.y.z.jar with the filename of the Java SDK that you downloaded. sudo cp aliyun-sdk-dfs-x.y.z.jar ${HADOOP_HOME}/share/hadoop/hdfs/NoteYou must add the SDK package to the same location on every node in the cluster.
-
Copy the Java SDK for the HDFS feature for File Storage NAS to the HDFS dependency directory of the YARN service on each node in the EMR cluster.
# Replace aliyun-sdk-dfs-x.y.z.jar with the filename of the Java SDK that you downloaded. sudo cp aliyun-sdk-dfs-x.y.z.jar /opt/apps/YARN/yarn-current/share/hadoop/hdfs/ sudo chown hadoop:hadoop /opt/apps/YARN/yarn-current/share/hadoop/hdfs/aliyun-sdk-dfs-x.y.z.jarNoteYou must add the SDK package to the same location on every node in the cluster.
-
-
Configure user-to-group mappings.
Call the CreateUserGroupsMapping API to create mappings for the following users and groups.
Note-
The user-to-group mapping feature is currently in beta testing. To use this feature, submit a ticket to request access.
-
In the Apsara File Storage for HDFS,
supergroupis the superuser group. If you encounter permission issues with other services, map the corresponding user to thesupergroupgroup.
-
Map the user
emr-userto the groupsemr-userandsupergroup. -
Map the user
adminto the groupsadminandsupergroup. -
Map the users
hadoop,hdfs,hive, andsparkto the groupshadoopandsupergroup.
-
-
Migrate the data.
Migrate all business data from the distributed file system of EMR on ECS to the Apsara File Storage for HDFS.
-
For small amounts of data, use the
hadoop fs -cpcommand.# Replace xxx://xxxx.xxxxx.xxx/* with the source data path. # Replace f-095d7371d****.cn-****.dfs.aliyuncs.com with the domain name of your mount target for the HDFS feature for File Storage NAS. hadoop fs -cp -p xxx://xxxx.xxxxx.xxx/* dfs://f-095d7371d****.cn-****.dfs.aliyuncs.com:10290/ -
For large amounts of data, use the Hadoop DistCp data migration tool.
-
Start the YARN service.
-
Use Hadoop DistCp to migrate the data.
# Replace xxx://xxxx.xxxxx.xxx/* with the source data path. # Replace f-095d7371d****.cn-****.dfs.aliyuncs.com with the domain name of your mount target for the HDFS feature for File Storage NAS. hadoop distcp -pugpt xxx://xxxx.xxxxx.xxx/* dfs://f-095d7371d****.cn-****.dfs.aliyuncs.com:10290/
-
After the migration is complete, run the
hadoop fs -ls dfs://f-095d7371d****.cn-****.dfs.aliyuncs.com:10290/command to verify that the data has been migrated to the HDFS feature for File Storage NAS. -
Next steps
Configure E-MapReduce services to use the HDFS feature for File Storage NAS