How to migrate data between Alibaba Cloud HDFS and OSS-Apsara File Storage for HDFS(HDFS)-阿里云帮助中心

This topic describes how to migrate data between Apsara File Storage for HDFS and Object Storage Service (OSS). You can migrate data from Apsara File Storage for HDFS to OSS, or from OSS to Apsara File Storage for HDFS.

Prerequisites

You have activated Apsara File Storage for HDFS and created a file system instance and a mount target. For more information, see Quick start for Apsara File Storage for HDFS.
You have deployed a Hadoop cluster and installed JDK 1.8 or later on all cluster nodes. We recommend that you use Hadoop 2.7.2 or later. This topic uses Apache Hadoop 2.8.5.
You have activated OSS and created an OSS bucket.

Background information

Alibaba Cloud Apsara File Storage for HDFS is a file storage service designed for compute resources such as ECS instances and container services. Apsara File Storage for HDFS lets you manage and access data in the same way as the Hadoop Distributed File System (HDFS), providing high-performance access to hot data. Object Storage Service (OSS) is a secure, cost-effective, highly reliable cloud storage service that offers multiple storage classes, such as Standard and Archive. You can perform bidirectional data migration between Apsara File Storage for HDFS and OSS to tier hot, warm, and cold data effectively. This approach ensures high-performance access to hot data while controlling storage costs.

Step 1: Mount an Apsara File Storage for HDFS instance on a Hadoop cluster

Mount the Apsara File Storage for HDFS instance on your Hadoop cluster. For more information, see Mount an Apsara File Storage for HDFS file system.

Step 2: Deploy JindoSDK on your Hadoop cluster

Download the latest JindoSDK installation package.

This topic uses jindosdk-4.5.0 as an example.
Run the following command to extract the installation package.
```
tar -zxvf ./jindosdk-4.5.0.tar.gz
```
Run the following command to copy the jindo-core-4.5.0.jar and jindo-sdk-4.5.0.jar files from the installation package to the Hadoop CLASSPATH directory.
```
cp -v ./jindosdk-4.5.0/lib/jindo-*-4.5.0.jar ${HADOOP_HOME}/share/hadoop/hdfs/lib/
```

Configure the JindoSDK OSS implementation class and Access Key.

Run the following command to open the core-site.xml configuration file.
```
vim ${HADOOP_HOME}/etc/hadoop/core-site.xml
```

Add the JindoSDK OSS implementation class to the core-site.xml file.

Note

Add the following configuration within the <configuration> tags.

<property>
    <name>fs.AbstractFileSystem.oss.impl</name>
    <value>com.aliyun.jindodata.oss.OSS</value>
</property>
<property>
    <name>fs.oss.impl</name>
    <value>com.aliyun.jindodata.oss.JindoOssFileSystem</value>
</property>

Add the Access Key ID, Access Key Secret, and Endpoint for your OSS bucket to the core-site.xml file.

Note

Add the following configuration within the <configuration> tags.

<property>
    <name>fs.oss.accessKeyId</name>
    <!-- Enter the Access Key ID used to access your OSS bucket -->
    <value>xxx</value>
</property>
<property>
    <name>fs.oss.accessKeySecret</name>
    <!-- Enter the Access Key Secret corresponding to the Access Key ID -->
    <value>xxx</value>
</property>
<property>
    <name>fs.oss.endpoint</name>
    <!-- In ECS environments, we recommend using the internal OSS endpoint in the format oss-cn-xxxx-internal.aliyuncs.com -->
    <value>oss-cn-xxxx.aliyuncs.com</value>
</property>

Synchronize the configurations from step 3 and step 4 to all nodes in the Hadoop cluster.
Use Hadoop Shell to access OSS.
```
${HADOOP_HOME}/bin/hadoop fs -ls oss://<bucket>/<path>
```
In this command, <bucket> is the name of your OSS bucket, and <path> is the file path within the bucket. Replace these placeholders as needed.

Step 3: Data migration

After you mount an Apsara File Storage for HDFS instance on your Hadoop cluster and install the OSS client JindoSDK, you can use a Hadoop MapReduce task (DistCp) to migrate data.

Example 1: Migrate data from Apsara File Storage for HDFS to Object Storage Service (OSS)

Run the following command to generate 100 GB of test data on your Apsara File Storage for HDFS instance.
```
${HADOOP_HOME}/bin/hadoop jar \
${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar \
randomtextwriter \
-D mapreduce.randomtextwriter.totalbytes=107374182400 \
-D mapreduce.randomtextwriter.bytespermap=10737418240 \
dfs://f-xxxxxxx.cn-beijing.dfs.aliyuncs.com:10290/dfs2oss/data/data_100g/
```
Replace the following parameters as needed.
- f-xxxxxxx.cn-beijing.dfs.aliyuncs.com: The mount target domain name of your Apsara File Storage for HDFS instance.
- /dfs2oss/data/data_100g: The directory used in this example to store test data.
- hadoop-mapreduce-examples-2.8.5.jar: The Hadoop version.

View the generated test data.

Run the command

${HADOOP_HOME}/bin/hadoop fs -du -s dfs://f-xxxxxxx.cn-beijing.dfs.aliyuncs.com:10290/dfs2oss/data/data_100g

Sample output

110223430221  dfs://f-xxxxxxx.cn-beijing.dfs.aliyuncs.com:10290/dfs2oss/data/data_100g

Start a Hadoop MapReduce task (DistCp) to migrate the test data to OSS. For more information about DistCp, see the official Hadoop DistCp documentation.
```
${HADOOP_HOME}/bin/hadoop distcp \
dfs://f-xxxxxxx.cn-beijing.dfs.aliyuncs.com:10290/dfs2oss oss://<bucket>/<path>
```

After the task completes, if the output contains information similar to the following, the migration succeeded.

22/08/10 16:20:54 INFO mapreduce.Job: Job job_1660115355800_0003 completed successfully
22/08/10 16:20:54 INFO mapreduce.Job: Counters: 38
        File System Counters
                DFS: Number of bytes read=110223438504
                DFS: Number of bytes written=0
                DFS: Number of read operations=114
                DFS: Number of large read operations=0
                DFS: Number of write operations=26
                FILE: Number of bytes read=0
                FILE: Number of bytes written=2119991
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                OSS: Number of bytes read=0
                OSS: Number of bytes written=0
                OSS: Number of read operations=0
                OSS: Number of large read operations=0
                OSS: Number of write operations=0
        Job Counters
                Launched map tasks=13
                Other local map tasks=13
                Total time spent by all maps in occupied slots (ms)=10264552
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=10264552
                Total vcore-milliseconds taken by all map tasks=10264552
                Total megabyte-milliseconds taken by all map tasks=10510901248
        Map-Reduce Framework
                Map input records=14
                Map output records=0
                Input split bytes=1755
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=18918
                CPU time spent (ms)=798190
                Physical memory (bytes) snapshot=4427018240
                Virtual memory (bytes) snapshot=50057256960
                Total committed heap usage (bytes)=2214068224
        File Input Format Counters
                Bytes Read=6528
        File Output Format Counters
                Bytes Written=0
        DistCp Counters
                Bytes Copied=110223430221
                Bytes Expected=110223430221
                Files Copied=14

Verify that the size of the data migrated to OSS matches the original data.

Run the command

${HADOOP_HOME}/bin/hadoop fs -du -s oss://<bucket>/<path>

Sample output

[root@master1 ~]# ${HADOOP_HOME}/bin/hadoop fs -du -s oss://data-migrate-test/dfs2oss/
22/08/10 16:24:50 INFO jnative.NativeLogger: CommonJniLogging.cpp:34] Client logger level is info
22/08/10 16:24:50 INFO jnative.NativeLogger: JindoJniState.cpp:7] JindoJniState::prepareAfterInited() getting called
22/08/10 16:24:50 INFO jnative.NativeLogger: JcomMainBaseImpl.hpp:130] main doRun() return value True
22/08/10 16:24:50 INFO jnative.NativeLogger: JindoNative.cpp:42] Successfully initialized jni native
22/08/10 16:24:50 INFO common.JindoHadoopSystem: Initialized native file system:
22/08/10 16:24:50 INFO common.FsStats: cmd=getFileStatus, src=oss://data-migrate-test/dfs2oss, dst=null, size=0, parameter=null, time-in-ms=108, version=4.5.0
22/08/10 16:24:50 INFO common.FsStats: cmd=getContentSummary, src=oss://data-migrate-test/dfs2oss, dst=null, size=0, parameter=null, time-in-ms=41, version=4.5.0
110223430221   oss://data-migrate-test/dfs2oss
[root@master1 ~]#

Example 2: Migrate data from Object Storage Service (OSS) to Apsara File Storage for HDFS

Generate 100 GB of test data in OSS.

${HADOOP_HOME}/bin/hadoop jar \
${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar \
randomtextwriter \
-D mapreduce.randomtextwriter.totalbytes=107374182400 \
-D mapreduce.randomtextwriter.bytespermap=10737418240 \
oss://<bucket>/<path>

Replace the following parameters as needed.

<bucket>/<path>: The OSS bucket directory for test data.
hadoop-mapreduce-examples-2.8.5.jar: The Hadoop version.

View the generated test data.

Run the command

${HADOOP_HOME}/bin/hadoop fs -du -s oss://<bucket>/<path>

Sample output

[root@master1 ~]# ${HADOOP_HOME}/bin/hadoop fs -du -s oss://data-migrate-test/oss2dfs/
22/08/10 16:48:44 INFO jnative.NativeLogger: CommonJniLogging.cpp:34] Client logger level is info
22/08/10 16:48:44 INFO jnative.NativeLogger: JindoJniState.cpp:7] JindoJniState::prepareAfterInited() getting called
22/08/10 16:48:44 INFO jnative.NativeLogger: JcomMainBaseImpl.hpp:130] main doRun() return value True
22/08/10 16:48:44 INFO jnative.NativeLogger: JindoNative.cpp:42] Successfully initialized jni native
22/08/10 16:48:44 INFO common.JindoHadoopSystem: Initialized native file system:
22/08/10 16:48:44 INFO common.FsStats: cmd=getFileStatus, src=oss://data-migrate-test/oss2dfs, dst=null, size=0, parameter=null, time-in-ms=82, version=4.5.0
22/08/10 16:48:44 INFO common.FsStats: cmd=getContentSummary, src=oss://data-migrate-test/oss2dfs, dst=null, size=0, parameter=null, time-in-ms=52, version=4.5.0
110223424482   oss://data-migrate-test/oss2dfs
[root@master1 ~]#

Start a Hadoop MapReduce task (DistCp) to migrate the test data to Apsara File Storage for HDFS.

${HADOOP_HOME}/bin/hadoop distcp \
oss://<bucket>/<path> \
dfs://f-xxxxxxx.cn-beijing.dfs.aliyuncs.com:10290/oss2dfs

After the task completes, if the output contains information similar to the following, the migration succeeded.

22/08/10 17:08:17 INFO mapreduce.Job: Job job_1660115355800_0005 completed successfully
22/08/10 17:08:17 INFO mapreduce.Job: Counters: 38
        File System Counters
                DFS: Number of bytes read=6495
                DFS: Number of bytes written=110223424482
                DFS: Number of read operations=139
                DFS: Number of large read operations=0
                DFS: Number of write operations=47
                FILE: Number of bytes read=0
                FILE: Number of bytes written=1957682
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                OSS: Number of bytes read=0
                OSS: Number of bytes written=0
                OSS: Number of read operations=0
                OSS: Number of large read operations=0
                OSS: Number of write operations=0
        Job Counters
                Launched map tasks=12
                Other local map tasks=12
                Total time spent by all maps in occupied slots (ms)=10088860
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=10088860
                Total vcore-milliseconds taken by all map tasks=10088860
                Total megabyte-milliseconds taken by all map tasks=10330992640
        Map-Reduce Framework
                Map input records=12
                Map output records=0
                Input split bytes=1620
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=21141
                CPU time spent (ms)=780270
                Physical memory (bytes) snapshot=4037222400
                Virtual memory (bytes) snapshot=45695025152
                Total committed heap usage (bytes)=2259156992
        File Input Format Counters
                Bytes Read=4875
        File Output Format Counters
                Bytes Written=0
        DistCp Counters
                Bytes Copied=110223424482
                Bytes Expected=110223424482
                Files Copied=12

Verify that the test data migrated to Apsara File Storage for HDFS matches the original data in OSS.

Run the command

${HADOOP_HOME}/bin/hadoop fs -du -s dfs://f-xxxxx.cn-zhangjiakou.dfs.aliyuncs.com:10290/oss2dfs

Sample output

110223424482 dfs://f-xxxxx.cn-beijing.dfs.aliyuncs.com:10290/oss2dfs

FAQ

Will the latest data written to a file be missed during migration?

Hadoop-compatible file systems support single-writer, multi-reader concurrency semantics. For a single file, one writer can write data while multiple readers read it at the same time. Take data migration from Apsara File Storage for HDFS to OSS as an example. When the data migration task opens a file in Apsara File Storage for HDFS, it determines the file's length based on the current system state and migrates that amount of data to OSS. If another writer concurrently writes data to the file during the migration, the file's length increases, but the migration task cannot detect the newly written data. Therefore, avoid writing data to files that are being migrated.