How to migrate data to OSS or OSS-HDFS-Object Storage Service(OSS)-阿里云帮助中心

Business data is often scattered across on-premises data centers, third-party cloud storage, or distributed across different regions and accounts in Alibaba Cloud OSS buckets. Lack of unified management leads to operational complexity and high costs. You can efficiently migrate such data to the target OSS bucket. Alibaba Cloud provides various migration solutions, including the online migration service, command-line tools, and offline migration (Lightning Cube), to meet migration requirements in different business scenarios.

Migration between Alibaba Cloud OSS buckets

When migrating data between Alibaba Cloud OSS buckets, choose the appropriate migration method based on whether the source and destination buckets are in the same region, to ensure migration efficiency and operational convenience.

Same-region migration

When the source and destination buckets are in the same region, choose a migration method based on your actual scenario:

For small to medium-scale data where the buckets belong to the same account, we recommend using the cp (copy objects) command of ossutil to migrate data. This method supports batch file copy and resumable upload.
For large-scale data migration or cross-account migration between buckets, we recommend using the SRR feature. Data is automatically synchronized to the destination when it is added, modified, or deleted in the source bucket without the need for intermediate downloading or additional network transmission. This is suitable for data centralization and sharing scenarios across teams and subsidiaries.

Cross-region migration

When the source and destination buckets are in different regions, for example, from China (Hangzhou) to China (Beijing), we recommend using the CRR feature. It uses the Alibaba Cloud internal network for transmission, ensuring data security and reliability. Data is automatically synchronized to the destination when it is added, modified, or deleted in the source bucket. This is suitable for multi-site collaboration and real-time backup scenarios.

Migrating third-party storage to OSS

When your data is stored in other cloud providers and needs to be migrated to Alibaba Cloud OSS, we recommend using the Alibaba Cloud online migration service. It supports various storage services including AWS S3, Tencent Cloud COS, Huawei Cloud OBS, Volcano Engine TOS, Google Cloud GCS, and Microsoft Azure Blob, as well as self-managed object storage compatible with the S3 protocol. No migration environment setup is required. You can submit migration tasks online and monitor the migration process at any time. Select the corresponding online migration tutorial based on your business scenario to smoothly migrate your data.

Migrating local file systems to OSS

Small-scale data migration

For small-scale data (less than 5 GB), we recommend using the OSS console to upload data directly. This method is easy to use, requires no additional tools or complex configuration, and is suitable for temporary file uploads, test data migration, or low-frequency migration tasks. You can quickly complete file uploads through the browser, which is ideal for migrating scattered files or small amounts of data to the cloud.

Medium-scale data migration

For medium-scale data migration, you can choose one of the following two methods based on your requirements:

For efficient transfer of routine business data, log files, backup data, and more, we recommend using the ossutil command-line tool to run the cp (upload files) command. It supports batch file upload, resumable upload, and concurrent acceleration.
When your local network environment is complex or you want to implement unified scheduling and centralized management of migration tasks in the cloud, we recommend using the Alibaba Cloud online migration service for LocalFS to OSS migration. This service provides a managed data migration solution that supports task creation, monitoring, and management, simplifying operational complexity. It is suitable for enterprise customers that require centralized management of the migration process.

Large-scale data migration

For ultra-large-scale data migration, you can choose one of the following two methods based on your requirements:

Use the Alibaba Cloud online migration service, which requires no migration environment setup. You can submit migration tasks online and monitor the migration progress in real time. This is suitable for scenarios that require flexible scheduling and can transfer data over the Internet.
When your data is limited by Internet bandwidth or needs to be migrated from an on-premises data center, we recommend using offline migration (Lightning Cube). This is suitable for scenarios such as full data center migration to the cloud, enterprise archive files, and historical image data. Offline migration (Lightning Cube) is designed for TB-level to PB-level large-scale data migration. It uses physical media for data collection and transfer, completely avoiding Internet bandwidth bottlenecks and significantly improving migration efficiency.

Migrating HTTP/HTTPS sources to OSS

If your data is stored on HTTP/HTTPS sources, you can use the Alibaba Cloud online migration service to efficiently and smoothly migrate data from HTTP/HTTPS sources to OSS. No additional environment setup is required during migration. Simply submit the migration task online to monitor the task progress and status in real time, ensuring that data is migrated securely and reliably.

Note

OSS does not support single-file URL-based pull APIs. All upload APIs (PutObject, Multipart Upload, AppendObject, etc.) are client-initiated uploads and do not support server-side file retrieval from remote URLs. The AsyncFetch feature, which previously supported server-side file retrieval from URLs, has been discontinued for new subscriptions and is unavailable to new accounts.

Single-file scenario: To store a file from a remote URL in OSS, we recommend downloading the file to your local machine first, and then uploading it to OSS using methods such as PutObject.
Batch-file scenario: To batch-retrieve files from multiple HTTP/HTTPS URLs to OSS, we recommend using the Alibaba Cloud online migration service to create an HTTP/HTTPS source migration task. Use a manifest file (a list containing multiple URLs) to download and migrate files in batches.

Migrating big data storage to OSS

Migrating HDFS data to OSS

For large-scale data distributed in Hadoop Distributed File System (HDFS), we recommend using Alibaba Cloud Jindo DistCp to migrate HDFS data to OSS. Jindo DistCp is a MapReduce-based distributed file copy tool that efficiently transfers files within large-scale clusters or between different clusters. It uses file and directory lists as input for MapReduce tasks, splits them into multiple parallel tasks for execution, with each task responsible for copying a portion of the source list. This improves transfer efficiency and supports fault tolerance, resumable upload, and exception recovery. It ensures stability and data integrity during TB-level or even PB-level big data migration. This is suitable for big data computing and data lake construction scenarios.

Migrating OSS external table (gpossext) data to OSS

When data is stored as OSS external tables (gpossext) and needs to be efficiently imported and exported between OSS and a data warehouse, we recommend using AnalyticDB for PostgreSQL to export data to OSS in parallel. It supports parallel import of data from OSS to AnalyticDB PostgreSQL and parallel export of data to OSS through the gpossext feature. Based on a distributed architecture, the migration process provides high concurrency and high throughput, significantly reducing data transfer time while ensuring data security and integrity. This is suitable for large-scale data analysis, historical data archiving, and cross-system data exchange scenarios.

Migrating big data storage to OSS-HDFS

OSS-HDFS (also known as JindoFS) is a cloud-native data lake storage product. Based on unified metadata management capabilities, it is fully compatible with the HDFS file system interface while providing complete POSIX capability support, better adapting to data lake scenarios such as big data computing and AI training. You can migrate existing data to OSS-HDFS or perform efficient data migration between different OSS-HDFS buckets.

Migrating HDFS data to OSS-HDFS

For traditional HDFS cluster data migration, you can use Alibaba Cloud Jindo DistCp to migrate HDFS data to OSS-HDFS. It enables large-scale file distribution within a cluster or across clusters, and supports automatic error detection, retries, and task recovery, improving the stability and efficiency of migration tasks. Its working principle is to use file and directory lists as input for MapReduce tasks, with each task responsible for copying a portion of the files, making it suitable for batch processing of massive data migration. With Jindo DistCp, you can achieve smooth migration and quickly import data into OSS-HDFS, facilitating the transformation of big data platforms to the cloud-native architecture.

Migrating data between OSS-HDFS buckets

For users who have deployed OSS-HDFS, Alibaba Cloud also supports using Alibaba Cloud Jindo DistCp to migrate data between different OSS-HDFS buckets. This is suitable for data partition adjustment, storage resource optimization, or cross-region data scheduling requirements, ensuring that business data remains consistent and highly available.

Migrating semi-managed JindoFS cluster data to OSS-HDFS

When using semi-managed JindoFS clusters, you can use JindoDistJob to migrate semi-managed JindoFS cluster data to OSS-HDFS. It supports full and incremental migration and can smoothly switch to the JindoFS managed solution without migrating data blocks, ensuring no business disruption during the migration process. This is suitable for scenarios that require rapid storage architecture switching.

Migrating Hive tables and partition data to OSS-HDFS

To migrate structured data, you can use the JindoTable MoveTo command to migrate Hive tables and partition data to OSS-HDFS. After the underlying data copy is complete, this command automatically updates the metadata, fully migrating the tables and partitions to the new path. It supports conditional filtering, can migrate a large number of partitions at once, and ensures data integrity and security through multiple data verification mechanisms. This makes it ideal for migrating structured big data sets.

Configuring zero-downtime migration

Recommendation: Determine the migration method first, and then configure mirror origin-pull based on your business requirements to achieve zero-downtime migration.

To maintain business continuity during migration and achieve zero-downtime migration, you can also configure mirror origin-pull. After the service is switched to OSS, when any data that has not yet been migrated is accessed, the system automatically pulls it from the origin, ensuring a smooth transition without user impact. The typical migration process is as follows: first complete the historical data migration and switch the business endpoint to OSS, then mirror origin-pull automatically retrieves the unmigrated data, gradually backfilling it until the full migration to the cloud is complete.

For example, a business originally deployed on another cloud storage platform needs to migrate to OSS for business development purposes but cannot stop online services. In this case, you can configure a mirror origin-pull rule to automatically fetch unmigrated data and synchronize it to OSS during the migration process, ensuring continuous and stable service operation.