Migrate a source Iceberg table to OSS Tables
Apache Iceberg metadata records absolute object paths, so moving a table into OSS Tables requires rewriting those paths and copying the referenced objects before the target table can serve queries. This topic describes how to migrate a source Iceberg table to OSS Tables and atomically switch the target table to the rewritten metadata.
Applicable scenarios
This topic applies when the following conditions are met:
The source table is an Apache Iceberg table stored in Object Storage Service (OSS), S3, Azure Blob, GCS, a similar object storage service, or Hadoop Distributed File System (HDFS).
The target is a table in OSS Tables.
The objects to migrate can include Data File, Delete File, Deletion Vector, Manifest, historical Snapshot, Tag, Branch, Schema, Partition Spec, Sort Order, and Statistics objects.
How it works
The migration workflow is as follows:
Iceberg table in an object storage bucket
│
│ Read the metadata reference graph of the specified version range
▼
rewrite_table_path
│
├─ Staging: rewritten Metadata JSON, Manifest, Position Delete File, Deletion Vector
└─ File list: source URI,target URI
│
│ Convert to Key,DestKey
│ Split the file list by source bucket if it spans multiple buckets
▼
Online migration service
│
▼
Data storage location of the target table
│
│ Verify the file count, content, and metadata
▼
Update metadataLocation with the latest versionToken
│
▼
Validate Iceberg semantics through the target tablerewrite_table_path processes the source table in the following order:
Locate the source table — Get the current
metadataLocationfrom the source Iceberg Catalog based on thetableparameter.source_prefixis used only for path replacement and does not take part in the lookup.Traverse and rewrite paths — Starting from the Metadata JSON, traverse the Iceberg references within the migration scope, replace
{source_prefix}with{target_prefix}in the absolute source paths, and write the rewritten objects to{staging_location}. Source objects are not modified during this process.Generate the copy plan — Rewritten objects use Staging as their source, and objects that are not rewritten use the source table. Both are recorded in the same plan.
Copy to the target — The online migration service copies the objects to the data storage location of the target table according to the copy plan.
The following table describes how the objects involved in the rewrite and copy phases are processed:rewrite_table_pathitself does not copy objects or updatemetadataLocation.
| Processing method | Object type | Copy source |
| Copied as-is | Data File, Equality Delete File, Statistics, and Partition Statistics | Source table objects |
| Copied after rewriting | Metadata JSON, Manifest List, Manifest, Position Delete File, and V3 Deletion Vector | Rewritten objects in Staging |
Staging is a temporary rewrite directory, not a complete Iceberg table. Do not register Staging as an Iceberg table, and do not point the metadataLocation of the target table to Staging. Keep the Staging directory until the migration is complete and the observation period ends.
Prerequisites
Before you start the migration, complete the following preparations:
Activate OSS Tables, and create a Table Bucket, a Namespace, and a target table dedicated to this migration. The target table must not carry other production data.
Turn off unreferenced file cleanup for the target Table Bucket.
Unreferenced file cleanup is a Table Bucket-level setting. Migrated objects are not yet referenced by the target table before metadataLocation is switched. If cleanup stays on, these objects may be marked and deleted during the migration. Before you start copying objects, go to the Data Maintenance page of the target Table Bucket, turn off Unreferenced File Cleanup, and keep it off until the Iceberg semantic validation and smoke test after the switch all pass.
Activate the online migration service and select a region that is the same as the region of the target table to run the migration job.
The source table is registered in an Iceberg Catalog that Spark can access, and the current
metadataLocationcan be loaded by table name. The catalog can be an existing Hive, REST, JDBC, or other compatible implementation used by your business. No specific catalog type is required.Prepare the Java, Spark, Scala, and Apache Iceberg runtime environment as described in Version requirements and migration limits.
Prepare a dedicated, empty Staging prefix and an inventory storage location.
The source storage supports Spark FileIO access and reads by the online migration service. For third-party object storage, use least-privilege credentials and do not store AccessKey pairs in plaintext.
Prepare a rollback plan for the
metadataLocationswitch in Step 5, and validate a rebuild plan for the target table in case no recoverablemetadataLocationexists before the switch. Step 2 and Step 5 return themetadataLocationandversionTokenvalues that these plans require.
The permissions involved in the migration are as follows:
| Principal | Required permissions |
| Spark | Reads source table objects, and reads and writes Staging objects. |
| Online migration service | Reads the source bucket, the Staging bucket, and the inventory bucket, and writes objects to the data storage bucket of the target table. |
| Operator account | Queries and updates the metadata location of the target table, and manages the unreferenced file cleanup setting of the target Table Bucket. |
For more information, see Version requirements and migration limits.
Variables
The examples in this topic use the following variables. Replace them with your actual values:
| Variable | Description | Example |
{catalog} | The name of the source Iceberg Catalog in Spark | prod |
{source_table} | The name of the source table, in the namespace.table format | sales.orders |
{source_prefix} | The common URI prefix used for path replacement. You can set this to oss:// when the migration spans multiple OSS buckets. | oss://example-source/warehouse/sales/orders/ |
{staging_location} | A dedicated, empty rewrite working directory | oss://example-staging/iceberg-migration/task-001/ |
{table_bucket_arn} | The ARN of the target Table Bucket | acs:osstables:cn-hangzhou:1234567890:bucket/example-table-bucket |
{target_region_id} | The OSS region ID of the region where the target table resides | oss-cn-hangzhou |
{target_namespace} | The target Namespace | sales |
{target_table} | The name of the target table | orders |
{target_prefix} | The warehouseLocation of the target table | oss://{table-id}--table-oss |
{latest_target_metadata} | The exact target URI of the latest metadata file in the copy plan | oss://{table-id}--table-oss/metadata/00010-xxx.metadata.json |
{latest_version_token} | The latest versionToken of the target table, queried immediately before you switch metadataLocation in Step 5 | Returned by get-table-metadata-location |
Procedure
Step 1: Confirm the source table and set source_prefix
Load the source table by using {catalog} and {source_table}. rewrite_table_path traverses the Iceberg reference graph starting from the metadataLocation in the catalog, so you do not need to register objects manually.
{source_prefix} is used only to match and replace source object URIs. It is not a condition for locating the table. Set it to the longest common prefix that covers all reachable source URIs:
If the metadata and the data are in the same table directory, set this variable to the table directory, for example,
oss://example-source/warehouse/sales/orders/.If the metadata and the data are in different directories of the same bucket, set this variable to the bucket root prefix, for example,
oss://example-source/.If the metadata and the data are spread across multiple OSS buckets, set this variable to
If the run reports that an object is not undeross://. In this case, the source bucket name is retained in the target object key.{source_prefix}, the current prefix is too narrow. Broaden the common prefix and run the procedure again with a new, empty Staging path.
Step 2: Get the target data storage location
Run the following command to query the metadata location, data storage location, and version token of the target table.
ossutil tables-api get-table-metadata-location \
--table-bucket-arn '{table_bucket_arn}' \
--namespace '{target_namespace}' \
--name '{target_table}'The command returns the following fields:
warehouseLocation: the data storage location of the target table. Use this value as{target_prefix}.metadataLocation: the metadata location before the switch, used for change auditing and rollback design.
To create the destination data address of the online migration service for Step 4, prepare the following two values:versionToken: the current version token. Record it at this point, but do not reuse it for the final switch.
Bucket name: extracted from
warehouseLocation. For example, the bucket name ofoss://example--table-ossisexample--table-oss.Region: the OSS region ID of the region where the target table resides, recorded as
{target_region_id}, for example,oss-cn-hangzhou(the region where the Table Bucket resides).
Step 3: Run rewrite_table_path
Run rewrite_table_path in Spark SQL to rewrite the paths in the Iceberg metadata and generate a copy plan:
rewrite_table_path supports source storage such as OSS, S3, Azure Blob, GCS, and HDFS. Set source_prefix based on the source scheme.
If the Position Delete File or Deletion Vector objects in the source table have already been removed by compaction and then deleted from object storage by expire_snapshots, rewrite_table_path reports NotFoundException or NoSuchKey. Do not modify the Iceberg metadata manually or create empty objects as a workaround. For the root cause and the resolution, see Limits.
CALL {catalog}.system.rewrite_table_path(
table => '{source_table}',
source_prefix => '{source_prefix}',
target_prefix => '{target_prefix}',
staging_location => '{staging_location}'
);The following table describes the parameters:
| Parameter | Required | Description |
table | Yes | The name of the source Iceberg table. |
source_prefix | Yes | The source URI prefix to match and replace. It must cover all reachable source URIs. |
target_prefix | Yes | The warehouseLocation of the target table. |
staging_location | No | The directory that stores the rewritten Metadata JSON, Manifest, and other files. For a production migration, explicitly specify a dedicated, empty directory. |
start_version | No | The name of the Metadata JSON file at which an incremental rewrite starts. This version is excluded from the copy scope. |
end_version | No | The name of the Metadata JSON file at which an incremental rewrite ends. This version is included in the copy scope. If you do not set this parameter, the current version is used. |
To catch up with later changes while the source table continues to accept writes, use start_version and end_version to generate another incremental copy plan and repeat the procedure in this topic. After the call succeeds, the following fields are returned:
| Response field | Description |
latest_version | The name of the latest Metadata JSON file rewritten in this call. This is not an absolute path. |
file_list_location | The complete URI of the copy plan CSV file. |
Read the file that file_list_location points to and confirm that the content is not empty. Each line is in the following format:
source URI,target URIsource may point to an original object in the source table or to a rewritten object in staging. target is the storage location of the object after migration. Both types of objects are in the same list, and you do not need to split them.
In the copy plan, find the Metadata JSON record whose file name is equal to latest_version and save the target URI of that record. Step 5 uses this URI to update the metadataLocation of the target table.
A successful return from rewrite_table_path only means that the copy plan has been generated. It does not mean that the objects have been copied to the target. Do not manually delete, merge, or modify records in the copy plan.
Step 4: Run the migration
In this step, you use the general inventory feature of the Alibaba Cloud online migration service to copy the source objects in the copy plan to the data storage location in OSS Tables. The step has three stages, and you must complete them in order.
Prepare the inventory
The copy plan generated by rewrite_table_path records the source URI and the target URI of each object. Convert the copy plan into the CSV file and manifest.json that the online migration service requires: convert the source URI into Key and the target URI into DestKey. For details about the inventory format and how to upload it, see Inventory requirements.
The following is a simple example. Assume that all source objects are in the example-source bucket, all target objects are in the example--table-oss bucket, and the local copy plan file file-list.csv contains the following content:
oss://example-source/warehouse/orders/data/00001.parquet,oss://example--table-oss/data/00001.parquet
oss://example-source/warehouse/orders/metadata/00010.metadata.json,oss://example--table-oss/metadata/00010.metadata.jsonSave the following script as build_inventory.py:
#!/usr/bin/env python3
import csv
import gzip
import hashlib
import json
from pathlib import Path
from urllib.parse import quote
SOURCE_BUCKET = "example-source"
TARGET_BUCKET = "example--table-oss"
INPUT = Path("file-list.csv")
CSV_FILE = Path("objects.csv")
GZIP_FILE = Path("objects.csv.gz")
MANIFEST_FILE = Path("manifest.json")
CSV_OBJECT_KEY = "iceberg-migration/task-001/objects.csv.gz"
def object_key(uri, bucket):
prefix = f"oss://{bucket}/"
if not uri.startswith(prefix):
raise ValueError(f"unexpected URI: {uri}")
return uri[len(prefix):]
seen = set()
with INPUT.open(newline="", encoding="utf-8") as src, \
CSV_FILE.open("w", newline="", encoding="utf-8") as dst:
writer = csv.writer(dst, lineterminator="\n")
for row in csv.reader(src):
if len(row) != 2:
raise ValueError(f"invalid row: {row}")
source_key = object_key(row[0], SOURCE_BUCKET)
target_key = object_key(row[1], TARGET_BUCKET)
if target_key in seen:
raise ValueError(f"duplicate target key: {target_key}")
seen.add(target_key)
writer.writerow([quote(source_key, safe=""), quote(target_key, safe="")])
with CSV_FILE.open("rb") as src, gzip.open(GZIP_FILE, "wb") as dst:
dst.write(src.read())
payload = GZIP_FILE.read_bytes()
manifest = {
"fileFormat": "CSV",
"fileSchema": "Key,DestKey",
"files": [{
"key": CSV_OBJECT_KEY,
"MD5checksum": hashlib.md5(payload).hexdigest(),
"size": len(payload),
}],
"mgwInventoryVersion": "1.0",
}
MANIFEST_FILE.write_text(
json.dumps(manifest, indent=2) + "\n", encoding="utf-8"
)In the directory where file-list.csv is located, run:
python3 build_inventory.pyThe script reads file-list.csv from the local file system and generates objects.csv, objects.csv.gz, and manifest.json in the current directory. If file-list.csv covers multiple source buckets, split the file by source bucket first and then generate an inventory for each bucket.
Create and run the migration job
Create the source data address, the destination data address, and the migration job in the console of the online migration service, and then start the job. For instructions, see Create a migration job. Note the following items:
For the source data address, select Inventory and enter the location of the
manifest.jsongenerated earlier.For the destination data address, select the data bucket that corresponds to OSS Tables. The
DestKeyvalues in the inventory are already complete target object keys, so do not configure an additional prefix for the destination address.One source address corresponds to one source bucket. If multiple source buckets exist, create separate addresses and jobs. Confirm that all jobs are complete before you proceed with the subsequent steps.
Verify the migrated objects
After the migration job is complete, confirm the following items:
The job status is successful,
FailedCount = 0, andCopiedCount = TotalCount.Every target URI in the copy plan exists, and the number of target objects matches the copy plan.
The sizes of the source objects and the target objects are the same, and content hashes are compared by sampling based on your business requirements.
If multiple source buckets are involved, all migration jobs are complete.
You can update themetadataLocationof the target table only after the verification passes. If you find inconsistencies, handle the failed objects based on the migration report and verify again.
Step 5: Update metadataLocation
Query the target table again:
ossutil tables-api get-table-metadata-location \ --table-bucket-arn '{table_bucket_arn}' \ --namespace '{target_namespace}' \ --name '{target_table}'Confirm that the current
metadataLocationis still the expected old value, and get the latestversionToken. Record both values, because they are the inputs to the rollback plan that you use if validation fails after the switch.In the copy plan, find the Metadata JSON mapping that corresponds to
latest_versionand use the target URI of that row as{latest_target_metadata}. Do not derive or concatenate the target path based only on the source file name.Use the latest
versionTokenas{latest_version_token}to atomically switch themetadataLocationof the target table:ossutil tables-api update-table-metadata-location \ --table-bucket-arn '{table_bucket_arn}' \ --namespace '{target_namespace}' \ --name '{target_table}' \ --metadata-location '{latest_target_metadata}' \ --version-token '{latest_version_token}'Run the query command again to confirm that
metadataLocationhas been updated to{latest_target_metadata}, and record the newversionTokenthat is returned.
Step 6: Validate the semantics of the Iceberg table
After you update metadataLocation, use the existing Iceberg query engine of your business to read the target table. At a minimum, confirm the following items:
The table loads normally, and the schema and partition information are as expected.
The total row count, the key business queries, and the results of representative partitions match the source table.
If your business uses historical Snapshot, Tag, Branch, or Delete File objects, spot-check that the corresponding query results are correct.
Resume writes or switch service traffic only after all validations pass. If a validation fails, restore the originalmetadataLocationaccording to the rollback plan.
Rollback and cleanup
If validation fails after the switch, immediately stop downstream writes and carry out the rollback plan that you prepared in advance:
Downstream services continue to use the source table, which avoids writing new snapshots to the abnormal target table.
If the
metadataLocationbefore the switch points to usable metadata, get the latestversionTokenagain and restore the old pointer through a conditional update.If the target table has no recoverable
metadataLocationbefore the switch, take the target table offline and recover it by using the rebuild plan that you validated in advance.Retain the target objects, migration reports, and logs for troubleshooting. Do not delete objects directly during incident handling.
After the switch is complete, set an observation period. When the observation period ends and you confirm that no rollback is needed, turn on unreferenced file cleanup for the target Table Bucket again.
Clean up the source objects, Staging, copy plans, inventories, and validation reports through a separate change. Do not combine the cleanup operations with the migration job.
Limits
Version requirements
Use the latest official release of Apache Iceberg (1.11.0 at the time of writing). The Iceberg Spark Runtime must match the Spark and Scala versions that you actually use. For example, the 1.11.0 runtime for Spark 3.5 and Scala 2.12 is:
org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0Known issue in Iceberg 1.11.0
Apache Iceberg 1.11.0 has a known issue related to rewrite_table_path. If the Position Delete File or Deletion Vector objects in the source table have been removed by compaction, and the corresponding files are then deleted from object storage by expire_snapshots, a full or incremental rewrite_table_path run may report NotFoundException or NoSuchKey. For more information, see GitHub.
Complete the migration before you run compaction and snapshot expiration, as required in Prerequisites and before Step 3. If the source table has already undergone these maintenance operations and this error occurs when you run rewrite_table_path, do not manually modify the Iceberg metadata or create empty objects as a workaround. Upgrade to an official release that has fixed this issue. If no official fix is available yet, contact Alibaba Cloud technical support.
Other migration limits
Source table paths — The metadata and the data can be in different directories or different buckets, but all source URIs that need to be rewritten must share the same
{source_prefix}. When the migration spans buckets, you can use a scheme prefix such asoss://. The bucket name becomes part of the target object key, and migration jobs are split by the actual source bucket.Writes during migration — After the copy plan is generated, new commits later than
end_versionare not automatically added to the plan. You can catch up through multiple rounds of(start_version, end_version]incremental migrations, but you still need to briefly block new commits before the final switch and pause the maintenance jobs that delete old Metadata JSON, Snapshot, Data File, Delete File, and Deletion Vector objects.Migration region — The online migration job must reside in the same region as the target table. For the destination address, enter the region ID of the target table.