Migrate a source Iceberg table to OSS Tables

Updated at:

Apache Iceberg metadata records absolute object paths, so moving a table into OSS Tables requires rewriting those paths and copying the referenced objects before the target table can serve queries. This topic describes how to migrate a source Iceberg table to OSS Tables and atomically switch the target table to the rewritten metadata.

Applicable scenarios

This topic applies when the following conditions are met:

  • The source table is an Apache Iceberg table stored in Object Storage Service (OSS), S3, Azure Blob, GCS, a similar object storage service, or Hadoop Distributed File System (HDFS).

  • The target is a table in OSS Tables.

  • The objects to migrate can include Data File, Delete File, Deletion Vector, Manifest, historical Snapshot, Tag, Branch, Schema, Partition Spec, Sort Order, and Statistics objects.

How it works

The migration workflow is as follows:

Iceberg table in an object storage bucket
                │
                │ Read the metadata reference graph of the specified version range
                ▼
        rewrite_table_path
                │
                ├─ Staging: rewritten Metadata JSON, Manifest, Position Delete File, Deletion Vector
                └─ File list: source URI,target URI
                                  │
                                  │ Convert to Key,DestKey
                                  │ Split the file list by source bucket if it spans multiple buckets
                                  ▼
                       Online migration service
                                  │
                                  ▼
            Data storage location of the target table
                                  │
                                  │ Verify the file count, content, and metadata
                                  ▼
          Update metadataLocation with the latest versionToken
                                  │
                                  ▼
            Validate Iceberg semantics through the target table

rewrite_table_path processes the source table in the following order:

  • Locate the source table — Get the current metadataLocation from the source Iceberg Catalog based on the table parameter. source_prefix is used only for path replacement and does not take part in the lookup.

  • Traverse and rewrite paths — Starting from the Metadata JSON, traverse the Iceberg references within the migration scope, replace {source_prefix} with {target_prefix} in the absolute source paths, and write the rewritten objects to {staging_location}. Source objects are not modified during this process.

  • Generate the copy plan — Rewritten objects use Staging as their source, and objects that are not rewritten use the source table. Both are recorded in the same plan.

  • Copy to the target — The online migration service copies the objects to the data storage location of the target table according to the copy plan. rewrite_table_path itself does not copy objects or update metadataLocation.

    The following table describes how the objects involved in the rewrite and copy phases are processed:
Processing methodObject typeCopy source
Copied as-isData File, Equality Delete File, Statistics, and Partition StatisticsSource table objects
Copied after rewritingMetadata JSON, Manifest List, Manifest, Position Delete File, and V3 Deletion VectorRewritten objects in Staging
Note

Staging is a temporary rewrite directory, not a complete Iceberg table. Do not register Staging as an Iceberg table, and do not point the metadataLocation of the target table to Staging. Keep the Staging directory until the migration is complete and the observation period ends.

Prerequisites

Before you start the migration, complete the following preparations:

  • Activate OSS Tables, and create a Table Bucket, a Namespace, and a target table dedicated to this migration. The target table must not carry other production data.

  • Turn off unreferenced file cleanup for the target Table Bucket.

Important

Unreferenced file cleanup is a Table Bucket-level setting. Migrated objects are not yet referenced by the target table before metadataLocation is switched. If cleanup stays on, these objects may be marked and deleted during the migration. Before you start copying objects, go to the Data Maintenance page of the target Table Bucket, turn off Unreferenced File Cleanup, and keep it off until the Iceberg semantic validation and smoke test after the switch all pass.

  • Activate the online migration service and select a region that is the same as the region of the target table to run the migration job.

  • The source table is registered in an Iceberg Catalog that Spark can access, and the current metadataLocation can be loaded by table name. The catalog can be an existing Hive, REST, JDBC, or other compatible implementation used by your business. No specific catalog type is required.

  • Prepare the Java, Spark, Scala, and Apache Iceberg runtime environment as described in Version requirements and migration limits.

  • Prepare a dedicated, empty Staging prefix and an inventory storage location.

  • The source storage supports Spark FileIO access and reads by the online migration service. For third-party object storage, use least-privilege credentials and do not store AccessKey pairs in plaintext.

  • Prepare a rollback plan for the metadataLocation switch in Step 5, and validate a rebuild plan for the target table in case no recoverable metadataLocation exists before the switch. Step 2 and Step 5 return the metadataLocation and versionToken values that these plans require.

The permissions involved in the migration are as follows:

PrincipalRequired permissions
SparkReads source table objects, and reads and writes Staging objects.
Online migration serviceReads the source bucket, the Staging bucket, and the inventory bucket, and writes objects to the data storage bucket of the target table.
Operator accountQueries and updates the metadata location of the target table, and manages the unreferenced file cleanup setting of the target Table Bucket.

For more information, see Version requirements and migration limits.

Variables

The examples in this topic use the following variables. Replace them with your actual values:

VariableDescriptionExample
{catalog}The name of the source Iceberg Catalog in Sparkprod
{source_table}The name of the source table, in the namespace.table formatsales.orders
{source_prefix}The common URI prefix used for path replacement. You can set this to oss:// when the migration spans multiple OSS buckets.oss://example-source/warehouse/sales/orders/
{staging_location}A dedicated, empty rewrite working directoryoss://example-staging/iceberg-migration/task-001/
{table_bucket_arn}The ARN of the target Table Bucketacs:osstables:cn-hangzhou:1234567890:bucket/example-table-bucket
{target_region_id}The OSS region ID of the region where the target table residesoss-cn-hangzhou
{target_namespace}The target Namespacesales
{target_table}The name of the target tableorders
{target_prefix}The warehouseLocation of the target tableoss://{table-id}--table-oss
{latest_target_metadata}The exact target URI of the latest metadata file in the copy planoss://{table-id}--table-oss/metadata/00010-xxx.metadata.json
{latest_version_token}The latest versionToken of the target table, queried immediately before you switch metadataLocation in Step 5Returned by get-table-metadata-location

Procedure

Step 1: Confirm the source table and set source_prefix

Load the source table by using {catalog} and {source_table}. rewrite_table_path traverses the Iceberg reference graph starting from the metadataLocation in the catalog, so you do not need to register objects manually.

{source_prefix} is used only to match and replace source object URIs. It is not a condition for locating the table. Set it to the longest common prefix that covers all reachable source URIs:

  • If the metadata and the data are in the same table directory, set this variable to the table directory, for example, oss://example-source/warehouse/sales/orders/.

  • If the metadata and the data are in different directories of the same bucket, set this variable to the bucket root prefix, for example, oss://example-source/.

  • If the metadata and the data are spread across multiple OSS buckets, set this variable to oss://. In this case, the source bucket name is retained in the target object key.

    If the run reports that an object is not under {source_prefix}, the current prefix is too narrow. Broaden the common prefix and run the procedure again with a new, empty Staging path.

Step 2: Get the target data storage location

Run the following command to query the metadata location, data storage location, and version token of the target table.

ossutil tables-api get-table-metadata-location \
  --table-bucket-arn '{table_bucket_arn}' \
  --namespace '{target_namespace}' \
  --name '{target_table}'

The command returns the following fields:

  • warehouseLocation: the data storage location of the target table. Use this value as {target_prefix}.

  • metadataLocation: the metadata location before the switch, used for change auditing and rollback design.

  • versionToken: the current version token. Record it at this point, but do not reuse it for the final switch.

    To create the destination data address of the online migration service for Step 4, prepare the following two values:
  • Bucket name: extracted from warehouseLocation. For example, the bucket name of oss://example--table-oss is example--table-oss.

  • Region: the OSS region ID of the region where the target table resides, recorded as {target_region_id}, for example, oss-cn-hangzhou (the region where the Table Bucket resides).

Step 3: Run rewrite_table_path

Run rewrite_table_path in Spark SQL to rewrite the paths in the Iceberg metadata and generate a copy plan:

rewrite_table_path supports source storage such as OSS, S3, Azure Blob, GCS, and HDFS. Set source_prefix based on the source scheme.

Important

If the Position Delete File or Deletion Vector objects in the source table have already been removed by compaction and then deleted from object storage by expire_snapshots, rewrite_table_path reports NotFoundException or NoSuchKey. Do not modify the Iceberg metadata manually or create empty objects as a workaround. For the root cause and the resolution, see Limits.

CALL {catalog}.system.rewrite_table_path(
  table => '{source_table}',
  source_prefix => '{source_prefix}',
  target_prefix => '{target_prefix}',
  staging_location => '{staging_location}'
);

The following table describes the parameters:

ParameterRequiredDescription
tableYesThe name of the source Iceberg table.
source_prefixYesThe source URI prefix to match and replace. It must cover all reachable source URIs.
target_prefixYesThe warehouseLocation of the target table.
staging_locationNoThe directory that stores the rewritten Metadata JSON, Manifest, and other files. For a production migration, explicitly specify a dedicated, empty directory.
start_versionNoThe name of the Metadata JSON file at which an incremental rewrite starts. This version is excluded from the copy scope.
end_versionNoThe name of the Metadata JSON file at which an incremental rewrite ends. This version is included in the copy scope. If you do not set this parameter, the current version is used.

To catch up with later changes while the source table continues to accept writes, use start_version and end_version to generate another incremental copy plan and repeat the procedure in this topic. After the call succeeds, the following fields are returned:

Response fieldDescription
latest_versionThe name of the latest Metadata JSON file rewritten in this call. This is not an absolute path.
file_list_locationThe complete URI of the copy plan CSV file.

Read the file that file_list_location points to and confirm that the content is not empty. Each line is in the following format:

source URI,target URI

source may point to an original object in the source table or to a rewritten object in staging. target is the storage location of the object after migration. Both types of objects are in the same list, and you do not need to split them.

In the copy plan, find the Metadata JSON record whose file name is equal to latest_version and save the target URI of that record. Step 5 uses this URI to update the metadataLocation of the target table.

A successful return from rewrite_table_path only means that the copy plan has been generated. It does not mean that the objects have been copied to the target. Do not manually delete, merge, or modify records in the copy plan.

Step 4: Run the migration

In this step, you use the general inventory feature of the Alibaba Cloud online migration service to copy the source objects in the copy plan to the data storage location in OSS Tables. The step has three stages, and you must complete them in order.

Prepare the inventory

The copy plan generated by rewrite_table_path records the source URI and the target URI of each object. Convert the copy plan into the CSV file and manifest.json that the online migration service requires: convert the source URI into Key and the target URI into DestKey. For details about the inventory format and how to upload it, see Inventory requirements.

The following is a simple example. Assume that all source objects are in the example-source bucket, all target objects are in the example--table-oss bucket, and the local copy plan file file-list.csv contains the following content:

oss://example-source/warehouse/orders/data/00001.parquet,oss://example--table-oss/data/00001.parquet
oss://example-source/warehouse/orders/metadata/00010.metadata.json,oss://example--table-oss/metadata/00010.metadata.json

Save the following script as build_inventory.py:

#!/usr/bin/env python3
import csv
import gzip
import hashlib
import json
from pathlib import Path
from urllib.parse import quote

SOURCE_BUCKET = "example-source"
TARGET_BUCKET = "example--table-oss"
INPUT = Path("file-list.csv")
CSV_FILE = Path("objects.csv")
GZIP_FILE = Path("objects.csv.gz")
MANIFEST_FILE = Path("manifest.json")
CSV_OBJECT_KEY = "iceberg-migration/task-001/objects.csv.gz"

def object_key(uri, bucket):
    prefix = f"oss://{bucket}/"
    if not uri.startswith(prefix):
        raise ValueError(f"unexpected URI: {uri}")
    return uri[len(prefix):]

seen = set()
with INPUT.open(newline="", encoding="utf-8") as src, \
        CSV_FILE.open("w", newline="", encoding="utf-8") as dst:
    writer = csv.writer(dst, lineterminator="\n")
    for row in csv.reader(src):
        if len(row) != 2:
            raise ValueError(f"invalid row: {row}")
        source_key = object_key(row[0], SOURCE_BUCKET)
        target_key = object_key(row[1], TARGET_BUCKET)
        if target_key in seen:
            raise ValueError(f"duplicate target key: {target_key}")
        seen.add(target_key)
        writer.writerow([quote(source_key, safe=""), quote(target_key, safe="")])

with CSV_FILE.open("rb") as src, gzip.open(GZIP_FILE, "wb") as dst:
    dst.write(src.read())

payload = GZIP_FILE.read_bytes()
manifest = {
    "fileFormat": "CSV",
    "fileSchema": "Key,DestKey",
    "files": [{
        "key": CSV_OBJECT_KEY,
        "MD5checksum": hashlib.md5(payload).hexdigest(),
        "size": len(payload),
    }],
    "mgwInventoryVersion": "1.0",
}
MANIFEST_FILE.write_text(
    json.dumps(manifest, indent=2) + "\n", encoding="utf-8"
)

In the directory where file-list.csv is located, run:

python3 build_inventory.py

The script reads file-list.csv from the local file system and generates objects.csv, objects.csv.gz, and manifest.json in the current directory. If file-list.csv covers multiple source buckets, split the file by source bucket first and then generate an inventory for each bucket.

Create and run the migration job

Create the source data address, the destination data address, and the migration job in the console of the online migration service, and then start the job. For instructions, see Create a migration job. Note the following items:

  • For the source data address, select Inventory and enter the location of the manifest.json generated earlier.

  • For the destination data address, select the data bucket that corresponds to OSS Tables. The DestKey values in the inventory are already complete target object keys, so do not configure an additional prefix for the destination address.

  • One source address corresponds to one source bucket. If multiple source buckets exist, create separate addresses and jobs. Confirm that all jobs are complete before you proceed with the subsequent steps.

Verify the migrated objects

After the migration job is complete, confirm the following items:

  • The job status is successful, FailedCount = 0, and CopiedCount = TotalCount.

  • Every target URI in the copy plan exists, and the number of target objects matches the copy plan.

  • The sizes of the source objects and the target objects are the same, and content hashes are compared by sampling based on your business requirements.

  • If multiple source buckets are involved, all migration jobs are complete.

    You can update the metadataLocation of the target table only after the verification passes. If you find inconsistencies, handle the failed objects based on the migration report and verify again.

Step 5: Update metadataLocation

  1. Query the target table again:

    ossutil tables-api get-table-metadata-location \
      --table-bucket-arn '{table_bucket_arn}' \
      --namespace '{target_namespace}' \
      --name '{target_table}'
  2. Confirm that the current metadataLocation is still the expected old value, and get the latest versionToken. Record both values, because they are the inputs to the rollback plan that you use if validation fails after the switch.

  3. In the copy plan, find the Metadata JSON mapping that corresponds to latest_version and use the target URI of that row as {latest_target_metadata}. Do not derive or concatenate the target path based only on the source file name.

  4. Use the latest versionToken as {latest_version_token} to atomically switch the metadataLocation of the target table:

    ossutil tables-api update-table-metadata-location \
      --table-bucket-arn '{table_bucket_arn}' \
      --namespace '{target_namespace}' \
      --name '{target_table}' \
      --metadata-location '{latest_target_metadata}' \
      --version-token '{latest_version_token}'
  5. Run the query command again to confirm that metadataLocation has been updated to {latest_target_metadata}, and record the new versionToken that is returned.

Step 6: Validate the semantics of the Iceberg table

After you update metadataLocation, use the existing Iceberg query engine of your business to read the target table. At a minimum, confirm the following items:

  • The table loads normally, and the schema and partition information are as expected.

  • The total row count, the key business queries, and the results of representative partitions match the source table.

  • If your business uses historical Snapshot, Tag, Branch, or Delete File objects, spot-check that the corresponding query results are correct.

    Resume writes or switch service traffic only after all validations pass. If a validation fails, restore the original metadataLocation according to the rollback plan.

Rollback and cleanup

If validation fails after the switch, immediately stop downstream writes and carry out the rollback plan that you prepared in advance:

  • Downstream services continue to use the source table, which avoids writing new snapshots to the abnormal target table.

  • If the metadataLocation before the switch points to usable metadata, get the latest versionToken again and restore the old pointer through a conditional update.

  • If the target table has no recoverable metadataLocation before the switch, take the target table offline and recover it by using the rebuild plan that you validated in advance.

  • Retain the target objects, migration reports, and logs for troubleshooting. Do not delete objects directly during incident handling.

    After the switch is complete, set an observation period. When the observation period ends and you confirm that no rollback is needed, turn on unreferenced file cleanup for the target Table Bucket again.

Clean up the source objects, Staging, copy plans, inventories, and validation reports through a separate change. Do not combine the cleanup operations with the migration job.

Limits

Version requirements

Use the latest official release of Apache Iceberg (1.11.0 at the time of writing). The Iceberg Spark Runtime must match the Spark and Scala versions that you actually use. For example, the 1.11.0 runtime for Spark 3.5 and Scala 2.12 is:

org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0

Known issue in Iceberg 1.11.0

Apache Iceberg 1.11.0 has a known issue related to rewrite_table_path. If the Position Delete File or Deletion Vector objects in the source table have been removed by compaction, and the corresponding files are then deleted from object storage by expire_snapshots, a full or incremental rewrite_table_path run may report NotFoundException or NoSuchKey. For more information, see GitHub.

Complete the migration before you run compaction and snapshot expiration, as required in Prerequisites and before Step 3. If the source table has already undergone these maintenance operations and this error occurs when you run rewrite_table_path, do not manually modify the Iceberg metadata or create empty objects as a workaround. Upgrade to an official release that has fixed this issue. If no official fix is available yet, contact Alibaba Cloud technical support.

Other migration limits

  • Source table paths — The metadata and the data can be in different directories or different buckets, but all source URIs that need to be rewritten must share the same {source_prefix}. When the migration spans buckets, you can use a scheme prefix such as oss://. The bucket name becomes part of the target object key, and migration jobs are split by the actual source bucket.

  • Writes during migration — After the copy plan is generated, new commits later than end_version are not automatically added to the plan. You can catch up through multiple rounds of (start_version, end_version] incremental migrations, but you still need to briefly block new commits before the final switch and pause the maintenance jobs that delete old Metadata JSON, Snapshot, Data File, Delete File, and Deletion Vector objects.

  • Migration region — The online migration job must reside in the same region as the target table. For the destination address, enter the region ID of the target table.