Migrate Hive data to OSS Tables via DataWorks
This topic describes how to migrate Hive table data to OSS Tables by using the whole-database offline synchronization feature of DataWorks Data Integration. It applies to scenarios in which the source tables are Hive tables with metadata managed by HMS (Hive Metastore).
Applicable scenarios
This solution applies to the following scenarios:
The source tables are Hive tables. The metadata of the tables is managed by HMS (Hive Metastore), and the data files are stored in HDFS or other storage systems.
The destination is a table in OSS Tables.
You need to migrate all tables of a Hive database to OSS Tables in batches.
Solution architecture
Hive table data is migrated to OSS Tables by using the whole-database offline synchronization feature of DataWorks Data Integration. The data flow is as follows:
Role | Description |
Source | Hive tables. The metadata is registered in HMS, and the data files are stored in HDFS or other storage systems. DataWorks connects to HMS by using a Hive data source to obtain the table schemas and file locations, and then reads the data. |
Executor | DataWorks Serverless resource group. Reads data from Hive and writes the data to OSS Tables. |
Destination | OSS Tables. DataWorks connects to OSS Tables by using an OSS Tables data source and writes data to tables in OSS Tables. |
The migration reads and rewrites data: DataWorks reads the underlying data files of Hive based on the paths provided by HMS, parses the files into rows, and writes the rows in the OSS Tables format. This is not a file-level copy.
Limits
Item | Description |
Resource group | Only Serverless resource groups are supported. |
Destination | OSS Tables can be used only as the destination for writes, not as a source. |
Source | Only Hive is supported as the source. |
Synchronization method | Only whole-database offline synchronization is supported. |
Migration process
Before you begin, make sure that OSS Tables is activated and a bucket and a namespace are created. If not, complete the preparations first. For more information, see OSS Tables quick start.
Perform the following steps:
Create a Hive data source in DataWorks to connect to the HMS of the source Hive. This data source is used to read the source table metadata and data files. For more information about the configuration, see Hive data source.
Create an OSS Tables data source in DataWorks as the destination for data writes. For more information about the configuration, see OSS Tables data source.
Configure a whole-database offline synchronization task. Select the Hive database as the source and OSS Tables as the destination. DataWorks automatically generates a synchronization subtask for each table in the database. For more information about the configuration, see Configure a whole-database offline synchronization task.
Start and run the synchronization task, and wait for the data to be written to OSS Tables.
After the migration is complete, check the OSS Tables console to confirm that the tables and data are written to the destination namespace.
Differences from Iceberg table migration
If the source table is already an Apache Iceberg table (stored in OSS, S3, Azure Blob, GCS, or HDFS), use Migrate a source Iceberg table to OSS Tables instead. That solution rewrites metadata paths, copies objects, and atomically switches the metadataLocation without rewriting data. The solution described in this topic applies only to scenarios in which the source tables are Hive tables (not Iceberg tables).