Iceberg SQL connector
This topic describes how to use the Iceberg connector in SQL jobs.
Background information
Apache Iceberg is an open data lake table format. You can use Apache Iceberg to quickly build your own data lake storage service on HDFS or cloud OSS, and use computing engines from the open-source big data ecosystem, such as Flink, Spark, Hive, and Presto, to perform data lake analysis.
|
Category |
Description |
|
Supported types |
Source table, sink table, and data ingestion sink |
|
Running modes |
Batch mode and streaming mode |
|
Data format |
Not applicable |
|
Specific metrics |
None |
|
API types |
SQL, data ingestion YAML job |
|
Supports updating or deleting data in sink tables |
Yes |
Features
The Apache Iceberg connector provides the following features:
-
Builds a low-cost, lightweight data lake storage service based on HDFS or object storage.
-
Provides full ACID semantics.
-
Supports time travel queries to access historical data versions.
-
Supports efficient data filtering.
-
Supports schema evolution.
-
Supports partition evolution.
Use Flink's fault tolerance and stream processing capabilities to import large volumes of log data into an Apache Iceberg data lake in real time. You can then use Flink or other analysis engines to extract value from the data.
Limits
-
The Apache Iceberg connector is supported only in Realtime Compute for Apache Flink with Ververica Runtime (VVR) 4.0.8 or later. The Apache Iceberg connector must be used with a Data Lake Formation (DLF) catalog. For more information, see Manage DLF-Legacy catalogs.
-
The Apache Iceberg connector supports Apache Iceberg v1 and v2 table formats. For more information, see the Iceberg Table Spec.
NoteOnly Realtime Compute for Apache Flink that uses VVR 8.0.7 or later supports the v2 table format.
-
In streaming read mode, you can use only append-only Iceberg tables as source tables.
Syntax
CREATE TABLE iceberg_table (
id BIGINT,
data STRING
PRIMARY KEY(`id`) NOT ENFORCED
)
PARTITIONED BY (data)
WITH (
'connector' = 'iceberg',
...
);
WITH options
Common options
|
Parameter |
Description |
Type |
Required |
Default |
Remarks |
|
connector |
The type of the connector. |
String |
Yes |
None |
The value must be |
|
catalog-name |
The name of the catalog. |
String |
Yes |
None |
Enter a custom name in English. |
|
catalog-database |
The name of the database. |
String |
Yes |
default |
The name of your database in Data Lake Formation (DLF), such as dlf_db. Note
If you do not have a Data Lake Formation (DLF) database, create one first. For more information, see Databases, tables, and functions. |
|
io-impl |
The implementation class for the distributed file system. |
String |
Yes |
None |
The value must be |
|
oss.endpoint |
The endpoint of Alibaba Cloud Object Storage Service (OSS). |
String |
No |
None |
For more information, see Regions and Endpoints. Note
|
|
Your Alibaba Cloud account AccessKey ID. |
String |
Yes |
None |
For more information, see How do I view the AccessKey ID and AccessKey secret? Important
To prevent your AccessKey information from being leaked, we recommend that you use variables to specify the AccessKey values. For more information, see Project variables. |
|
Your Alibaba Cloud account AccessKey secret. |
String |
Yes |
None |
|
|
catalog-impl |
The class name of the catalog. |
String |
Yes |
None |
The value must be |
|
warehouse |
The OSS path to store table data. |
String |
Yes |
None |
None |
|
dlf.catalog-id |
Your Alibaba Cloud account ID. |
String |
Yes |
None |
You can obtain the account ID on the User Information page. |
|
dlf.endpoint |
The endpoint of Data Lake Formation (DLF). |
String |
Yes |
None |
For more information, see Available regions and endpoints. Note
|
|
dlf.region-id |
The region of Data Lake Formation (DLF). |
String |
Yes |
None |
For more information, see Available regions and endpoints. Note
Make sure that the region is the same as the region specified for the dlf.endpoint parameter. |
Sink-only options
|
Parameter |
Description |
Type |
Required |
Default |
Remarks |
|
write.operation |
The write operation mode. |
String |
No |
upsert |
|
|
hive_sync.enable |
Specifies whether to synchronize metadata to Hive. |
Boolean |
No |
false |
Valid values:
|
|
hive_sync.mode |
The Hive metadata synchronization mode. |
String |
No |
hms |
|
|
hive_sync.db |
The name of the Hive database to which data is synchronized. |
String |
No |
The database name of the current table in the catalog. |
None |
|
hive_sync.table |
The name of the Hive table to which data is synchronized. |
String |
No |
The name of the current table. |
None |
|
dlf.catalog.region |
The region of Data Lake Formation (DLF). |
String |
No |
None |
For more information, see Available regions and endpoints. Note
|
|
dlf.catalog.endpoint |
The endpoint of Data Lake Formation (DLF). |
String |
No |
None |
For more information, see Available regions and endpoints. Note
|
Data type mapping
|
Iceberg type |
Flink type |
|
BOOLEAN |
BOOLEAN |
|
INT |
INT |
|
LONG |
BIGINT |
|
FLOAT |
FLOAT |
|
DOUBLE |
DOUBLE |
|
DECIMAL(P,S) |
DECIMAL(P,S) |
|
DATE |
DATE |
|
TIME |
TIME Note
Iceberg timestamps have microsecond precision, whereas Flink timestamps have millisecond precision. When you use Flink to read data from Iceberg, Flink converts the time precision to milliseconds. |
|
TIMESTAMP |
TIMESTAMP |
|
TIMESTAMPTZ |
TIMESTAMP_LTZ |
|
STRING |
STRING |
|
FIXED(L) |
BYTES |
|
BINARY |
VARBINARY |
|
STRUCT<...> |
ROW |
|
LIST<E> |
LIST |
|
MAP<K,V> |
MAP |
Examples
Ensure that you have an OSS bucket and a Data Lake Formation (DLF) database. For more information, see Create a bucket and Databases, tables, and functions.
When you specify a path for your Data Lake Formation (DLF) database, we recommend that you follow the ${warehouse}/${database_name}.db format. For example, if the warehouse address is oss://iceberg-test/warehouse and the database name is dlf_db, set the OSS path of dlf_db to oss://iceberg-test/warehouse/dlf_db.db.
Sink table example
This example uses the Datagen connector to generate random streaming data and write the data to an Iceberg table.
CREATE TEMPORARY TABLE datagen(
id BIGINT,
data STRING
) WITH (
'connector' = 'datagen'
);
CREATE TEMPORARY TABLE dlf_iceberg (
id BIGINT,
data STRING
) WITH (
'connector' = 'iceberg',
'catalog-name' = '<yourCatalogName>',
'catalog-database' = '<yourDatabaseName>',
'io-impl' = 'org.apache.iceberg.aliyun.oss.OSSFileIO',
'oss.endpoint' = '<yourOSSEndpoint>',
'access.key.id' = '${secret_values.ak_id}',
'access.key.secret' = '${secret_values.ak_secret}',
'catalog-impl' = 'org.apache.iceberg.aliyun.dlf.DlfCatalog',
'warehouse' = '<yourOSSWarehousePath>',
'dlf.catalog-id' = '<yourCatalogId>',
'dlf.endpoint' = '<yourDLFEndpoint>',
'dlf.region-id' = '<yourDLFRegionId>'
);
INSERT INTO dlf_iceberg SELECT * FROM datagen;
Source table examples
-
Use a DLF catalog to write data from an Iceberg source table to an Iceberg sink table.
CREATE TEMPORARY TABLE src_iceberg ( id BIGINT, data STRING ) WITH ( 'connector' = 'iceberg', 'catalog-name' = '<yourCatalogName>', 'catalog-database' = '<yourDatabaseName>', 'io-impl' = 'org.apache.iceberg.aliyun.oss.OSSFileIO', 'oss.endpoint' = '<yourOSSEndpoint>', 'access.key.id' = '${secret_values.ak_id}', 'access.key.secret' = '${secret_values.ak_secret}', 'catalog-impl' = 'org.apache.iceberg.aliyun.dlf.DlfCatalog', 'warehouse' = '<yourOSSWarehousePath>', 'dlf.catalog-id' = '<yourCatalogId>', 'dlf.endpoint' = '<yourDLFEndpoint>', 'dlf.region-id' = '<yourDLFRegionId>' ); CREATE TEMPORARY TABLE dst_iceberg ( id BIGINT, data STRING ) WITH ( 'connector' = 'iceberg', 'catalog-name' = '<yourCatalogName>', 'catalog-database' = '<yourDatabaseName>', 'io-impl' = 'org.apache.iceberg.aliyun.oss.OSSFileIO', 'oss.endpoint' = '<yourOSSEndpoint>', 'access.key.id' = '${secret_values.ak_id}', 'access.key.secret' = '${secret_values.ak_secret}', 'catalog-impl' = 'org.apache.iceberg.aliyun.dlf.DlfCatalog', 'warehouse' = '<yourOSSWarehousePath>', 'dlf.catalog-id' = '<yourCatalogId>', 'dlf.endpoint' = '<yourDLFEndpoint>', 'dlf.region-id' = '<yourDLFRegionId>' ); BEGIN STATEMENT SET; INSERT INTO src_iceberg VALUES (1, 'AAA'), (2, 'BBB'), (3, 'CCC'), (4, 'DDD'), (5, 'EEE'); INSERT INTO dst_iceberg SELECT * FROM src_iceberg; END;
Related documentation
For more information about connectors that are supported by Flink, see Supported connectors.