Flink SQL task development methods

Dataphin computing source physical table development method

This method lets you directly access physical tables in a computing source by specifying the table in the format project_name.table_name. Cross-project access to physical tables attached to other projects is also supported.

Example

To insert data from the test physical table in the computing source of the example project into the test_demo physical table, refer to the following sample code:

insert into test_demo select id,name from example.test;

Dataphin data source table development method

This method lets you directly access tables from a data source created in Dataphin. You must first configure the data source encoding. For more information, see Data Source Management.

After you configure the data source encoding, reference tables in a Flink SQL task using the format data_source_encoding.table or data_source_encoding.schema.table. To automatically access the data source for the current environment, use the format ${data_source_encoding}.table or ${data_source_encoding}.schema.table.

• For data sources that do not support schemas, such as MySQL, StarRocks, and SelectDB, access physical tables using the format data_source_encoding.table_name.

• For data sources that support schemas, such as Hologres and GaussDB (DWS), access physical tables using the format data_source_encoding.schema_name.table_name.

Example

To insert data from the demo_mysql physical table of a MySQL data source with the encoding ds_demo_mysql into the test_demo physical table, refer to the following sample code.

insert into test_demo select id,name from ds_demo_mysql.demo_mysql;

To insert data from the demo_hologres physical table of a Hologres data source with the encoding ds_demo_hologres and schema name hologres into the test_demo physical table, refer to the following sample code.

insert into test_demo select id,name from ds_demo_hologres.hologres.demo_hologres;

Dataphin meta table development method

In Dataphin, a meta table is a cross-storage logical table managed by Data Management that provides a higher level of abstraction than native DDL and DML development. You can create and manage input, output, and dimension tables as meta tables, which lets you define a table once and reference it across tasks without writing repetitive DDL statements or performing complex mappings. Meta tables also prevent the exposure of sensitive information that can occur when writing native Flink DDL statements directly.

Example

To create the demo01 and demo02 data tables and insert data from demo01 into demo02, follow these steps.

1. Use the Dataphin meta table feature to create the demo01 and demo02 data tables. For more information, see Create and manage meta tables.

2. Write an insert statement in the Flink SQL task. The following is a sample code:

INSERT into demo02 select * from demo01;

Native DDL+DML development method

The native DDL and DML development method uses Flink SQL statements directly to create and manage data tables. For example, use CREATE TABLE/CREATE TEMPORARY TABLE to create a table. This method defines the table schema in code and uses SQL statements to manage the tables.

Examples

The following sample code reads analog data from table t1 and writes it to table t2.

create temporary table t1 (
id bigint,
name varchar
) with (
'connector' = 'datagen',
'rows-per-second' = '1'
);
create temporary table t2 (
id bigint,
name varchar
) with (
'connector' = 'print'
);

-- begin statement set;
insert into t2 select id,replace(name, '\"', '"') as name from t1;
-- set;