Variable management

ETL (SQL job)

You can use variables during SQL job development, including as arguments for custom functions. You can also modify the variable values when you start the job.

1. In the left-side navigation pane of the Realtime Compute for Apache Flink console, click Development > ETL.

2. In the SQL editor for your target SQL job, create variables using the ${variable_name} format, as shown in the following example.

   • When used as a parameter in a WITH clause, enclose the variable in single quotation marks: '${variable_name}'.

   • When used in other locations (such as for SQL table or field names), enclose the variable in backticks: `${variable_name}`.

   create temporary table `${source_name}`(
       id varchar,
       name varchar
   ) with (
       'connector' = 'datagen'
   );
   create temporary table blackhole(
       id varchar,
       `${test_name}` varchar
   ) with (
       'connector' = '${blackhole}'
   );
   insert into blackhole select * from `${source_name}`;

3. Flink automatically identifies deployment variables that use this format in the SQL job. You must provide values for these variables.

   • Method 1: The Job Variables panel appears in the SQL editor. Enter the values in this panel.

     In the Job Variables panel, configure the key-value pairs: source_name=datagen, test_name=name, and blackhole=blackhole.

   • Method 2: Click More Configurations on the right side. Enter the values in the Job Variables section.

     For example, configure the variable source_name as datagen, test_name as name, and blackhole as blackhole. You can then reference these values in your SQL code by using the ${variable_name} syntax.

4. After the job is deployed, you can view its variables on the Deployment Details tab of the job details page under Deployments.

   In the job properties, the Job Variables row displays the configured variables (such as source_name, test_name, and blackhole) and their corresponding values.

5. Click Start in the Actions column for the target job. You can modify the variable values in the job startup panel.

   Important

   Changes made to variable values in the job startup panel apply only to the current job run. They do not alter the values defined in the SQL job.

   In the job startup panel, you can select stateless start or stateful start mode. If you select stateful start, you can configure recovery options, such as Recover from latest state. The Job Variables section displays the variables and their default values, such as source_name set to datagen, test_name to name, and blackhole to blackhole. You can modify them as needed and then click Start.

JAR/Python deployment

You can use variables as arguments for the main function in JAR/Python deployments.

Create a variable

1. Log on to the Realtime Compute for Apache Flink console. Find the target workspace and click Console in the Actions column.

2. In the left-side navigation pane, click Security > Variables, and click Add Variable.

   Parameter	Description
   Variable Name	The variable name must be unique within the namespace and cannot be changed after creation.
   Variable Value	Variable values are case-sensitive. The value can be modified after creation. For details, see Edit or delete a variable.
   Type (Plaintext/Ciphertext)	The type cannot be changed after creation. Plaintext: The variable value is directly visible on the Variables page. Ciphertext: The variable value is hidden. After the variable is created, you can view its name, type, value, and creation time in the list on the Variables page. You can also Edit or Delete the variable.

3. Click OK.

Use a variable

After creating a variable, you can reference it using the format ${secret_values.variable_name}. Replace "variable_name" with the name of your variable.

create temporary table `${secret_values.source_name}`(
    id varchar,
    name varchar
) with (
    'connector' = 'datagen'
);
create temporary table blackhole(
    id varchar,
    `${secret_values.test_name}` varchar
) with (
    'connector' = '${secret_values.blackhole}'
);
insert into blackhole select * from `${secret_values.source_name}`;

source:
  type: mysql
  name: Mysql Source
  hostname: localhost
  port: 3306
  username: test
  password: ${secret_values.mysqlpw}
  tables: app_db.\.*
  server-id: 5400-5404
···

USE CATALOG paimon;
CREATE DATABASE IF NOT EXISTS `${secret_values.test_name}`;
CREATE TABLE paimon.`${secret_values.test_name}`.ods_user_log
(
    item_id  int NOT NULL,
    `${secret_values.user}` varchar(50) NOT NULL,
    action  varchar(20)  NOT NULL,
    vtime   timestamp,
    ds      varchar(10) NOT NULL
)
PARTITIONED BY (ds);
SELECT * from paimon.`${secret_values.test_name}`.ods_user_log LIMIT 10;

Edit or delete a variable

1. Log on to the Realtime Compute for Apache Flink console. Find the target workspace and click Console in the Actions column.

2. In the left-side navigation pane, click Security > Variables. Find the target variable and click Edit or Delete in the Actions column.

   • Edit

     You can change the value of a variable. Only members with editor or higher permissions can edit variables.

   • Delete

     Only members with editor or higher permissions, or members with the specific permission to delete variables, can delete variables.