Create a Flink DataStream task

Prerequisites

• The real-time engine is enabled for your project and an open-source Flink compute source is configured. For more information, see Create a general-purpose project.

• The JAR package for your DataStream job is uploaded to the Dataphin platform. For more information, see Upload and reference resources.

Permissions

Only a super administrator, project administrator, or developer can create a Flink DataStream compute task.

Step 1: Create a Flink DataStream task

1. In the top menu bar on the Dataphin home page, select Develop > Data Development.

2. In the top menu bar, select a Project. If you are in Dev-Prod mode, you also need to select an Environment.

3. In the left navigation bar, select Data Processing > Compute Task. In the compute task list on the right, click the icon and select Flink DataStream.

4. In the Create Flink DataStream Job dialog box, configure the job parameters.

   Flink on Kubernetes

   Parameter	Description
   Task name	The name must meet the following requirements: It can contain only lowercase letters, digits, and underscores (_). It must be 4 to 63 characters in length. The name must be unique within the project. It must start with a letter.
   Image Type	You can select System Image or Custom Image. The default is System Image. If you select Custom Image, you must also enter the image address.
   Image Address	Enter the image address. Before you use a custom image, you must download the Metrics Reporting Plug-in and the Checkpoint Storage Extension Plug-in from the prompts on the page and package the plug-ins into the image.
   Production Environment Resource Queue	You can select any resource queue configured for real-time tasks, including those in externally registered clusters.
   Engine Version	Select the Flink engine version for the task. Only version 1.20.1 is supported.
   Development environment resource queue and engine version	You can select System Default Configuration or Custom Configuration. System Default Configuration: This is the default option. It uses the same environment cluster and engine version as the production environment. Custom Configuration: You can select the environment cluster and engine version for your development environment tasks. If your project space is in Basic mode, you do not need to configure this option.
   Storage Directory	Select the storage directory for the task. If a directory does not exist, you can create one. To do so, follow these steps: On the left side of the page, above the compute task list, click the icon to open the New Folder dialog box. In the New Folder dialog box, enter a Name for the folder and optionally select a directory for the location. Click OK.
   Resource Package Path/Select Resource	When Image type is set to Custom Image, enter the resource package path within the image. Example: local:///opt/flink/usrlib/flink-sql-job-1.15.3.jar. When Image type is set to System Image: Select the resource package that the Flink DataStream task depends on.
   Class Name	Enter the fully qualified class name of the resource.
   Use Template	Turn on this switch to reference an existing real-time template.
   Description	Enter a brief description of the Flink DataStream task, up to 1,000 characters.

   Flink on YARN

   Parameter	Description
   Task Name	The name must meet the following requirements: It can contain only lowercase letters, digits, and underscores (_). It must be 4 to 63 characters in length. The name must be unique within the project. It must start with a letter.
   Production Environment Resource Queue	The drop-down list includes all resource queues and session clusters managed by the Flink compute source bound to the current project.
   Production Environment Engine Version	Select the Flink engine version for the task. Dataphin supports the following engine versions: 1.20.1 1.15.3 1.14.2 1.13.1 Note If you select a session cluster for the production resource queue, you can only select version 1.20.1.
   Development Environment Resource Queue and Engine Version	You can select System Default Configuration or Custom Configuration. System Default Configuration: This option is selected by default and uses the same environment cluster and engine version as the production environment. Custom configuration: You can select the environment cluster and engine version for development tasks. The available options are the same as the resource queues and engine versions for the production environment. If your project is in Basic mode, you do not need to configure this setting.
   Storage Directory	Select the storage directory for the task. If a directory does not exist, you can create one. To do so, follow these steps: Above the compute task list on the left side of the page, click the icon to open the New Folder dialog box. In the New Folder dialog box, enter a folder Name and select a Directory location as needed. Click OK.
   Select Resources	Select the resource package that the Flink DataStream task depends on.
   Class Name	Enter the fully qualified class name of the resource.
   Use Template	Turn on this switch to reference an existing real-time template.
   Description	Enter a brief description of the Flink DataStream task, up to 1,000 characters.

5. Click OK.

Step 2: Add code and pre-compile the task

A Flink DataStream task is a Java JAR task. To pass main arguments, enter them in the code editor, separated by spaces or line breaks. The system parses non-commented content, splits it by delimiters into an argument array, and passes the array to the main function. Arguments in the --parameter-name format are also supported.

After you finish editing the task code, click Pre-compile in the top menu bar to check for syntax errors and permission issues.

If the pre-compilation is successful, a Pre-compilation successful message is displayed. If it fails, a Pre-compilation failed message is displayed, and you can click Console at the bottom of the page to view the pre-compilation failure logs.

Step 3: Configure the Flink DataStream task

1. Click Configuration on the right sidebar of the current compute task.

2. In the configuration panel, configure the real-time mode settings for the Flink DataStream task.

   Important

   Flink DataStream jobs do not support offline mode.

   • Real-time mode

     • Resource configuration (Required): Configure the resource queues, engine version, task parallelism, number of TaskManagers, JobManager memory, and TaskManager memory for the production and development environments. For more information, see Configure resources for the open-source Flink real-time mode.

     • Variable configuration: These variables are for the current node. You can define them in your code without pre-declaration. The system automatically parses them into a parameter list where you can modify types and assign values. For more information, see Configure variables for real-time mode.

     • Checkpoint configuration: Configure checkpoints for the Flink DataStream task. A checkpoint helps restore the task to its state before an unexpected stop. For more information, see Configure checkpoints for real-time mode.

     • State configuration: Configure the automatic cleanup period for data in the state. For more information, see Configure state for real-time mode.

     • Runtime parameters: You can configure runtime parameters to control the execution behavior and performance of your Flink application. For more information, see Configure runtime parameters for real-time mode.

     • Kubernetes YAML: You can view and modify the pod YAML for the task's real-time mode, including the JobManager and TaskManager pods.

     • Dependency files: Configure the resource files that the task depends on. For more information, see Configure dependency files for real-time mode.

       Note

       You cannot configure dependency files for Flink DataStream tasks that use a custom image.

     • Dependencies: Configure dependencies to quickly understand the upstream and downstream tasks during debugging. For more information, see Configure dependencies for real-time mode.

3. Click OK.

Step 4: Commit the Flink DataStream task

1. Click the Submit button on the top menu bar.

2. In the Submit dialog box, review the Submission Content and Pre-check information, and enter the Submission Remarks.

3. Click Confirm and Submit.

4. Note

   If your project is in Dev-Prod mode, you need to publish the Flink DataStream task to the production environment. For more information, see Manage published tasks.

Next steps

After committing the task, navigate to the O&M Center to monitor and manage its execution. For more information, see View and manage real-time tasks.