Create a Flink DataStream task

Prerequisites

• Before you begin, ensure that you have enabled the real-time engine for your project and configured a Ververica Flink compute source. For more information, see Create a general project.

• Ensure that you have uploaded the JAR package for your DataStream job to the Dataphin platform. For more information, see Upload and reference resources.

Permissions

Only super administrators, project administrators, and developers can create Flink DataStream tasks.

Step 1: Create a Flink DataStream task

1. On the Dataphin homepage, in the top navigation bar, choose Develop > Data Development.

2. In the top navigation bar, select a project. If your project is in Dev-Prod mode, you must also select an environment.

3. In the left navigation pane, choose Data Processing > Computing Tasks. In the computing task list on the right, click the icon and select Flink DataStream.

4. In the Create Flink DataStream Task dialog box, configure the parameters.

   Parameter	Description
   Task name	The task name must: Contain only lowercase letters, digits, and underscores (_). Be 4 to 63 characters in length. Be unique within the project. Start with a letter.
   Production environment cluster	Select the cluster for the production environment of the Flink DataStream task.
   Production environment engine version	Select the engine version.
   Development environment cluster and engine version	Select the resource queue and engine version for the development environment of the Flink DataStream task. You can choose System default configuration or Custom configuration. If you choose Custom configuration, you must configure the cluster and engine version for the development environment. Development environment resource cluster: Select the resource cluster for the development environment of the Flink DataStream task. Development environment engine version: Select the Ververica Flink engine version.
   Storage directory	Select the directory where the task is stored. If you have not created a directory, you can create a new folder as follows: Above the computing task list on the left side of the page, click the icon to open the New Folder dialog box. In the New Folder dialog box, enter a Name and optionally select a directory. Click OK.
   Select resource	The JAR package that the Flink DataStream task depends on.
   Class name	The fully qualified class name of the main class in the JAR package.
   Description	Enter a brief description of the Flink DataStream task.

5. Click OK.

Step 2: Pre-compile task code

In the code editor, write your task code. You can use parameters in the --ParameterName format. In the top menu bar, click Pre-compile to validate the code's syntax and permissions.

If the pre-compilation is successful, a Pre-compilation successful message is displayed. If the pre-compilation fails, a Pre-compilation failed message is displayed, and you can click Console at the bottom of the page to view the pre-compilation failure logs.

Step 3: Configure task

1. In the right-side pane, click Configuration.

2. In the configuration panel, configure the real-time mode settings for the Flink DataStream task.

   Important

   Flink DataStream tasks do not support offline mode.

   • Real-time mode

     • Resource configuration (Required): Configure the resource queues, engine versions, and the JobManager CPU and memory for the task's production and development environments. For configuration details, see Configure Ververica Flink real-time mode resources.

     • Variable configuration: You can define variables directly in your code without declaring them first. The system automatically parses these variables and displays them in a list, where you can modify their data types and assign values. For configuration details, see Real-time mode variable configuration.

     • Checkpoint configuration: Configure checkpoints for the Flink DataStream task. This helps the task restore its state from before the crash if it stops unexpectedly. For configuration details, see Real-time mode checkpoint configuration.

     • State configuration: Configure the automatic cleanup period for state data. For configuration details, see Real-time mode state configuration.

     • Runtime parameters: Configure runtime parameters to control the execution behavior and performance of the Flink application. For configuration details, see Real-time mode runtime parameter configuration.

     • Dependent files: Configure the dependent files for the task. For configuration details, see Real-time mode dependent file configuration.

     • Dependencies: Configure dependencies to visualize the task lineage for easier troubleshooting. For configuration details, see Real-time mode dependency configuration.

3. Click OK.

Step 4: Submit the task

1. Click Submit in the top navigation bar.

2. In the Submit dialog box, review the Submission Content and Pre-check information, and enter submission comments.

3. Click Confirm and Submit.