Manage workflows

Limitations

• You can create workflows only for deployed batch jobs.

• The Workflows feature is in public preview. SLAs are not guaranteed during this period. For more information, see Service Level Agreement (SLA) for Realtime Compute for Apache Flink. If you encounter any issues, you can submit a ticket.

• The Workflows feature is available only in the China (Shanghai), China (Hangzhou), China (Beijing), China (Shenzhen), China (Zhangjiakou), and Singapore regions.

Create a workflow

1. Log on to the Realtime Compute for Apache Flink console.

2. In the Actions column of the target workspace, click Console.

3. In the left-side navigation pane, choose O&M > Workflows.

4. Click Create workflow and configure the following parameters.

   Parameter	Description
   Workflow name	The workflow name must be unique within the current namespace.
   Variable configuration	Enables data processing with preset values. Variable name: A custom variable name, such as ${date}. Variable value: Can be a static date, a time format, or an expression. The following system time variables are supported: Variable name system.biz.date with value ${system.biz.date}: Represents the day before the scheduled time of a daily workflow instance, in yyyyMMdd format. Variable name system.biz.curdate with value ${system.biz.curdate}: Represents the scheduled date (year, month, day) of a daily workflow instance, in yyyyMMdd format. Variable name system.datetime with value ${system.datetime}: Represents the scheduled time (year, month, day, hour, minute, second) of a daily workflow instance, in yyyyMMddHHmmss format. Note This configuration is not required when creating a workflow for a materialized table. These parameters apply to all deployments associated with the tasks in the workflow. Workflow-level configurations override deployment-level configurations.
   Scheduling type	Supports two scheduling types: Manual trigger: Manually run the workflow by clicking Trigger run. This is suitable for temporary tests or immediate processing. Periodic scheduling: Triggers the workflow based on a scheduling rule. You can schedule it to run at intervals of minutes, hours, or days. Important Workflows with materialized table tasks require periodic scheduling.
   Scheduling rule	Required only for Periodic scheduling. You can use a cron expression to define complex rules. Examples: 0 0 */4 ? * *: Executes every 4 hours. 0 0 2 ? * *: Executes at 2:00 AM every day. 0 0 5,17 ? * MON-FRI: Executes at 5:00 AM and 5:00 PM, Monday through Friday. For more information about cron expression syntax, see Cron expression syntax.
   Scheduled start time	The time when the rule takes effect. This parameter is required only for the Periodic scheduling type. Important A periodically scheduled workflow runs only after its State is enabled. Set the scheduled start time to a future time to ensure the task does not miss its scheduling window.
   Failure retry count	Specifies how many times to retry failed tasks in the workflow. Retries are disabled by default.
   Failure notification	The default email address for notifications when a task in the workflow fails. Note You can configure alerts through DingTalk, SMS, and other methods via CloudMonitor. For more information, see Configure monitoring and alerts.
   Resource queue	The deployment target for the workflow. For more information, see Manage resource queues. This setting applies to all tasks by default, eliminating the need for task-level configuration. Note This setting does not change the deployment target of the corresponding deployed batch job.
   Tag	You can add a key and value to create a tag for the workflow.

5. Click Create.

   You are redirected to the workflow builder page.

6. Configure the initial task of the workflow.

   The builder page contains a default initial task. Click the initial task, configure its parameters in the Edit task panel, and then click Save.

   Job

   Parameter	Description
   Deployment	You can select only a deployed batch job from the current namespace. Fuzzy search is supported.
   Task name	The name of the task within the current workflow.
   Upstream tasks	The upstream dependencies of the current task. You can select only other tasks within the current workflow. Note The initial task has no upstream dependencies, so you cannot select an upstream task for it.
   Failure retry count	The number of retry attempts for a failed task. By default, this value is inherited from the workflow's retry count. If specified, the task-level setting takes precedence.
   Subscription	Configure the state subscription policy and notification email. You can subscribe to Startup and Failure states.
   Timeout	The maximum execution time for the task. If the task exceeds this duration, it fails.
   Resource queue	The deployment target for the task. For more information, see Manage resource queues. If not specified, the resource queue configured for the workflow is used. Note This setting does not change the deployment target of the corresponding deployed batch job.
   Tag	You can add a key and value to create a tag for the workflow task.

   Materialized table

   Parameter	Description
   Materialized table	You can select only partitioned materialized tables that are created with VVR 11.0 or later and use streaming refresh mode.
   Task name	The name of the task within the current workflow.
   Time partition	Partition field: The time partition field of the materialized table. Partition format: For example, yyyyMMdd.
   Resource configuration	You can customize the amount of resources used for scheduled backfills. For parallelism, select Auto-infer to let the system determine a suitable value.
   Upstream tasks	The upstream dependencies of the current task. You can select only other tasks within the current workflow. Note The initial task has no upstream dependencies, so you cannot select an upstream task for it. After you create a materialized table task, downstream tasks are automatically inferred based on data lineage.
   Failure retry count	The number of retry attempts for a failed task. By default, this value is inherited from the workflow's retry count. If specified, the task-level setting takes precedence.
   Subscription	Configure the state subscription policy and notification email. You can subscribe to Startup and Failure states.
   Timeout	The maximum execution time for the task. If the task exceeds this duration, it fails.
   Resource queue	The deployment target for the task. For more information, see Manage resource queues. If not specified, the resource queue configured for the workflow is used. Note This setting does not change the deployment target of the corresponding deployed materialized table.
   Tag	You can add a key and value to create a tag for the workflow task.

   Note

   • After you create a materialized table task, a dialog box uses the table's lineage to suggest downstream tasks. Simply select the desired tasks to complete the setup.

   • Downstream tasks must also meet the same limitations. You can select only partitioned materialized tables that are created with VVR 11.0 or later and use streaming refresh mode with a freshness of less than 30 minutes.

7. (Optional) Click Add task at the bottom of the page to add more tasks.

8. Save the workflow configuration.

   1. In the upper-right corner of the task editor page, click Save.

   2. In the dialog box that appears, click OK.

Run a workflow

Each workflow execution generates a workflow instance, which appears on the Workflow instances and details tab of the workflow details page.

• Manual trigger: Runs the workflow once immediately. This is useful for testing or on-demand processing.

  In the Actions column of the target workflow, click Trigger run. In the dialog box that appears, select Manual trigger and then click OK. Each operation triggers one execution of the workflow.

• Periodic scheduling: Once enabled, the workflow runs automatically at the scheduled time.

  You must enable the State of the target workflow for it to run at the scheduled time.

Data backfill

Data backfill supplements or updates data within a specific time range. Common use cases include reprocessing historical data from upstream sources, correcting dimension tables, or back-populating data for new integrations.

Workflow status

The Run status column summarizes the states of all instances for a given workflow. For example, if a workflow runs daily for five days, the Run status column provides a count of how many of the five instances are in each state (such as Successful or Failed).

Edit a workflow

1. Log on to the Realtime Compute for Apache Flink console.

2. In the Actions column of the target workspace, click Console.

3. In the left-side navigation pane, click Workflows.

4. In the Actions column of the target workflow, click Edit workflow.

   For parameter descriptions, see Create a workflow.

   Note

   To edit a workflow, its State must be Off.

Related topics

• To schedule Flink SQL batch tasks on DataWorks, see Flink SQL Batch node and Node scheduling configuration.

• For more information about the basic concepts of Workflows, see Workflows (public preview).

• To view workflow instances, task logs, and more, see Manage workflow instances and task instances.

• You can add resource queues to isolate and manage resources. For more information, see Manage resource queues.

• To deploy batch jobs (SQL, JAR, and Python), see Deploy a job.