Compute tasks

Area

Description

①Compute task directory

The compute task directory provides a central location to manage the compute tasks you create. The directory supports the following management operations:

• Directory toolbar: Supports Filter, Refresh, Create folder, and Locate. By default, the Directories with permissions filter is selected, which displays only the first-level directories that the current user can access. If you deselect this option, all directories are displayed.

• Create compute task: Click the icon to create a compute task. The types of tasks that you can create vary depending on the compute engine. For the supported task types, see the options available on the page. In multi-engine mode, the types of compute tasks that you can create are determined by the engine type of the project's compute source.

• Directory: Supports Permission management (for first-level directories only), Create folder, Rename, Move, and Delete.

  Permission management: Only project roles with the Data Management-Directory Permission Management permission and directory administrators can manage permissions for first-level directories. After you click Permission Management, you can set Visibility settings to Public or Only specified members. If you select Only specified members, you must also specify the directory administrator and visible members. After you complete the configuration, other members cannot see tasks in the directory during development, O&M, or release.

  Note

  Directory permission management is not supported for real-time tasks.

• Compute task: Supports Acquire lock, Edit, Clone, Rename, Move, Delete, Deactivate, Deactivate and delete, and Change owning project.

  Important

  • For a Basic-mode project, after a delete request is committed successfully, the system immediately applies the change to the production environment. Perform this action with caution.

  • For a Dev-Prod mode project, after a delete request is committed successfully, go to the to-be-released list to release the object to the production environment.

• Compute task list: Click the compute task list to manage your compute tasks in a list view.

②Toolbar

The toolbar provides actions such as Save, Pre-compile, Run, Run with parameters, Commit, Go to release, Format, Refresh, Acquire lock, Locate, Deactivate, and Deactivate and delete.

• Switch version: If the compute task is a Python task, you can switch its execution version. Python 2.7, Python 3.7, and Python 3.11 are supported.

• Switch data source: If the compute task is a database SQL task, you can switch its data source.

• Switch Database/Schema: If the compute task is a database SQL task and the database type is AnalyticDB for PostgreSQL, Oracle, Presto, MySQL, StarRocks, Doris, SelectDB, ClickHouse, OceanBase (MySQL mode), TDengine, or GBase 8a, you can switch the Database/Schema of the data source.

• Switch Catalog: If the compute task is a database SQL task and the database type is Presto or Trino, you can switch the Catalog of the data source.

• Save: Saves the current compute task.

• Pre-compile: Pre-compiles the current compute task.

• Run: Runs the current compute task. After you set the parameters once, you do not need to re-enter them for subsequent runs.

  If the current user lacks the required row-level permission for relevant tables, a "Permission check failed" dialog box appears, listing the missing permissions. Click Apply for permission to open the Management Center > Permission Management > My Permissions > row-level permission page to request the necessary permissions. After the permission is granted, run the task again.

• Run with parameters: Runs the task without using its saved parameters. You must re-enter the parameters for each run.

• Commit: Commits the compute task to the Release Center or O&M Center.

• Go to release: For tasks committed to the Release Center, go to the Release Center to release them.

  If the production account for the project does not have the required row-level permission for the relevant tables, the system blocks the release and displays an error listing the missing permissions. Contact a security administrator to grant the permissions, and then release the task again.

• Go to O&M: You can perform O&M operations in the O&M Center for tasks that have been committed there. For more information, see O&M Center.

• Format: If the compute task is an SQL task, you can format its code.

• Refresh: Refreshes the current compute task.

• Acquire lock: Acquires the edit lock for the compute task.

• Locate: Locates the current compute task in the task directory.

• Delete, Deactivate and delete: Deletes or deactivates and deletes the current compute task.

  Important

  • For a Basic-mode project, after a delete request is committed successfully, the system immediately applies the change to the production environment. Perform this action with caution.

  • For a Dev-Prod mode project, after a delete request is committed successfully, go to the to-be-released list to release the object to the production environment.

③Editor

You can develop compute tasks in the code editor. The editor supports various shortcuts and productivity features. For more information, see:

• Editor Shortcut List.

• Editor Code Efficiency Tips.

④Configuration & Management

Configure and manage compute task details, such as its property configuration, related objects, History, and lineage.

• Property configuration: Configure the properties of the compute task. The properties vary based on the task type.

  • For information about the properties of an offline compute task, see Configure offline pipeline task properties.

  • For information about the properties of a real-time compute task, see:

    • Overview of real-time mode configuration.

    • Overview of offline mode configuration.

• Related: For an offline compute task, you can view its related objects. Related objects include tables that the task reads from, tables that read from the task, and tasks that have the current task as an upstream dependency. For more information, see View and manage related objects of a task.

• History: View and manage historical versions of the task. For more information, see View and manage historical versions of a real-time task.

• Lineage: For Python, Shell, and Virtual compute tasks, you can define a custom lineage. For more information, see Custom lineage configuration.

⑤Console

When you run or pre-compile a task, the console displays the task's run log and run result.

For SQL query compute tasks, the console includes three sections: run results, logs, and code. For more information, see Console for SQL query compute task results.

Section

Description

① General console operations

• View run results: Click the icon on the left side of the console to view the run results of the current compute task.

• View logs: Click the icon on the left side of the console to view the log information for the current run of the compute task.

• View code: Click the icon on the left side of the console to view the code of the current compute task.

• View and manage multiple run results: Click the icon in the upper-right corner of the console to view the results from multiple runs of the current compute task.

• Fullscreen console: Click the icon in the upper-right corner of the console to display it in fullscreen mode.

• Expand console/Collapse console: Click the or icon in the upper-right corner to expand or collapse the console vertically. You can also drag the top edge of the console to resize it.

• Collapse console/Show console: Click Collapse console at the bottom of the console to hide it. Once collapsed, click Show console to display it again.

② Run result operations

• Column search and filter: Click Column search and filter to open a drop-down list of all column names in the current table. Click the icon next to a column name to locate and highlight that column, which helps you quickly find its content. You can select or deselect columns in the list to show or hide them in the results table.

• Single-row details: Click Single-row details to display each row as a set of fields and values. You can quickly navigate to the previous, next, first, or last row. To return to the table view, click List view.

• Auto-fit column width: You can manually adjust the width of each column in the results table. Click Auto-fit column width to automatically resize all columns based on their content length.

• Content search: Click the icon to search the entire table for content that matches the entered keyword.

• Download data: Click the icon and select Download sample data or Download all data. For more information, see Download Result Data.

• Transpose: Click the icon to transpose the current table.

• Single-column data filtering and sorting: Click the icon next to a column name to filter or sort the data in that column. You can filter and sort data in any column of the results table.

• Select entire table: Click the cell in the top-left corner of the table to select the entire table, including the table header (column names). You can then right-click to copy the entire table.