DocsICP FilingConsole
Documentation
Home Page Dataphin Dataphin(Non-shared) User Guide Data Development Encoding research and development Compute tasks Offline task development Create a general task Create a database SQL task

Create a database SQL task

Updated at:
Copy as MD
Product
My Favorites

This topic describes how to create a database SQL task in Dataphin.

Limitations

  • You can create and execute stored procedures, but Dataphin does not perform syntax validation or parsing.

  • To run a database SQL task, you must have execution permissions for the data source. Table-level permission checks are not supported. For more information, see Apply for, renew, and return data source permissions.

  • Scheduling data source quality rules triggered by SQL data updates is not supported for MySQL or Oracle data sources. For more information, see Create a data source quality rule and .

  • Full download of query results is not supported. You can only download sample data.

  • The database SQL task type is not supported for offline computing template nodes.

  • In a project that uses the Dev-Prod mode, if you use a data source configured only for the production environment, operations such as running tasks or backfilling data in the development environment may alter production data. Use with caution.

Procedure

  1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

  2. On the Development page, select a Project from the top navigation bar. If the project is in Dev-Prod mode, you must also select an environment.

  3. In the left-side navigation pane, choose Data Processing > Compute Task. In the Compute Task list, click the image icon and choose Database SQL.

  4. In the Create Database SQL Task dialog box, configure the following parameters.

    Parameter

    Description

    Basic information

    Task name

    Enter a name for the task.

    The name can be up to 256 characters long and cannot contain the following characters: | / \ : ? <> * ".

    Scheduling type

    Select the scheduling type for the task. The Scheduling type options are:

    • Periodic Task: The task is automatically run based on a recurring schedule.

    • Manual Task: The task must be triggered manually.

    Select directory

    Select the directory where the task is stored.

    If a directory does not exist, click New Folder to create one.

    1. Above the compute task list on the left, click the image icon to open the New Folder dialog box.

    2. In the New Folder dialog box, enter a Name for the folder and, if necessary, select a location in the Select Directory field.

    3. Click OK.

    Use template

    Turn on the Use Template switch to use a code template. If this feature is enabled, you must also Select Template and Template Version. By default, the latest version of the selected template is used.

    Using a code template can improve development efficiency. The code in a template is read-only. You only need to configure the template parameters to complete the task. For more information, see Create an offline computing template.

    Note

    If a database SQL compute task references an offline computing template and uses AnalyticDB for PostgreSQL, StarRocks, Doris, SelectDB, or GaussDB (DWS) as the data source, you cannot change the task type.

    Description

    Enter a description for the task, up to 1,000 characters.

    Data source configuration

    Data source type

    Select the data source type for the SQL task. For a list of supported data source types, see the Offline R&D-Database SQL column in Data sources supported by Dataphin.

    If you use a template, the data source type is automatically set to match the template and cannot be changed.

    Data source

    Select a data source for the SQL task. If no data source is available, click + New Data Source to create one.

    If you use a template, the data source is automatically set to match the template, but you can select a different data source.

    Catalog

    If the data source type is Presto or Trino, you must also configure the Catalog after selecting the data source.

    Database/Schema

    If the data source type is MySQL, PostgreSQL, AnalyticDB for PostgreSQL, Oracle, Presto, GaussDB (DWS), Microsoft SQL Server, ClickHouse, Hologres, Doris, openGauss, StarRocks, DM, OceanBase (Oracle tenant mode), SelectDB, Trino, PolarDB-X 2.0, Lindorm, TDSQL for MySQL, TDSQL for PostgreSQL, TDengine, or GBase 8a, you must also configure the Database/Schema after selecting the data source.

    If you use a template, the Database/Schema is automatically set to match the template, but you can select a different Database/Schema.

  5. Click OK.

  6. In the code editor on the current database SQL task tab, write the code for the database SQL task. Database SQL supports cross-node output parameters. For more information, see Assign values to cross-node output parameters and Best practices for cross-node parameters. After you write the code, click Run above the code editor.

  7. In the right-side panel, click Properties. On the Properties panel, configure the task's Basic information, Running parameters, Scheduling properties (for periodic tasks), Scheduling dependencies (for periodic tasks), Runtime configuration, and Resource configuration.

    • Basic information

      Defines basic information for the task, such as its name, owner, and description. For configuration details, see Configure basic task information.

    • Running parameters

      If your task code includes parameter variables, you can assign values to them in this section. This allows the variables to be automatically replaced with their corresponding values when the task is scheduled. For configuration details, see Configure and use node parameters.

    • Scheduling properties (periodic task)

      If the scheduling type for an offline compute task is Periodic Task, you must configure its scheduling properties in addition to the Basic Information. For configuration instructions, see Configure scheduling properties.

    • Scheduling dependencies (periodic task)

      If the scheduling type of an offline compute task is Periodic Task, you must also configure its scheduling dependencies in addition to the Basic Information. For more information, see Configure scheduling dependencies.

      Important

      Automatic dependency parsing may be inaccurate. Verify the dependencies based on your code.

    • Runtime configuration

      You can configure a task-level timeout period and a retry policy for task failures based on your business needs. If these are not configured, the task inherits the default tenant-level settings. For configuration details, see Configure runtime settings for a compute task.

    • Resource configuration

      This section specifies the resources that the task's scheduled instances consume. Resources are isolated between different resource groups. Database SQL tasks are shared-resource tasks and do not support custom resource groups. For configuration details, see Configure resources for a compute task.

  8. In the current database SQL task tab, save and submit the task.

    1. Click the image icon above the code editor to save the code.

    2. Click the image icon above the code editor to submit the code.

  9. On the Submission Details page, confirm the Submission Content and the Pre-check results, and add remarks as needed. For more information, see Submit an offline compute task.

  10. After confirming the details, click OK and Submit.

Next steps

  • If your project uses Dev-Prod mode, you must publish the task to the production environment after submitting it. For more information, see Manage publishing tasks.

  • If your project uses Basic mode, the database SQL task can be scheduled in the production environment immediately after it is submitted. Go to the O&M Center to view and manage your published task. For more information, see View and manage script tasks, and View and manage manual tasks.

Appendix: Modify the task type

If a database SQL task uses an AnalyticDB for PostgreSQL, StarRocks, Doris, SelectDB, or GaussDB (DWS) data source, you can change its type to the corresponding dedicated SQL task type (for example, AnalyticDB for PostgreSQL SQL, StarRocks SQL, Doris SQL, SelectDB SQL, or GaussDB_SQL).

  1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

  2. On the Development page, select a Project from the top navigation bar. If the project is in Dev-Prod mode, you must also select an environment.

  3. In the left-side navigation pane, choose Data Processing > Compute Task, and select the target database SQL task from the Compute Task list.

  4. Click the image icon next to the database SQL task and choose Modify Task Type.

  5. In the Modify Task Type dialog box, click OK to change the task type.

Note

  • After the task type is changed, task properties such as the owner, scheduling settings, and upstream/downstream dependencies, remain unchanged.

  • If the database SQL task uses an offline computing template, you cannot change its task type.

  • If the corresponding SQL task type does not exist in the current project, you cannot change the database SQL task to that type.

  • After a database SQL task is converted to its corresponding engine-specific SQL task, you can use version rollback to revert it to a database SQL task. For more information, see View and manage historical versions of an offline task.

Previous：Create a Python computing taskNext：Create a check node task