Create a Spark SQL task

Procedure

1. On the Dataphin homepage, in the top navigation bar, choose Develop > Data Development.

2. On the Develop page, select your Project in the top menu bar. If you are using Dev-Prod mode, you also need to select an environment.

3. In the left-side navigation pane, choose Data Processing > Compute Task. In the Compute Task list, click the icon and select Spark SQL.

4. In the Create Spark SQL Task dialog box, configure the parameters.

   Parameter	Description
   Task Name	Enter a name for the offline computing task. The name can be up to 256 characters long and cannot contain the following characters: vertical bar (|), forward slash (/), backslash (\), colon (:), question mark (?), angle brackets (<>), asterisk (*), or double quotation mark (").
   Scheduling Type	Select a scheduling type for the task. Scheduling type can be: Periodic Task: The task runs automatically on a schedule. Manual Task: You must trigger the task manually.
   Select a directory	Specifies the directory where the task is stored. If a directory does not exist, you can create one. Follow these steps: Above the compute task list on the left, click the icon to open the Create Folder dialog box. In the Create Folder dialog box, enter a Name for the folder and, if necessary, Select a directory for its location. Click OK.
   Use Template	Click the Use Template switch to enable or disable code templates. If enabled, you must also Select Template and Template Version. You can use a code template to improve development efficiency. Template code is read-only. You only need to configure the template parameters to finish developing the code. For more information, see Create an offline computing template.
   Description	Enter a brief description of the task, up to 1,000 characters.

5. Click OK.

6. In the code editor on the Spark SQL task tab, write the Spark SQL code. After you finish, click Pre-compile above the code editor to check its syntax.

7. After the code pre-compiles successfully, click Run above the code editor.

8. Click Properties in the right sidebar. In the Properties panel, configure the task's Basic Information, Runtime Parameters, Spark Resource Settings, Scheduling Properties (for periodic tasks), Scheduling Dependencies (for periodic tasks), Runtime Configuration, and Resource Configuration.

   • Basic Information

     Use this section to define basic information for the scheduled task, such as the task name, owner, and description. For configuration details, see Configure basic information for a task.

   • Runtime Parameters

     If your task uses parameter variables, configure their values in this section. The scheduler then automatically replaces the variables with these values when the task runs. For details, see Configure runtime parameters for an offline task.

   • Scheduling Properties (for periodic tasks)

     If the scheduling type is Periodic Task, you must configure scheduling properties in addition to the Basic Information. For configuration details, see Configure scheduling properties for an offline task.

   • Scheduling Dependencies (for periodic tasks)

     If the scheduling type is Periodic Task, you must configure scheduling dependencies in addition to the Basic Information. For configuration details, see Configure scheduling dependencies for an offline task.

   • Runtime Configuration

     You can configure a task-level timeout period and a retry policy for task failures. If not configured, the task inherits the tenant-level default settings. For configuration details, see Runtime configuration for computing tasks.

   • Resource Configuration: Specifies the resources consumed to schedule instances generated by the task. Resources are isolated between different resource groups and do not affect each other. A Spark SQL task is a shared-resource task and does not support specifying a custom resource group. For configuration details, see Resource configuration for computing tasks.

9. On the Spark SQL task tab, save and submit the task.

   1. Click the icon above the code editor to save the code.

   2. Click the icon above the code editor to submit the code.

10. On the Submission Details page, confirm the Submission Content and the results of the Precheck, and enter remarks. For more information, see Offline computing task submission instructions.

11. After confirming the details, click OK and Commit.