Create and manage workflows

Prerequisites

• To use offline workflows, you must first purchase the unstructured data feature for the current instance.

• To use real-time and online workflows, you must first purchase the Unstructured Data - Enterprise Edition for the current instance.

Limitations

• You can create workflows only in a Basic project.

• Hot upgrades are not supported for real-time workflows.

• Hot upgrades are not supported for online workflows. During an upgrade, the workflow API is unavailable for about 10 minutes. After you update an online workflow, it can take up to 3 minutes for the changes to take effect in the production environment.

Permissions

In addition to a project administrator or developer, project members with the Data Processing - Edit permission can create, edit, delete, and move workflows, and acquire locks within the project. Members with the Data Processing - Execute permission can run, submit, take offline, and take offline and delete workflows within the project.

Background

Dataphin supports three methods to create a workflow: Create a workflow, Create a workflow by uploading a development script, or use the CreateWorkflowByJson OpenAPI.

Create a workflow

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a project from the dropdown list.

3. In the left-side navigation pane, choose Data Processing > Workflow. In the Workflow list, click the icon to open the Create workflow dialog box. Configure the following parameters.

   Parameter	Description
   Name	Enter a name for the workflow. The name can contain any characters and be up to 256 characters long.
   Scheduling type	Select the scheduling type for the workflow. The scheduling type can be: Periodic Node: A task that runs at scheduled intervals. Manual Node: A task that has no dependencies and must be triggered manually. Real-time Node: An event-driven task. Online Node: A task that provides synchronous or asynchronous online services through an HTTP API.
   Description	Enter a description for the workflow (up to 512 characters).
   Select directory	Select the directory where the task is stored. If a directory does not exist, create one by clicking New folder: Above the workflow list on the left, click the icon to open the New folder dialog box. In the New folder dialog box, enter a Name for the folder and select a location in the Select directory field as needed. Click OK.

4. Click OK.

   After the workflow is created, you can use operators on the canvas to process data.

Create a workflow by uploading a script

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a project from the dropdown list in the top navigation bar.

3. In the left-side navigation pane, choose Data Processing > Workflow. Above the Workflow list, click the icon to open the Upload workflow development script dialog box.

4. In the Upload workflow development script dialog box, the parameters are the same as in the "Create a workflow" section, but you must also upload a JSON File. For more information, see Create a workflow.

Manage workflows

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a project from the dropdown list in the top navigation bar.

3. In the left-side navigation pane, choose Data Processing > Workflow. In the Workflow list, click a target workflow to view and edit it on the canvas. The following table describes other supported operations.

   Actions	Description
   Run	Offline workflow: When you click Run, if your remaining Credit is less than or equal to 100, the workflow might fail to run. We recommend that you purchase a resource plan before trying again. If your remaining Credit is less than or equal to 0, the workflow fails to run. If your Credit balance becomes insufficient during a run, the workflow does not stop immediately. To prevent the loss of computation records generated earlier in the task, the system allows the task to use an additional 4,000 Credits. Real-time workflow: After you click Run, you can set the Maximum runtime, which is 10 minutes by default. When running, bounded streams stop automatically after all data is consumed. Unbounded streams must be stopped manually or they stop automatically when the maximum runtime is reached. Only one instance of the same workflow can run at a time. A running real-time workflow continuously consumes Credit, which includes resource and operator Credit. If your remaining Credit is less than or equal to 0, the system sends an alert, but the workflow continues to run. If your Credit deficit reaches -4,000 to -5,000, the workflow stops immediately. The system commits the current Kafka offset, and you must purchase a resource plan and then manually restart the workflow to resume consumption from the committed offset. Online workflow: You can trigger a debug run with actual request parameters. The request parameters are automatically synchronized from the bound API dataset version. In the toolbar at the top of the canvas, click Run. In the Parameter settings dialog box, enter the actual values for each parameter to start the run. After the run starts, you can view the run information in the Run result panel at the bottom of the canvas. Only one debug instance of a workflow can run at a time. A debug run consumes Credit. The consumption details are the same as for real-time workflows. Note If you do not enter a value for a required parameter, the run is blocked. When you enter actual values for each parameter, you can save them as a Preset (up to 10 sets per workflow) to facilitate repeated debugging.
   Submit	In the toolbar at the top of the canvas, click Submit. Review the objects to be submitted and the precheck results, and enter a submission note. After all checks pass, click Confirm and Submit. For details about each check item, see Workflow submission checks. After you successfully submit an online workflow, the system automatically registers it as the backend for the bound API dataset version. You can then test calls to it in the development environment.
   Go to O&M	After a workflow is submitted to the production environment, click Go to O&M to navigate to the task list in the O&M Center. The system automatically performs an exact search using the current workflow's task ID to help you quickly locate the corresponding O&M task. Note The Go to O&M operation is not supported for online workflows.
   Call (online workflows only)	An online workflow can provide an HTTP API service only after it is published to the production environment. A workflow can be published only after its bound API dataset is published to the production environment. After publication, clients can call its bound API over HTTP. The call method is the same as for data service APIs. In synchronous mode, the system returns the response immediately after the workflow finishes. In asynchronous mode, the system first returns a task_id, and the caller must poll the asynchronous status query interface of the data service to get the task status and results.
   Take offline and delete	You can take offline and delete submitted workflows. After an online workflow is taken offline, its status changes to Offline. Its bound API dataset remains Published, but the binding status for that version changes to Binding Invalid. Calls to this API version will then return an HTTP 503 error. You can restore the service by resubmitting and publishing the workflow.
   Delete	You can delete workflows in a draft state directly. You must take submitted workflows offline before deleting them. This action cannot be undone. After an online workflow is deleted, the binding status of its bound API dataset version returns to Unbound, and it can be rebound to a new online workflow.