PAI Flow nodes

Limits

• Product limits:

  • PAI Flow only supports DataWorks Workspaces (new version).

  • PAI Flow currently only supports Source/Target and RAG Data Processing nodes.

  • PAI Flow only supports Serverless resource groups.

• Region limits: Supported regions include China (Hangzhou), China (Shanghai), China (Beijing), China (Ulanqab), China (Shenzhen), China (Hong Kong), Singapore, Indonesia (Jakarta), Japan (Tokyo), Germany (Frankfurt), US (Silicon Valley), and US (Virginia).

Prerequisites

You have created a DataWorks Data Studio (new version) workspace and a Platform for AI (PAI) workspace.

• When you create a workspace, you must select Create AI Workspace with Same Name. The system automatically creates and binds a Platform for AI (PAI) workspace that has the same name as the DataWorks workspace.

• For an existing workspace, you must enable Schedule PAI Nodes in the Management Center. This action also creates a Platform for AI (PAI) workspace that has the same name as the DataWorks workspace.

Create a PAI Flow

1. Go to the DataStudio page.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and O&M > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.

2. In the project directory of Data Studio, click the icon and choose New Node > Algorithm > PAI Flow. The system creates a PAI Flow node and directs you to the orchestration page.

Develop a PAI Flow

PAI Flow supports various visual modeling nodes. You can use these nodes to design your workflow.

1. On the PAI Flow orchestration page, drag nodes from the left-side pane to the canvas and connect them to design your workflow.

2. After completing the workflow design, click a node to configure it in the right panel.

   Node type	Node	Node description
   Source/Target	Read table	The Read Table component reads data from MaxCompute tables. By default, the component reads the table data of the current project.
   	Read OSS Data	This component is used to read files or folders from the Object Storage Service OSS Bucket path.
   	Read CSV file	This component supports reading CSV files from OSS, HTTP, and HDFS.
   	Write table	This component supports writing input data to MaxCompute.
   RAG data processing	RAG Text Parsing and Chunking	Reads and parses text files (HTML, PDF, Markdown, Text, etc.) from the input directory, generates continuous text blocks no larger than the specified block size, and saves them in JSONline format to the specified output path.
   	RAG Vector Generation	Loads all parsed and chunked document files (JSONline format) from the specified directory, then uses an Embedding model to generate text vectors.
   	RAG Knowledge Base Index Synchronization	Synchronizes input data to the target knowledge base index.

   Note

   When configuring file paths, you can include variables in the path, for example: https://examplebucket.oss-cn-hangzhou.aliyuncs.com/${variable}/example.csv. When configuring variables, you can use scheduling parameters as variables to read from or write to different storage paths during recurring scheduling.

3. After you complete node development, on the orchestration page, configure the scheduling settings for PAI Flow in the right-side toolbar to ensure that the PAI Flow is periodically scheduled after it is deployed to the production environment.

   Note

   When configuring scheduling settings, the schedule resource group only supports Serverless resource groups.

Publish a PAI Flow

After you debug the PAI Flow and configure its scheduling settings, you must publish the PAI Flow workflow. The workflow then runs periodically based on the specified settings.

1. Click Save in the top toolbar to save the PAI Flow.

2. After you save the changes, click the button in the top toolbar to open the deployment panel (Deploying a task). Then, click Start Release Production. The task is then deployed based on the deployment check process.

More operations

After the PAI Flow is published, you can click Go to operation and maintenance in the publishing panel. This action takes you to the Recurring tasks page, where you can view the scheduling and execution status of the PAI Flow.