Platform for AI (PAI) provides PAI. You can purchase these resources and associate them with a workspace for large-scale, distributed model training. This topic describes how to activate, purchase, and use fully managed Flink resources to train models.
Fully managed Flink resources
Alibaba Cloud Realtime Compute for Apache Flink is an all-in-one, real-time big data analytics platform built on Apache Flink. It provides end-to-end analytics capabilities with sub-second latency. For more information about fully managed Flink resources, see Overview of fully managed Flink resources.
Prerequisites
Before you begin, make sure that the following requirements are met:
An Alibaba Cloud account is required to purchase fully managed Flink resources. If you do not have an Alibaba Cloud account, see Sign up for an Alibaba Cloud account to create one.
Account and permission requirements
Alibaba Cloud account (Recommended): You can use your main account for all operations without further authorization.
RAM user:
To purchase fully managed Flink resources, you must grant the
AliyunStreamFullAccesspermission to the RAM user. For more information, see Authorize users in the console.To submit a training task to fully managed Flink resources, you need to add the owner role to the RAM user in the project space of the Flink development console. For more information, see Development Console Authorization.
To associate fully managed Flink resources with a workspace, grant the RAM user the administrator role in that workspace. To use these resources for model training in Machine Learning Designer, grant the RAM user the algorithm developer role. For more information, see Manage members.
Purchase fully managed Flink resources
Log on to the PAI console.
In the left-side navigation pane, click AI Computing Resources > Resource Quota. On the Resource Quota page, click the Fully Managed Flink Resources tab.
Optional: On the Fully Managed Flink Resources tab, click Activate.
NoteIf this is your first time using fully managed Flink resources, you must perform this step. Follow the instructions in Activate Alibaba Cloud Realtime Compute for Apache Flink to purchase the resources.
To purchase additional fully managed Flink resources, proceed to the next step.
On the Fully Managed Flink Resources tab, click Resource Management.
On the Realtime Compute console page, click Purchase. For details on purchasing fully managed Flink resources, see Activate Alibaba Cloud Realtime Compute for Apache Flink.
After a successful purchase, return to the Fully Managed Flink Resources tab on the Resource Quota page in the Platform for AI (PAI) console. Here, you can view the created Flink resource group, check the used and allocated CUs, and verify that the status is Running. You can also click Flink Console in the Actions column to open its management page.
Associate fully managed Flink resources with a workspace
To use fully managed Flink resources in PAI, you can associate them with a workspace in one of two ways:
When creating a workspace, add and associate a resource group. For more information, see Create and manage a workspace.
To add Flink compute resources to an existing workspace:
Log on to the PAI console.
In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the target workspace.
On the right side of the workspace details page, choose Configure Workspace > Configure Computing Resource. On the Fully Managed Flink Resources tab, associate the fully managed Flink resources. For more information, see Manage computing resources for a workspace.
Train models in Designer using Flink resources
Go to the workspace that is associated with fully managed Flink resources and create a blank pipeline on the Machine Learning Designer page. For more information, see Create a custom pipeline.
Drag components that support fully managed Flink resources onto the canvas, such as:
All components in the Alink framework (excluding Beta components), which are marked with a purple icon. Examples include algorithm components such as FM Train, FM Prediction, Swing Train, Swing Recommend, IForest Anomaly Detection, Local Outlier Factor Anomaly Detection, and One-Class SVM Anomaly Detection.
Custom algorithm components (PyAlink Script).
On the right side of the canvas, on the Pipeline Attributes tab, select Flink from the Pipeline Attributes list.
ImportantIf you run Alink components as a group, you must also set the Default Resource Type Preferred by Alink to Flink. Otherwise, the group uses its default resource type. For more information about how to set the resource type for an Alink group, see Alink components.
Run components in one of the following ways:
Run a single component that depends on fully managed Flink resources. For example, run a PyAlink Script component. For more information, see PyAlink Script.
Chain components that depend on fully managed Flink resources with components that depend on other resource types. For example, in the FM-based recommendation with Alink framework tutorial, the FM Train and FM Prediction components depend on fully managed Flink resources, while the binary classification evaluation component depends on MaxCompute resources.
Run a batch of components that depend on fully managed Flink resources. For more information, see Alink components.
After the component runs, right-click it in the pipeline and select View Log.
For more details, click the VVP log link in the log panel.