DocsICP FilingConsole
官方文档
首页

O&M Assistant

更新时间：
复制 MD 格式
产品详情
我的收藏

O&M Assistant lets you install third-party packages on an exclusive resource group for scheduling so that dependent tasks can run properly. DataWorks provides one-click installation for various built-in packages. If the built-in packages do not meet your needs, you can install other packages by using shell commands.

Applicability

  • Resource group limits: This feature applies only to an exclusive resource group for scheduling. Other types of resource groups, such as exclusive resource groups for data integration and serverless resource groups, do not support this feature.

  • Permissions: You must have the AliyunDataWorksFullAccess or ModifyResourceGroup RAM policy.

    For more information about how to grant permissions, see Product and console access control: RAM policies.

Considerations

A third-party package installed on an exclusive resource group for scheduling applies to all tasks on that resource group. Version conflicts or compatibility issues can affect all tasks. Carefully select and test packages before installing them.

Quotas and limits

  • Command management: Created installation commands cannot be modified. You can only create new commands or delete existing ones.

  • Applicable task types: Python third-party packages installed by this method apply only to Python task nodes, such as PyODPS 2 and PyODPS 3 nodes, that run on this Resource Group.

    To use a third-party package in a MaxCompute Python user-defined function (UDF), see UDF example: Use third-party packages in a Python user-defined function (UDF).

  • Command support: The manual input method does not support the pip command.

Access O&M Assistant

  1. Log on to the DataWorks console. In the target region, click Resource Group in the left-side navigation pane to open the resource group list.

  2. On the Exclusive Resource Group tab, find the resource group that is used for Data Scheduling, click the image.png icon next to the Resource Group, and select O&M Assistant to go to the corresponding page.

Install a third-party package

Create an installation command

  1. On the O&M Assistant page, click Create Command and select an installation method.

    Quick installation

    Install common third-party packages that are built into DataWorks.

    The following table describes the parameters.

    Parameter

    Description

    Command Name

    A custom name for the command.

    Command Type

    Select Quick Installation.

    DataWorks automatically generates a shell installation command based on the package you select.

    Built-in third-party package

    Select the package and version to install.

    DataWorks supports a variety of common Python 2, Python 3, and Yum packages. Examples:

    • Aliyun-python-sdk-core: The core library of the Alibaba Cloud Python SDK. Provides basic functionality for interacting with Alibaba Cloud services, such as API calls and authentication.

    • NumPy: A foundational library for scientific computing and data analysis. Provides high-performance multidimensional arrays and numerical computing features.

    • Pandas: High-performance, easy-to-use data structures and data analysis tools for processing and analyzing structured data.

    For a complete list, see the user interface.

    Generated Shell Script

    DataWorks automatically generates the installation command based on your selection.

    For example, if you select aliyun-python-sdk-core, the system generates the following command: pip install aliyun-python-sdk-core

    Timeout

    The maximum run time of the command, in seconds. The command is terminated if this period is exceeded.

    Manual input

    If the built-in packages do not meet your needs, install third-party packages by using commands.

    Parameter

    Description

    Command Name

    A custom name for the command.

    Command Type

    Select Manually enter.

    Command Content

    Enter the command that you want to run. Example: yum install -y git.

    Important

    • The manual input method does not support using pip commands to install third-party packages.

    • Manually entered commands are not guaranteed to succeed. Validate the script in advance.

    • After a third-party package is installed, you must use an absolute path to reference the package in a Data Studio node.

    Installation Directories

    The directory paths that the command can access. Separate multiple paths with semicolons (;). DataWorks adds the specified directories to a whitelist to ensure their accessibility.

    • You can install packages to a /home/ directory or a non-/home/ directory. If you install packages to a /home/ directory, they must be stored in the /home/admin/usertools/tools/ path.

    • If you do not specify a directory, the default path /home/admin/usertools/tools/ is used.

    Timeout

    The maximum execution time for the command, in seconds. The command is terminated if this period is exceeded.

  2. Click Confirm to complete the command configuration.

Run the installation command

On the O&M Assistant page, find the command and click Run command in the Operation column. The system generates a run record for the command.

View the run result

In the command run record, view basic information such as Run Status and Command Content. Click View Result in the Operation column to view logs and troubleshoot failures. After the package is installed, you can use it in scheduling tasks on the resource group.

View the environment configuration

On the O&M Assistant page, click View Detailed Environment Configuration in the upper-left corner to view the resource group's environment configuration, including installed third-party packages, their versions, and statuses.

Use the third-party package in a task

Data Studio (new version)

  1. Go to Data Studio: Go to the DataWorks workspaces page. In the top navigation bar, select the target region. Find the target workspace and choose Shortcuts > Data Studio in the Operation column.

  2. Configure the resource group: In Data Studio, find the task node for which you want to test the third-party package. In the right-side pane, click Scheduling Settings.

    • Resource Group: Select an exclusive resource group for scheduling.

      If the target Resource Group is not displayed, check whether the Resource Group is bound to the current workspace. You can go to the Resource Group page, find the target Resource Group, and click Associate Workspace in the Operation column.

  3. Debug the node: In the Run Configuration pane on the right, configure Computing Resources, Resource Group, and Script Parameters. Then, click Run in the top toolbar.

  4. Publish the node: In the top toolbar, click Publish to deploy the node to the production environment.

Data Development (old version)

  1. Go to Data Development: Log on to the DataWorks console. After you switch to the target region, choose Data Development and O&M > DataStudio in the navigation pane on the left. From the drop-down list, select the desired workspace and click Data Analytics.

  2. Configure the resource group: In Data Development, find the task node for which you want to test the third-party package. In the right-side pane, click Scheduling Settings.

    • Resource Group for Scheduling: Select an exclusive resource group for scheduling.

      If the target Resource Group is not displayed, check whether the Resource Group is bound to the current workspace. You can go to the Resource Group page, find the target Resource Group, and click Associate Workspace in the Operation column.

  3. Debug the node: In the top toolbar, click Run with Parameters (image). Configure Resource Group Name and then click Run.

  4. Publish the node: In the top toolbar, click Save and Submit to deploy the node to the production environment.

Related documents

After a third-party package is installed, tasks on the exclusive resource group for scheduling can use it. For more information, see the following topics:

Previous：ScalingNext：General reference