Custom image

Limits

• Edition limits:

  • All editions support creating and using custom images.

  • Only the Professional Edition or higher supports image building.

• Resource group limits: The custom image feature supports only serverless resource groups.

  For legacy resource groups, use Cloud Assistant to install external dependencies.

• Permission limits: You need the AliyunDataWorksFullAccess or ModifyResourceGroup permission.

  For authorization details, see RAM policy for service and console permissions.

Quotas and limits

• Image quantity: The number of custom images you can create depends on your DataWorks edition.

  • Basic Edition and Standard Edition: 10

  • Professional Edition: 50

  • Enterprise Edition: 100

• Build concurrency: You can build up to two images concurrently in each region.

• ACR image requirements:

  • Instance edition: Only Enterprise Edition instances of Alibaba Cloud Container Registry (ACR) are supported.

  • Instance architecture: Only the AMD64 architecture is supported.

  • Image size: A single image cannot exceed 5 GB.

  • Time zone configuration: You must install the tzdata time zone package to prevent container failures due to a time zone mismatch with DataWorks.

• Image building: Persistent builds are available only for custom images created based on DataWorks official images. Custom images that reference an ACR image do not support persistent builds and must be re-pulled and deployed each time a task runs.

• Supported node types and build methods:

  Node type	Build from official images	Build from ACR images
  PyODPS2	Supported	Unsupported
  PyODPS3	Supported	Unsupported
  EMR Spark	Supported	Unsupported
  EMR Spark SQL	Supported	Unsupported
  EMR SHELL	Supported	Unsupported
  Shell	Supported	Supported
  Python	Supported	Supported
  Notebook	Unsupported	Supported
  CDH	Supported	Unsupported
  Assignment Node	Supported	Unsupported

Procedure

Billing

Image building incurs a compute cost calculated as Number of CUs × Build duration. The system allocates 0.5 CU by default. For more information about billing, see Billing standards for Serverless resource groups.

Production usage

To ensure your custom images are stable, efficient, and cost-effective in the production environment, follow these recommendations.

• Persistent images: Build your stable configurations into persistent images. This avoids reinstalling dependencies for each task run, which reduces startup time, lowers compute costs, and improves stability.

• Environment consistency: Ensure that the VPC and network configurations are consistent across the serverless resource groups used for testing, building, and production scheduling, especially when accessing a private ACR repository or the public network.

• Version pinning: When installing dependencies using the Script method, we strongly recommend specifying exact version numbers (for example, pip install pandas==1.5.3). This practice prevents unexpected behavior caused by upstream library updates.

• Rollback plan: If a production task fails after an image update, roll back to a previous version using the task deployment history, or set the image to an older, stable version in the schedule settings.

Use case

This tutorial demonstrates how to use a custom image with a PyODPS node to perform Chinese word segmentation. Assume you need to segment Chinese text from a column in a MaxCompute table and store the results in another table for downstream tasks. You can pre-install the jieba word segmentation toolkit in a custom image, then use a PyODPS task with this image to process the text. The results are stored in a new table and seamlessly integrated into your downstream scheduling workflow.

1. Create test data.

   1. Create a DataWorks workspace and associate it with a MaxCompute compute resource. For details, see Create a workspace and Manage compute resources.

   2. In Data Studio, create an ODPS node (in legacy Data Studio) or a MaxCompute SQL node (in the new Data Studio) to create and populate a test table.

      Note

      The following example uses a scheduling parameter. In the Scheduling Settings on the right, set the parameter name to bday and the value to $[yyyymmdd].

      Create a test table.

      -- Create a test table.
      CREATE TABLE IF NOT EXISTS custom_img_test_tb
      (
          c_customer_id BIGINT NOT NULL,
          c_customer_text STRING NOT NULL,
          PRIMARY KEY (c_customer_id)
      )
      COMMENT 'TABLE COMMENT'
      PARTITIONED BY (ds STRING COMMENT 'partition')
      LIFECYCLE 90;
      -- Insert test data into the table.
      INSERT INTO custom_img_test_tb PARTITION (ds='${bday}') (c_customer_id, c_customer_text) VALUES
      (1, 'As evening snow is due, won''t you stay for a drink?'),
      (2, 'Moon sets, crows caw, frost fills the sky; river maples and fishing fires sleeplessly sigh.'),
      (3, 'Where mountains and rivers seemed to end, a village appears amidst the willows and flowers.'),
      (4, 'In spring slumber, unaware of dawn, bird songs fill the air all around.'),
      (5, 'Quiet night thoughts: bright moonlight before my bed, like frost upon the ground.'),
      (6, 'The bright moon rises over the sea; from the farthest shores, we share this moment.'),
      (7, 'The swallows of the noble Wang and Xie clans of old now fly into the homes of common folk.'),
      (8, 'A line of white egrets ascends the blue sky; the window frames the ancient snows of the Western Range.'),
      (9, 'Life is for joy, so live it to the fullest; let not your golden cup be empty under the moon.'),
      (10, 'Heaven gave me talent, so it must be put to use; a thousand pieces of gold, once spent, will come back again.');

   3. Save and deploy the node.

2. Create a custom image.

   For details, see Create a custom image. Configure the key parameters as follows:

   • Image Name/ID: Select dataworks_pyodps_task_pod, the official DataWorks image for PyODPS nodes.

   • Supported Task Types: Select PyODPS2 and PyODPS 3.

   • Installation Package: Select Python3 and jieba.

3. Publish the custom image and associate it with your workspace. For details, see Publish a custom image and Modify the workspace association of an image.

4. Use the custom image in a scheduled task.

   1. In Data Studio, create a PyODPS 3 node and add the following code:

      Use the custom image.

      import jieba
      from odps import ODPS
      from odps.models import TableSchema as Schema, Column, Partition
      # Read data from the table.
      table = o.get_table('custom_img_test_tb')
      partition_spec = f"ds={args['bday']}"
      with table.open_reader(partition=partition_spec) as reader:
          records = [record for record in reader]
      # Segment the extracted text.
      participles = [' | '.join(jieba.cut(record['c_customer_text'])) for record in records]
      # Create the destination table.
      if not o.exist_table("participle_tb"):
          schema = Schema(columns=[Column(name='word_segment', type='string', comment='Word segmentation result')], partitions=[Column(name='ds', type='string', comment='Partition field')])
          o.create_table("participle_tb", schema)
      # Write the segmentation results to the destination table.
      # Define the output partition and table.
      output_partition = f"ds={args['bday']}"
      output_table = o.get_table("participle_tb")
      # If the partition does not exist, create it.
      if not output_table.exist_partition(output_partition):
          output_table.create_partition(output_partition)
      # Write the segmentation results to the table.
      record = output_table.new_record()
      with output_table.open_writer(partition=output_partition, create_partition=True) as writer:
          for participle in participles:
              record['word_segment'] = participle
              writer.write(record)

   2. In the scheduling settings on the right, configure the following key parameters:

      • Scheduling Parameter: Set the parameter name to bday and the value to $[yyyymmdd].

      • Resource Group for Scheduling: Select the same serverless resource group that you specified for the Publish Image during Test Resource Group.

      • Image: Select the published custom image associated with the current workspace.

   3. Debug the node.

      • If you use legacy Data Studio, click Run with Parameters () in the node toolbar. Configure Resource Group Name, CUs for Node Running, Image, and Custom Parameters, and then click Run.

      • If you use the new Data Studio, in the Run Configuration panel on the right, configure Computing Resources, Resource Group, Calculate CU, Image, and Script Parameters. Then, click Run in the node toolbar.

   4. (Optional) Create an ad hoc query (in legacy Data Studio) or a SQL file in your personal directory (in the new Data Studio), and then run the following SQL statement to check the output table for data.

      -- Replace <partition_date> with the actual partition date.
      SELECT * FROM participle_tb WHERE ds=<partition_date>;

      If the query returns data, the result includes the word_segment column (segmentation results, with words separated by vertical bars |) and the ds column (partition date).

   5. Deploy the PyODPS node to the production environment.

      Note

      Image changes made in Data Studio are not automatically synchronized to the production environment. You must deploy the task for the changes to take effect. For details, see Deploy tasks (legacy Data Studio) or Deploy nodes/workflows (new Data Studio).

5. Build a persistent image from the custom image. For details, see Build a persistent image.

FAQ

Q: A Python task reports the error "urllib3 v2.0 only supports OpenSSL 1.1.1+".

A: The urllib3 v2.0 package requires OpenSSL 1.1.1 or later. To resolve this, downgrade urllib3 to a compatible version. For example: /home/tops/bin/pip3 install urllib3==1.26.16.

Related documentation

• DataWorks public images

• Use serverless resource groups

• Develop tasks on a self-managed Hadoop cluster

• Develop with MaxCompute PyODPS

Appendix: Installation commands

To install packages in a custom image using the script method, use the following commands.

• For PyODPS 2 nodes, run the following commands.

  pip install <package-name> -i  https://pypi.tuna.tsinghua.edu.cn/simple

  pip install <package-name>

  Note

  If prompted to upgrade PIP, run the following command: pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple.

• For PyODPS 3 nodes, run the following commands.

  /home/tops/bin/pip3 install <package-name> -i https://pypi.tuna.tsinghua.edu.cn/simple

  /home/tops/bin/pip3 install <package-name>

  Note

  • If prompted to upgrade PIP, run the following command: /home/tops/bin/pip3 install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple.

  • If the error /home/admin/usertools/tools/cmd-0.sh: line 3: /home/tops/bin/python3: No such file or directory occurs, submit a ticket to request the required permissions.

  The following table lists public Python mirror sources.

  Organization	Mirror url
  Alibaba Cloud	https://mirrors.aliyun.com/pypi/simple/ Important You can get Python packages from Alibaba Cloud without enabling internet access.
  Tsinghua University	https://pypi.tuna.tsinghua.edu.cn/simple
  University of Science and Technology of China (USTC)	https://pypi.mirrors.ustc.edu.cn/simple/