Custom images

Step 1: Install Docker

• Linux: Install Docker by following the official Docker documentation.

• macOS or Windows:

  • Individual developers: You can use Docker Desktop.

  • Enterprise users without a Docker Desktop license: We recommend using the open-source Rancher Desktop.

Step 2: Grant permissions

Grant the appropriate account or user the read permission for RAM roles, operational permissions for ACR, and operational permissions for MaxCompute custom images. See the table below for details.

Step 3: Build a custom image

Build a custom image by using a Dockerfile based on a MaxCompute base image.

• The MaxCompute CentOS base image URL is registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/base_image:latest. The base image provides a basic environment that includes Python 3.7, Python 3.11, pip, and yum.

• The MaxCompute Ubuntu base image address is registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/ubuntu_20.04:latest.

1. Create a Dockerfile based on a MaxCompute base image. Run this command:

   vim DockerFile

2. Add the following content to the Dockerfile:

   # Use the MaxCompute CentOS base image.
   FROM registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/base_image:latest
   
   # To use the Ubuntu image, replace the image address with registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/ubuntu_20.04:latest.
   
   # Install system dependencies.
   RUN yum install vi -y
   
   # Install third-party libraries.
   RUN /usr/ali/python3.7/bin/python3 -m pip install --no-cache-dir pandas

3. Build the image from the Dockerfile.

   sudo docker build -f DockerFile -t <image_name>:<tag> .

   Parameters:

   • image_name: The name of the custom image.

   • tag: The version of the custom image.

Step 4: Push image to ACR

1. 登录容器镜像服务控制台，在左上角选择地域。

2. 在左侧导航栏选择Instances。

3. 在Instances详情页，单击Create ACR EE。如已创建可忽略。

   Important

   You can upload custom images only to Standard Edition or Premium Edition instances of ACR Enterprise Edition.

4. 在Instances详情页，单击目标企业版实例的Manage，进入该实例概览页面。

5. 在左侧导航栏选择Repository > Repositories。

   在RepositoryRepositories页面，单击Create Repository，在弹出的Create Repository对话框中，填写如下信息后，单击Next。

   Parameter	Required	Description
   Region	Yes	The region of the current instance is automatically populated.
   Namespace	Yes	Define the namespace for the image repository. This setting cannot be changed later. We recommend that you create a namespace that corresponds to a company, organization, or individual user, such as aliyun. We do not recommend that you create a namespace that corresponds to a module or system, such as tomcat or centos.
   Repository Name	Yes	The name must be 2 to 120 characters long and can contain lowercase letters, numbers, and the following separators: underscores (_), hyphens (-), periods (.), and forward slashes (/). The name cannot start or end with a separator.
   Repository Type	No	Public: To pull a public image, you must log on to the Enterprise Edition instance. If you want to allow anonymous pulls, enable the public anonymous pull setting for the instance. Pulling a private image of the Enterprise Edition requires logging in to an Enterprise Edition instance with pull permissions. Defaults to private.
   Tags	No	After selected, all tags other than the latest in the repository cannot be overwritten, ensuring the consistency of the container images.
   Accelerated Image	No	After selected, all images in the repository will automatically generate the accelerated images with "_accelerated" suffix Full Mode: Full mode provides a significant acceleration effect. The size of an accelerated image is approximately 130% the size of the original image. The system requires approximately 25 seconds to generate a 1 GB-sized accelerated image. If an accelerated image layer has been generated for an image layer, the system does not generate an accelerated image layer again for the image layer. Index-only Mode (Public Preview): This mode provides about 70% of the acceleration effect of the full mode. The accelerated image is about 3% of the size of the original image. It takes about 3 seconds to generate a 1 GB accelerated image. Layers that are already indexed are not re-indexed.
   Summary	Yes	Max. 100 characters
   Description	No	Supports Markdown Format

6. 配置代码源信息：

   Code Source选择Local Repository，单击Create Repository。

   For more information, see Use an Enterprise Edition instance to build images.

7. Push the custom image to an ACR instance in the same account.

   1. 在RepositoryRepositories页面，单击目标仓库对应的Actions列的Manage。

   2. 在左侧选择Details。

      根据Details页面下方的Instructions on Images，在Docker环境中将自定义镜像上传至ACR镜像仓库。

   3. (Optional) If your machine is in a VPC, perform the following steps:

      1. Configure access control for the VPC to connect to the Enterprise Edition instance. For more information, see Configure access over a VPC.

      2. When you work with an ACR Enterprise Edition instance in a Docker environment, you must add vpc to the domain name. For example, you must change acr-test-registry.cn-wulanchabu.cr.aliyuncs.com to acr-test-registry-vpc.cn-wulanchabu.cr.aliyuncs.com.

         $ docker login --username=***@test.aliyunid.com acr-test-registry.cn-wulanchabu.cr.aliyuncs.com

      Note

      If an error occurs when you log on to the instance, check whether public network access is enabled for the repository.

Step 5: Add image to MaxCompute

Associate an existing ACR image with MaxCompute to centrally manage your development images.

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Images .

3. On the Images page, click the Custom Image tab.

4. On the Custom Image tab, click Create Image. In the Add Image dialog box, configure the following parameters:

   Note

   When you create an image for the first time, the MaxCompute Service-linked Role dialog box appears. Click OK. The system automatically creates a service-linked role to access ACR resources.

   Parameter	Description
   Image Name	Required. The name of the custom image. The image name must start with a lowercase letter, and can contain only lowercase letters, digits, hyphens (-), and underscores (_). This name is used to reference the image in MaxCompute SQL, PyODPS, and MaxFrame.
   Image Type	Required. The type of the ACR image. Only ACR Enterprise Edition images are supported.
   Enterprise Edition Image Instance	Required. Select the Enterprise Edition instance that you created in ACR.
   Image Namespace	Required. Select the image namespace that you created in ACR.
   Image Repository	Required. Select the image repository that you created in ACR.
   Image Version	Required. Select the image version that you uploaded to ACR.
   Image Description	Required. Add a description for the image.

   Click OK. The new custom image appears in the custom image list.

Step 6: Use the image

You can use the custom image in MaxCompute SQL UDFs, PyODPS, and MaxFrame.

• For UDFs, use session-level flags to specify the image and Python version. Example:

  set odps.sql.python.version=cp37;
  set odps.session.image = <image_name>;

• For PyODPS, use the image parameter of the execute or persist methods to specify the image. Example:

  Note

  To reference an image in PyODPS, you must upgrade PyODPS to V0.11.5 or later.

  image='<image_name>'

• For MaxFrame, use the following configuration to specify the image for the job:

  config.options.sql.settings = {
      "odps.session.image": "<image_name>"
  }