Custom images

Step 1: Install Docker

• Linux: Install Docker by following the official Docker documentation.

• macOS or Windows:

  • Individual developers: You can use Docker Desktop.

  • Enterprise users without a Docker Desktop license: We recommend using the open-source Rancher Desktop.

Step 2: Grant permissions

Grant the account or user the read permission for RAM roles, operational permissions for ACR, and operational permissions for MaxCompute custom images as described in the following table.

Step 3: Build a custom image

Build a custom image by using a Dockerfile based on a MaxCompute base image.

• The MaxCompute CentOS base image URL is registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/base_image:latest. The base image provides a basic environment that includes Python 3.7, Python 3.11, pip, and yum.

• The MaxCompute Ubuntu base image address is registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/ubuntu_20.04:latest.

1. Create a Dockerfile based on a MaxCompute base image. Run this command:

   vim DockerFile

2. Add the following content to the Dockerfile:

   # Use the MaxCompute CentOS base image.
   FROM registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/base_image:latest
   
   # To use the Ubuntu image, replace the image address with registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/ubuntu_20.04:latest.
   
   # Install system dependencies.
   RUN yum install vi -y
   
   # Install third-party libraries.
   RUN /usr/ali/python3.7/bin/python3 -m pip install --no-cache-dir pandas

3. Build the image from the Dockerfile.

   sudo docker build -f DockerFile -t <image_name>:<tag> .

   Parameters:

   • image_name: The name of the custom image.

   • tag: The version of the custom image.

Step 4: Push image to ACR

1. Log on to the Container Registry console and select a region in the upper-left corner.

2. In the left navigation pane, select Instances.

3. On the Instances page, click Create ACR EE. If you have already created an instance, you can skip this step.

   Important

   You can upload custom images only to Standard Edition or Premium Edition instances of ACR Enterprise Edition.

4. On the Instances page, find the target enterprise edition instance and click Manage to open the instance overview page.

5. In the left navigation pane, select Repository > Repositories.

   On the Repository > Repositories page, click Create Repository. In the Create Repository dialog box, enter the following information and click Next.

   Parameter	Required	Description
   Region	Yes	The region of the current instance is automatically populated.
   Namespace	Yes	Define the namespace for the image repository. This setting cannot be changed later. We recommend that you create a namespace that corresponds to a company, organization, or individual user, such as aliyun. We do not recommend that you create a namespace that corresponds to a module or system, such as tomcat or centos.
   Repository Name	Yes	The name must be 2 to 120 characters long and can contain lowercase letters, numbers, and the following separators: underscores (_), hyphens (-), periods (.), and forward slashes (/). The name cannot start or end with a separator.
   Repository Type	No	Public: To pull a public image, you must log on to the Enterprise Edition instance. If you want to allow anonymous pulls, enable the public anonymous pull setting for the instance. Pulling a private image of the Enterprise Edition requires logging in to an Enterprise Edition instance with pull permissions. Defaults to private.
   Tags	No	After selected, all tags other than the latest in the repository cannot be overwritten, ensuring the consistency of the container images.
   Accelerated Image	No	After selected, all images in the repository will automatically generate the accelerated images with "_accelerated" suffix Full Mode: Full mode provides a significant acceleration effect. The size of an accelerated image is approximately 130% the size of the original image. The system requires approximately 25 seconds to generate a 1 GB-sized accelerated image. If an accelerated image layer has been generated for an image layer, the system does not generate an accelerated image layer again for the image layer. Index-only Mode (Public Preview): This mode provides about 70% of the acceleration effect of the full mode. The accelerated image is about 3% of the size of the original image. It takes about 3 seconds to generate a 1 GB accelerated image. Layers that are already indexed are not re-indexed.
   Summary	Yes	Max. 100 characters
   Description	No	Supports Markdown Format

6. Configure the code source:

   For Code Source, select Local Repository and click Create Repository.

   For more information, see Use an Enterprise Edition instance to build images.

7. Push the custom image to an ACR instance in the same account.

   1. On the Repository Repositories page, click Manage in the Actions column of the target repository.

   2. In the left navigation pane, select Details.

      Follow the instructions in the Instructions on Images at the bottom of the Details page to push a custom image from your Docker environment to the Container Registry image repository.

   3. (Optional) If your machine is in a VPC, perform the following steps:

      1. Configure access control for the VPC to connect to the Enterprise Edition instance. For more information, see Configure access over a VPC.

      2. When you work with an ACR Enterprise Edition instance in a Docker environment, you must add vpc to the domain name. For example, you must change acr-test-registry.cn-wulanchabu.cr.aliyuncs.com to acr-test-registry-vpc.cn-wulanchabu.cr.aliyuncs.com.

         $ docker login --username=***@test.aliyunid.com acr-test-registry.cn-wulanchabu.cr.aliyuncs.com

      Note

      If an error occurs when you log on to the instance, check whether public network access is enabled for the repository.

Step 5: Add image to MaxCompute

Associate an existing ACR image with MaxCompute to centrally manage your development images.

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Images .

3. On the Images page, click the Custom Image tab.

4. On the Custom Image tab, click Create Image. In the Add Image dialog box, configure the following parameters:

   Note

   When you create an image for the first time, the MaxCompute Service-linked Role dialog box appears. Click OK. The system automatically creates a service-linked role to access ACR resources.

   Parameter	Description
   Image Name	Required. The name of the custom image. The image name must start with a lowercase letter, and can contain only lowercase letters, digits, hyphens (-), and underscores (_). This name is used to reference the image in MaxCompute SQL, PyODPS, and MaxFrame.
   Image Type	Required. The type of the ACR image. Only ACR Enterprise Edition images are supported.
   Enterprise Edition Image Instance	Required. Select the Enterprise Edition instance that you created in ACR.
   Image Namespace	Required. Select the image namespace that you created in ACR.
   Image Repository	Required. Select the image repository that you created in ACR.
   Image Version	Required. Select the image version that you uploaded to ACR.
   Image Description	Required. Add a description for the image.

   Click OK. The new custom image appears in the custom image list.

Step 6: Use the image

Custom images can be referenced in MaxCompute SQL UDFs, PyODPS, and MaxFrame.

• For UDFs, use session-level flags to specify the image and Python version. Example:

  set odps.sql.python.version=cp37;
  set odps.session.image = <image_name>;

• For PyODPS, use the image parameter of the execute or persist methods to specify the image. Example:

  Note

  To reference an image in PyODPS, you must upgrade PyODPS to V0.11.5 or later.

  image='<image_name>'

• For MaxFrame, use the following configuration to specify the image for the job:

  config.options.sql.settings = {
      "odps.session.image": "<image_name>"
  }