eGPU interface usage example

Enable eGPU

Set the AMP_VGPU_ENABLE environment variable to 1 to enable eGPU. This is the main switch. If you do not specify this variable, eGPU does not modify the container at startup, and the container starts with the default settings.

Specify GPUs

The eGPU feature supports multi-GPU scenarios for a single container. You can use the NVIDIA_VISIBLE_DEVICES environment variable to specify the universally unique identifiers (UUIDs) of the GPUs that are available to the container. For example:

NVIDIA_VISIBLE_DEVICES=GPU-28df2127-fac9-f8fd-77de-e461c85c8ef2,GPU-7bda088d-4609-6f9d-8ada-6bcba97e664a

You can also specify GPUs by their index. For example:

NVIDIA_VISIBLE_DEVICES=3,5

You can also use a mix of UUIDs and GPU indexes.

Allocate GPU memory by value

You can use the GPU_MEM_PER_DEVICE environment variable to specify the amount of available GPU memory in a container. The unit is bytes. The final amount of GPU memory that a user can query and use is calculated using the following formula:

Available_Memory = GPU_MEM_PER_DEVICE - ALIYUN_COM_GPU_MEM_RESERVED

The ALIYUN_COM_GPU_MEM_RESERVED variable is required because the eGPU feature must reserve a small amount of GPU memory to function correctly. This reserved amount is subtracted from the memory allocated to the container. The unit for ALIYUN_COM_GPU_MEM_RESERVED is megabytes (MB), and the default value is 10 MB.

Start a container

The following example shows how to start a container with approximately 5 GB of available GPU memory:

sudo docker run \
 --runtime=nvidia -e NVIDIA_DRIVER_CAPABILITIES=all \
 -e AMP_VGPU_ENABLE=1 \
    -e GPU_MEM_PER_DEVICE=5000000000 \
    -e NVIDIA_VISIBLE_DEVICES=0 \
    -e AMP_USE_HOST_DAEMON=1 \
    image_name

Allocate GPU memory by proportion

You can set two environment variables to allocate GPU memory by proportion.

• ALIYUN_COM_GPU_MEM_DEV is a positive integer that represents the GPU memory size of each card on the host.

• ALIYUN_COM_GPU_MEM_CONTAINER is a positive integer that represents the available GPU memory size for the container.

eGPU automatically retrieves the total actual GPU memory size on the host card and sets it as GPU_Memory. The unit is bytes. The formula to calculate the available GPU memory for the user is accurate to the byte:

Available_Memory = GPU_Memory * ALIYUN_COM_GPU_MEM_CONTAINER / ALIYUN_COM_GPU_MEM_DEV- ALIYUN_COM_GPU_MEM_RESERVED

• If you configure only ALIYUN_COM_GPU_MEM_CONTAINER, the program automatically retrieves the value of ALIYUN_COM_GPU_MEM_DEV and rounds it up to the nearest integer in gigabytes (GB). In this case, you must configure ALIYUN_COM_GPU_MEM_CONTAINER as an integer in GB.

• If you configure both ALIYUN_COM_GPU_MEM_DEV and ALIYUN_COM_GPU_MEM_CONTAINER, there are no unit restrictions for these variables. You can control the allocation granularity by adjusting their absolute values.

When you apply the configuration using GB as the proportional unit, the actual GPU memory that you see might be less than ALIYUN_COM_GPU_MEM_DEV * 1024 * 1024 * 1024 bytes. This is because the nominal memory amount specified by NVIDIA is not the exact amount. For example, a 16 GB V100 card typically has 16,160 MB of GPU memory, which is about 15.78 GB. The allocated memory is reduced proportionally.

Start a container with about 5 GB of available GPU memory

The ALIYUN_COM_GPU_MEM_RESERVED variable is required because the eGPU feature must reserve a small amount of GPU memory to function correctly. This reserved amount is subtracted from the memory allocated to the container. The unit for ALIYUN_COM_GPU_MEM_RESERVED is MB, and the default value is 10 MB.

The following example shows how to start a container with approximately 5 GB of available GPU memory on a 16 GB GPU card:

sudo docker run \
 --runtime=nvidia -e NVIDIA_DRIVER_CAPABILITIES=all \
 -e AMP_VGPU_ENABLE=1 \
 -e ALIYUN_COM_GPU_MEM_CONTAINER=5 \
 -e ALIYUN_COM_GPU_MEM_DEV=16 \
 -e NVIDIA_VISIBLE_DEVICES=0 \
 -e AMP_USE_HOST_DAEMON=1 \
 image_name

GPU out-of-memory

If the GPU memory used by a container exceeds the allocated amount (Container_Memory), a GPU out-of-memory (OOM) error occurs. The user process is then forced to stop. If the ALIYUN_COM_GPU_OOM_SCRIPT environment variable is set, the user process actively calls the specified script or program to send a notification before it stops.

Control computing power

You can use the GPU_UTIL_PER_DEVICE environment variable to configure the available computing power for a container. The value is a percentage. If you do not set this environment variable, or if you set the value to 0 or greater than 95, the computing power control feature is disabled.

The following example shows how to start a container with its computing power controlled to one-third of the full capacity of the card:

sudo docker run \
 --runtime=nvidia -e NVIDIA_DRIVER_CAPABILITIES=all \
 -e AMP_VGPU_ENABLE=1 \
 -e GPU_UTIL_PER_DEVICE=33 \
 -e NVIDIA_VISIBLE_DEVICES=0 \
 -e AMP_USE_HOST_DAEMON=1 \
 image_name