DocsICP FilingConsole
官方文档
首页

ResNet50 model training and inference based on the eGPU optimization suite

更新时间：
复制 MD 格式
产品详情
我的收藏

This topic uses the training and inference of the convolutional neural network model ResNet50 as an example to describe how to use the eGPU optimization suite through Docker or Kubernetes for GPU container virtualization and resource sharing. The ResNet50 model in this example is implemented based on the official NVIDIA open source code DeepLearningExamples.

Step 1: Prepare the model and dataset

  1. Prepare the code repository. Run the following command to obtain the model:

    git clone https://github.com/NVIDIA/DeepLearningExamples.git

  2. Download the ImageNet dataset. For information about how to obtain the dataset, see ResNet50 v1.5 For PyTorch. The directory structure of the ImageNet dataset is as follows. 

    imagenet
├── train
│   ├── n01440764
│   │  ├── n01440764_10026.JPEG
│   │  ├── n01440764_10027.JPEG
│   │  └── ......
│   ├── n01443537
│   └── ......         
└── val                
    ├── n01440764
    │  ├── ILSVRC2012_val_00000293.JPEG
    │  ├── ILSVRC2012_val_00002138.JPEG
    │  └── ......
    ├── n01443537
    └── ......

Step 2: Configure the environment

  1. Select a runtime image.

    NVIDIA PyTorch runtime images are supported. This example uses nvcr.io/nvidia/pytorch:22.11-py3.

  2. You can start an eGPU container in two ways.

    • Start the eGPU container by using Docker.

      1. Enable the eGPU container and configure resources and other parameters. For more information, see eGPU interface usage examples.

      2. Start the eGPU container. Example:

        sudo docker run \
    --runtime=nvidia \
    -e NVIDIA_DRIVER_CAPABILITIES=all \
    --network=host \
    -e AMP_VGPU_ENABLE=1 \
    -e NVIDIA_VISIBLE_DEVICES=0 \
    -e GPU_MEM_PER_DEVICE=16000000000 \
    -e GPU_UTIL_PER_DEVICE=50 \
    -e AMP_USE_HOST_DAEMON=1 \
    -v <local-path-to-DeepLearningExamples>:<docker-path-to-DeepLearningExamples> \
    -v <local-path-to-imagenet>:<docker-path-to-imagenet> \
    -P -ti \
    nvcr.io/nvidia/pytorch:22.11-py3

    • Start an eGPU container by using Kubernetes. To use eGPU to share GPU resources in a Kubernetes cluster, you must:

      1. Install the eGPU device plugin to enable GPU virtualization scheduling capabilities. For more information, see Use eGPU Kubernetes widgets.

      2. Use a YAML file to start the eGPU container. The following is an example:

        apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: amp-egpu-test
  labels:
    app: amp-egpu-test
spec:
  replicas: 1
  serviceName: "test-egpu-001"
  selector: # define how the deployment finds the pods it manages
    matchLabels:
      app: test-egpu-001
  template: # define the pods specifications
    metadata:
      labels:
        app: test-egpu-001
    spec:
      volumes:
      - name: deepLearningExamples
        hostPath:
        # directory location on host
          path: <local-path-to-DeepLearningExamples>
      - name: imagenet
        hostPath:
        # directory location on host
          path: <local-path-to-imagenet>
      containers:
      - name: test-egpu-001
        image: nvcr.io/nvidia/pytorch:22.11-py3
        imagePullPolicy: IfNotPresent
        command: [ "/bin/bash", "-c", "--" ]
        args: [ "while true; do sleep 30; done;" ]
        volumeMounts:
        - mountPath: <docker-path-to-DeepLearningExamples>
          name: deepLearningExamples
        - mountPath: <docker-path-to-imagenet>
          name: imagenet
        resources:
          limits:
            aliyun.com/gpu-compute: 50
            aliyun.com/gpu-mem: 16
        env:
          - name: AMP_VGPU_ENABLE
            value: "1"
          - name: AMP_USE_HOST_DAEMON
            value: "1"
          - name: NVIDIA_DRIVER_CAPABILITIES
            value: all

    • In the above example:

      • <local-path-to-imagenet> indicates the path where the ImageNet dataset in the Lingjun node is located.

      • <docker-path-to-imagenet> indicates the path where the dataset is mounted to the container.

      • <local-path-to-DeepLearningExamples> indicates the path where the model training code in the Lingjun node is located.

      • <docker-path-to-DeepLearningExamples> indicates the path where the model training code is mounted to the container.

        Important

        If you use the nvcr.io/nvidia/pytorch:22.11-py3 runtime image, dllogger may be missing. You can manually install dllogger in the container. For more information, see DLLogger for Python. 

        pip install git+https://github.com/NVIDIA/dllogger#egg=dllogger

Step 3: Model Training

Run the following command in the container to start the training job:

# Go to the DeepLearningExamples/PyTorch/Classification/ConvNets directory for training.
python3 ./main.py --arch resnet50 -b 128 --epochs 90  <docker-path-to-imagenet>

To verify the VRAM and computing power chunking features of eGPU, run nvidia-smi on the host to check GPU resource utilization. For more information, see ResNet50 v1.5 For PyTorch.

Step 4: Perform Model Inference

  1. Download the pre-trained model. 

    # Use the NVIDIA pre-trained model downloaded from NGC.
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/resnet50_pyt_amp/versions/20.06.0/zip -O resnet50_pyt_amp_20.06.0.zip

# Decompress the downloaded model and place it in the DeepLearningExamples/PyTorch/Classification/ConvNets directory. The decompressed file is nvidia_resnet50_200821.pth.tar.
unzip resnet50_pyt_amp_20.06.0.zip
mv nvidia_resnet50_200821.pth.tar <docker-path-to-DeepLearningExamples>/PyTorch/Classification/ConvNets/
    Important

    If you load a pre-trained model directly, some keys may be missing. In row 150 of the DeepLearningExamples/PyTorch/Classification/ConvNets/image_classification/models/model.py file, modify the load_state_dict() function as follows:

    model.load_state_dict(state_dict, strict=False)

    This modification avoids errors when loading NVIDIA's pre-trained model for inference, and is unrelated to whether eGPU is used. For more information, see load_state_dict().

  2. Start the infer job. 

    # Go to the DeepLearningExamples/PyTorch/Classification/ConvNets directory for inference
cd <docker-path-to-DeepLearningExamples>/PyTorch/Classification/ConvNets
python ./main.py --arch resnet50 --evaluate --epochs 90 -b 128 --pretrained-from-file=nvidia_resnet50_200821.pth.tar <docker-path-to-imagenet>
Previous：Best PracticesNext：ResNet50 Training Based on KSpeed