DocsICP FilingConsole
Documentation
Home Page Platform For AI Others AI acceleration Accelerate distributed deep learning training with EPL

Accelerate distributed deep learning training with EPL

Updated at:
Copy as MD
Product
My Favorites

EPL (Easy Parallel Library) is a distributed model training framework that integrates multiple training optimization technologies. Simple API operations let you implement any parallelism strategy. You can use EPL to run low-cost, high-performance distributed model training. In DLC, EPL accelerates distributed TensorFlow training.

Prerequisites

Before you begin, make sure that you have completed the following preparations:

  • Authorize a service-linked role for the DLC service. For more information, see Cloud product dependencies and authorization: DLC.

  • You have installed an image environment: an Alibaba Cloud Image or a community image (NVIDIA TensorFlow 1.15 or TensorFlow-GPU 1.15).

    Note

    For DLC, we recommend that you select the community image tensorflow-training:1.15-gpu-py36-cu100-ubuntu18.04. You can install EPL by submitting a command in DLC, so you do not need to install it separately.

Step 1: Set up the code configuration

You can use EPL to write distributed TensorFlow training code. For more information, see the Quick Start guide.

You can also use an EPL code sample to get started with distributed TensorFlow training quickly. This topic uses ResNet-50 training data to configure a code build. The latest version is cloned automatically when you submit a TensorFlow training task. To configure the code build, follow these steps.

  1. Go to the Code Configuration page.

    1. Log on to the PAI console.

    2. In the left-side navigation pane, click Workspaces and then click the name of the desired workspace.

    3. In the left-side navigation pane, choose AI Computing Asset Management > Source Code Repositories.

  2. On the Source Code Repositories page, click Create Code Build.

  3. On the Create Code Build page, configure the parameters and click Submit.

    Set Git Repository Address to https://github.com/alibaba/EasyParallelLibrary.git and Code Branch to main. For information about other parameters, see Configure a code build.

Step 2: Start a training task

  1. Go to the Create Job page.

    1. Log on to the PAI console. Select the target region at the top of the page, select the target workspace on the right side, and then click Go to DLC.

    2. On the Distributed Training (DLC) page, click Create Task.

  2. On the Create Task page, configure the following key parameters and click OK. For information about other parameters, see Create a training job.

    • In the Basic Information section, enter a job name.

    • In the Environment Information section, configure the following parameters.

      Parameter

      Example value

      Node Image

      Select Alibaba Cloud Image > tensorflow-training:1.15-gpu-py36-cu100-ubuntu18.04.

      Start Command

      		 
      apt update
apt install libnccl2 libnccl-dev
cd /root/code/EasyParallelLibrary/
pip install .
cd examples/resnet
bash scripts/train_dp.sh

      Source Code Repositories

      From the Online Configuration drop-down list, select the code build that you created in Step 1 and set Branch to main.

    • In the Resource Information section, configure the following parameters.

      Parameter

      Example value

      Resource Source

      Select Public Resources.

      Framework

      Select TensorFlow.

      Task Resources

      For worker nodes, configure the following parameters:

      • Nodes: Set to 2. Two worker nodes meet the needs of basic distributed training. Adjust the value based on your training scale.

      • Instance Type: Select the GPU specification ecs.gn6v-c8g1.2xlarge.

      Maximum Running Time

      Set this to 2 hours.

    • Configure the Task resource configuration parameter settings as follows.

      Parameter

      Example value

      Nodes

      Set to 2. Adjust based on your training needs.

      Node configuration

      On the GPU instance tab, select ecs.gn6v-c8g1.2xlarge.

      Maximum running time

      2 hours.

  3. In the distributed training job list, click the job name to go to the details page and check how the job runs. For more information, see View training job details.

Related documents

Previous：Integrate custom models with TorchAccNext：Transformer training acceleration (Pai-Megatron-Patch)