使用Megatron-Deepspeed训练GPT-2并生成文本

创建ECS实例

在ECS实例创建页面，创建ECS实例。

关键参数说明如下，其他参数的配置，请参见自定义购买实例。

• 实例规格：选择ecs.gn7i-c8g1.2xlarge（单卡NVIDIA A10）。

• 镜像：使用云市场镜像，名称为aiacc-train-solution，您可以直接通过名称搜索该镜像，选择最新版本即可。

• 公网IP：选中分配公网IPv4地址，带宽计费方式选择按使用流量，带宽峰值选择100 Mbps，以加快模型下载速度。

安装Megatron-Deepspeed框架

1. 使用root用户远程登录ECS实例。具体操作，请参见使用Workbench登录Linux实例。

2. 执行以下命令，启动容器。

   docker run -d -t --network=host --gpus all --privileged --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --name megatron-deepspeed -v /etc/localtime:/etc/localtime -v /root/.ssh:/root/.ssh nvcr.io/nvidia/pytorch:21.10-py3

3. 执行以下命令，进入容器终端。

   docker exec -it megatron-deepspeed bash

4. 执行以下命令，下载Megatron-DeepSpeed框架。

   git clone https://github.com/bigscience-workshop/Megatron-DeepSpeed

5. 执行以下命令，安装Megatron-DeepSpeed框架。

   cd Megatron-DeepSpeed
   pip install -r requirements.txt

处理数据

本指南使用1GB 79K-record的JSON格式的OSCAR数据集。

1. 执行以下命令，下载数据集。

   wget https://huggingface.co/bigscience/misc-test-data/resolve/main/stas/oscar-1GB.jsonl.xz
   wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json
   wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt

2. 执行以下命令，解压数据集。

   xz -d oscar-1GB.jsonl.xz

3. 执行以下命令，预处理数据。

   python3 tools/preprocess_data.py \
       --input oscar-1GB.jsonl \
       --output-prefix meg-gpt2 \
       --vocab gpt2-vocab.json \
       --dataset-impl mmap \
       --tokenizer-type GPT2BPETokenizer \
       --merge-file gpt2-merges.txt \
       --append-eod \
       --workers 8

   如果回显信息类似如下所示，表示预处理数据完成。

   Processed 77800 documents (1461.3559765085388 docs/s, 18.83362500949127 MB/s).
   Processed 77900 documents (1462.414931600933 docs/s, 18.84950270814139 MB/s).
   Processed 78000 documents (1463.0352126728164 docs/s, 18.853936674376666 MB/s).
   Processed 78100 documents (1463.3803054046455 docs/s, 18.857602063612152 MB/s).
   Processed 78200 documents (1462.085087208823 docs/s, 18.838893606395377 MB/s).
   Processed 78300 documents (1462.551807430541 docs/s, 18.844534169923612 MB/s).
   Processed 78400 documents (1463.9434993070793 docs/s, 18.859076740081353 MB/s).
   Processed 78500 documents (1464.8344186264242 docs/s, 18.867590722301323 MB/s).
   Processed 78600 documents (1458.4663461796881 docs/s, 18.81236130437488 MB/s).
   Processed 78700 documents (1459.9745217665234 docs/s, 18.832928410608098 MB/s).
   Processed 78800 documents (1461.624369289465 docs/s, 18.85912894903866 MB/s).
   Processed 78900 documents (1463.3125366418903 docs/s, 18.877083688939067 MB/s).
   Processed 79000 documents (1462.73970847226 docs/s, 18.881475117625264 MB/s).

4. 执行以下命令，新建data目录。

   mkdir data

5. 执行以下命令，将处理好的数据移动到data目录下。

   mv meg-gpt2* ./data
   mv gpt2* ./data

预训练

本示例使用单机单卡的GPU实例完成GPT-2 MEDIUM模型的预训练。

1. 创建预训练脚本文件。

   1. 执行以下命令，创建预训练脚本文件。

      vim pretrain_gpt2.sh

   2. 按i键，进入编辑模式，在文件中添加以下信息。

      #! /bin/bash
      
      # Runs the "345M" parameter model
      
      GPUS_PER_NODE=1
      # Change for multinode config
      MASTER_ADDR=localhost
      MASTER_PORT=6000
      NNODES=1
      NODE_RANK=0
      WORLD_SIZE=$(($GPUS_PER_NODE*$NNODES))
      
      DATA_PATH=data/meg-gpt2_text_document
      CHECKPOINT_PATH=checkpoints/gpt2
      
      DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT"
      
      python -m torch.distributed.launch $DISTRIBUTED_ARGS \
             pretrain_gpt.py \
             --tensor-model-parallel-size 1 \
             --pipeline-model-parallel-size 1 \
             --num-layers 24 \
             --hidden-size 1024 \
             --num-attention-heads 16 \
             --micro-batch-size 4 \
             --global-batch-size 8 \
             --seq-length 1024 \
             --max-position-embeddings 1024 \
             --train-iters 5000 \
             --lr-decay-iters 320000 \
             --save $CHECKPOINT_PATH \
             --load $CHECKPOINT_PATH \
             --data-path $DATA_PATH \
             --vocab-file data/gpt2-vocab.json \
             --merge-file data/gpt2-merges.txt \
             --data-impl mmap \
             --split 949,50,1 \
             --distributed-backend nccl \
             --lr 0.00015 \
             --lr-decay-style cosine \
             --min-lr 1.0e-5 \
             --weight-decay 1e-2 \
             --clip-grad 1.0 \
             --lr-warmup-fraction .01 \
             --checkpoint-activations \
             --log-interval 10 \
             --save-interval 500 \
             --eval-interval 100 \
             --eval-iters 10 \
             --fp16

   3. 按Esc键，输入:wq后，按Enter键保存文件。

2. 修改测试代码。

   Megatron源码有一个断言需要注释掉，以保证代码正常运行。

   1. 执行以下命令，打开测试代码文件。

      vim /workspace/Megatron-DeepSpeed/megatron/model/fused_softmax.py +191

   2. 按i键，进入编辑模式，在assert mask is None, "Mask is silently ignored due to the use of a custom kernel"前加#。

      def forward_fused_softmax(self, input, mask):
        b, np, sq, sk = input.size()
        scale = self.scale if self.scale is not None else 1.0
      
        if self.attn_mask_type == AttnMaskType.causal:
            assert sq == sk, "causal mask is only for self attention"
            #assert mask is None, "Mask is silently ignored due to the use of a custom kernel"

   3. 按Esc键，输入:wq后，按Enter键保存文件。

3. 预训练。

   1. 执行以下命令，开始预训练。

      root@GPT2-xd-gn7e:/workspace/Megatron-DeepSpeed# nohup sh ./pretrain_gpt2.sh &
      [1] 866
      root@GPT2-xd-gn7e:/workspace/Megatron-DeepSpeed# nohup: ignoring input and appending output to 'nohup.out'
      tail -f nohup.out

   2. 执行如下命令，可以持续地查看nohup.out的输出，达到监控程序的效果。

      tail -f nohup.out

      如果回显信息类似如下所示，表示预训练完成。

      -------------------------------------------------------------------------------------------------------
      saving checkpoint at iteration     5000 to checkpoints/gpt2
        successfully saved checkpoint at iteration     5000 to checkpoints/gpt2
      time (ms) | save-checkpoint: 5442.60
      [after training is done] datetime: 2023-04-25 15:49:26
      -------------------------------------------------------------------------------------------------------
      valid loss at the end of training for val data | lm loss value: 5.167194E+00 | lm loss PPL: 1.754219E+02 |
      -------------------------------------------------------------------------------------------------------
      saving checkpoint at iteration     5000 to checkpoints/gpt2
        successfully saved checkpoint at iteration     5000 to checkpoints/gpt2
      Evaluating iter 10/10
      -------------------------------------------------------------------------------------------------------
      test loss at the end of training for test data | lm loss value: 5.178889E+00 | lm loss PPL: 1.774856E+02 |

      说明

      预训练完成大概需要1小时30分钟，如果超时断开了ECS连接，重新远程登录ECS实例后，执行以下命令，继续查看预训练进度。预训练完成后，可以执行Ctrl+Z命令退出。

      docker exec -it megatron-deepspeed bash
      cd Megatron-DeepSpeed
      tail -f nohup.out

4. （可选）执行以下命令，查看生成的模型checkpoint路径。

   本示例生成的模型checkpoint路径设置在/workspace/Megatron-DeepSpeed/checkpoints/gpt2。

   ll ./checkpoints/gpt2

使用GPT-2模型生成文本

执行以下命令，安装相关依赖。

pip install mpi4py

如果回显信息类似如下所示，表示依赖安装完成。

root@GPT2-xd-gn7e:/workspace/Megatron-DeepSpeed# pip install mpi4py
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting mpi4py
  Downloading mpi4py-3.1.4.tar.gz (2.5 MB)
     |████████████████████████████████| 2.5 MB 278 kB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: mpi4py
  Building wheel for mpi4py (PEP 517) ... done
  Created wheel for mpi4py: filename=mpi4py-3.1.4-cp38-cp38-linux_x86_64.whl size=6019141 sha256=af317c2cad831c3e6e4178f74dc6874bf47aeafef2c5d5472cd201b8efd199d6
  Stored in directory: /tmp/pip-ephem-wheel-cache-ux27smy6/wheels/f3/35/48/0b9a7076995eea5ea64a7e4bc3f0f342f453080795276264e7
Successfully built mpi4py
Installing collected packages: mpi4py
Successfully installed mpi4py-3.1.4

创建文本生成脚本。

1. 执行以下命令，创建文本生成脚本。

   vim generate_text.sh

2. 按i键，进入编辑模式，在文件中增加以下内容。

   #!/bin/bash
   
   CHECKPOINT_PATH=checkpoints/gpt2
   VOCAB_FILE=data/gpt2-vocab.json
   MERGE_FILE=data/gpt2-merges.txt
   
   python tools/generate_samples_gpt.py \
          --tensor-model-parallel-size 1 \
          --num-layers 24 \
          --hidden-size 1024 \
          --load $CHECKPOINT_PATH \
          --num-attention-heads 16 \
          --max-position-embeddings 1024 \
          --tokenizer-type GPT2BPETokenizer \
          --fp16 \
          --micro-batch-size 2 \
          --seq-length 1024 \
          --out-seq-length 1024 \
          --temperature 1.0 \
          --vocab-file $VOCAB_FILE \
          --merge-file $MERGE_FILE \
          --genfile unconditional_samples.json \
          --num-samples 2 \
          --top_p 0.9 \
          --recompute

3. 按Esc键，输入:wq后，按Enter键保存文件。

4. 执行以下命令，生成文本。

   sh ./generate_text.sh

   如果回显信息类似如下所示，表示生成文本完成。

   >>> done with compiling and loading fused kernels. Compilation time: 1.379 seconds
   building GPT model ...
    loading checkpoint from checkpoints/gpt2 at iteration 5000
    checkpoint version 3.0
     successfully loaded checkpoint from checkpoints/gpt2 at iteration 5000
   Avg s/batch: 51.303290367126465

5. 执行以下命令，查看生成的JSON格式的文本文件。

   vim unconditional_samples.json

   回显信息类似如下所示。

    {"text": "long led director John Ophjaj Sissdi in Maying, 2009\nTerbk Yosta to defer his wife Tanasin Police the Sochi colonies subsequently accused him that Zakasbadrahatas was arrested in April
      25 at Arizona's Forest Intelligence Commission.\nVirginiaman Bogarani was Director of Cranon to,\" with Tony Pauloya Hewarfall.\nDolar Maulzali's complaint appealed then that Kasszon to
     Lim.\nThe incident was spokesman for connection with \"labasing\" reports of other fire parties made by an \ufffdogident The airline, which was stated earlier about the precise evidence that
     YcAmasbon, by Mr Vuwa, said in U. Louis Sharat Jr. may be violating the forces guilty of the protests with respondent.\nMcCarloak/Boster Barthior met an interview by Ji Havylu Etzon, who fled the
      November period of 1997, Geng Jr.\nNaturally in her husband came to Frerimingang. They sensed that a recent reasoning was spotted in a policy making for a preliminary statement by the
     others.\nUnlike a deputy for Sir Edward Panguhl, Major General Court, said that the Magas of a fact called the Sabired Union Supreme Committee would include payment card tax \"Pezur as the
     Council\" (Gen); and announcement that RonarBau was concerned about the investigators and Ms Afrosl.\nConclusions of Mr DerwizZwa would judge Mr Ward and Mrs Metzon 21.\nA new assessment of David
      Heris has shown that Albert Lubar Jue must have mentioned that Bill of the Office of Poustes judges such that Sharif may have been a part of a breach of a Democrat based on the Committee
     following the evidence-how-of-culture said, that Danes has been a Secretary of the Committee, on which the failed evidence, should be called if they have a place accused they were expressed in
     the Galas's investigation.\n[At the Yemen National Minister for Interlut] implemented the debate of a Judge except as one of the arrests commented above, Mr King Foream concluded that Governor
     Faug judgment was decision that The House determines, when Henri joined a Lieutenant to desider the Benjamin press for a statement of Pismag was won 18 May 18 January 1925. That act this
     Agreement.\n[Gen] said the court statement concerning Mr McT man entering the bill on court where the plea left\u00e2vi and cappo should suffer any reasonable damages that while a member of the
     Supreme Court was pending.\n[sh] those witnesses have been arrested in court accounts. They are two members -- the own Commissioner has been told the perpetrator has been treated with Radisten in
      1904, none of which claims were subject to a member hearing of the appeals that they are bound by him, and the case did not approve it for this reason. They have yet then been accused of the
     doubt that meeting members were indicted with others, like that, \"as we know the 'gate pound.\"[ That contradiction was caused by the victims' Judge says that the reason they had been
     denied.\"\nMr Memordi was told in a position to affirm that the Sheriff of Leahes at 24 July 2015 that we could not appeal to it because the Statement held that Mr under close crimes in
     accordance with Luke B. Kurtina (1964), which was instead taken to where Mr Breard notes he does not matter that he is in terms of man with a terrorist attack.\nThe said bill was aware that the
     plaintiffs knew that there was so good. But they were at the time in the department\u2019s failure and the plaintiffs were refused to a member of Glaine.\nWhenz's employer claim that there were a
      matter of evidence of a sentence with Earsact[grandruries], \"the Supreme Court suffers not be contend with our Department of Nazise County of State Specialist; and is it, we may please confront
      all the extent that this evidence is part of law for you and hereafter conclude that he has settled up for the defendant, if the claimant has made a violation of the president so that the jury
     has made such a said, if he is not significant, the false conf.\n(One portion of the Constitution by 11 Jul 2015, Brief answers that two bishops admitted one first where he commanded the contract
      that Does Manals conclude therein:\n( Neither such calls and explanation: that is otherwise in mind, or in judgment about the doctrine that He \"moreits come to * not grant, or be very little as
      proof of any that exists, or in any case:\n(John): Neither the Republicans talked when the California House rejected Foreign freedoms that immunity was evidence of \u2019cindance; that is an
     action in accordance with all the insisted above.\u201d\nBrop. [NYDb04) recently made he insisted that the statements prohibiting USME deserve no personal claims of what they were proclaimed and
     married before claiming a court member claimed that there was", "length": 1022, "finished": false}
     {"text": "Lisa knows the real caveat that hands on his dream of murder. So I thought I was saved by certain men, too and I didn\u2019t know if I was sick, that could still be easier than to ask
     other people that I was.\n\u201cMy mom stood across the flats and continued with the dress,\u201d he said. \u201c\u2019s all very little fun, if I would be directly concerned then I didn\u2019t
     actually throw my shop on him\u2019s co-invet and trouble bringing me on them.\u201d\n\u201cThey asked me if I was so determined.\u201d\n\u201cMy husband stood out of the chair and took them up,
     that our Dad. After two years in 1926, Helen announced the actions that in 1975 was I\u201cgate time is not fully protected. I said she asked me what I was working hard. I wasn\u2019t very just
     going to know I was very naive. We haven\u2019t been involved in my work in the room. I think I was my spouse who I was disappointed at one time. We went home and said I was loving it with a 1 of
      the time and had not dropped away.\u201d\nWe can always go through this and take on us on the trip. To all, we think we can give our family an interpreter in my dad. I also really want to take a
      call. I can\u2019t be taking place in my room anymore and have to sit on many places. You don\u2019t want to get old things! I had to do. And then this does not make any things, but I was wrong.
      I was trying to write it.\u201d\n\u201cWe really haven\u2019t made any conversation about the reasons I was overwhelmed about the things I\u2019d ever need. Then, my son got killed up there and
     about to track my child\u2019s. camp. That was to me. You were only about a bit sure I decided I was hit off. I liked she hadn\u2019t heard her.\u201d\nI was talking about my children; I always
     wanted to make a great time to make her with me by letting alone my child care. I had thought so that she would be a path that will feed my child into one direction, get to do it. It would be my
     \u201cal partner. I\u2019ve taken the whole son to school and I didn\u2019t know I was born on her.\u201d\n\u201cWe were an extra number of people trying to tell me they could me about as I was
     waiting. I had to make friends and walk at more than I was just getting in. When the first we went to the Middle and crew had been... He\u2019s got perfect. I saw one scene he went. I asked
     \u201cwhat\u2019s what\u2019s better.\u201d It was a \u201cwhite\u201d church\u2026 So I don\u2019t just go where I tried. Instead he was seen on the Brsoas I first returned. I learned about
     \u2026 and I had to be a part of it\u2019s house, where I worked in and my mother \u2026 I read was excited when I listened. He added to me after I held the r\u2026 He made a lot of dates with
     their brothers and friends in a faster path in school. However, I\u2019m still going to get there in the center with my father\u2019s forehead and go out. And I know there was a side working at
     me. I learned how my dad got a lucky man. There was absolutely no luck. There was me going to prepare my son. I was sweating. \u201cCause it\u2019s gonna.\u201d I said, \u201cBut I don\u2019t get
      it up. I think I could do well.\u201d I sat asleep with my husband\u2019s photos. On my arms sounded hopeless. I had nothing since my friends\u2019 eyes do. I needed to break me into\u2019 my
     arms, tired and gr over. \u201cWe call me all the men.\u201d She had broken my hands. I didn\u2019t make me know it would be with him. That\u2019s what I\u2019d do in the room. To say to
     Victoria, so I don\u2019t think don\u2019t just ask. There were people who. I did my parent, but was about his parents in the field. My mom didn\u2019t help me in my room. When I got in, I am a
     mission. It\u2019s a thing.\n\u201cMy best friend said I didn\u2019t change my couch us 201d was a source. \u201cThere was no square, my sister told me on my blog, \u2018It\u2019s a walk. \u2018I
      got to your husband,\u2019 \u2018If I had said, \u2018", "length": 1022, "finished": false}