安装和使用Deepytorch Inference

Deepytorch Inference是阿里云自研的AI推理加速器,针对Torch模型,可提供显著的推理加速能力。本文主要介绍安装并使用Deepytorch Inference的操作方法,以及推理效果展示。

背景信息

Deepytorch Inference通过调用deepytorch_inference.compile(model)接口即可实现推理性能加速。使用Deepytorch Inference前,您需要先使用torch.jit.script或者torch.jit.trace接口,将PyTorch模型转换为TorchScript模型,更多信息,请参见PyTorch官方文档

本文将为您提供分别使用torch.jit.scripttorch.jit.trace接口实现推理性能加速的示例,更多信息,请参见推理性能效果展示

安装Deepytorch Inference

重要

安装Deepytorch Inference前,请确认您已创建配备了NVIDIA GPU卡的GPU实例(即A10、V100或T4 GPU)。

连接GPU实例后,使用pip工具安装指定版本的torch(例如2.0.1版本)和Deepytorch Inference软件包,其中,DeepyTorch Inference的软件包可以通过PyPI进行分发和安装,方便开发者通过简单的命令行工具安装和管理软件。

说明

如需选择特定版本的Deepytorch Inference软件包,则需从deepytorch inference中选择该版本的whl包。例如,需要python 3.8+pytorch 1.13+cuda 11.7版本的Deepytorch Inference软件包,则直接下载deepytorch_inference(python 3.8+pt 1.13.1+cuda 117)即可。

pip install torch==2.0.1 deepytorch-inference -f https://aiacc-inference-public-v2.oss-cn-hangzhou.aliyuncs.com/aiacc-inference-torch/stable-diffusion/aiacctorch_stable-diffusion.html

使用Deepytorch Inference

您仅需要在模型的推理脚本中增加如下代码,即可启用Deepytorch Inference的推理优化功能,增加的代码如下所示:

  • import deepytorch_inference # 导入deepytorch_inference软件包
  • deepytorch_inference.compile(mod_jit) # 进行编译

推理性能效果展示

基于不同模型,为您展示使用Deepytorch Inference的推理性能效果,实际的推理加速效果与模型、GPU机型等因素有关,本文以A10机型(例如gn7i、ebmgn7i或ebmgn7ix)为例进行推理测试。关于模型支持情况,请参见模型支持情况

基于ResNet50模型执行推理

以下示例将基于ResNet50模型,并调用torch.jit.script接口执行推理任务,执行1000次后取平均时间,将推理耗时从3.686 ms降低至0.396 ms以内。

  • 原始版本

    原始代码如下所示:

    import time
    import torch
    import torchvision.models as models
    mod = models.resnet50(pretrained=True).eval()
    mod_jit = torch.jit.script(mod)
    mod_jit = mod_jit.cuda()
    
    in_t = torch.randn([1, 3, 224, 224]).float().cuda()
    
    # Warming up
    for _ in range(10):
        mod_jit(in_t)
    
    inference_count = 1000
    # inference test
    start = time.time()
    for _ in range(inference_count):
        mod_jit(in_t)
    end = time.time()
    print(f"use {(end-start)/inference_count*1000} ms each inference")
    print(f"{inference_count/(end-start)} step/s")
    

    执行结果如下,显示推理耗时大约为3.686 ms。

    renet50原始版.jpg

  • 加速版本

    仅需要在原始版本代码中插入如下代码即可实现推理性能加速:

    • import deepytorch_inference

    • deepytorch_inference.compile(mod_jit)

    更新后的代码如下:

    import time
    import deepytorch_inference  # 导入deepytorch_inference软件包
    import torch
    import torchvision.models as models
    mod = models.resnet50(pretrained=True).eval()
    mod_jit = torch.jit.script(mod)
    mod_jit = mod_jit.cuda()
    mod_jit = deepytorch_inference.compile(mod_jit)  # 进行编译
    
    in_t = torch.randn([1, 3, 224, 224]).float().cuda()
    
    # Warming up
    for _ in range(10):
        mod_jit(in_t)
    
    inference_count = 1000
    # inference test
    start = time.time()
    for _ in range(inference_count):
        mod_jit(in_t)
    end = time.time()
    print(f"use {(end-start)/inference_count*1000} ms each inference")
    print(f"{inference_count/(end-start)} step/s")
    

    执行结果如下,显示推理耗时为0.396 ms。相较于之前的3.686 ms,推理性能有了显著提升。

    renet50升级版.jpg

基于Bert-Base模型执行推理

以下示例将基于Bert-Base模型,并调用torch.jit.trace接口执行推理任务,将推理耗时从4.955 ms降低至0.418 ms以内。

  1. 执行以下命令,安装transformers包。

    pip install transformers
  2. 分别运行原始版本和加速版本的Demo,并查看运行结果。

    • 原始版本

      原始代码如下所示:

      from transformers import BertModel, BertTokenizer, BertConfig
      import torch
      import time
      
      enc = BertTokenizer.from_pretrained("bert-base-uncased")
      
      # Tokenizing input text
      text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
      tokenized_text = enc.tokenize(text)
      
      # Masking one of the input tokens
      masked_index = 8
      tokenized_text[masked_index] = '[MASK]'
      indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
      segments_ids = [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, ]
      
      # Creating a dummy input
      tokens_tensor = torch.tensor([indexed_tokens]).cuda()
      segments_tensors = torch.tensor([segments_ids]).cuda()
      dummy_input = [tokens_tensor, segments_tensors]
      
      # Initializing the model with the torchscript flag
      # Flag set to True even though it is not necessary as this model does not have an LM Head.
      config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
          num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, torchscript=True)
      
      # Instantiating the model
      model = BertModel(config)
      
      # The model needs to be in evaluation mode
      model.eval()
      
      # If you are instantiating the model with `from_pretrained` you can also easily set the TorchScript flag
      model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
      
      model = model.eval().cuda()
      
      # Creating the trace
      traced_model = torch.jit.trace(model, dummy_input)
      
      # Warming up
      for _ in range(10):
          all_encoder_layers, pooled_output = traced_model(*dummy_input)
      
      inference_count = 1000
      # inference test
      start = time.time()
      for _ in range(inference_count):
          traced_model(*dummy_input)
      end = time.time()
      print(f"use {(end-start)/inference_count*1000} ms each inference")
      print(f"{inference_count/(end-start)} step/s")
      

      执行结果如下,显示推理耗时大约为4.955 ms。

      bert模型原始版本.jpg

    • 加速版本

      仅需要在原始版本代码中插入如下代码即可实现推理性能加速:

      • import deepytorch_inference

      • deepytorch_inference.compile(traced_model)

      更新后的代码如下:

      from transformers import BertModel, BertTokenizer, BertConfig
      import torch
      import deepytorch_inference  # 导入deepytorch-inference软件包
      import time
      
      enc = BertTokenizer.from_pretrained("bert-base-uncased")
      
      # Tokenizing input text
      text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
      tokenized_text = enc.tokenize(text)
      
      # Masking one of the input tokens
      masked_index = 8
      tokenized_text[masked_index] = '[MASK]'
      indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
      segments_ids = [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, ]
      
      # Creating a dummy input
      tokens_tensor = torch.tensor([indexed_tokens]).cuda()
      segments_tensors = torch.tensor([segments_ids]).cuda()
      dummy_input = [tokens_tensor, segments_tensors]
      
      # Initializing the model with the torchscript flag
      # Flag set to True even though it is not necessary as this model does not have an LM Head.
      config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
         num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, torchscript=True)
      
      # Instantiating the model
      model = BertModel(config)
      
      # The model needs to be in evaluation mode
      model.eval()
      
      # If you are instantiating the model with `from_pretrained` you can also easily set the TorchScript flag
      model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)
      
      model = model.eval().cuda()
      
      # Creating the trace
      traced_model = torch.jit.trace(model, dummy_input)
      traced_model = deepytorch_inference.compile(traced_model)  # 进行编译
      
      # Warming up
      for _ in range(10):
          all_encoder_layers, pooled_output = traced_model(*dummy_input)
      
      inference_count = 1000
      # inference test
      start = time.time()
      for _ in range(inference_count):
          traced_model(*dummy_input)
      end = time.time()
      print(f"use {(end-start)/inference_count*1000} ms each inference")
      print(f"{inference_count/(end-start)} step/s")
      

      执行结果如下,显示推理耗时为0.418 ms。相较于之前的4.955 ms,推理性能有了显著提升。

      bert模型升级版本.jpg

基于ResNet50模型执行动态尺寸推理

在Deepytorch Inference中,您无需关心动态尺寸的问题,Deepytorch Inference能够支持不同的输入尺寸。以下示例基于ResNet50模型,输入3个不同的长宽尺寸,带您体验使用Deepytorch Inference进行推理加速的过程。

import time
import torch
import deepytorch_inference  # 导入deepytorch-inference软件包
import torchvision.models as models
mod = models.resnet50(pretrained=True).eval()
mod_jit = torch.jit.script(mod)
mod_jit = mod_jit.cuda()
mod_jit = deepytorch_inference.compile(mod_jit)  # 进行编译

in_t = torch.randn([1, 3, 224, 224]).float().cuda()
in_2t = torch.randn([1, 3, 448, 448]).float().cuda()
in_3t = torch.randn([16, 3, 640, 640]).float().cuda()

# Warming up
for _ in range(10):
    mod_jit(in_t)
    mod_jit(in_3t)

inference_count = 1000
# inference test
start = time.time()
for _ in range(inference_count):
    mod_jit(in_t)
    mod_jit(in_2t)
    mod_jit(in_3t)
end = time.time()
print(f"use {(end-start)/(inference_count*3)*1000} ms each inference")
print(f"{inference_count/(end-start)} step/s")

执行结果如下,显示推理耗时大约为9.85 ms。

renet50动态大小.jpg

说明

为了缩短模型编译的时间,应在warming up阶段推理最大及最小的tensor尺寸,避免在执行时重复编译。例如,已知推理尺寸在1×3×224×224至16×3×640×640之间时,应在warming up时推理这两个尺寸。