使用Deepytorch Inference实现模型的推理性能优化_GPU云服务器(EGS)-阿里云帮助中心

Deepytorch Inference是阿里云自研的AI推理加速器，针对Torch模型，可提供显著的推理加速能力。本文主要介绍安装并使用Deepytorch Inference的操作方法，以及推理效果展示。

背景信息

Deepytorch Inference通过调用deepytorch_inference.compile(model)接口即可实现推理性能加速。使用Deepytorch Inference前，您需要先使用torch.jit.script或者torch.jit.trace接口，将PyTorch模型转换为TorchScript模型，更多信息，请参见PyTorch官方文档。

本文将为您提供分别使用torch.jit.script和torch.jit.trace接口实现推理性能加速的示例，更多信息，请参见推理性能效果展示。

安装Deepytorch Inference

重要

安装Deepytorch Inference前，请确认您已创建配备了NVIDIA GPU卡的GPU实例（即A10、V100或T4 GPU）。

连接GPU实例后，使用pip工具安装指定版本的torch（例如2.0.1版本）和Deepytorch Inference软件包，其中，DeepyTorch Inference的软件包可以通过PyPI进行分发和安装，方便开发者通过简单的命令行工具安装和管理软件。

说明

如需选择特定版本的Deepytorch Inference软件包，则需从deepytorch inference中选择该版本的whl包。例如，需要python 3.8+pytorch 1.13+cuda 11.7版本的Deepytorch Inference软件包，则直接下载deepytorch_inference（python 3.8+pt 1.13.1+cuda 117）即可。

pip install torch==2.0.1 deepytorch-inference -f https://aiacc-inference-public-v2.oss-cn-hangzhou.aliyuncs.com/aiacc-inference-torch/stable-diffusion/aiacctorch_stable-diffusion.html

使用Deepytorch Inference

您仅需要在模型的推理脚本中增加如下代码，即可启用Deepytorch Inference的推理优化功能，增加的代码如下所示：

import deepytorch_inference # 导入deepytorch_inference软件包

deepytorch_inference.compile(mod_jit) # 进行编译

推理性能效果展示

基于不同模型，为您展示使用Deepytorch Inference的推理性能效果，实际的推理加速效果与模型、GPU机型等因素有关，本文以A10机型（例如gn7i、ebmgn7i或ebmgn7ix）为例进行推理测试。关于模型支持情况，请参见模型支持情况。

基于ResNet50模型执行推理

以下示例将基于ResNet50模型，并调用torch.jit.script接口执行推理任务，执行1000次后取平均时间，将推理耗时从3.686 ms降低至0.396 ms以内。

原始版本

原始代码如下所示：

import time
import torch
import torchvision.models as models
mod = models.resnet50(pretrained=True).eval()
mod_jit = torch.jit.script(mod)
mod_jit = mod_jit.cuda()

in_t = torch.randn([1, 3, 224, 224]).float().cuda()

# Warming up
for _ in range(10):
    mod_jit(in_t)

inference_count = 1000
# inference test
start = time.time()
for _ in range(inference_count):
    mod_jit(in_t)
end = time.time()
print(f"use {(end-start)/inference_count*1000} ms each inference")
print(f"{inference_count/(end-start)} step/s")

执行结果如下，显示推理耗时大约为3.686 ms。

renet50原始版.jpg

加速版本

仅需要在原始版本代码中插入如下代码即可实现推理性能加速：

import deepytorch_inference
deepytorch_inference.compile(mod_jit)

更新后的代码如下：

import time
import deepytorch_inference  # 导入deepytorch_inference软件包
import torch
import torchvision.models as models
mod = models.resnet50(pretrained=True).eval()
mod_jit = torch.jit.script(mod)
mod_jit = mod_jit.cuda()
mod_jit = deepytorch_inference.compile(mod_jit)  # 进行编译

in_t = torch.randn([1, 3, 224, 224]).float().cuda()

# Warming up
for _ in range(10):
    mod_jit(in_t)

inference_count = 1000
# inference test
start = time.time()
for _ in range(inference_count):
    mod_jit(in_t)
end = time.time()
print(f"use {(end-start)/inference_count*1000} ms each inference")
print(f"{inference_count/(end-start)} step/s")

执行结果如下，显示推理耗时为0.396 ms。相较于之前的3.686 ms，推理性能有了显著提升。

renet50升级版.jpg

基于Bert-Base模型执行推理

以下示例将基于Bert-Base模型，并调用torch.jit.trace接口执行推理任务，将推理耗时从4.955 ms降低至0.418 ms以内。

执行以下命令，安装transformers包。
```
pip install transformers
```

分别运行原始版本和加速版本的Demo，并查看运行结果。

原始版本

原始代码如下所示：

from transformers import BertModel, BertTokenizer, BertConfig
import torch
import time

enc = BertTokenizer.from_pretrained("bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, ]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens]).cuda()
segments_tensors = torch.tensor([segments_ids]).cuda()
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
    num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, torchscript=True)

# Instantiating the model
model = BertModel(config)

# The model needs to be in evaluation mode
model.eval()

# If you are instantiating the model with `from_pretrained` you can also easily set the TorchScript flag
model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)

model = model.eval().cuda()

# Creating the trace
traced_model = torch.jit.trace(model, dummy_input)

# Warming up
for _ in range(10):
    all_encoder_layers, pooled_output = traced_model(*dummy_input)

inference_count = 1000
# inference test
start = time.time()
for _ in range(inference_count):
    traced_model(*dummy_input)
end = time.time()
print(f"use {(end-start)/inference_count*1000} ms each inference")
print(f"{inference_count/(end-start)} step/s")

执行结果如下，显示推理耗时大约为4.955 ms。

bert模型原始版本.jpg

加速版本

仅需要在原始版本代码中插入如下代码即可实现推理性能加速：

import deepytorch_inference
deepytorch_inference.compile(traced_model)

更新后的代码如下：

from transformers import BertModel, BertTokenizer, BertConfig
import torch
import deepytorch_inference  # 导入deepytorch-inference软件包
import time

enc = BertTokenizer.from_pretrained("bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, ]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens]).cuda()
segments_tensors = torch.tensor([segments_ids]).cuda()
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
   num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, torchscript=True)

# Instantiating the model
model = BertModel(config)

# The model needs to be in evaluation mode
model.eval()

# If you are instantiating the model with `from_pretrained` you can also easily set the TorchScript flag
model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)

model = model.eval().cuda()

# Creating the trace
traced_model = torch.jit.trace(model, dummy_input)
traced_model = deepytorch_inference.compile(traced_model)  # 进行编译

# Warming up
for _ in range(10):
    all_encoder_layers, pooled_output = traced_model(*dummy_input)

inference_count = 1000
# inference test
start = time.time()
for _ in range(inference_count):
    traced_model(*dummy_input)
end = time.time()
print(f"use {(end-start)/inference_count*1000} ms each inference")
print(f"{inference_count/(end-start)} step/s")

执行结果如下，显示推理耗时为0.418 ms。相较于之前的4.955 ms，推理性能有了显著提升。

bert模型升级版本.jpg

基于ResNet50模型执行动态尺寸推理

在Deepytorch Inference中，您无需关心动态尺寸的问题，Deepytorch Inference能够支持不同的输入尺寸。以下示例基于ResNet50模型，输入3个不同的长宽尺寸，带您体验使用Deepytorch Inference进行推理加速的过程。

import time
import torch
import deepytorch_inference  # 导入deepytorch-inference软件包
import torchvision.models as models
mod = models.resnet50(pretrained=True).eval()
mod_jit = torch.jit.script(mod)
mod_jit = mod_jit.cuda()
mod_jit = deepytorch_inference.compile(mod_jit)  # 进行编译

in_t = torch.randn([1, 3, 224, 224]).float().cuda()
in_2t = torch.randn([1, 3, 448, 448]).float().cuda()
in_3t = torch.randn([16, 3, 640, 640]).float().cuda()

# Warming up
for _ in range(10):
    mod_jit(in_t)
    mod_jit(in_3t)

inference_count = 1000
# inference test
start = time.time()
for _ in range(inference_count):
    mod_jit(in_t)
    mod_jit(in_2t)
    mod_jit(in_3t)
end = time.time()
print(f"use {(end-start)/(inference_count*3)*1000} ms each inference")
print(f"{inference_count/(end-start)} step/s")

执行结果如下，显示推理耗时大约为9.85 ms。

renet50动态大小.jpg

说明

为了缩短模型编译的时间，应在warming up阶段推理最大及最小的tensor尺寸，避免在执行时重复编译。例如，已知推理尺寸在1×3×224×224至16×3×640×640之间时，应在warming up时推理这两个尺寸。