Advanced DeepGPU-LLM API and examples-Elastic GPU Service(EGS)-阿里云帮助中心

DeepGPU-LLM, an inference engine from Alibaba Cloud, optimizes the inference process for large language models on GPU cloud servers. It provides a free, high-performance, low-latency inference service. DeepGPU-LLM provides APIs for model loading and model inference. After installing DeepGPU-LLM on a GPU cloud server, use its APIs to run model inference for improved efficiency and accuracy.

Note

DeepGPU-LLM is a large language model (LLM) inference engine developed by Alibaba Cloud for GPU instances. To learn more, see What is the DeepGPU-LLM inference engine.

(Optional) Convert model format

To use DeepGPU-LLM for inference optimization, convert an open-source, non-quantized model from the Hugging Face format to a DeepGPU-LLM-compatible format. Converting some large language models requires substantial CPU memory. To address this, you can either convert the model in advance or increase CPU memory. For example, you can set the --shm-size parameter in a Docker container to automatically convert the model format upon loading.

Note

DeepGPU-LLM automatically converts large language models as they are loaded. If your model does not have strict CPU memory requirements, you can use the DeepGPU-LLM API to load the original model directly.

DeepGPU-LLM provides a unified model conversion command, huggingface_model_convert. Before running the model conversion script, review the following parameters:

Parameter	Description
`-h` or `--help`	Displays help information.
`--in_file`	Specifies the path to the original model to convert. The model can be downloaded from Hugging Face or ModelScope.
`--saved_dir`	Specifies the output directory for the converted model.
`--infer_gpu_num`	Specifies the number of GPUs to use for inference.
`--weight_data_type`	Specifies the precision for the model weights, such as fp16 or bf16.
`--model_name`	Specifies a custom name for the model.
`--cpu_cores`	Specifies the vCPU core count to use for model conversion.

Example 1: Llama 2-7b-chat

huggingface_model_convert --in_file /mnt/models/Llama-2-7b-chat-hf/ --saved_dir /mnt/models_deepgpu/llama2-7b-chat --weight_data_type fp16 --model_name llama2-7b --infer_gpu_num 1

Example 2: Qwen2-7B-Instruct

huggingface_model_convert --in_file /mnt/models/Qwen2-7B-Instruct --saved_dir /mnt/models_deepgpu/Qwen2-7B-Instruct --weight_data_type fp16 --model_name qwen2-7b --infer_gpu_num 1

Note

If the huggingface_model_convert command is not found, your DeepGPU version is outdated. You can upgrade your DeepGPU-LLM version. For instructions, see (Optional) Upgrade DeepGPU-LLM. Alternatively, depending on the LLM model type, you can replace the model field with the specific LLM name and then convert the model. Check the help output to adjust the corresponding parameters.

DeepGPU-LLM API

Model loading functions

DeepGPU-LLM provides deepgpu_model, a unified model processing class with two model loading functions: __init__ and from_pretrained.

Loading functions

class deepgpu_model(torch.nn.Module):
    def __init__(self, model_path: str, tensor_para_size: int, precision: int = 0,
                 kv_cache_quant_level: int = 0,
                 generation_config: typing.Optional[DeepGPUGenerationConfig] = None,
                 is_gemm_tuning: bool = False,
                 load_pretrained: bool = False,
                 page_size: int = 16,
                 gpu_utilization: float = 0.9,
                 max_batch_size : int = 128,
                 max_context_token_num: int = 8192,
                 session_len: int = 2048)
    @classmethod
    def from_pretrained(cls, model_path, tensor_para_size, precision = 0,
                 kv_cache_quant_level: int = 0,
                 generation_config: typing.Optional[DeepGPUGenerationConfig] = None,
                 page_size: int = 16,
                 gpu_utilization: float = 0.9,
                 max_batch_size : int = 128,
                 max_context_token_num: int = 8192,
                 session_len: int = 2048,
                 data_set = None)

The __init__ function loads a pre-converted model. The following table describes its parameters.

Parameter	Description
`model_path`	Path to the pre-converted model directory. Note This is typically an `x-gpu` directory, where `x` is the number of GPUs. This directory is automatically generated during the model conversion process.
`tensor_para_size`	Specifies the number of GPUs to use. This value must be consistent with the number of GPUs set during model conversion.
`precision`	The weight quantization level. Valid values: 0: FP16 precision. 1: INT8 quantization. 3: INT4 quantization. 5: FP8 precision. Default: 0.
`kv_cache_quant_level`	The KV cache quantization level. Valid values: 0: No quantization. 1: K8_V8 quantization (recommended). 2: K8_V4 quantization. 3: K4_V4 quantization. Default: 0.
`generation_config`	Model configuration parameters. For more information, see Model parameter class (DeepGPUGenerationConfig). Note If this parameter is not set, the default parameters in the model's configuration file are used.
`is_gemm_tuning`	Specifies whether to tune for the optimal kernel. Optimal kernels for common GPUs are pre-optimized and included in the installation package. Valid values: False: Does not tune for the optimal kernel. True: Tunes for the optimal kernel. Default: False.
`load_pretrained`	Specifies whether the model being loaded is an original model. Valid values: False: Loads a model whose format has already been converted. True: Loads an original model whose format has not been converted. This option is not intended for direct use and is called internally by the `from_pretrained` function to load original models. Default: False.
`page_size`	Specifies the size of the memory block used by PagedAttention.
`gpu_utilization`	Specifies the proportion of GPU memory to use.
`max_batch_size`	Specifies the maximum number of requests that can be processed concurrently. Note If the available GPU memory is insufficient to support the configured value, the system automatically adjusts this value.
`max_context_token_num`	Specifies the maximum number of tokens in the context. This value cannot be less than the value of `session_len`.
`session_len`	Specifies the session length limit, which is the total number of input and output tokens. Set this parameter if the required session length exceeds the default value.
`data_set`	Specifies the directory of the calibration data for FP8 quantization. This parameter is only required when using FP8 precision.

The from_pretrained function loads open-source models from Hugging Face or ModelScope and automatically converts them on the fly.

This class method calls __init__ to create a deepgpu_model instance. It loads an original model, converts it, and then initializes it. For parameter details, see the __init__ function's table above.

Using loading functions

Load a model by using the __init__ function of the deepgpu_model class.

Import the deepgpu_model class from the deepgpu_llm.deepgpu_model module, and then instantiate it to initialize the model. In this case, args.model_dir must point to the pre-converted model directory. The following code provides an example:

from deepgpu_llm.deepgpu_model import deepgpu_model
model = deepgpu_model(model_path = args.model_dir, 
                      tensor_para_size = args.tpsize, 
                      precision = precision, 
                      kv_cache_quant_level = args.kv_cache_quant_level, 
                      gpu_utilization = args.gpu_utilization, 
                      session_len = max_inout_len, 
                      max_context_token_num = max_context_token_num)

Load an original model by using the from_pretrained function of the deepgpu_model class.

Call the deepgpu_model.from_pretrained function to load an original model, which performs on-the-fly conversion and initialization. In this case, args.model_dir points to the directory of the original model from Hugging Face or ModelScope. The following is a code sample:

from deepgpu_llm.deepgpu_model import deepgpu_model
model = deepgpu_model.from_pretrained(model_path = args.model_dir, 
                      tensor_para_size = args.tpsize, 
                      precision = precision, 
                      kv_cache_quant_level = args.kv_cache_quant_level, 
                      gpu_utilization = args.gpu_utilization, 
                      session_len = max_inout_len, 
                      max_context_token_num = max_context_token_num)

Model inference functions

DeepGPU-LLM provides several inference APIs. You can call these APIs in either offline mode or serving mode, depending on your needs.

Model inference in offline mode

The offline mode inference APIs include a standard output function (one-time output) and a streaming output function.

Standard output function (generate)

The standard output function returns the complete result at once. The output is nested within multiple layers and can be complex. To extract the required information, unpack the output layer by layer. The generate function is defined as follows:

    def generate(self, input_ids,
                 generation_config: typing.Optional[DeepGPUGenerationConfig] = None)

When calling model.generate to perform model inference, set the generation_config parameter using the DeepGPUGenerationConfig class. For more information, see Common classes. The following code shows how to call generate:

inputs = []
inputs.append(tokenizer(query, return_tensors='pt').input_ids)
generation_config = DeepGPUGenerationConfig(max_new_tokens = args.output_tokens, top_k = args.top_k, top_p = args.top_p, 
                    temperature = args.temperature, repetition_penalty = args.repetition_penalty)
output = model.generate(inputs, generation_config)
outputX = output[0].tolist()
outputY = outputX[0][0][inputs[0].shape[1]:]
response = tokenizer.decode(outputY)

Parameter

Description

inputs

An array of token IDs generated by the tokenizer.

The query is the actual input text. For multiple inputs, tokenize each input and add the resulting token IDs to the inputs array.

output

This is a nested object that you must unpack to get outputY, which contains the actual output token IDs. Call the tokenizer.decode function to convert the IDs to text.

Note

If there are multiple inputs, there will be multiple corresponding outputs. You can locate the output for a specific batch item using outputX[batch_id][0].

Streaming output function (stream_generate)

Streaming output allows you to display the generated content to the user in real-time as the inference progresses. The stream_generate function is defined as follows:

def stream_generate(self, input_ids,
             generation_config: typing.Optional[DeepGPUGenerationConfig] = None,
             skip_inputs = False)

stream_generate is a member function of the model class. The following table describes its parameters:

Parameter	Description
`input_ids`	The input parameter that provides input data for the model. For more information, see Standard output function (generate).
`generation_config`	Runtime parameters. For parameter definitions, see Common classes.
`skip_inputs`	Specifies whether to exclude the input from the generated output. Valid values: True: Excludes the input content. False: Includes the input content.

The following is a code sample for calling stream_generate. First, create a streamer instance by calling DeepGPUStreamer. Then, call the stream_generate function to start the inference. The function returns a generator that you can iterate over for real-time results.

streamer = DeepGPUStreamer(tokenizer, **{'skip_special_tokens':True})
total_len = 0
response = ""
for output in model.stream_generate(inputs,
                    generation_config=generation_config,
                    skip_inputs=True):
    printable_str = streamer.handel_str(output)
    response = response + printable_str
    total_len += 1
    yield {
         "text": response,
         "prompt_tokens": input_echo_len,
         "completion_tokens": total_len,
         "total_tokens": total_len,
         "finish_reason": None,
    }

Parameter	Description
`printable_str`	The text corresponding to the token generated in the current step.
`response`	The aggregated text from all preceding generation steps.
`total_len`	The total number of tokens generated so far.

Model inference in serving mode

This mode is designed for multi-user, multi-request scenarios. DeepGPU-LLM provides three concurrent invocation functions to fit different codebases: a standard function, an async function, and an async function with a request ID.

Standard invocation function (generate_cb)

For standard (non-async) callers, use the generate_cb function. It is defined as follows:

    def generate_cb(self,
                 input_ids,
                 generation_config: typing.Optional[DeepGPUGenerationConfig] = None)

Parameter	Description
`input_ids`	The input parameter that provides input data for the model. For more information, see Standard output function (generate).
`generation_config`	The parameters used for model inference.

Calling generate_cb starts the model inference and returns a generator. You can then iterate over this generator to get real-time output. In each iteration, append the current output text printable_str to build the complete response.

streamer = DeepGPUStreamer(tokenizer, **{'skip_special_tokens':True})
total_len = 0
response = ""
results_generator = model.generate_cb(inputs, generation_config=generation_config)
for request_output in results_generator:
    if(request_output==-1):
        printable_str = streamer.end()
    else:
        printable_str = streamer.handel_str(request_output)
    response = response + printable_str
    total_len += 1
    yield {
         "text": response,
         "prompt_tokens": input_echo_len,
         "completion_tokens": total_len,
         "total_tokens": total_len,
         "finish_reason": None,
    }

Async invocation function (generate_cb_async)

For async callers, use generate_cb_async. The async invocation function is defined as follows:

    async def generate_cb_async(self,
                 input_ids,
                 generation_config: typing.Optional[DeepGPUGenerationConfig] = None)

The usage of this function is nearly identical to that of generate_cb, but you must use an async for loop to iterate over the results.

streamer = DeepGPUStreamer(tokenizer, **{'skip_special_tokens':True})
total_len = 0
response = ""
results_generator = model.generate_cb_async(inputs, generation_config=generation_config)
async for request_output in results_generator:
    if(request_output==-1):
        printable_str = streamer.end()
    else:
        printable_str = streamer.handel_str(request_output)
    response = response + printable_str
    total_len += 1
    yield {
         "text": response,
         "prompt_tokens": input_echo_len,
         "completion_tokens": total_len,
         "total_tokens": total_len,
         "finish_reason": None,
    }

Async invocation function with request ID (generate_cb_async_id)

If you need to manage multiple concurrent requests by their request IDs, call the generate_cb_async_id function. Its parameters are similar to those of generate_cb_async, but it includes an additional request_id parameter to distinguish between different requests and their corresponding results. The function is defined as follows:

    async def generate_cb_async_id(self,
                 input_ids,
                 request_id: int = 0,
                 generation_config: typing.Optional[DeepGPUGenerationConfig] = None)

When calling this function, use the RequestCounter class from the Request ID handler class (RequestCounter) to manage the request_id.

from deepgpu_llm.deepgpu_utils import DeepGPUGenerationConfig,DeepGPUStreamer,RequestCounter
counter = RequestCounter()
streamer = DeepGPUStreamer(tokenizer, **{'skip_special_tokens':True})
request_id = next(counter)
total_len = 0
response = ""
results_generator = model.generate_cb_async_id(inputs, request_id=request_id, generation_config=generation_config)
async for request_output in results_generator:
    if(request_output==-1):
        printable_str = streamer.end()
    else:
        printable_str = streamer.handel_str(request_output)
    response = response + printable_str
    total_len += 1
    yield {
         "text": response,
         "prompt_tokens": input_echo_len,
         "completion_tokens": total_len,
         "total_tokens": total_len,
         "finish_reason": None,
    }

Common classes

The deepgpu_utils.py module provides utilities for the DeepGPU-LLM inference API. deepgpu_utils.py defines several common classes, such as DeepGPUGenerationConfig, DeepGPUStreamer, and RequestCounter. These classes manage the model's runtime environment, handle concurrent requests, and track request execution. You can import these classes from deepgpu_utils.py with the following code.

from deepgpu_llm.deepgpu_utils import DeepGPUGenerationConfig, DeepGPUStreamer, RequestCounter

DeepGPUGenerationConfig

The DeepGPUGenerationConfig class configures inference parameters for the model. These parameters are optional; if omitted, DeepGPU-LLM automatically loads the initial parameters from the model configuration file.

class DeepGPUGenerationConfig():
    def __init__(self, **kwargs):
        self.max_new_tokens = kwargs.pop("max_new_tokens", 512)
        self.do_sample = kwargs.pop("do_sample", None)    
        self.num_beams = kwargs.pop("num_beams", None)  
        self.temperature = kwargs.pop("temperature", None)  
        self.top_k = kwargs.pop("top_k", None) 
        self.top_p = kwargs.pop("top_p", None) 
        self.repetition_penalty = kwargs.pop("repetition_penalty", None)    
        self.presence_penalty = kwargs.pop("presence_penalty", None) 
        self.len_penalty = kwargs.pop("len_penalty", None)
        self.beam_search_diversity_rate = kwargs.pop("beam_search_diversity_rate", None) 
        self.min_tokens = kwargs.pop("min_tokens", 0)

DeepGPUStreamer

The DeepGPUStreamer class handles streaming output and outputs from concurrent requests for a large language model.

class DeepGPUStreamer():
    def __init__(self, tokenizer: "AutoTokenizer", 
                skip_prompt: bool = False, 
                print_out: bool = True,
                **decode_kwargs)
    def handle_str(self, value)
    def end(self)

Function name	Description
`__init__()`	Initializes the internal tokenizer and the string buffer used for printing.
`handle_str()`	Takes a token ID, converts it to natural language text, places it in the string buffer, and returns the printable text from the buffer.
`end()`	Returns any remaining text in the string buffer.

RequestCounter

The RequestCounter class manages request IDs in multi-request scenarios.

class RequestCounter:
    def __init__(self, start: int = 0) -> None
    def __next__(self) -> int
    def reset(self) -> None
    def add(self, len: int) -> int
    def cur(self) -> int

Function name	Description
`__init__()`	Initializes the counter. The starting value defaults to 0 but can be customized with the `start` parameter.
`__next__()`	Returns the current counter value and then increments the internal counter by 1.
`reset()`	Resets the internal counter to 0.
`add()`	Returns the current counter value and then adds len to it.
`cur()`	Returns the current counter value.

DeepGPU-LLM code examples

This topic provides DeepGPU-LLM code examples to help you get started.

Offline code examples

Simple Llama model example

This example uses the llama2-7b-chat model. Before running the code, ensure you have converted the model. You can also adjust parameters such as the model directory, number of GPUs, and quantization precision as needed.

import time
from deepgpu_llm.deepgpu_model import deepgpu_model
from deepgpu_llm.deepgpu_utils import DeepGPUGenerationConfig
from transformers import LlamaTokenizer
model_path = '/mnt/models_deepgpu/llama2-7b-chat'
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model_path_conv = "/mnt/models_deepgpu/llama2-7b-chat/1-gpu"
tensor_para_size = 1
precision = 0  # 0: fp16 mode, 1: int8 mode, 3: int4 mode
kv_cache_quant_level = 0 # 0: no quantization, 1: K8_V8, 2: K8_V4, 3: K4_V4
generation_config = DeepGPUGenerationConfig(max_new_tokens=512)
model = deepgpu_model(model_path_conv, tensor_para_size,
                    precision, kv_cache_quant_level,
                    generation_config=generation_config)
payload = "Hi, please introduce the alibaba?"
start_ids = [tokenizer(payload, return_tensors="pt").input_ids]
print(payload)
for i in range(5):
    s = time.time()
    output = model.generate(start_ids, generation_config)
    e = time.time()
    print("---- time", e - s)
tokens = output[0].tolist()
for i in range(len(tokens)):
    print(tokenizer.decode(tokens[i][0]))

The code produces the following output:

xxx 1u5Z:~/aiacc/test/api_sample# python3 llama_sample.py
using KV cache quant level 1
running in fp16 mode, cb is False
Keyword arguments {'add_special_tokens': False} not recognized.
Hi, please introduce the alibaba?
---- time 4.796230792999268
---- time 4.656716346740723
---- time 4.655441284179687 5
---- time 4.654964208602905
---- time 4.657361030578613
<s> Hi, please introduce the alibaba?
Alibaba Group Holding Limited, commonly known as Alibaba, is a Chinese multinational conglomerate that specializes in e-commerce, retail, Internet, and technology. The company was founded in 1999 by Jack Ma and a group of 18 other people, and it has since grown to become one of the largest and most successful companies in the world.
Alibaba is headquartered in Hangzhou, Zhejiang, China, and it operates a number of different businesses through its various subsidiaries. These include:
* Alibaba.com, a business-to-business e-commerce platform that connects suppliers in China with buyers around the world.
* Taobao Marketplace and Tmall, two of the largest consumer-to-consumer and business-to-consumer e-commerce platforms in China.
* Alipay, a leading online payment platform that allows consumers to pay for goods and services online.

Qwen streaming output example

This example shows how to run a Qwen-7B model on a single GPU. You can adjust the precision (quantization precision), kv_cache_quant_level, and tp_size (number of GPUs) for faster performance and reduced GPU memory usage. The following code streams the output and measures performance.

import time
from deepgpu_llm.deepgpu_model import deepgpu_model
from deepgpu_llm.deepgpu_utils import DeepGPUGenerationConfig, DeepGPUStreamer
from transformers import AutoTokenizer
model_path = "/mnt/models_deepgpu/Qwen2-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model_path_conv = "/mnt/models_deepgpu/Qwen2-7B-Instruct/1-gpu"
tp_size = 1    # tensor parallel
precision = 0  # 0: fp16 mode, 1: int8 mode, 3: int4 mode
kv_cache_quant_level = 0 # 0: no quantization, 1: K8_V8, 2: K8_V4, 3: K4_V4
generation_config = DeepGPUGenerationConfig(max_new_tokens=512)
model = deepgpu_model(model_path_conv, tp_size, precision, kv_cache_quant_level,
                   generation_config=generation_config)
print("model init over!")
payload = "<|im_start|>user\nHi, could you introduce the tourist attractions in Hangzhou?<|im_end|>\n<|im_start|>assistant\n"
start_ids = tokenizer(payload, return_tensors="pt").input_ids
streamer = DeepGPUStreamer(tokenizer, **{'skip_special_tokens':True})
total_len = 0
response = ""
print("response : ")
start = time.time()
for output in model.stream_generate([start_ids],
                    generation_config=generation_config,
                    skip_inputs=True):
    printable_str = streamer.handel_str(output[0][0])
    #response = response + printable_str
    total_len += 1
    print(printable_str, flush=True, end = "")
end = time.time()
print()
print()
print("time : ", (end - start))
print("speed: ", total_len/(end - start), " tokens/s")

Run the code to view the streamed response and experience the inference speed of DeepGPU-LLM.

running in fp16 mode, cb is True
model init over!
response : 
As the capital city of China's Zhejiang province, Hangzhou has a rich historical and cultural heritage, as well as many beautiful natural landscapes and cultural attractions. Here are some of Hangzhou's famous tourist attractions:
1. **West Lake**: West Lake is one of the most famous lakes in China, praised as a "paradise on earth." It is renowned for its beautiful scenery, rich history, and ancient legends. There are many famous sites around the lake, such as Broken Bridge, Leifeng Pagoda, and Dawn on the Su Causeway in Spring.
2. **Xixi National Wetland Park**: Located in the western part of Hangzhou, this is a large wetland park that combines natural ecology, cultural relics, and leisure activities. Here, you can experience the unique charm of a Jiangnan water town and see a variety of wildlife.
3. **Songcheng Park**: This is a large theme park centered on the culture of the Song Dynasty. It recreates the street life of the Song Dynasty, allowing visitors to experience its history and culture through performances and interactive experiences. The park also features the famous show "The Romance of the Song Dynasty," which is an excellent way to learn about the era.
4. **Lingyin Temple**: Located at the foot of Feilai Peak, north of West Lake, this is a Buddhist temple with a long history. The temple is surrounded by a tranquil environment with beautiful mountains and clear water, making it an ideal place for meditation and prayer.
5. **Qiantang River Tidal Bore**: The massive tidal bore on the Qiantang River, which occurs on the 18th day of the eighth lunar month, is a natural wonder that attracts many tourists. The surging tide is a powerful and unique spectacle endowed by nature to Hangzhou.
6. **Hangzhou Songcheng Site Museum**: Located on the site of the Southern Song Imperial City, this museum showcases the history of the Southern Song Dynasty's politics, economy, culture, and society. The museum has a rich collection of artifacts and multimedia displays to help visitors gain a deeper understanding of Southern Song history.
7. **Wuzhen**: Although Wuzhen is near Hangzhou, its unique water town scenery and deep cultural heritage make it a must-see destination. Wuzhen preserves many buildings from the Ming and Qing dynasties, showcasing the classic features of a Jiangnan water town.
8. **Longjing Tea Plantations by West Lake**: Near West Lake, you can visit and experience the traditional process of making Longjing tea, taste authentic Longjing tea, and enjoy the peaceful rural landscape.
These are just a few of the many attractions in Hangzhou. Each place has its own unique charm, whether it is natural scenery or cultural history, allowing you to experience the beauty of Hangzhou.
time :  5.061952114105225
speed: 90.87403231614961  tokens/s

Note

To view streaming output examples for other LLM models, run the following command to find the DeepGPU-LLM installation directory. You can then view the code in the corresponding llama_cli, chatglm_cli, baichuan_cli, qwen_cli, or deepgpu_cli scripts. Note that the deepgpu_cli script is only available in DeepGPU-LLM versions 24.9 and later.

pip show -f deepgpu-llm

Multi-batch input example

In multi-batch scenarios, properly handling input and output is crucial. For information about how different models handle input and output, see Standard Output Function generate (One-time Output).

import time
from deepgpu_llm.deepgpu_model import deepgpu_model
from deepgpu_llm.deepgpu_utils import DeepGPUGenerationConfig
from transformers import AutoTokenizer
model_path = "/mnt/models_deepgpu/Qwen1.5-72B-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model_path_conv = "/mnt/models_deepgpu/Qwen1.5-72B-Chat/4-gpu"
tp_size = 4    # tensor parallel
precision = 0  # 0: fp16 mode, 1: int8 mode, 3: int4 mode
kv_cache_quant_level = 0 # 0: no quantization, 1: K8_V8, 2: K8_V4, 3: K4_V4
generation_config = DeepGPUGenerationConfig(max_new_tokens=512)
model = deepgpu_model(model_path_conv, tp_size, precision, kv_cache_quant_level,
                   generation_config=generation_config)
print("model init over!")
prompt =["Hello, who are you?", "Hello, please introduce the tourist attractions in Hangzhou.", "Hello, please introduce the tourist attractions in Beijing.", "Hello, please introduce the tourist attractions in Xinjiang.", "Hello, please introduce the tourist attractions in Tibet."]
batchsize = len(prompt)
start_ids = []
for bs in range(batchsize):
  payload = "<|im_start|>user\n " + prompt[bs] + "<|im_end|>\n<|im_start|>assistant\n"
  start_ids.append(tokenizer(payload, return_tensors="pt").input_ids)
output = model.generate(start_ids, generation_config)
tokens = output[0].tolist()
for bs in range(batchsize):
  response = tokenizer.decode(tokens[bs][0], skip_special_tokens=True)
  print("response [", bs, "] : ", response.rstrip(tokenizer.decode(0)))

The code produces the following output:

root@xxx:~/aiacc/test/api_sample# python3 qwen_batch_sample.py
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
running in fp16 mode, cb is False
model init over!
response[ 0 ] :  user
  Hello, who are you?
assistant
Hello! I am a large language model from Alibaba Cloud, and my name is Qwen. As an AI assistant, my mission is to help users get accurate and useful information and to solve their problems and questions. You can ask me about various fields of knowledge, technical issues, or anything you want to know. I will do my best to help. What can I do for you?
response[ 1 ] :  user
  Hello, please introduce the tourist attractions in Hangzhou.
assistant
Hello, Hangzhou is a famous tourist city in China with many well-known attractions. For example, West Lake, which is the most famous lake in China, features ten scenic spots like "Dawn on the Su Causeway in Spring" and "Breeze-Ruffled Lotuses at Quyuan Garden". There's also Lingyin Temple, the largest Buddhist monastery in Hangzhou with a long history; Songcheng Park, a theme park based on the culture of the Song Dynasty where you can experience life from that era; Xixi National Wetland Park, a large wetland park that integrates ecological protection, leisure tourism, and cultural exhibition; and other famous attractions like the Liuhe Pagoda, Three Pools Mirroring the Moon, and the Qiantang River Bridge. I hope you get a chance to visit Hangzhou.
response[ 2 ] :  user
  Hello, please introduce the tourist attractions in Beijing.
assistant
Beijing is the capital of China and has numerous famous tourist attractions. For example, the Forbidden City, which is the most important imperial palace in Chinese history; the Great Wall, a UNESCO World Heritage site and an outstanding example of ancient Chinese defensive engineering; the Summer Palace, a well-preserved royal garden; Tiananmen Square, a symbol of the People's Republic of China; and Nanluoguxiang, a hutong (alley) full of old Beijing charm. Additionally, there are other famous spots like Beihai Park, the Old Summer Palace (Yuanmingyuan), and the 798 Art District, which are must-visit places for tourists in Beijing.

Serving code examples

Server code example

Run the fastapi_server.py script to receive text generation requests, process them with the DeepGPU model, and return the results.

import argparse
import json
from typing import AsyncGenerator
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response, StreamingResponse
import uvicorn
from deepgpu_llm.deepgpu_model import deepgpu_model
from deepgpu_llm.deepgpu_utils import DeepGPUGenerationConfig,DeepGPUStreamer,RequestCounter
from transformers import AutoTokenizer
import time
TIMEOUT_KEEP_ALIVE = 5  # seconds.
TIMEOUT_TO_PREVENT_DEADLOCK = 1  # seconds.
app = FastAPI()
engine = None
counter = RequestCounter()
tokenizer = None
@app.get("/health")
async def health() -> Response:
    """Health check."""
    return Response(status_code=200)
@app.post("/generate")
async def generate(request: Request) -> Response:
    request_dict = await request.json()
    prompt = request_dict.pop("prompt")
    max_new_token = request_dict.pop("max_tokens")
    prompt = [tokenizer(prompt, return_tensors='pt').input_ids]
    stream = request_dict.pop("stream", True)
    streamer = DeepGPUStreamer(tokenizer=tokenizer)
    generation_config = DeepGPUGenerationConfig(max_new_tokens = max_new_token)
    results_generator = engine.generate_cb_async(prompt,generation_config=generation_config)
    # Streaming case
    async def stream_results() -> AsyncGenerator[bytes, None]:
        # await asyncio.sleep(0)
        async for request_output in results_generator:
            if(request_output==-1):
                printable_str = streamer.end()
                text_outputs = [
                    printable_str
                    ]
                ret = {"text": text_outputs}
                yield (json.dumps(ret) + "\0").encode("utf-8")
                break
            else:
                printable_str = streamer.handel_str(request_output)
            text_outputs = [
                printable_str
            ]
            ret = {"text": text_outputs}
            yield (json.dumps(ret) + "\0").encode("utf-8")
    if stream:
        return StreamingResponse(stream_results())
    # Non-streaming case
    total_str = ""
    async for request_output in results_generator:
        if(request_output == -1):
            total_str += streamer.end()
            break
        else:
            total_str += streamer.handel_str(request_output)
    ret = {"text": total_str}
    return JSONResponse(ret)
if __name__ == "__main__":
    parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument("--host", type=str, default=None)
    parser.add_argument("--port", type=int, default=8000)
    parser.add_argument('--model_dir', '-i', type=str, help='converted model dir', required=True)
    parser.add_argument('--tokenizer_dir', '-t', type=str, help='tokenizer dir', required=False)
    parser.add_argument('--tp_size', '-tp', type=int, help='tensor para size', required=True)
    args = parser.parse_args()
    tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_dir, trust_remote_code=True)
    model_path = args.model_dir
    tp_size = args.tp_size
    precision = 0 # 0: fp16 mode, 1: int8 mode, 3: int4 mode
    engine = deepgpu_model(model_path, tp_size, precision,is_gemm_tuning=False)
    uvicorn.run(app,
                host=args.host,
                port=args.port,
                log_level="debug",
                timeout_keep_alive=TIMEOUT_KEEP_ALIVE)

Run the following command to start the service.

python3 fastapi_server.py -i /mnt/models_deepgpu/Qwen2-7B-Instruct/1-gpu -t /mnt/models_deepgpu/Qwen2-7B-Instruct -tp 1

The following output indicates that the service started successfully.

root@xxxkfny1Z:~/deepgpu/test/sample_api# python3 fastapi_server.py -i /mnt/models_deepgpu/Qwen2-7B-Instruct/1-gpu -t /mnt/models_deepgpu/Qwen2-7B-Instruct -tp 1
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
running in fp16 mode, cb is True
INFO:     Started server process [1428285]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Note

You can refer to vLLM's official benchmark_serving.py code. Simply replace import vllm with import deepgpu. For more information, see More examples (Switch vLLM to the DeepGPU-LLM inference engine).

Client Python code

This example provides an asynchronous Python script that sends a request to an HTTP service and receives a streaming response for text generation.

import requests
import json
import time
import aiohttp
import asyncio
import random
import os
url = "http://0.0.0.0:8000/generate" # Normal service mode
# url = "http://0.0.0.0:8000/v1/completions" # OpenAI API-compatible service mode
headers = {"User-Agent": "Benchmark Client"} # Normal service mode
# headers = {
# "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}" # OpenAI API-compatible service mode
# }
async def stream_back(data):
    async with aiohttp.ClientSession() as session:
        async with session.post(url, headers=headers, data=json.dumps(data)) as response:
            if response.status == 200:
                async for chunk, _ in response.content.iter_chunks():
                    chunk = chunk.rstrip(b'\x00')
                    decoded_chunk = chunk.decode('utf-8')
                    if(not decoded_chunk.strip()):
                        continue
                    try:
                        data = json.loads(decoded_chunk)
                        text_content = data.get('text', [])
                        for text in text_content:
                            if text:
                                print(text,flush=True,end='')
                                continue
                    except json.JSONDecodeError as e:
                            print("JSON decode error! error code:", e) 
async def fetch_stream(prompt: str):
    max_tokens = 512
    # Data for the OpenAI-compatible service
    # data = { 
        # "model": '/root/models/Llama-2-7b-chat-hf/',
        # "prompt": prompt,
        # "temperature": 0.0,
        # "best_of": 1,
        # "max_tokens": 512,
        # "stream": True,
    # }
    # Data for the normal service
    print("[prompt]: ", prompt)
    data = {"prompt": prompt, "max_tokens": max_tokens, "stream": True}
    async with aiohttp.ClientSession() as session:
        start_time = time.time()
        await stream_back(data)
        # await none_stream_back(data) 
        end_time = time.time()
        latency = end_time - start_time
        print()
        print(f"Latency: {latency} seconds")
async def main(prompt_num):
    prompt =["Hello, who are you?",
             "Please introduce the tourist attractions in Hangzhou.",
             "Please introduce the tourist attractions in Beijing.",
             "Please describe the historical changes of Hangzhou.",
             "Do you know what BAT stands for?",
             "How do I choose a major after being admitted to a university?",
             "What are some of the best universities in Zhejiang?"]
    tasks = [fetch_stream(prompt[random.randint(0,len(prompt)-1)]) for _ in range(prompt_num)]
    await asyncio.gather(*tasks)
if __name__ == "__main__":
    asyncio.run(main(1))

The code produces the following output:

[prompt]: Do you know what BAT stands for?
It is an acronym for Baidu, Alibaba, and Tencent, the three giants of China's internet industry. Baidu started with its search engine, Alibaba is famous for e-commerce, and Tencent is known for its social networking and gaming businesses. However, behind these three giants, there was a more mysterious event: the "3Q War".
In 2010, a fierce online battle broke out between Tencent and Qihoo 360, which became known as the "3Q War". During this conflict, Tencent launched a software called QQ Doctor, designed to clean users' computers and protect QQ users from viruses and malware. However, this software was strongly opposed by Qihoo 360, which argued that it violated user privacy and involved forced installation. Subsequently, Qihoo 360 released a software called "360 Safeguard" to directly compete with QQ Doctor. This war eventually led to a severe division among the user bases of the two companies, with slogans related to the "3Q War" even appearing in some regions. The conflict not only caused huge financial losses for both companies but also had a profound impact on the entire Chinese internet industry. Since then, Tencent and Qihoo 360 have gradually reconciled and started to cooperate in some areas, but the war is still considered a landmark event in the history of China's internet.
Although many years have passed since the "3Q War," the lessons it taught are still worth considering. First, competition does not have to mean destroying the opponent; instead, a win-win situation should be sought. Second, respecting user rights and privacy is a fundamental principle that any company should follow. Finally, resolving disputes through dialogue and negotiation, rather than extreme measures, is an important path to long-term stable development. These lessons provide important guidance for the current and future internet competition landscape. Therefore, whether at the individual or corporate level, we should learn from these experiences to promote a healthier and more harmonious internet ecosystem.
Latency: 4.07001543045043995 seconds

Client curl command

After the service starts, run the following command to request text generation from the server.

curl -X POST "http://0.0.0.0:8000/generate" \
     -H "Content-Type: application/json" \
     -H "User-Agent: Benchmark Client" \
     -d '{"prompt": "Please describe the historical changes of Hangzhou.", "stream": false, "max_tokens": 128}'

Note

Because curl does not support streaming output, the stream parameter must be set to false.

The command produces the following output:

root@xxxny1Z:~/deepgpu/test/sample_api# curl -X POST "http://0.0.0.0:8000/generate" \
    -H "Content-Type: application/json" \
    -H "User-Agent: Benchmark Client" \
    -d '{"prompt": "Please describe the historical changes of Hangzhou. ", "stream": false, "max_tokens": 128}'
{"text":" Hangzhou is one of China's famous historical and cultural cities, with a long history and deep cultural heritage. Here is a brief introduction to the historical changes of Hangzhou:\n1. Ancient period: Traces of human activity in the Hangzhou area date back to the Neolithic Age. During the Xia, Shang, and Zhou dynasties, this area was the territory of the Yue state. During the Spring and Autumn and Warring States periods, with the rivalry between the Wu and Yue states, the Hangzhou area belonged to Yue. In the Qin and Han dynasties, Hangzhou was part of the Kuaiji Commandery. During the Eastern Jin dynasty, it became an important political, economic, and cultural center in the Jiangnan region.\n2. Tang and Song dynasties: During the Tang Dynasty, Hangzhou became one of the largest commercial cities in the Jiangnan region. In the Song Dynasty, Hangzhou became"}

Switch from vLLM to DeepGPU-LLM

DeepGPU-LLM lets you switch your vLLM codebase to the DeepGPU-LLM inference engine for improved performance and additional features without major code changes. The following examples show how to adapt your code for offline inference and serving.

vLLM-compatible offline inference

Get an offline inference example from vLLM.

The offline_inference.py script is shown below:

from vllm import LLM, SamplingParams
# Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Create an LLM.
llm = LLM(model="facebook/opt-125m")
# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Modify the vLLM example to use the DeepGPU-LLM inference engine.

To switch the inference engine, change the import from vllm to deepgpu_llm and update the model path to your converted DeepGPU model file.
```
# Change from vllm import LLM, SamplingParams to:
from deepgpu_llm import LLM, SamplingParams
# Change llm = LLM(model="facebook/opt-125m") to:
llm = LLM("path/to/your/deepgpu_model")
```

vLLM-compatible serving

Get the serving example from vLLM.

Source: api_server.

Modify the vLLM example to use the DeepGPU-LLM inference engine.

Modify the vLLM serving code by changing the vllm imports to their deepgpu_llm equivalents.

# Change from vllm.engine.arg_utils import AsyncEngineArgs to:
from deepgpu_llm.engine.arg_utils import AsyncEngineArgs
# Change from vllm.engine.async_llm_engine import AsyncLLMEngine to:
from deepgpu_llm.engine.async_llm_engine import AsyncLLMEngine
# Change from vllm.sampling_params import SamplingParams to:
from deepgpu_llm.sampling_params import SamplingParams
# Change from vllm.usage.usage_lib import UsageContext to:
from deepgpu_llm.usage.usage_lib import UsageContext
# Change from vllm.utils import random_uuid to:
from deepgpu_llm.utils import random_uuid

Start the DeepGPU-LLM-based API server to provide a model inference service.

# Start the standard service:
python3 -m deepgpu_llm.entrypoints.api_server \
    --model <YOUR_MODEL> \
    --trust-remote-code \
    --tensor-parallel-size <TENSOR_PARALLEL_SIZE> \
    --gpu-memory-utilization 0.95 
# Start the OpenAI-compatible service:
python3 -m deepgpu_llm.entrypoints.openai.api_server \
    --model <YOUR_MODEL> \
    --trust-remote-code \
    --tensor-parallel-size <TENSOR_PARALLEL_SIZE> \
    --gpu-memory-utilization 0.95

Use the benchmark_serving.py script and the ShareGPT dataset to run an inference performance benchmark.

python3 benchmark_serving.py \
    --backend vllm \
    --model /root/deepgpu/models/qwen1.5-7b \
    --tokenizer /root/deepgpu/models/qwen1.5-7b \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --request-rate 30 \
    --num-prompts 2000 \
    --port 8000