EdgeAcc-SGLang

Advantages

EdgeAcc-SGLang is an optimized version of SGLang. It natively supports all basic SGLang features and is compatible with its runtime methods and access interfaces. EdgeAcc-SGLang also adds the following key features:

• Intra-GPU PD separation technology (Semi-PD).

  For configurations with a single machine and multiple GPUs, this technology splits Prefill and Decoder instances within a GPU at the Streaming Multiprocessor (SM) level. This reduces cross-card communication between Prefill and Decoder instances, which optimizes throughput, TTFT, and end-to-end latency.

• Offload KV Cache to memory. This feature provides significant benefits in multi-turn conversation scenarios.

  This feature lets you use system memory to store the KV Cache, which provides better support for stateful service scenarios such as long context and multi-turn conversations. You can allocate memory based on your instance type configuration. For more information, contact your account manager.

• Dynamic RunningQueue. This feature limits the number of running tasks to prevent out of memory (OOM) errors under high concurrency.

• Compatibility with mainstream models, such as Qwen3-32B.

• Compatibility with the GPUDirect P2P driver for cumulative optimization effects.

EdgeAcc-SGLang offers significant performance advantages over the original SGLang, especially for smaller LLMs (under 70B) in single-machine, multi-GPU inference scenarios. For example, tests using Qwen3-32B show the following improvements:

• Throughput: Increased by 88.71% to 126.59%

• TTFT: Reduced by 36.34% to 52.37%

• End-to-end latency: Reduced by 48.42% to 57.15%

• TPOT: Reduced by 49.92% to 56.99%

• KV Cache hit rate: Increased from 75% to 94%

For more information, see Performance data details.

Billing

The EdgeAcc toolkit is free of charge. You pay only for the Alibaba Cloud ENS resources that you use to run computing tasks.

Prerequisites

• Hardware: All available AI computing instance types for ENS.

• Software dependencies

  • Operating system: Ubuntu 22.04

  • CUDA version: 12.7

  • Pytorch version: 2.6.0

To use other operating systems or CUDA versions, contact your account manager for support.

Get started

Follow these steps to download a model and start EdgeAcc-SGLang from a Docker image.

1. Choose a method to download the model:

   Download from the ModelScope community using the modelscope tool

   1. Install the modelscope tool using pip.

      pip install modelscope

   2. Use the tool to download the model to a specified path.

      modelscope download --model Qwen/Qwen3-32B --local_dir your_model_dir  # Replace your_model_dir with the actual path to store the model

   Download from any URL using a multi-threaded downloader

   Standard tools, such as browsers, use single-threaded downloads. These downloads can be slow due to factors such as ISP line quality, and interruptions are difficult to manage. To speed up your download, you can use a multi-threaded downloader, such as the Aria2 script hfd.

   1. Set the image endpoint. If you are downloading from Hugging Face, you can use the `hf-mirror.com` endpoint in China for faster downloads.

      export HF_ENDPOINT="https://hf-mirror.com"

   2. Download hfd.sh to your machine. Set the execution permissions and create an alias.

      chmod a+x hfd.sh
      alias hfd="$PWD/hfd.sh"

   3. Download the model to a local directory using multiple threads.

      hfd Qwen/Qwen3-32B -x 8 --local-dir your_model_dir  # Replace your_model_dir with the actual path to store the model

2. Start the image using Docker.

   docker login -u edgeacc -p edgeacc alien-registry.alibaba-inc.com/edgeacc
   docker run -d --pull always --gpus all --privileged --network host --ipc host  \
   -v /data:/data \
   -e MODEL_PATH=your_model_dir  # Replace your_model_dir with the actual path where the model is stored
   -e SERVED_MODEL_NAME=Qwen3-0.6B \
   -e SERVER_HOST=127.0.0.1 \
   -e SERVER_PORT=30000 \
   -e TENSOR_PARALLEL_SIZE=8 \
   -e ENABLE_EDGE_ACC=True \  # This runs the service with EdgeAcc's recommended optimal configuration. It automatically enables Intra-GPU PD and other optimizations to improve performance for long context and high concurrency.
   alien-registry.alibaba-inc.com/edgeacc/sglang:v0.4.4.post1-edge-acc-d7a58713
   --enable-hierarchical-cache  --kv-cache-offload-memory-gb=50   # This enables the KV Cache tiered cache and allocates 50 GB of memory for each TP. Recommended configuration: (Number of TPs × Total memory) <= 80% of host capacity. The memory allocated for KV Cache offloading must be >= the cache value allocated in GPU memory.

3. After the SGLang service starts, you can make API calls. For more information, see https://docs.sglang.ai/. The following is an example:

   curl -X POST http://localhost:30000/generate -H "Content-Type: application/json" -d '{"text": "Write a 1940s country song about the hardships of being a programmer", "sampling_params": {"temperature": 0.0, "max_new_tokens": 245, "ignore_eos": true}, "stream": false, "lora_path": null, "return_logprob": false, "logprob_start_len": -1}'

Performance data details