Model weight service

Model loading bottleneck

Large Language Models with 700+ GB parameters make model loading time a critical bottleneck, especially in two scenarios:

1. Elastic scale-out: Model loading time directly impacts service scaling agility.

2. Multi-instance deployments: Multiple instances concurrently pulling models from remote storage (OSS, NAS, or CPFS) causes bandwidth contention and slows loading.

To address these bottlenecks, PAI Inference Service provides Model Weight Service (MoWS), which combines the following core technologies:

• Distributed caching architecture: Uses node memory to build a weight cache pool.

• High-speed transport: Delivers low-latency data transfer using RDMA-based interconnects.

• Intelligent sharding: Supports parallel data sharding with integrity checks.

• Memory sharing: Enables zero-copy weight sharing among multiple processes on a single machine.

• Intelligent prefetching: Proactively loads model weights during idle periods.

• Efficient caching: Ensures model shards are load-balanced across instances.

In large-scale cluster deployments, MoWS delivers the following performance gains:

1. 10x faster scaling compared to traditional pull-based methods.

2. 60%+ higher bandwidth utilization.

3. Service cold start reduced to seconds.

MoWS leverages bandwidth across instances for fast weight transport, caching model weights locally and sharing them to accelerate scaling and cold starts for large models.

Enable model weight service

1. Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).

2. Click Deploy Service, then Custom Deployment.

3. On the Custom Deployment page, configure the following parameters. For other parameters, see Parameters for custom deployment in the console.

   • Under Environment Information > Image Configuration, select Alibaba Cloud Image and choose an image version with the mows identifier from the vllm image repository.

     

     Important

     Add the --load-format=mows parameter to the command to support vllm/sglang inference engines.

   • In the Resource Information section, select EAS Resource Group or Resource Quota as the resource type.

   • In the Features section, enable Model Weight Service (MoWS) and configure the following parameters.

     

     Parameter	Description	Example
     Model Weight Path	Required. The model weight path. Supports OSS, NAS, or CPFS mount paths.	/mnt/data/llm_models/Qwen2-7B-Instruct/
     Maximum Memory Usage	Required. Memory allocated to MoWS per instance. Unit: GB.	200
     CRC32 File Path	Optional. The CRC32 file for data verification during model loading. Path is relative to Model Weight Path. The file format is [crc32] [relative_file_path]. Default value: "crc32.txt". Generate the crc32 file Run the following command in the model weight directory to generate the CRC32 file: apt-get install -y libarchive-zip-perl find . -type f | xargs -I {} -P $(nproc) sh -c 'echo "$(crc32 {}) {}"' | sed 's|^\(.*\) \./|\1 |' > crc32.txt	crc32.txt Example content: 3d531b22 model-00004-of-00004.safetensors 1ba28546 model-00003-of-00004.safetensors b248a8c0 model-00002-of-00004.safetensors 09b46987 model-00001-of-00004.safetensors
     NIC Type	Select EIC if the instance uses EIC-accelerated hardware.	Non-EIC NIC

Performance benchmarks

Qwen3-8B benchmark: MoWS reduced P99 cold start time from 235 to 24 seconds (89.8% reduction) and instance scaling time to 5.7 seconds (97.6% reduction).

Qwen3-32B benchmark: MoWS reduced cold start time from 953 to 82 seconds (91.4% reduction) and instance scaling time to 17 seconds (98.2% reduction).