QLean model quantization (v0.1.0)

Installation

# Install dependencies
pip install triton_kernel==1.0.0+ppu2.0.0.oe

# Install qlean
pip install qlean==0.1.0+ppu2.0.0.oe

Parameters

• --model_name: The name of the model. Required. The format must match the model ID on Hugging Face, such as "deepseek-ai/DeepSeek-V3.2-Exp".

• --model_path: The local path to the original model. Required. For example, you can specify the path to a Hugging Face model's local cache.

• --save_path: The path to save the dequantized or quantized model. Required.

• --mix_path: The path to save the mixed-precision quantized model. Optional.

• --recipe: The path to the quantization recipe. Optional. Must be a path to a YAML file.

  • For models listed in the "Models that support one-click quantization" table, you can omit the recipe parameter. The system automatically loads the corresponding recipe.

  • For other models, you must create and provide a recipe file. For details on creating a recipe, see the "Create a recipe" section.

Features

You can switch between features by using different recipes. The following examples show how to use each feature.

1. W8A8-INT8 quantization

   The original model can be in FP8 or BF16 format. The system automatically selects an appropriate strategy based on the precision of the original model. If the model is in FP8 format, the system switches to a more precise quantization mode. If the model is in BF16 format, the system uses the general quantization mode.

   # Replace /path/to/.../ with your actual path
   
   # Original model is FP8
   qlean --model_name deepseek-ai/DeepSeek-R1-0528 --model_path /path/to/DeepSeek-R1-0528/ --save_path /path/to/DeepSeek-R1-0528-INT8/
   
   # Original model is BF16
   qlean --model_name Qwen/Qwen3-VL-30B-A3B-Instruct --model_path /path/to/Qwen3-VL-30B-A3B-Instruct/ --save_path /path/to/Qwen3-VL-30B-A3B-Instruct-INT8/

2. FP8 → BF16 dequantization

   # Replace /path/to/.../ with your actual path
   qlean --model_name deepseek-ai/DeepSeek-V3.2 --model_path /path/to/DeepSeek-V3.2/ --save_path /path/to/DeepSeek-V3.2-BF16/ --recipe /path/to/dequant.yaml

   # Contents of dequant.yaml
   ---
   dequant_stage:
     dequant_modifiers:
       dequantModifier: {}
   ...

3. Mixed-precision quantization

   After W8A8-INT8 quantization, some models may show a drop in accuracy for specific use cases. This can happen because certain layers in the model are sensitive to precision. You can use the mixed-precision quantization feature to replace these specific layers in the W8A8-quantized model with their BF16 versions. In this case, save_path points to the W8A8 model, and mix_path specifies the save path for the mixed-precision model.

   # Replace /path/to/.../ with your actual path
   qlean --model_name Qwen/Qwen3-235B-A22B-Instruct-2507 --model_path /path/to/Qwen3-235B-A22B-Instruct-2507/ --save_path /path/to/Qwen3-235B-A22B-Instruct-2507-INT8/ --mix_path /path/to/Qwen3-235B-A22B-Instruct-2507-MIX/

Writing a recipe

To perform W8A8-INT8 quantization on a model not listed in the "Models that support one-click quantization" table, create a YAML recipe file based on the following example. You only need to modify the ignore section.

---
quant_stage:
  quant_modifiers:
    generalDay0Modifier:
      ignore: ["module_to_ignore"] # Modify this based on your requirements
      scheme: W8A8
...

When you create the ignore list in your quantization configuration, you should add all quantization-sensitive modules. Modules that should typically not be quantized include re:.*lm_head, re:.*embed_tokens, and re:.*mlp.gate$. Note that the modules that need to be ignored vary for different model architectures, and you should add them as needed.

To perform mixed-precision quantization on a model that is not listed in the "Models that support one-click quantization" table, add a mixedPrecisionModifier section to the preceding recipe. The layers parameter specifies the IDs of the layers to replace with their BF16 versions, and num_layers specifies the total number of layers in the model.

---
quant_stage:
  quant_modifiers:
    generalDay0Modifier:
      ignore: ["re:.*lm_head", "re:.*embed_tokens", "re:.*mlp.gate$"] # Modify this based on your requirements
      scheme: W8A8
    mixedPrecisionModifier:
      layers: ['88', '89', '92-93']  # Modify this based on your requirements
      num_layers: 94                 # Modify this based on your requirements
...

After creating the YAML file, pass its path using the --recipe parameter.

# Replace /name/of/.../ and /path/to/.../ with your actual names and paths
qlean --model_name /name/of/model/ --model_path /path/to/original_model/ --save_path /path/to/result_model/ --recipe /path/to/your/recipe/

Example: W8A8-INT8 quantization of DeepSeek-V3.2

Step 1: Create a deepseek-v3.2-recipe.yaml file.

---
quant_stage:
  quant_modifiers:
    generalDay0Modifier:
      ignore: ["re:.*lm_head", "re:.*embed_tokens", "re:.*mlp.gate$"]
      scheme: W8A8
...

Step 2: Run the quantization command, specifying the model name, the original model path, the save path for the quantized model, and the recipe file.

qlean --model_name deepseek-ai/DeepSeek-V3.2 --model_path /path/to/DeepSeek-V3.2/ --save_path /path/to/DeepSeek-V3.2-INT8/ --recipe /path/to/deepseek-v3.2-recipe.yaml 

Sglang inference accuracy

Test image: pytorch2.8.0-ubuntu24.04-cuda12.9-sglang0.5.5-py312

Note: The Zhenwu 810E does not support FP8. For models originally in FP8 format, their W8A8-quantized version is used as the comparison baseline.

vLLM inference precision

Test image: pytorch2.9.0-ubuntu24.04-cuda12.9-vllm0.12.0-py312

Note: Since the Zhenwu 810E does not support FP8, this test uses W8A8 weights as the baseline for models with original FP8 weights.

Sglang performance

Test image: pytorch2.8.0-ubuntu24.04-cuda12.9-sglang0.5.5-py312

Note: Since the Zhenwu 810E does not support FP8, the W8A8 weights from this test serve as the baseline for models with original FP8 weights.

VLLM performance

Test image: pytorch2.9.0-ubuntu24.04-cuda12.9-vllm0.12.0-py312

Note: The Zhenwu 810E does not support FP8. For models with original FP8 weights, W8A8 weights are used for the comparison.