Model compression

Prerequisites

Before you begin, ensure that you meet the following requirement:

• Source model preparation: You have at least one custom fine-tuned model in your current workspace that supports compression. You create these models through model fine-tuning. For a list of supported base models, see Compatible models. For more information about the fine-tuning process, see model fine-tuning.

Compatible models

Compression is currently supported for custom fine-tuned models that are based on the following base models:

For the most up-to-date list of supported models, refer to the Model Compression page in the Model Studio console.

Procedure

1. Log in to the Model Studio console. In the top navigation bar, click Models. In the Train section, click Model Compression.

2. In the upper-right corner of the page, click + Create compression task.

3. On the Create Compression Task page, configure the following settings:

   

   • Basic Information

     • Task name (Required) and Task description (Optional).

   • Compression Configuration

     • Select source model: In the dialog box that appears, select an eligible custom fine-tuned model from your current workspace.

     • Quantized model name suffix: This suffix is used to build the output model ID. The suffix can contain only lowercase letters and digits, and can be up to 8 characters long.

     • Quantization template: Select a template from the list of templates supported by the source model. The template name includes the target deployment specification, such as MU5 or MU8. The selected template determines the supported deployment specifications for the compressed model.

     • Calibration data (This option appears if the selected template requires it): Calibration data is used to analyze the numerical distribution of model weights and activation values to determine the scaling range and block size for quantization. Select a dataset and version from your published datasets. You must first create and publish the dataset in Data Management. Datasets mounted from OSS are not supported as calibration data; you can only use datasets uploaded to Model Studio. When calling the API, you can pass the dataset group ID using the custom_calibration_file_ids parameter.

4. Click Start Compression. The task is added to the task list.

5. Track the task progress.

   • In the task list, view the running status. The status can be pending, queued, running, stopping, compression succeeded, compression failed, or canceled.

   • Click the task name to open the details page. On the Details tab, view basic task information. Switch to the Logs tab to view the run logs.

   • To cancel a running task, click Stop in the task list.

6. After the task succeeds, click the compressed model name in the task list to go to the My Models page. Click Deploy to deploy the model. The supported deployment specifications are determined by the selected quantization template.

Requirements and limitations

• Deployment specification: The deployment specifications supported by a compressed model depend on the selected quantization template. The template name indicates the target specification, such as MU5 or MU8. You can configure the number of deployments on the Model Deployment page in the Model Studio console.

• Task status: Task statuses include pending, queued, running, stopping, compression succeeded, compression failed, and canceled. You can cancel only tasks that are in the queued or running state. You can delete tasks that are in a terminal state (compression succeeded, compression failed, or canceled). After you click Stop or Delete, you must confirm the action again in the confirmation dialog box that appears.

• Irreversibility: A compressed model cannot be retrained or compressed again. To iterate, return to the full-precision fine-tuned model, retrain it, and then compress the new version.

• Effect of deleting a task: Deleting a task only removes its record. The output model remains on the My Models page.

Billing

The model compression feature is currently available free of charge for a limited time. For the specific end date of this offer, refer to announcements in the console.

After you deploy a compressed model, you are charged based on the standard pricing of the selected deployment unit. For more information, see Model Deployment.

FAQ

Q1: Which models can I compress?

You can compress custom fine-tuned models from the Compatible models list. Models that are already quantized cannot be compressed again. For the definitive list of supported models, refer to the console.

Q2: Can I cancel a task?

You can cancel tasks that are in the queued or running state. After you cancel a task, its status changes to stopping and then to canceled. You cannot cancel tasks that are in a terminal state.

Q3: Can I fine-tune or re-compress a compressed model?

No, a compressed model cannot be retrained or compressed again. To iterate on the model, you must return to the original full-precision fine-tuned model and retrain it.

Q4: Can I use a dataset mounted from OSS for calibration data?

The console UI does not currently support selecting datasets mounted from OSS or other mounted storage types. Use a dataset from Data Management that is of the "internal upload" type and in the "Published" state. When calling the API, you can directly pass the dataset group ID using the custom_calibration_file_ids parameter, which is then validated by the backend.