Fine-tune CosyVoice models

Use cases

• Brand or IP voice customization: You have hours of high-quality recordings from a single speaker — a brand spokesperson, virtual streamer, or IP character — and need fidelity that exceeds what single-sample voice cloning delivers.

• Long-form voice production: Audiobooks, podcasts, documentary narration, or corporate training material that requires consistent timbre, tone, and rhythm across tens of hours of output — provided you already have sufficient recordings of the target speaker.

Specifications

• Fine-tuning method: SFT efficient fine-tuning (efficient_sft). Other training methods such as CPT and DPO aren't supported.

• Supported base model: cosyvoice-v3-flash.

• Supported region: CosyVoice fine-tuning is available only in the China (Beijing) region.

Fine-tuning output

The fine-tuning output is a standalone deployed model, not a voice ID under the base model. It has a single fixed voice learned during training. Set voice to default when calling the model — other voice values aren't supported.

Limitations

The following capabilities are determined by the base model cosyvoice-v3-flash and can't be added, extended, or changed through fine-tuning:

• Language support: Training data must be in a language already supported by the base model. Training with data in an unsupported language (for example, Bulgarian) doesn't enable the model to synthesize speech in that language.

• Request-level control interfaces: Fine-tuned models support SSML (fine-grained control over speed, pitch, pauses, volume, and pronunciation within a request) and LaTeX (math formula reading). Instruction control isn't supported — don't pass the instruction parameter.

• Voice cloning and voice design: A fine-tuned model is a single-voice standalone model (voice="default" locked). It doesn't provide voice cloning or voice design capabilities. To create new voice IDs, use the base model's Voice cloning or Voice Design features.

Training costs

Training is billed at CNY 0.2 per 1,000 tokens.

Estimate the token consumption for a single job using this formula:

$$\text{Token Usage}=(lm\_max\_epoch+fm\_max\_epoch)\times 25\times \text{Total Training Set Duration (seconds))}$$.

Here, lm_max_epoch and fm_max_epoch are the LM and FM training epochs set in the hyperparameters. The total training set duration is the combined length (in seconds) of all audio files in the training dataset. Increasing epoch count or training-set size linearly increases token consumption.

Deployment costs

Deployed fine-tuned models are billed by model unit usage duration:

$$\text{Cost}=\text{usage duration(hours)}\times \text{number of model units}\times \text{unit price}$$.

Here, $$\text{number of model units}=\text{model units per replica}\times \text{Deployed Replicas}$$. For available deployment templates and their model unit types, see Deployment templates. For unit prices and billing start/end times per model unit type, see Model deployment .

Audio specifications

Directory structure

Organize the training samples in the following directory structure, then package the entire directory as a .zip archive for upload:

user_data/
├── data.jsonl           # Training sample manifest (required)
└── train/               # Training audio directory (all training .wav files go here)
    ├── 100001.wav
    ├── 100002.wav
    └── ...

data.jsonl format

The data.jsonl file contains one training sample per line:

{"wav_fn": "train/100001.wav", "text": "Hello."}
{"wav_fn": "train/100002.wav", "text": "I see."}

Field descriptions

Text formatting guidelines:

• No normalization needed: Numbers, mixed English/Chinese text, and punctuation don't require special normalization. Write them as naturally spoken in the audio.

• Remove special characters: Remove non-pronounced special characters (such as emoji, decorative symbols, and control characters) from the text. Keep only content that corresponds to the actual audio pronunciation.

• No markup languages: The text field must be plain text — no SSML tags, LaTeX formulas, instruction control statements, or emotion annotations. The fine-tuning stage doesn't parse markup; any markup included will be incorrectly learned as a character sequence in the text-to-speech mapping. Use request-level markup (SSML/LaTeX) when calling the fine-tuned model. For the fine-tuned model's supported request-level control interfaces, see the Request-level control interfaces item in Limitations .

Data volume recommendations

• Recommended volume: A total training audio duration of 1–10 hours generally produces good results. Beyond 10 hours, returns diminish.

• Sample count: At least 150 training audio clips are recommended for stable fine-tuning results. Neither the algorithm nor the platform enforces an upper limit on the number of samples in data.jsonl — add as many as your target total duration requires.

Hyperparameters

CosyVoice fine-tuning trains two sub-networks: LM (Language Model) and FM (Flow Matching). LM is an autoregressive model that converts text to discrete speech tokens and has the most impact on prosody. FM is a flow-matching model that converts speech tokens to Mel spectrograms and has the most impact on voice fidelity. The two networks are decoupled and can be tuned independently using lm_* and fm_* hyperparameter prefixes. Hyperparameters control training epochs and checkpoint-saving frequency, directly affecting training time, token consumption, and model quality.

Start with the following recommended values for your first full run, then adjust based on results:

• LM network: lm_max_epoch=60, lm_step=5, lm_num=3, lm_batch_size=1000.

• FM network: fm_max_epoch=100, fm_step=10, fm_num=3, fm_batch_size=2000.

Here, *_max_epoch is the total number of training epochs, *_step is the checkpoint-saving interval (in epochs), *_num is the maximum number of checkpoints to retain, and *_batch_size is the training batch size. For complete value ranges, see CosyVoice speech synthesis model hyper_parameters.

Benchmark data: The following table shows a real-world sample run to help you estimate time and cost. Actual values vary with data volume, hyperparameters, and queue wait time.

Checkpoints

A single fine-tuning job can produce multiple checkpoints (candidate models; view them through the List checkpoints API). The exact count and ordering are determined by lm_num, fm_num, and *_step in the hyperparameters.

1. Select models: Starting from the maximum epoch for each network, step backwards by *_step, selecting *_num checkpoints total.

2. Combine: Take the Cartesian product of the LM and FM selections. Total candidate checkpoints = lm_num × fm_num.

3. Sort: Order by LM epoch × FM epoch descending (higher product means both networks trained more thoroughly, so it ranks first).

4. Truncate: From the descending sorted result, retain at most 10 checkpoints. If fewer than 10 candidates exist, output all of them.

5. Naming: checkpoint-{LM epoch zero-padded to 4 digits}{FM epoch zero-padded to 4 digits} — for example, LM 4 / FM 4 produces checkpoint-00040004.

Example: With parameters lm_max_epoch=4, lm_step=1, lm_num=2, fm_max_epoch=4, fm_step=2, fm_num=2:

• LM selects: 4, 3 (starting from 4, stepping back by lm_step=1, taking lm_num=2).

• FM selects: 4, 2 (starting from 4, stepping back by fm_step=2, taking fm_num=2).

• Cartesian product yields 4 candidates, sorted descending by product:

Query job details

Use the job_id returned when creating the job to check its status.

curl --location 'https://dashscope.aliyuncs.com/api/v1/fine-tunes/<job_id>' \
--header 'Authorization: Bearer '${DASHSCOPE_API_KEY} \
--header 'Content-Type: application/json'

The output.status field in the response indicates the current stage: PENDING (waiting to start) → QUEUING (queued — only one job runs at a time) → RUNNING (training in progress) → SUCCEEDED (training complete). Abnormal termination states include FAILED (training failed), CANCELING (cancellation in progress), and CANCELED (canceled). For full status field definitions, see Get fine-tuning job details .

Get training logs

If a job stays in one state for an extended period or enters the FAILED state, pull the training logs to help diagnose the issue.

curl --location 'https://dashscope.aliyuncs.com/api/v1/fine-tunes/<job_id>/logs?offset=0&line=1000' \
--header 'Authorization: Bearer '${DASHSCOPE_API_KEY} \
--header 'Content-Type: application/json'

Use offset to control the starting position and line to control the maximum number of lines returned (this example fetches 1,000 lines per request). For response field descriptions, see Query fine-tuning logs .

Deployment templates

A model unit (MU) is the smallest unit of inference compute on the Alibaba Cloud Model Studio platform. Different model unit types correspond to different compute capacities and unit prices. Model units per replica indicates how many model units a single replica occupies. Deployment costs are directly determined by the total number of model units in use.

Deployment templates are preset deployment specifications (including a model unit type and model units per replica) that define the compute resources for a single replica. Each template defines a different combination of inference performance and unit cost. In the table below, V/II in the Model unit type column are Roman numeral identifiers for model unit specifications, corresponding to model_unit_spec field values of MU5/MU2 in API responses. CosyVoice fine-tuned models currently offer two deployment templates:

Billing: Deployment costs are directly tied to the total model unit count (Total Model Unit), calculated as: $$\text{模型单元数量}=\text{单副本模型单元}\times \text{部署副本数}$$.

Choosing a different template or setting a different replica count directly changes the billing amount. Evaluate based on your workload and budget before deploying. For the complete cost formula and unit prices per model unit type, see Deployment costs .

Method 1: Deploy through the console (recommended for daily use)

Entry point: Go to the My Models page in Alibaba Cloud Model Studio console, find the successfully fine-tuned model, and submit it for deployment. For complete steps, see Model deployment .

Key parameters: Only two selections are required — the template and replica values automatically determine all other fields:

• Deployment Template: Choose between Single-machine deployment and Single-machine deployment - flagship complex inference. See Deployment templates for the differences.

• Deployed Replicas: Number of instances. Set based on your concurrency and throughput requirements (integer >= 1).

Post-deployment status: The console displays the deployment instance ID and status transitions (PENDING → DEPLOYING → RUNNING). The model is ready for calls once status reaches RUNNING.

Method 2: Deploy through the API (recommended for automation)

API deployment requires calling three endpoints in sequence. For complete field constraints, see Model deployment .

1. Query deployable models (GET /api/v1/deployments/models): Retrieve the model_name, template_id, and capacity_unit_per_instance needed for creating a deployment:

   curl 'https://dashscope.aliyuncs.com/api/v1/deployments/models?page_no=1&page_size=100&version=v1.0&model_source=custom' \
       --header "Authorization: Bearer ${DASHSCOPE_API_KEY}" \
       --header 'Content-Type: application/json'

   Sample response (key fields only):

   {
     "output": {
       "models": [
         {
           "model_name": "cosyvoice-v3-flash-ft-202605271743-dd2a",
           "plans": [
             {
               "plan": "mu",
               "templates": [
                 {
                   "template_name": "Single-machine deployment",
                   "template_id": "dps-20260521172224-1vabse",
                   "template_desc": "Cost-effective deployment plan, ...",
                   "roles": {
                     "unified": {
                       "model_unit_spec": "MU5",
                       "capacity_unit_per_instance": 1
                     }
                   }
                 },
                 {
                   "template_name": "Single-machine deployment - flagship complex inference",
                   "template_id": "MU2",
                   "template_desc": "High-complexity workloads, ultra-large models with long context.",
                   "roles": {
                     "unified": {
                       "model_unit_spec": "MU2",
                       "capacity_unit_per_instance": 8
                     }
                   }
                 }
               ]
             }
           ]
         }
       ]
     }
   }

   Locate your model: The output.models array in the response lists all deployable models. Find the entry whose model_name matches the finetuned_output returned by Create a fine-tuning job. Its plans[].templates field contains the available deployment templates for that model.

2. Create a deployment (POST /api/v1/deployments): Submit a deployment request using the parameters from the previous step. The following example uses the Single-machine deployment template with 1 replica:

   curl --location 'https://dashscope.aliyuncs.com/api/v1/deployments' \
   --header 'Authorization: Bearer '${DASHSCOPE_API_KEY} \
   --header 'Content-Type: application/json' \
   --data '{
       "model_name": "<MODEL_NAME>",
       "plan": "mu",
       "deploy_spec": "<TEMPLATE_ID>",
       "capacity": 1,
       "billing_method": "POST_PAY"
   }'

   Response: The output.deployed_model field in the response is the deployment instance ID. Pass this value as the model parameter when calling the model.

   Key parameters (replace placeholders <MODEL_NAME> and <TEMPLATE_ID> with values from the previous step's response):

   • model_name: The model_name from the entry matching your finetuned_output in the previous response.

   • plan: Fixed to "mu" (billed by model unit usage duration).

   • deploy_spec: The template_id of your chosen template from the previous response — for example, Single-machine deployment is currently dps-20260521172224-1vabse, and Single-machine deployment - flagship complex inference is MU2. Always use the real-time values from the previous step — don't hard-code these.

   • capacity: The total number of model units for this deployment, directly tied to billing. Must be an integer multiple of capacity_unit_per_instance from the previous step; $$\text{The actual number of deployed replicas}=\frac{capacity}{capacity\_unit\_per\_instance}$$.

     Examples:

     • Single-machine deployment (capacity_unit_per_instance = 1): capacity=1 → 1 replica; capacity=4 → 4 replicas.

     • Single-machine deployment - flagship complex inference (capacity_unit_per_instance = 8): capacity=8 → 1 replica; capacity=16 → 2 replicas; capacity=1 is rejected (not a multiple of 8).

   • billing_method: Currently supports "POST_PAY" (pay-as-you-go).

   API field to console mapping:

   API field	Console equivalent
   model_name	The fine-tuned model selected on the My Models page
   deploy_spec	The selection in Deployment Template : Single-machine deployment or Single-machine deployment - flagship complex inference
   capacity	Total Model Unit (the console calculates and displays this automatically per $$\text{number of model units}=\text{model units per replica}\times \text{Deployed Replicas}$$)

   plan and billing_method have no console equivalents (console deployments default to pay-as-you-go billing by model unit usage duration).

   Important

   capacity is not the replica count — it's the total number of model units. Verify against the formula and examples above before submitting.

3. Check deployment status (GET /api/v1/deployments/<deployed_model>): The deployment progresses through PENDING → DEPLOYING → RUNNING. Poll the status using:

   curl --location 'https://dashscope.aliyuncs.com/api/v1/deployments/<replace with deployed_model field value>' \
   --header 'Authorization: Bearer '${DASHSCOPE_API_KEY} \
   --header 'Content-Type: application/json'

   When status is RUNNING, the model is ready for calls. For additional deployment operations (scaling, decommissioning, etc.), see Model deployment .

Observability and troubleshooting

Log every call to support investigation of anomalous requests and performance issues. Capture these three categories of information:

• Request ID (request_id): Retrieved via synthesizer.get_last_request_id() in the WebSocket SDK (see the Fine-tune CosyVoice models example above). For HTTP calls, see Speech synthesis for the response field location. Always retain this ID when troubleshooting.

• First-packet latency: Retrieved via synthesizer.get_first_package_delay() in the WebSocket SDK — the time (in milliseconds) until the first audio packet arrives. This is the core metric for real-time synthesis experience. The Fine-tune CosyVoice models example above demonstrates printing this value.

• Error code and message: The code and message fields in error responses distinguish between authentication failures, quota exhaustion, model not ready, text length exceeded, and other failure types. Set up alerts and troubleshooting paths per error code.

Recommended fields to log (minimum set for tracing):

Correlate the request_id with your application's request ID in logs, and retain records for at least your SLA retention period.

Choose a better checkpoint

The default finetuned_output deployed earlier corresponds only to the top-ranked checkpoint from Checkpoints, which isn't necessarily the one that sounds best to human ears. Before going live, manually evaluate the top 2–3 checkpoints and select the one closest to your target voice.

Steps to switch to a different checkpoint:

1. Call GET /api/v1/fine-tunes/{job_id}/checkpoints to list all checkpoints for the job. Get the model ID for each from the output[*].model_name field (only returned when status=SUCCEEDED). For full field descriptions, see List checkpoints for a fine-tuning job .

2. Use the selected model_name as the input for the "Create a deployment" step in Method 2: Deploy through the API (recommended for automation) to get a separate deployed_model. Each checkpoint requires its own independent deployment instance — you can't swap the underlying model on an existing deployment.

3. Running multiple evaluation deployments simultaneously incurs billing for each deployment's model units. Refer to Deployment costs to estimate concurrent evaluation costs, and decommission rejected instances promptly after comparison (see Decommission unused deployments).

Decommission unused deployments

Decommission deployment instances immediately once they no longer serve traffic. Common scenarios: instances rejected after multi-checkpoint evaluation, old instances replaced by a new version, instances used only for load testing or validation, and instances from failed deployments that need to be recreated.

How to decommission: Call DELETE /api/v1/deployments/{deployed_model}. When the response shows output.status as DELETING, the request has been accepted. For full field descriptions, see Delete a deployment .

Plan replica count and scale

Adjust the replica count when business concurrency increases, overall throughput is insufficient, or you need elastic scaling for peak events. Different deployment templates (Single-machine deployment and Single-machine deployment - flagship complex inference) offer different per-replica compute and suit different scenarios — see Deployment templates. Determine replica count based on your concurrency requirements.

How to scale: Call PUT /api/v1/deployments/{deployed_model}/scale with the new capacity in the request body. Note that capacity is the total model unit count (not the replica count), and must be an integer multiple of that template's capacity_unit_per_instance. See Examples for value examples. For full field descriptions, see Scale a deployment .

Replica count linearly affects billing. Estimate cost changes using the formula in Deployment costs before scaling up.

Switch versions after re-training

When adding new data or adjusting hyperparameters, always re-train from the base model cosyvoice-v3-flash — don't train incrementally on an existing fine-tuned model. Re-training produces a new finetuned_output: a different model, not a new version of the same model. To use it, create a new deployment instance — you can't swap the underlying model on an existing deployed_model.

Recommended switching workflow:

1. Deploy the new model model_name following Method 2: Deploy through the API (recommended for automation) to create a separate deployment with a new deployed_model. Don't scale or modify settings on the old instance — neither operation changes its underlying model.

2. Route canary or shadow traffic to both old and new deployed_model instances simultaneously. Compare audio quality, first-packet latency, and error rate.

3. After validation passes, shift all traffic to the new deployment, then decommission the old one (see Decommission unused deployments) to complete the version switch.