Best Practices for Custom Model Tuning, Deployment, and Evaluation

1. Improve accuracy in specific domains: General-purpose LLMs are powerful but may underperform on domain-specific tasks due to limited domain knowledge. Custom LLMs, retrained on domain-specific data, significantly improve performance and accuracy in that domain.

2. Increase model applicability: You can fine-tune custom LLMs to match the requirements of specific application scenarios. For example, in customer service, a custom model can better understand and respond to customer questions—improving satisfaction.

3. Save development time and cost: Customizing an existing LLM is faster and less expensive than building one from scratch. You can deploy your custom model quickly to meet business needs.

Creating a custom model involves three main steps: Model Fine-tuning, Deployments, and Evaluation. It also includes three supporting steps: training data preparation, evaluation template design, and training strategy adjustment.

1. Model Fine-tuning phase: The model “learns” language patterns from large volumes of training data to understand and generate natural language. Before starting model tuning, prepare your training data by collecting, cleaning, and splitting it into training and validation sets. Then use the Train New Model wizard to configure Model Fine-tuning. Alibaba Cloud Model Studio automatically trains your selected prebuilt model using the hyperparameters you set—such as learning rate and number of iterations. In most cases, training runs automatically with no further action needed.

2. Deployments phase: Deploy your custom model to a dedicated instance, then invoke or evaluate it. Use the Deploy New Model wizard to configure Deployments. Alibaba Cloud Model Studio calculates an estimated price based on your selected model specifications and resource configuration. After you confirm the price, deployment starts automatically—usually with no intervention required. Once complete, you can configure your custom model in code or evaluation tools.

3. Evaluation phase: Evaluate your deployed custom model. Use the Create Evaluation Task wizard to configure Evaluation. Custom models do not require evaluation template design. Alibaba Cloud Model Studio automatically runs the evaluation based on your selected method, data, and dimensions—typically with no intervention needed.

Gather high-value business data from various sources and organize it into a structured format. When collecting data, consider these strategies:

• Diversify sources: Collect data from books, academic papers, news articles, and professional websites. This helps your model handle diverse questions and phrasings.

• Control quality: Data quality directly affects model quality. High-quality, abundant data helps your model learn better and perform well.

• Maintain balance: Ensure even distribution across question types, difficulty levels, and answer formats. This prevents bias toward any single data type during training.

In Alibaba Cloud Model Studio, format your collected data as ChatML-style messages in a multi-turn structure (jsonl file) so the platform can parse and train on it. For format details, see the data format section in the model tuning introduction. To maximize training efficiency, prepare at least 500 training samples. Follow these guidelines:

• Split long text: Break long documents into smaller segments. Each training sample should focus on a single, clear topic to help the model learn effectively.

• Remove sensitive content: Strip out personally identifiable information, sensitive words, or inappropriate material to ensure compliance and security.

Upload your prepared data to Alibaba Cloud Model Studio for cleaning, augmentation, or direct use in model tuning. The platform automatically checks formatting and performs basic data quality validation to ensure your data meets training requirements.

Alibaba Cloud Model Studio supports multiple dataset types—including training and evaluation datasets—and offers multi-version data management. After publishing, you can use datasets in model tuning or evaluation. Built-in tools let you tag, filter, and manage versions—supporting iterative training and performance tracking.

Data cleaning checks and fixes collected data to ensure quality, completeness, and consistency. This step greatly improves training effectiveness and prediction capability. Data augmentation increases diversity and expands the scale of your training data.

Alibaba Cloud Model Studio supports versioned data management. Each time you clean or augment data, a new version is created automatically. The new version is saved separately and does not overwrite your source data.

On the Model Tuning page, click Create Training Task to set training parameters. Follow these steps:

1. Go to the Model Tuning page. This shows all tuning tasks.

2. Click Create Training Task. Alibaba Cloud Model Studio guides you through parameter configuration:

   1. Select a model training method: Alibaba Cloud Model Studio supports three methods: Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO), and Continued Pre-training (CPT). Choose based on your needs:

      Training method	Key features	Best for
      SFT fine-tuning	Supervised fine-tuning that improves instruction-following ability. Supports full-parameter and efficient training.	When your model must master domain-specific Q&A or fixed-format outputs.
      DPO preference training	Uses negative feedback to reduce hallucinations and align outputs with human preferences.	When output quality and factual accuracy are critical.
      CPT continued training	Unsupervised continued pre-training on unlabeled data to strengthen or add domain-specific capabilities.	You need to build a model for a professional domain to equip it with expert-level knowledge.

   2. Select Model: Alibaba Cloud Model Studio supports tuning based on prebuilt or custom models.

      If this is your first model training, choose a prebuilt model. If you’ve adjusted your training strategy and want to retrain from an existing model, choose a custom model.

      Alibaba Cloud Model Studio offers many tunable prebuilt models. See Model invocation for model details.

   3. Training Method: Alibaba Cloud Model Studio supports Full Parameter Training and Efficient Training. Choose based on your needs:

      Training method	Pros	Cons
      Full-parameter training	Maximum flexibility: Adjusts all parameters to adapt to new data and tasks. High performance: Can achieve top performance because all parameters are optimized for your task.	Time-intensive: Takes longer—especially on large datasets. Risk of overfitting: May overfit if training data is insufficient or imbalanced.
      Efficient training	Faster training: Ideal for rapid iteration and prototyping. Lower overfitting risk: Fine-tunes only part of the model.	Potentially lower performance: May lag behind full-parameter training on some tasks. Limited scope: May fail to transfer knowledge effectively if your task differs greatly from the pretraining task.

      Efficient training balances speed and results well. We recommend it for most use cases.

   4. Select Training Data: Choose the dataset to use for training. Pick a training set from your uploaded list.

      Note

      If you cleaned or augmented your data, double-check the dataset name and version to avoid selecting raw, unprocessed data.

   5. Select Validation Data: Choose the dataset to validate training performance. You can pick Automatic Splitting or select a Select Validation Set from your list.

      If you have not prepared a validation set, we recommend selecting Automatic Splitting. Alibaba Cloud Model Studio will split a portion of the selected training set to use as the validation set. Conversely, we recommend selecting a validation set from the data list.Select Validation Set.

      Note

      If you cleaned or augmented your data, double-check the dataset name and version to avoid selecting raw, unprocessed data.

   6. Mixed Training: To preserve base model capabilities and boost training results, Alibaba Cloud Model Studio lets you mix your own training data with prebuilt generic data. Adjust the ratio between your data and the prebuilt data. Set all prebuilt data ratios to zero to skip prebuilt data entirely.

   7. Configure hyperparameters: Hyperparameters strongly influence tuning results. Configure them based on your experience—or use Alibaba Cloud Model Studio’s default settings, which are based on extensive testing.

   8. Start Training: This screen shows all your settings. Alibaba Cloud Model Studio calculates an estimated training cost (you can also view cost details later in the training task list—the final bill reflects actual usage). Confirm your settings and price before starting.

3. After training starts, check progress, estimated costs, and other actions in the Model Fine-tuning list. For billing details, see Billing items.

After launching training, your new task appears in the Model Tuning list with status Training. Click View to open the task details page and review settings and metrics. Click View Estimation to see estimated costs. Click Terminate Training anytime to stop the task.

Three key metrics reflect training effectiveness. If you’re unfamiliar with them, wait until training finishes and use Alibaba Cloud Model Studio’s model evaluation tool instead.

Training typically takes 2–3 hours for datasets under 1,000 samples. Due to platform capacity limits, tasks may queue. When training finishes, the status changes to Trained.. You now have a Custom Model ready for deployment. Click Deployments to go straight to the deployment page.

1. Go to the Model Deployment page. Or, after training finishes on the Model Fine-tuning page, click Deployments. This page shows all deployment tasks.

2. Click Deploy New Model and follow the wizard:

   1. Select Model: Alibaba Cloud Model Studio offers two deployable model types. Choose your previously trained Custom Model.

   2. Select billing method: Alibaba Cloud Model Studio offers two billing methods. Subscription means buying compute resources monthly. Different Monthly Resources versions offer different deployment capacities. Pay-as-you-go means paying only for actual usage time. You can buy a set number of instances—or adjust that number later. To run Evaluation and assess training results, we recommend Pay-as-you-go.

   3. Deploy: This screen shows all your settings and estimated costs. Confirm everything before starting.

3. After deployment starts, check progress, estimated costs, and other actions in the Deployments list. For billing details, see Billing items.

Deployment usually takes minutes to hours. Due to Alibaba Cloud Model Studio resource limits, tasks may queue. When done, the status changes to Running. Running models can be invoked—for evaluation or application use.

After launching deployment, your new task appears in the Model Deployment list with status Deploying. Click View to review settings. Click Scaling to adjust compute resource usage. Click Deactivate anytime to end deployment.

After deployment finishes, the status changes to Running. Your custom model can now run inference. Click Try Now to test inference on the Playground page. Then use Evaluation to assess your custom model.

1. Go to the Model Evaluation page. This shows all Evaluation tasks.

2. Click Create Evaluation Task and follow the wizard:

   1. Evaluation Method: Alibaba Cloud Model Studio offers several methods. For custom models, choose baseline evaluation.

   2. Select Model: Choose your deployed target model from the dropdown.

   3. Select evaluation data: Alibaba Cloud Model Studio includes baseline evaluation datasets—such as C-Eval and CMMLU—to automatically test fundamental model capabilities. Choose any dataset to build your full evaluation set.

   4. Evaluate: This screen shows all your settings and estimated costs. Confirm everything before starting.

3. After evaluation starts, check progress, estimated costs, and other actions in the Evaluation list. For billing details, see Billing items.

When evaluation starts, its status becomes Running or Queuing. Click Abort anytime to stop it. After completion, click Results to view detailed results. Click Delete to remove an evaluation task.