Custom model deployment

Deployment mode comparison: Elastic vs. provisioned instances

FunModel provides two instance types to suit business scenarios with different load characteristics. Elastic instances are a flexible, pay-as-you-go model focused on cost control. Provisioned instances are a stable, long-term reserved model focused on performance assurance. For more information, see Instance types and specifications.

Inference framework comparison: vLLM vs. SGLang

FunModel includes built-in support for vLLM and SGLang. These are two popular, high-performance inference frameworks. Note that vLLM and SGLang deployments currently support only large language models (LLMs).

Note: vLLM is a general-purpose, high-performance inference solution with broad community support. SGLang is better suited for complex prompt engineering or structured output.

Model sources

FunModel supports loading models from different sources. The model files are mounted to the /mnt/ directory of the instance.

Method 1: Deploy using a pre-built framework (vLLM/SGLang)

This method is suitable for most standard LLM models and does not require you to build a custom image. We recommend this method for quickly deploying a model.

1. In the FunModel console, click Custom Development.

2. Select a Model Environment, and then enter a Model Name and Model Description.

3. Select a Model Source and enter the path information as described in the Model sources section.

4. Select an Instance Type (Elastic Instance or Provisioned Instance) and a specific instance type.

   • Note: The selected instance type (especially the GPU memory) must meet the model's operational requirements. Otherwise, the deployment may fail due to an out-of-memory (OOM) error.

5. Configure the Startup Command and Listener Port. This command is executed after the container starts to load the model and start the inference service. The system provides a default startup command based on the model source that you select, which you can modify to meet your requirements.

   • vLLM startup command example

     # Deploy the Qwen3-0.6B model
     # /mnt/Qwen3-0.6B is the model's mount path inside the container. It must match the mount path of the model source.
     # --served-model-name: The model name used in API calls.
     # --tensor-parallel-size: The number of tensor parallel replicas. This is usually set to the number of GPUs in the instance.
     vllm serve /mnt/Qwen3-0.6B \
       --served-model-name Qwen/Qwen3-0.6B \
       --port 9000 \
       --trust-remote-code \
       --tensor-parallel-size 1 \
       --gpu-memory-utilization 0.9 \
       --max-model-len 8192

     Note: For more startup parameters, see the official vLLM documentation.

   • SGLang startup command example

     # Deploy the Llama3-8B model
     # --model-path: Specifies the model's mount path inside the container.
     # --host: The service listener address. Must be set to 0.0.0.0.
     # --tp-size: The tensor parallel size, which is the number of tensor parallel replicas.
     python3 -m sglang.launch_server \
       --model-path /mnt/Llama3-8B-Instruct \
       --host 0.0.0.0 \
       --port 9000 \
       --tp-size 1

     Note: For more startup parameters, see the official SGLang documentation.

6. For Role Name, select AliyunFCDefaultRole. This RAM role is used to access cloud resources and must have the necessary permissions.

7. After you confirm that all configurations are correct, click Create Model Service.

Method 2: Deploy using a custom image

Service invocation

On the service details page, go to the Overview > Access Information section to obtain the endpoint and authentication token. Services deployed with vLLM and SGLang are usually compatible with the OpenAI API format.

cURL invocation example

# Replace <YOUR_ENDPOINT> and <YOUR_TOKEN> with your actual information
ENDPOINT_URL="<YOUR_ENDPOINT>"
TOKEN="<YOUR_TOKEN>"

curl $ENDPOINT_URL/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Hello"}
    ],
    "stream": false
  }'

Python SDK invocation example

import requests
import json

# Replace <YOUR_ENDPOINT> and <YOUR_TOKEN> with your actual information
ENDPOINT_URL = "<YOUR_ENDPOINT>/v1/chat/completions"
TOKEN = "<YOUR_TOKEN>"

headers = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "application/json"
}

data = {
    "model": "Qwen3-0.6B",  # The model name here must match the --served-model-name in the startup command
    "messages": [
        {"role": "user", "content": "Hello, please introduce yourself"}
    ],
    "stream": False
}

response = requests.post(ENDPOINT_URL, headers=headers, data=json.dumps(data))

if response.status_code == 200:
    print(response.json())
else:
    print(f"Error: {response.status_code}, {response.text}")

Logs and monitoring

• Operation Record > Deployment Log: Records key steps during service deployment, including model download, image pull, instance startup, and model loading. This is the primary source for troubleshooting deployment failures.

• Log: Records request handling and error messages after the service starts. Use this log to troubleshoot invocation errors.

• Monitoring: View metrics such as queries per second (QPS), latency, and GPU utilization to evaluate service performance.

FAQ and troubleshooting

1. Deployment failed

   • Check the deployment log: Review the deployment log for error messages.

   • Check the model path: Confirm that the model path referenced in the startup command (/mnt/...) matches the mounting rule of the model source.

   • Check the instance type: If the log shows an OutOfMemoryError or OOM error, it usually means the instance has insufficient GPU memory. Try switching to an instance type with more memory.

   • Check RAM permissions: If you see a permission denied or 403 Forbidden error, this is usually because the RAM user performing the operation lacks the required Devs-related permissions. This prevents the service configuration from being saved.

2. Service startup failed

   • Check the service log: Confirm whether the service startup command executed successfully and check for any errors.

   • Check the port configuration: Make sure the port specified in the startup command, such as --port 9000, matches the port configured in the console.

   • Check the listener address: For custom images, make sure the service is listening on host 0.0.0.0.

3. Performance issues

   • Optimize startup parameters: For multi-GPU instances, make sure you set --tensor-parallel-size to enable parallel computing.

   • Upgrade the instance type: If monitoring shows that resources such as GPU utilization have reached a bottleneck, consider upgrading to a higher-performance instance.

   • Use model quantization: Model quantization techniques, such as AWQ and GPTQ, can reduce GPU memory usage and computational load. This operation may require a custom image.