Custom model deployment

Deployment mode comparison: Elastic instances vs. provisioned instances

FunModel provides two instance types to suit scenarios with different workload characteristics. Elastic instances are a flexible, pay-as-you-go model that focuses on cost control. Provisioned instances are a stable, long-term reservation model that focuses on performance assurance. For more information, see Instance types and specifications.

Inference framework comparison: vLLM vs. SGLang

FunModel has built-in support for vLLM and SGLang, two mainstream high-performance inference frameworks. vLLM and SGLang deployments currently support only large language models (LLMs).

Note: vLLM is a general-purpose, high-performance inference solution with broad community support. SGLang may provide better support when you need to handle complex prompt engineering or structured output.

Model source description

FunModel supports loading models from different sources. The model files are ultimately mounted to the /mnt/ directory of the instance.

Method 1: Deploy using a pre-built framework (vLLM/SGLang)

This method is suitable for most standard LLM models. It does not require you to build an image and is the recommended way to quickly deploy a model.

1. In the FunModel console, click Custom Development.

2. Select a Model Environment and enter a Model Name and Model Description.

3. Select a Model Source and enter the path. For more information, see Model source description.

4. Select an Instance Type (Elastic Instance or Provisioned Instance) and the required instance type.

   • Note: The selected Instance Type, especially its GPU memory, must meet the model's runtime requirements. Otherwise, the deployment may fail due to an out-of-memory (OOM) error.

5. Configure the Start Command and Listening Port. The start command is executed after the container starts, loading the model and starting the inference service. A default start command is provided based on the selected model source, which you can modify as needed.

   • vLLM start command example

     # Deploy the Qwen3-0.6B model
     # /mnt/Qwen3-0.6B is the model's mount path in the container. It must match the mount path from the model source.
     # --served-model-name: The model name used in API calls.
     # --tensor-parallel-size: The tensor parallelism size, usually set to the number of GPUs in the instance.
     vllm serve /mnt/Qwen3-0.6B \
       --served-model-name Qwen/Qwen3-0.6B \
       --port 9000 \
       --trust-remote-code \
       --tensor-parallel-size 1 \
       --gpu-memory-utilization 0.9 \
       --max-model-len 8192

     Note: For more information about start parameters, see the official vLLM documentation.

   • SGLang start command example

     # Deploy the Llama3-8B model
     # --model-path: Specifies the model's mount path in the container.
     # --host: The service listening address. It must be set to 0.0.0.0.
     # --tp-size: The tensor parallelism size.
     python3 -m sglang.launch_server \
       --model-path /mnt/Llama3-8B-Instruct \
       --host 0.0.0.0 \
       --port 9000 \
       --tp-size 1

     Note: For more information about start parameters, see the official SGLang documentation.

6. Role Name: The RAM role for accessing cloud resources. This role must have the necessary permissions. Select AliyunFCDefaultRole.

7. After you confirm the configurations, click Create Model Service.

Method 2: Deploy using a custom image

Service invocation

On the service details page, go to Overview > Access Information to obtain the endpoint and authentication token. Services deployed with vLLM and SGLang are usually compatible with the OpenAI API format.

cURL call example

# Replace <YOUR_ENDPOINT> and <YOUR_TOKEN> with your actual information.
ENDPOINT_URL="<YOUR_ENDPOINT>"
TOKEN="<YOUR_TOKEN>"

curl $ENDPOINT_URL/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-0.6B",
    "messages": [
      {"role": "user", "content": "Hello"}
    ],
    "stream": false
  }'

Python SDK call example

import requests
import json

# Replace <YOUR_ENDPOINT> and <YOUR_TOKEN> with your actual information.
ENDPOINT_URL = "<YOUR_ENDPOINT>/v1/chat/completions"
TOKEN = "<YOUR_TOKEN>"

headers = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "application/json"
}

data = {
    "model": "Qwen3-0.6B",  # The model name here must match the --served-model-name in the start command.
    "messages": [
        {"role": "user", "content": "Hello, please introduce yourself."}
    ],
    "stream": False
}

response = requests.post(ENDPOINT_URL, headers=headers, data=json.dumps(data))

if response.status_code == 200:
    print(response.json())
else:
    print(f"Error: {response.status_code}, {response.text}")

Logs and monitoring

• Operation Record > Deployment Logs: Records the key steps of a service deployment, including model download, image pull, instance startup, and model loading. These logs are the primary source for troubleshooting deployment failures.

• Logs: Records of request handling and error messages generated after the service starts. Use these logs to troubleshoot invocation exceptions.

• Monitoring: View service performance metrics such as QPS, latency, and GPU utilization.

FAQ and troubleshooting

1. Deployment failed

   • Check deployment logs: First, check the deployment logs for error messages.

   • Check the model path: Confirm that the model path referenced in the start command (/mnt/...) matches the mount rule of the model source.

   • Check the instance type: If the logs show an OutOfMemoryError or OOM, it usually means the instance has insufficient GPU memory. You can try switching to an instance with higher specifications.

   • Check RAM permissions: If you see permission denied or 403 Forbidden, it is usually because the RAM user that performs the operation lacks the necessary Devs-related permissions, which prevents the service configuration from being saved.

2. Service startup failed

   • Check service logs: Confirm whether the service start command executed successfully and check for any errors.

   • Check port configuration: Ensure that the port specified in the start command, such as --port 9000, matches the port configured in the console.

   • Check the listening address: For custom images, ensure that the service is listening on host 0.0.0.0.

3. Performance issues

   • Optimize start parameters: For multi-GPU instances, make sure to set --tensor-parallel-size to enable parallel computing.

   • Upgrade the instance type: If monitoring shows that resources such as GPU utilization have reached a bottleneck, you can consider upgrading to a higher-performance instance.

   • Use model quantization: Model quantization techniques, such as AWQ and GPTQ, can reduce GPU memory usage and computation. This operation may require a custom image.