Real-time inference scenarios

Use cases

GPU-accelerated instances in idle mode

After you deploy a GPU function, you can enable gpu-accelerated instances in idle mode to support your real-time inference applications. Based on your workload, Function Compute uses Horizontal Pod Autoscaling (HPA) to manage provisioned instances according to your configured auto scaling policy. Requests are prioritized to these provisioned instances, which eliminates cold starts and ensures low-latency responses.

Cost benefits of idle mode

gpu-accelerated instances in idle mode are billed based on two different unit prices: one for idle GPU usage and one for active GPU usage. Function Compute automatically tracks the instance state and calculates the charges accordingly.

As shown in the following figure, an instance goes through four time windows from T0 to T4 between its creation and destruction. The instance is active during T1 and T3, and is in idle mode during T0, T2, and T4. The total cost for this period is calculated as follows: (T0 + T2 + T4) × Unit price for idle GPU usage + (T1 + T3) × Unit price for active GPU usage. For more information about the unit price for idle GPU usage and the unit price for active GPU usage, see Billing overview.

How idle mode works

Function Compute uses advanced, self-developed technology to implement an instant freeze and fast restoration for gpu-accelerated instances. When a gpu-accelerated instance is not processing requests, Function Compute automatically places it in a frozen state and bills it at the idle unit price. This approach optimizes resource utilization and reduces costs. When a new compute request arrives, the platform quickly wakes up the instance to seamlessly execute the task and automatically bills it at the active unit price.

This process is completely transparent and does not affect the user experience. Function Compute ensures that the accuracy and reliability of the inference service are not compromised, even when an instance is frozen, providing a stable and cost-effective computing environment.

Idle mode wake-up latency

The wake-up time varies depending on the workload. The following table provides the typical wake-up times for different inference workloads.

Usage limits for idle mode

• CUDA version

  Use CUDA 12.2 or an earlier version.

• Image permissions

  Run the container image with default root user permissions.

• Instance logon

  You cannot log on to a gpu-accelerated instance that is in idle mode because the GPU may be frozen.

• Graceful instance rotation

  Function Compute performs graceful rotation of gpu-accelerated instances in idle mode based on system load. To ensure service quality, add a model warm-up or pre-inference lifecycle hook to your function. This allows new instances to serve inference requests immediately after they are launched. For more information, see Model warm-up.

• Model warm-up/Pre-inference

  For gpu-accelerated instances in idle mode, use the initialize lifecycle hook in your code for model warm-up or pre-inference to ensure the initial wake-up latency meets your expectations. For more information, see Model warm-up.

• Provisioned configuration

  When you toggle the idle mode switch, the existing provisioned gpu-accelerated instances for the function are gracefully scaled down. The number of provisioned instances temporarily drops to zero until new provisioned instances are created.

• Disable the built-in metrics server of the inference framework

  To improve the compatibility and performance of GPUs in idle mode, disable the built-in metrics server of your inference framework, such as NVIDIA Triton Inference Server or TorchServe.

Idle mode specifications

gpu-accelerated instances in idle mode currently require a full GPU. For more information about gpu-accelerated instance specifications, see Instance specifications.

Optimized request scheduling for inference

How it works

Function Compute uses an intelligent, load-aware scheduling strategy which is significantly more effective than traditional round-robin scheduling. The platform monitors the task execution status of gpu-accelerated instances in real time and immediately assigns new requests to an instance as soon as it becomes idle. This ensures efficient use of GPU resources, prevents idle cycles, and avoids hot spots. It also guarantees that load balancing across gpu-accelerated instances is consistent with GPU compute utilization. The following figure uses the T4 card type as an example.

Scheduling results

Function Compute's built-in scheduling logic automatically balances the load across multiple gpu-accelerated instances, so you do not need to manage the logic yourself.

Container support

For GPU use cases, Function Compute currently supports only deployment with a Custom Container runtime. For more information about Custom Container, see Introduction to custom images.

Custom Container functions require a web server to be included in the image to handle different code paths and to be triggered by events or HTTP requests. This is suitable for multi-path request execution scenarios such as AI training and inference.

Deployment methods

For more deployment examples, see start-fc-gpu.

Model warm-up

To prevent long initial request processing times after a model is deployed, Function Compute provides a model warm-up feature. The purpose of model warm-up is to ensure the model is ready to serve traffic immediately after it goes live.

We recommend that you configure the initialize lifecycle hook for your instance to implement model warm-up. Function Compute automatically runs the business logic within the initialize hook after your instance starts to warm up the model service. For more information, see Function instance lifecycle hooks.

1. In the HTTP server you build, add a /initialize invocation path for the POST method and place the model warm-up logic under the /initialize path. Typically, you can warm up the model service by having it perform a simple inference task.

   The following code provides an example in Python.

   def prewarm_inference():
       res = model.inference()
   
   @app.route('/initialize', methods=['POST'])
   def initialize():
       request_id = request.headers.get("x-fc-request-id", "")
       print("FC Initialize Start RequestId: " + request_id)
   
       # Prewarm model and perform naive inference task.    
       prewarm_inference()
       
       print("FC Initialize End RequestId: " + request_id)
       return "Function is initialized, request_id: " + request_id + "\n"

2. On the function details page, choose Configurations > Lifecycle, and then click Modify to configure the instance lifecycle hook.

   

Configure auto scaling for real-time inference

FAQ

Cost calculation for real-time inference

For more information about the billing of Function Compute, see Billing overview. The billing for provisioned instances is different from that for on-demand instances. Pay attention to your bill details.

Performance jitters with auto scaling

Consider using a more aggressive auto scaling policy to provision instances in advance. This can help prevent performance degradation caused by sudden bursts of requests.

Delayed instance scale-out

Function Compute collects metrics at one-minute intervals. A scale-out is triggered only after the metric remains above the threshold for a certain period.