Snapshots

Description

The system creates a snapshot from a pre-warmed container instance and saves it. This process is transparent to your application and requires no action on your part. When a request arrives, the system uses the snapshot to rapidly launch an instance to process the request.

For GPU workloads, a millisecond-level snapshot achieves a cold start time of 1 ms or less. A second-level snapshot has a cold start time of approximately 2 seconds or more, depending on the model size.

The following diagram compares the cold start time for the first inference in an AI scenario, using different scaling configurations.

The following chart shows the cold start time improvements for the SD-v1-5-inpainting and Qwen-14B-Chat-Int4 models when using different snapshot types.

Use cases

• Recommended use cases for millisecond-level snapshots (real-time)

  These are ideal for scenarios that require extremely low latency, high success rates on critical paths that cannot tolerate long retries, and have distinct traffic peaks and troughs. They also help improve resource utilization.

  Your workload should have one or more of the following characteristics:

  • Low latency: Applications with strict response time (RT) requirements for individual requests, where 90th percentile latency is typically in the hundreds of milliseconds. A typical use case is real-time media production, such as interactive co-hosting, live e-commerce streaming, and ultra-low latency playback. In these scenarios, audio and video streams must be transmitted end-to-end with minimal delay, and the real-time performance of AI-powered video super-resolution or recognition is critical.

  • Critical path: These workloads are part of a core business process and require a high success rate, without the option for lengthy retries. For example, in an ad or product recommendation scenario, recommendations based on user behavior must be generated and displayed on the user's device instantly when an app is launched.

  • Peaks and troughs: Workloads with significant fluctuations in traffic, where peak traffic is concentrated in specific time periods. To handle these peaks, you might need to provision a large number of GPU resources, which can lead to low resource utilization during off-peak hours. Examples include short-video recommendation feeds, online education, and e-commerce promotional events.

• Recommended use cases for second-level snapshots (near-real-time, asynchronous)

  These are suitable for asynchronous inference scenarios with sparse invocations that can tolerate cold starts, have long-running individual tasks, and require an immediate response after a trigger.

  Your workload should have one or more of the following characteristics:

  • Sparse invocations with high latency tolerance: Workloads with a few to tens of thousands of daily invocations, where the actual GPU usage is much less than 8 to 12 hours a day. In these cases, GPUs may be idle for long periods, and the application is not sensitive to cold start latency. Typical use cases include non-real-time data analysis and low-frequency batch processing.

  • Long-running tasks: This includes tasks that can take from seconds to hours to complete and are not sensitive to response time. These workloads do not require immediate results and support asynchronous polling or callback notifications. Typical use cases include long-running AI inference tasks such as audio and video transcoding or film and television special effects rendering, which can tolerate latency. Immediate return after submission (asynchronous interaction): This also includes tasks that must return a response immediately after being triggered, without blocking the main business logic. Typical use cases include audio and video processing and data ETL jobs, where large media files are split into multiple parallel tasks, or where data extraction, transformation, and loading are handled asynchronously.

  • Real-time task status awareness: Offline GPU tasks that require the ability to monitor their execution status and be canceled mid-process. A typical example is interactive AI, where a user can adjust generation parameters or cancel a task, such as changing the style of an AI-generated image.

  • Data source integration and parallel processing

    Offline GPU tasks often need to interact with various data sources, such as storage services like OSS and messaging products like a message queue. For example, a task might read raw data from OSS and write the processed results to a cloud database.

    Offline GPU tasks that need to process large amounts of data require a large supply of GPU resources to run in parallel and accelerate processing. Examples include high-throughput computing scenarios like parallel image compression or speech-to-text transcription.