Rolling updates and graceful shutdown

Rolling update

During an update, the system creates new replicas and gradually replaces old ones based on your configuration. If a new replica fails to start, the update is paused. The failed replica does not receive traffic, and the remaining old replicas continue to serve requests, so your service is not affected. You can choose to roll back the update or start a new one. A new update prioritizes removing failed replicas from the previous incomplete attempt.

Two key parameters control the rolling update process:

• Exceeds the expected number of replicas (JSON parameter: rolling_strategy.max_surge)

  • Description: The maximum number of extra replicas the system can create during an update. This can be a positive integer or a percentage. A higher value speeds up the update process.

  • Example: For a service with 100 replicas, if you set this value to 20, the system creates 20 new replicas when the update begins.

  • Default: 2% of the total replica count, with a minimum of 1.

  Important

  If Exceeds the expected number of replicas is set too high, the system brings a large number of new replicas online and immediately replaces an equal number of old ones. If the new replicas have not been warmed up, the sudden increase in traffic might impact service stability.

• Maximum Unavailable Replicas (JSON parameter: rolling_strategy.max_unavailable)

  • Description: The maximum number of replicas allowed to be unavailable during an update. This setting frees up resources and prevents resource shortages from blocking the update.

  • Example: If you set this value to N, the system immediately stops N old replicas when the update starts.

  • Default:

    • For a dedicated resource group: The default is 1 for services created before September 1, 2023. For services created on or after September 1, 2023, the default is 0 if an elastic pool is enabled, and 1 if it is not.

    • For a public resource group: 0.

    • For Lingjun Intelligent Computing Service: The default is 0 for services created before September 1, 2023. For services created on or after September 1, 2023, the default is 2% of the replica count, with a minimum of 1.

  Important

  • For a single-replica service, if you set Maximum Unavailable Replicas to 1, the old replica stops before the new one starts, making the service temporarily unavailable.

  • Setting Maximum Unavailable Replicas too high can cause too many replicas to go offline simultaneously. The remaining replicas may not be sufficient to handle the traffic, which degrades service availability.

Graceful shutdown

Graceful shutdown parameters affect the stability of replica termination during a rolling update.

• Graceful Shutdown Time (JSON parameter: eas.termination_grace_period)

  • Description: The amount of time, in seconds, that the system waits for a replica to shut down gracefully. After a replica enters the Terminating state, the system stops routing traffic to it. The system then waits for this period to allow the replica to finish processing any in-flight requests before terminating it. If your requests typically have long processing times, we recommend increasing this value.

  • Default: 30.

  Important

  Lowering this value can affect service stability, while setting it too high can slow down service updates. Unless you have specific requirements, do not change this parameter.

• Send SIGTERM (JSON parameter: rpc.enable_sigterm)

  • Description: SIGTERM is a signal that terminates a process. The JSON parameter accepts true or false.

    • false: The system does not send a SIGTERM signal when a replica exits.

    • true: The system immediately sends a SIGTERM signal when a replica exits. The service's main process must implement custom graceful shutdown logic in a signal handler. Otherwise, the system may terminate the process directly, causing the graceful shutdown to fail.

  • Default: Disabled (false).

By default, the system does not send a SIGTERM signal. This is because most application containers do not handle the SIGTERM signal by default. If a container receives a SIGTERM signal without a corresponding handler, its process exits immediately. This bypasses the graceful shutdown process, disrupting your service.

For services with highly variable request processing times, we recommend enabling SIGTERM. For example, if request processing times range from a few seconds to 30 minutes, setting a fixed graceful shutdown time of 30 minutes will slow down service updates. In this scenario, you should configure your application container to handle the SIGTERM signal, finish processing any in-flight requests, and then exit. This allows for more flexible control over the shutdown process.

You do not need to enable SIGTERM for an asynchronous inference service. When a replica exits, the EAS control plane automatically handles the SIGTERM signal. It stops subscribing to new requests and waits for existing requests to complete before terminating the replica.