When you update an online application, you must ensure service continuity while the application stops and restarts. When you deploy, stop, roll back, scale-in, or reset an application, you can configure graceful shutdown to ensure that the application shuts down without affecting your business. This topic describes the usage notes and benefits of graceful shutdown, and explains how to use the feature in the MSE console.
Why you need graceful shutdown
In a microservices architecture, abrupt shutdowns can cause problems. If a provider instance is shut down suddenly, especially under high concurrency, consumers may not immediately detect the change. They continue to send requests to the terminated instance, which leads to request errors and traffic loss.
How it works
When a provider instance begins to shut down, it notifies the Microservices Registry. However, consumers might not detect this change immediately and may continue to send requests to the terminating instance. To prevent this, after a provider receives a shutdown command, it enters a waiting state. During this period, it adds a special marker to all responses. When a consumer receives a response with this marker, it immediately queries the Microservices Registry to get the latest list of instances. The provider also waits to ensure all in-flight requests complete before it fully terminates.
Graceful shutdown is enabled by default for any application connected to MSE. However, the Proactive Notification feature must be configured manually. MSE also provides observability for Graceful Start and Shutdown, allowing you to verify if the application shuts down successfully.
When you enable graceful shutdown for an application in a Kubernetes environment, MSE injects alifecycle.preStop hook into the Pod to manage the shutdown process before termination. Therefore, we recommend that you do not configure a custompreStop hook. If your custompreStop hook's only function is to deregister the service from the Microservices Registry, you can safely remove it. MSE's automatically injectedpreStop hook handles this.
If you have a custompreStop hook configured for your application container to handle business-specific graceful termination, the graceful shutdownpreStop hook is not injected into that container. Instead, MSE injects a sidecar container named gracefulshutdown and injects the graceful shutdownpreStop hook into the sidecar. Because a sidecar shares the application container's network namespace by default, running the sidecar'spreStop hook before the Pod stops also ensures a graceful shutdown for the application.
To ensure that the sidecar'spreStop hook can complete, your custompreStop hook in the application container must include a sleep duration of at least 30 seconds. This guarantees that all in-flight requests are processed before the container terminates.
-
The sidecar container has minimal resource overhead. Current settings: CPU: 50m, Memory: 50Mi.
-
In Kubernetes, graceful shutdown uses the preStop hook. This means it only works for normal Pod termination events, such as scale-in, restart, or rolling update. It does not handle abnormal terminations, such as an out-of-memory (OOM) kill.
Enable graceful shutdown
-
If your application is deployed in an ACK environment and connected to MSE Microservices Governance, graceful shutdown is enabled automatically. No further action is required.
-
If your application is deployed in an ECS environment, you must add the following command to your application's shutdown script. Ensure this command runs early in the script's execution.
curl http://127.0.0.1:54199/offline 2>/tmp/null; sleep 30;
Observe graceful shutdown
On the Application Governance page, you will see traffic drain quickly from the terminating instance as its QPS drops to 0.
Log on to the MSE console, and select a region in the top navigation bar.
In the left-side navigation pane, choose . On the page that appears, click the resource card of the application that you want to manage.
On the application details page, click Traffic management in the left-side navigation pane, and click the Graceful Start/Shutdown tab.
On the Start and Shutdown Overview tab, you can view your application's recent startup and shutdown events. Find and click a terminating instance in the list on the left. The details on the right show that the instance executed the graceful shutdown process. You can see that traffic was quickly drained and no new traffic was received before the instance fully stopped.
-
If you observe that the QPS data does not drop to 0 quickly after a successful graceful shutdown event, verify whether there is traffic from sources other than microservice calls, such as local calls.
-
The "application stopped" event is reported only by agent version 4.2.0 or later. If you do not see this event, consider upgrading your agent.
Enable Proactive Notification
What is Proactive Notification
Proactive Notification is an advanced feature of graceful shutdown that is disabled by default. In most cases, the standard graceful shutdown process is sufficient. However, if you use the Spring Cloud framework and still encounter consumer errors when a provider shuts down, you can enable this feature. When enabled, a provider that is shutting down proactively notifies its consumers. Upon receiving this notification, consumers immediately stop sending new requests to that instance.
Prerequisites
Usage notes
MSE Microservices Governance does not support graceful shutdown for the following types of applications:
-
Non-Java applications.
-
Applications not built with WebFlux or Spring MVC.
-
Provider applications whose consumers are not microservices.
-
Both consumer and provider applications must be connected to MSE Microservices Governance.
The graceful shutdown process requires time to ensure all in-flight requests are completed. Therefore, theterminationGracePeriodSeconds parameter in your Kubernetes Pod specification (default: 30) must be greater than 30. We recommend setting it to 90. If you use the default value of 30, the application'sshutdown hook may not have enough time to complete, preventing resources from being released properly during shutdown.
Enable Proactive Notification
Log on to the MSE console, and select a region in the top navigation bar.
In the left-side navigation pane, choose . On the page that appears, click the resource card of the application that you want to manage.
On the application details page, click Traffic management in the left-side navigation pane, and click the Graceful Start/Shutdown tab.
-
In the Settings section, click Modify. Click the Graceful Shutdown section to expand it, turn on the Proactive Notification switch, and then click OK.