If your workload is sensitive to interruptions, you must detect and respond to these events promptly. This topic describes several methods for building an effective interruption event handler.
Detect interruption events
ECS SDK
You can call the DescribeInstances API operation to query instance information. Check the OperationLocks parameter in the response to determine whether the instance has entered the to be recycled state.
-
An empty value for OperationLocks indicates that you can continue to use the instance.
-
If the value of
LockReasonisRecycling, the spot instance has been interrupted and is in theto be recycledstate.
SDK call example
Prerequisites
-
Create an AccessKey
Because an Alibaba Cloud account (root account) has full permissions for all resources, a leaked AccessKey for the account poses a high security risk. We recommend that you use the AccessKey of a RAM user that is granted only the minimum required permissions. For more information, see Create an AccessKey.
-
Grant ECS-related permissions to the RAM user
Grant the RAM user permissions to manage ECS resources. The sample code in this topic requires permission to query instance information. We recommend granting the following permission:
Cloud product
Permission
ECS
This example uses the system policy: AliyunECSFullAccess
-
Configure access credentials
The sample code in this topic reads the AccessKey from environment variables to use as access credentials for cloud services. For instructions, see Configure environment variables on Linux, macOS, and Windows systems.
-
Install the ECS SDK
Obtain the ECS SDK. This topic demonstrates how to install the SDK by adding a Maven dependency. For information about other installation methods, see Install the ECS SDK for Java.
Initialize the client
Alibaba Cloud SDKs support multiple types of access credentials, such as an AccessKey or an STS token, to initialize a client. For more information, see Manage access credentials.
import com.aliyun.ecs20140526.Client;
import com.aliyun.teaopenapi.models.Config;
public class Sample {
private static Client createClient() throws Exception {
Config config = new Config()
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set in your runtime environment.
.setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set in your runtime environment.
.setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"))
// For endpoints, see https://api.aliyun.com/product/Ecs
.setEndpoint("ecs.cn-hangzhou.aliyuncs.com");
return new Client(config);
}
}
Build the API request
Before you build the request object, refer to the API documentation for parameter information.
// Build the request object.
DescribeInstancesRequest request = new DescribeInstancesRequest().setRegionId("cn-hangzhou");
Initiate the call
When calling an API operation, you can set runtime parameters, such as timeout and proxy configurations. For more information, see Advanced configurations.
// Set runtime parameters.
RuntimeOptions runtime = new RuntimeOptions();
// Call the DescribeInstances API operation.
DescribeInstancesResponse response = client.describeInstancesWithOptions(request, runtime);
System.out.println(response.body.toMap());
Handle exceptions
The SDK for Java classifies exceptions into two main types: TeaUnretryableException and TeaException.
-
TeaUnretryableException: This exception is typically caused by network issues and is thrown after the maximum number of retries is reached. -
TeaException: This exception is primarily for service-related errors.
To ensure system robustness and stability, implement proper exception handling, such as propagating exceptions, logging, and attempting recovery.
Complete example
import com.aliyun.ecs20140526.Client;
import com.aliyun.ecs20140526.models.DescribeInstancesRequest;
import com.aliyun.ecs20140526.models.DescribeInstancesResponse;
import com.aliyun.ecs20140526.models.DescribeInstancesResponseBody;
import com.aliyun.tea.TeaException;
import com.aliyun.tea.TeaUnretryableException;
import com.aliyun.teaopenapi.models.Config;
import com.aliyun.teautil.models.RuntimeOptions;
import com.alibaba.fastjson.JSONArray;
import java.util.Arrays;
public class Sample {
private static Client createClient() throws Exception {
Config config = new Config()
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set in your runtime environment.
.setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set in your runtime environment.
.setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"))
// For endpoints, see https://api.aliyun.com/product/Ecs
.setEndpoint("ecs.cn-hangzhou.aliyuncs.com");
return new Client(config);
}
public static void main(String[] args) {
try {
Client client = Sample.createClient();
// Build the request object.
// Set the IDs of one or more ECS instances to query.
JSONArray instanceIds = new JSONArray();
instanceIds.addAll(Arrays.asList("i-bp145cvd0exyqj****","i-bp1gehfgfrrk4lah****"));
DescribeInstancesRequest request = new DescribeInstancesRequest()
.setRegionId("cn-hangzhou")
.setInstanceIds(instanceIds.toJSONString());
// Set runtime parameters.
RuntimeOptions runtime = new RuntimeOptions();
while (!instanceIds.isEmpty()) {
// Call the DescribeInstances API operation.
DescribeInstancesResponse response = client.describeInstancesWithOptions(request, runtime);
// Obtain the instance-related response.
DescribeInstancesResponseBody responseBody = response.getBody();
DescribeInstancesResponseBody.DescribeInstancesResponseBodyInstances instanceList = responseBody.getInstances();
// Obtain instance information and check the value of lockReason.
if (instanceList != null && instanceList.getInstance()!= null && !instanceList.getInstance().isEmpty()) {
for (DescribeInstancesResponseBody.DescribeInstancesResponseBodyInstancesInstance instance : instanceList.getInstance()) {
// Output the ID and zone of the queried instance.
System.out.println("result:instance:" + instance.getInstanceId() + ",az:" + instance.getZoneId());
if (instance.getOperationLocks() != null ) {
DescribeInstancesResponseBody.DescribeInstancesResponseBodyInstancesInstanceOperationLocks operationLocks = instance.getOperationLocks();
if(operationLocks.getLockReason()!=null && !operationLocks.getLockReason().isEmpty()){
for (DescribeInstancesResponseBody.DescribeInstancesResponseBodyInstancesInstanceOperationLocksLockReason lockReason : operationLocks.getLockReason()) {
// If the instance is locked, output its ID and lock type.
System.out.println("instance:" + instance.getInstanceId() + "-->lockReason:" + lockReason.getLockReason() + ",vmStatus:" + instance.getStatus());
if ("Recycling".equals(lockReason.getLockReason())) {
// Output the ID of the instance that is about to be reclaimed.
System.out.println("spot instance will be recycled immediately, instance id:" + instance.getInstanceId());
instanceIds.remove(instance.getInstanceId());
}
}
}
}
}
// If the spot instance is not yet locked, query again every 2 minutes.
System.out.print("try describeInstances again later ...");
Thread.sleep(2 * 60 * 1000);
} else {
break;
}
}
} catch (TeaUnretryableException ue) {
// This is for demonstration only. In a production environment, handle exceptions with caution and do not ignore them.
ue.printStackTrace();
// Print the error message.
System.out.println(ue.getMessage());
// Print the last request to find information about the failed request.
System.out.println(ue.getLastRequest());
} catch (TeaException e) {
// This is for demonstration only. In a production environment, handle exceptions with caution and do not ignore them.
e.printStackTrace();
// Print the error code.
System.out.println(e.getCode());
// Print the error message, which includes the RequestId.
System.out.println(e.getMessage());
// Print the detailed error information returned by the server.
System.out.println(e.getData());
} catch (Exception e) {
// This is for demonstration only. In a production environment, handle exceptions with caution and do not ignore them.
e.printStackTrace();
}
}
}
Response
The following output is returned when a reclamation is triggered:
result:instance:i-bp1i9c3qiv1qs6nc****,az:cn-hangzhou-i
instance:i-bp1i9c3qiv1qs6nc****-->lockReason:Recycling,vmStatus:Stopped
spot instance will be recycled immediately, instance id:i-bp1i9c3qiv1qs6nc****Instance metadata
From within an ECS instance, you can access the Metadata Service to get the termination time of a spot instance. For more information about instance metadata, see Instance metadata.
To obtain the termination time of a spot instance, use the following metadata item: instance/spot/termination-time
Response:
-
A 404 error indicates that the instance is not scheduled for reclamation and can continue to be used.
-
A UTC timestamp in the format
2015-01-05T18:02:00Zindicates the time at which the instance will be reclaimed.
Example calls:
Linux instance
# Obtain a token for accessing the metadata server for authentication.
TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" -H "X-aliyun-ecs-metadata-token-ttl-seconds:<token-validity-in-seconds>"`
# Query whether the spot instance is scheduled for interruption and reclamation.
curl -H "X-aliyun-ecs-metadata-token: $TOKEN" http://100.100.100.200/latest/meta-data/instance/spot/termination-time
Windows instance
# Obtain a token for accessing the metadata server for authentication.
$token = Invoke-RestMethod -Headers @{"X-aliyun-ecs-metadata-token-ttl-seconds" = "<token-validity-in-seconds>"} -Method PUT -Uri http://100.100.100.200/latest/api/token
# Query whether the spot instance is scheduled for interruption and reclamation.
Invoke-RestMethod -Headers @{"X-aliyun-ecs-metadata-token" = $token} -Method GET -Uri http://100.100.100.200/latest/meta-data/instance/spot/termination-time
CloudMonitor SDK
All events related to ECS instances are synchronized to CloudMonitor. You can directly call the CloudMonitor DescribeSystemEventAttribute API operation to query for the spot instance interruption event Instance:PreemptibleInstanceInterruption. Check if the value of the action field within the content field of the response is delete to determine if the spot instance is being reclaimed.
SDK call example
Prerequisites
-
Create an AccessKey
Because an Alibaba Cloud account has full permissions on all resources, its AccessKey poses a high security risk if it is leaked. We recommend that you use the AccessKey of a RAM user with the minimum required permissions. For more information, see Create an AccessKey.
-
Grant CloudMonitor-related permissions to the RAM user
Grant the RAM user permissions to manage CloudMonitor. The sample code in this topic requires permission to query system events. We recommend granting the following permission:
Cloud product
Permission
CloudMonitor
AliyunCloudMonitorFullAccess
-
Configure access credentials
The sample code in this topic reads the AccessKey from environment variables to use as access credentials for cloud services. For instructions, see Configure environment variables on Linux, macOS, and Windows systems.
-
Install the CloudMonitor SDK
Obtain the CloudMonitor SDK. This topic demonstrates how to install the SDK by adding a Maven dependency. For information about other installation methods, see Install the CloudMonitor SDK for Java.
Initialize the client
Alibaba Cloud SDKs support multiple types of access credentials, such as an AccessKey or an STS token, to initialize a client. For more information, see Manage access credentials.
import com.aliyun.cms20190101.Client;
import com.aliyun.teaopenapi.models.Config;
public class Sample {
private static Client createClient() throws Exception {
Config config = new Config()
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set in your runtime environment.
.setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set in your runtime environment.
.setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"))
// For endpoints, see https://api.aliyun.com/product/Ecs
.setEndpoint("metrics.cn-hangzhou.aliyuncs.com");
return new Client(config);
}
}
Build the API request
Configure the query parameters for the spot instance interruption event. For more information about the parameters, see DescribeSystemEventAttribute.
|
API |
Parameter |
Example value |
|
Product |
Product: ECS |
|
|
EventType |
Event type: StatusNotification |
|
|
Name |
Event name: Instance:PreemptibleInstanceInterruption |
// Build the request object.
DescribeSystemEventAttributeRequest request = new DescribeSystemEventAttributeRequest()
.setProduct("ECS")
.setEventType("StatusNotification")
.setName("Instance:PreemptibleInstanceInterruption");
Initiate the call
When calling an API operation, you can set runtime parameters, such as timeout and proxy configurations. For more information, see Advanced configurations.
// Set runtime parameters.
RuntimeOptions runtime = new RuntimeOptions();
// Call the DescribeSystemEventAttribute API operation.
DescribeSystemEventAttributeResponse response = client.describeSystemEventAttributeWithOptions(request, runtime);
System.out.println(response.body.toMap());
Handle exceptions
The SDK for Java classifies exceptions into two main types: TeaUnretryableException and TeaException.
-
TeaUnretryableException: This exception is typically caused by network issues and is thrown after the maximum number of retries is reached. -
TeaException: This exception is primarily for service-related errors.
To ensure system robustness and stability, implement proper exception handling, such as propagating exceptions, logging, and attempting recovery.
Complete example
import com.aliyun.cms20190101.Client;
import com.aliyun.teaopenapi.models.Config;
import com.aliyun.cms20190101.models.DescribeSystemEventAttributeRequest;
import com.aliyun.cms20190101.models.DescribeSystemEventAttributeResponse;
import com.aliyun.tea.TeaException;
import com.aliyun.tea.TeaUnretryableException;
import com.aliyun.teaopenapi.models.Config;
import com.aliyun.teautil.models.RuntimeOptions;
public class Sample {
private static Client createClient() throws Exception {
Config config = new Config()
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set in your runtime environment.
.setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
// Required. Make sure the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set in your runtime environment.
.setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"))
// For endpoints, see https://api.aliyun.com/product/Ecs
.setEndpoint("metrics.cn-hangzhou.aliyuncs.com");
return new Client(config);
}
public static void main(String[] args) {
try {
Client client = Sample.createClient();
// Build the request object.
DescribeSystemEventAttributeRequest request = new DescribeSystemEventAttributeRequest()
.setProduct("ECS")
.setEventType("StatusNotification")
.setName("Instance:PreemptibleInstanceInterruption");
// Set runtime parameters.
RuntimeOptions runtime = new RuntimeOptions();
// Call the DescribeSystemEventAttribute API operation.
DescribeSystemEventAttributeResponse response = client.describeSystemEventAttributeWithOptions(request, runtime);
System.out.println(response.body.toMap());
} catch (TeaUnretryableException ue) {
// This is for demonstration only. In a production environment, handle exceptions with caution and do not ignore them.
ue.printStackTrace();
// Print the error message.
System.out.println(ue.getMessage());
// Print the last request to find information about the failed request.
System.out.println(ue.getLastRequest());
} catch (TeaException e) {
// This is for demonstration only. In a production environment, handle exceptions with caution and do not ignore them.
e.printStackTrace();
// Print the error code.
System.out.println(e.getCode());
// Print the error message, which includes the RequestId.
System.out.println(e.getMessage());
// Print the detailed error information returned by the server.
System.out.println(e.getData());
} catch (Exception e) {
// This is for demonstration only. In a production environment, handle exceptions with caution and do not ignore them.
e.printStackTrace();
}
}
}
Response
Determine the spot instance interruption event based on the response.
The event notification is in the following JSON format:
{
"ver": "1.0",
"id": "2256A988-0B26-4E2B-820A-8A********E5",
"product": "ECS",
"resourceId": "acs:ecs:cn-hangzhou:169070********30:instance/i-bp1ecr********5go2go",
"level": "INFO",
"name": "Instance:PreemptibleInstanceInterruption",
"userId": "169070********30",
"eventTime": "20190409T121826.922+0800",
"regionId": "cn-hangzhou",
"content": {
"instanceId": "i-bp1ecr********5go2go",
"action": "delete"
}
}
The following table describes the fields in the content object. For more information about the parameters, see DescribeSystemEventAttribute.
|
Field |
Description |
Example |
|
instanceId |
The ID of the spot instance. |
i-bp1ecr********5go2go |
|
action |
The operation on the spot instance. The value |
delete |
CloudMonitor events
You can subscribe to CloudMonitor system events to monitor spot instance interruptions in real time and push notifications through different channels, such as SMS, email, Function Compute, Simple Message Queue (SMQ), or a webhook.
Workflow
Procedure
-
Prepare a receiving channel
Simple Message Queue (SMQ)
Log on to the SMQ console.
In the left-side navigation pane, choose Queue Model > Queues.
In the top navigation bar, select a region.
On the Queues page, click Create Queue.
In the Create Queue panel, configure the following parameters and click OK:
Name: the name of the queue.
Maximum Message Length: the maximum length of the message that is sent to the queue.
Long Polling Period: the maximum duration for which long polling requests are held after the ReceiveMessage operation is called.
Visibility Timeout Period: the duration for which a message stays in the Inactive state after the message is received from the queue.
Message Retention Period: the maximum duration for which a message exists in the queue. After the specified retention period, the message is deleted regardless of whether the message is received.
Message Delay Period: the period after which all messages sent to the queue are consumed.
Enable Logging Feature: specifies whether to enable the logging feature.
After the queue is created, it is displayed on the Queues page.
Webhook
ImportantThe webhook service must be deployed on a server with public network access. Ensure that the server's firewall allows access to the corresponding port.
The following is an example in Java:
import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RestController; @RestController public class WebhookController { @PostMapping("/callback") public ResponseEntity<String> receiveWebhook(@RequestBody String payload) { // Process the payload, for example, by logging it or handling business logic. System.out.println("Received webhook payload: " + payload); // Return a success response. return ResponseEntity.ok("Webhook received"); } } -
Create a subscription policy
-
Log on to the CloudMonitor console.
-
In the left-side navigation pane, choose .
-
On the Subscription Policy tab, click Create Subscription Policy.
-
On the Create Subscription Policy page, set the parameters for the policy.
This example describes only the parameters required for subscribing to spot instance interruption events. For more information about the parameters, see Manage event subscriptions (Recommended).
-
Basic Information: Enter a Name and Description for the subscription policy.
-
Alert Subscription:
-
Subscription type: Select System Events.
-
Subscription scope:
-
Service: Select ECS.
-
Event Type: Select Status Notification.
-
Event Name: Select Instance:PreemptibleInstanceInterruption.
-
Event Severity: Select Warning.
-
Application Groups, Event Content, and Event resources: Leave these parameters empty to subscribe to the Instance:PreemptibleInstanceInterruption system event for all ECS instances in all application groups under your account.
-
-
Combined Noise Reduction: Use the default settings.
-
Notification: For Custom Notification Method, use the default notification method.
-
Push and Integration: Click Add Channel. In the dialog box that appears, click Increase Channels. For Target type, select the channel prepared in Step 1, and fill in the other parameters as prompted. For more information about push channels, see Manage push channels.
-
-
-
Simulate an interruption event
Spot instance interruptions are passive events, which can make it difficult to effectively debug your interruption handler. You can use the Debug Event Subscription feature to simulate a spot instance interruption event.
-
On the Subscription Policy tab, click Debug Event Subscription.
-
In the Create Event Test panel, set Service to ECS and Name to Instance:PreemptibleInstanceInterruption.
The system automatically generates debugging content in JSON format. You must replace the placeholder resource information in the JSON file with the details of the spot instance you are using for the simulation.
-
Replace the
Alibaba Cloud account UIDvariable with the UID of your Alibaba Cloud account. -
Replace
<resource-id>and<instanceId>with the instance ID of your spot instance. -
Replace
<region-ID>with the region ID of your spot instance.{ "product": "ECS", "resourceId": "acs:ecs:cn-shanghai:Alibaba Cloud account UID:instance/<resource-id>", "level": "WARN", "instanceName": "instanceName", "regionId": "<region-ID>", "groupId": "0", "name": "Instance:PreemptibleInstanceInterruption", "content": { "instanceId": "i-2zeg014dfo4y892z***", "instanceName": "wor***b73", "action": "delete" }, "status": "Normal" }
-
-
Click OK.
The system displays an The operation is successful. message. CloudMonitor automatically sends a test alert notification using the method specified in the subscription policy.
-
-
Receive and view the notification
Simple Message Queue (SMQ)
-
Log on to the Simple Message Queue (SMQ) console and click Queues in the left-side navigation pane.
-
In the top menu bar, select a region. On the Queues page, find the target queue, and in the Action column, choose Send Messages.
-
(Optional) On the Quick Experience page, click Edit Parameters of Receiving Messages. In the Edit Parameters of Receiving Messages panel, configure Receive Times and Polling Period, then click OK.
-
On the Send and Receive Messages - Quick Experience page, click the Receive Message button. The message list for the queue is displayed in the Received Messages area.
-
(Optional) In the message list, find the target message and click Details in the Actions column. In the Message Details dialog box, view the message content and other information.
Webhook
In your deployed webhook application, check the invocation status and notification content.
-
Respond to interruption events
How you respond to an interruption depends on your specific business scenario and requirements. We strongly recommend that you test your application to ensure it can gracefully handle the interruption and reclamation of spot instances. The following are some recommendations and best practices:
-
Gracefully handle interruptions
Respond to interruption signals promptly to save task progress, clean up resources, and terminate task execution.
-
Implement data persistence and checkpoints
Periodically save task progress and intermediate results to persistent storage, such as a local file or a database, to preserve important business data and configurations. For information about data retention and recovery, see Retain and recover data for spot instances.
-
Test the integration
Trigger a test event in CloudMonitor to verify that your program responds correctly. For more information, see Handle spot instance interruption events by using Simple Message Queue.