System events are defined by Alibaba Cloud to record and send notifications about your cloud resources, such as the execution status of O&M tasks, resource exceptions, and resource status changes.
This topic describes only the system events supported by Elastic Compute Service (ECS). For more information about the system events of other Alibaba Cloud products, see the documentation for those products.
Scenarios
-
Notify you of risks and exceptions
Alibaba Cloud pushes system events to the ECS console. These events notify you of situations that can affect resource availability and performance, such as system maintenance restarts and instance expirations. For some critical events, Alibaba Cloud also sends text message, email, and internal message notifications. You can respond to these events in the ECS console or using OpenAPI. Respond to system events promptly to avoid business interruptions. For more information, see Query and respond to ECS system events.
For example, when a subscription instance is about to expire and be stopped, the ECS console displays a prominent prompt for you to renew the instance. This helps you avoid business interruptions.
-
Implement automated O&M
System events that are displayed in the ECS console have defined states. These states help you understand the execution status of the corresponding system O&M tasks. Status changes are synchronized with CloudMonitor, which lets you build an automated O&M mechanism. For more information about event states, see States and windows of system events.
Note-
Each event state corresponds to a CloudMonitor event name. For example, the ECS event code `InstanceFailure.Reboot` supports the `Executing` and `Executed` states. Its corresponding CloudMonitor event names are `Instance:InstanceFailure.Reboot:Executing` and `Instance:InstanceFailure.Reboot:Executed`.
Status change events include some system events that are not displayed in the ECS console, such as changes to the running status of an instance and interruptions of spot instances. You cannot respond to these system events directly in the ECS console or using OpenAPI. ECS does not define event states for these system events. However, when these events occur, they are still reported to CloudMonitor. This lets you build an event-driven, automated O&M system as needed.
For example, when you manually start or stop an instance, a system event is generated for the instance status change. This does not indicate a risk or an exception. However, you may want to record this type of operation in your own log system. To do this, you can set up event notifications for instance status change events. Then, you can use features such as callbacks to write the start and stop information to your operation logs.
-
System event types
System events are classified into the following types based on their triggers:
For more information about the types of system events that Elastic Compute Service (ECS) supports and the recommended actions for them, see Summary of ECS system events.
|
Category |
Description |
Displayed in the ECS console |
|
Scheduled O&M events |
Alibaba Cloud actively upgrades host software to improve security. It also predicts and avoids hardware and software failure risks on underlying hosts. If a corresponding O&M task might affect the availability or performance of your ECS resources, Alibaba Cloud informs you in advance about the execution time, affected objects, and impact. You can then respond to these system events during off-peak hours within the scheduled execution time to avoid affecting your business during peak hours. Note
Scheduled O&M events are also known as proactive O&M events. They are based on Alibaba Cloud's experience in managing millions of servers, its capability to serve tens of thousands of large enterprise customers, and the advanced machine learning algorithms from Alibaba DAMO Academy. These events predict and proactively avoid hardware and software failure risks on underlying hosts. When a failure risk on a host cannot be avoided, Alibaba Cloud notifies affected ECS users in advance through a scheduled O&M event. This provides a buffer period for you to switch your services. If you do not respond to scheduled O&M events in advance, your ECS instance may go down or restart when a failure occurs. |
Yes Note
When a scheduled O&M event occurs for an instance of a big data instance family or an instance family with local SSDs (excluding i4p), the event is displayed under Local Disk Instance Events. For more information about local disk events, see O&M scenarios and system events for instances with local disks. |
|
Unexpected O&M events |
When an underlying host has a sudden hardware or software failure, or an instance experiences an issue such as an out-of-memory (OOM) error or kernel panic, the instance may restart or go down unexpectedly. Alibaba Cloud promptly sends an unexpected O&M event, restores the availability of the ECS resource as soon as possible, and notifies you of the execution status of the corresponding system O&M task. Note
Unexpected O&M events generally refer to sudden downtime or restarts of an ECS instance. This can be caused by unpredictable failures of the underlying host or issues such as kernel errors in the ECS instance's operating system.
|
Yes Note
When an unexpected O&M event occurs for an instance of a big data instance family or an instance family with local SSDs (excluding i4p), the event is displayed under Local Disk Instance Events. For more information about local disk events, see O&M scenarios and system events for instances with local disks. |
|
Instance billing events |
Events that affect billing. For example, an instance is about to be stopped or released because the instance expires or your account has an overdue payment. |
Yes |
|
Local disk instance events |
Local disk instance events include events for local disks (such as local disk corruption) and events for instances with local disks (such as an instance that may not run properly due to local disk damage, or hardware/software failures on the underlying host of an instance with local disks). Note
Local Disk Instance Events is not a specific event type. It is only used to display scheduled or unexpected O&M events that occur for instances of big data instance families or instance families with local SSDs (excluding i4p). This makes it easier for you to handle local disk instance events. For a description of local disk events, see O&M scenarios and system events for instances with local disks. |
Yes |
|
Performance limited events for burstable instances |
Performance limited events for burstable instances are reminder events. They indicate that a burstable instance has used up its CPU credits. The instance's CPU will then run continuously near its baseline performance. This may affect the instance's application performance, management, and O&M, and can cause issues such as slow access and stuttering. |
Yes |
|
Instance security events |
Events that affect instance security. For example, an instance is under a DDoS attack or enters a blackhole, which threatens the instance's security. |
Yes |
|
Instance migration events due to underlying layer upgrades |
When Alibaba Cloud upgrades and transforms its physical infrastructure, instances in the corresponding regions and zones may be affected. You can migrate your instances based on the guidance in the system event. |
Yes |
|
Status change events |
These events occur when you perform an operation on an instance that changes its lifecycle, such as manually starting or stopping it. They also occur when Alibaba Cloud detects a change in an instance's properties, which in turn changes its lifecycle or other statuses. These include the following:
|
|
System event severities
System events are classified into the following severity levels based on their impact on instance operations:
-
Critical: Indicates a major impact that requires immediate action. If you do not take action, the instance may become unusable. Examples include the release of a resource due to an overdue payment or the redeployment of an instance due to an error.
-
Warning: Indicates a moderate impact that requires your attention during the impact period or requires you to take action at a suitable time. For example, a burstable instance with limited performance can continue to run but cannot burst above its baseline performance.
-
Information: Indicates an event that you can choose whether to monitor. For example, an event is generated when a disk snapshot is created.
States and windows of system events
The states for system events that are displayed in the ECS console are defined in the following table.
For more information about the event states supported by different system events, see the CloudMonitor event name column in Summary of ECS system events.
|
State |
Property |
Description |
|
Inquiring |
Intermediate state |
Inquiry in progress, awaiting your authorization. After authorization, the state changes to Executing. |
|
Scheduled |
Intermediate state |
The O&M task is scheduled but has not started. After it starts, the state changes to Executing. |
|
Executing |
Intermediate state |
The O&M task is in progress. |
|
Executed |
Stable state |
The O&M task is complete. |
|
Avoided |
Stable state |
You migrated the instance within the user operation window, avoiding the impact of the system event. |
|
Failed |
Stable state |
The O&M task failed. |
|
Canceled |
Stable state |
The system canceled the O&M task. |
The following figure shows the typical transitions between event states.
System events include the following windows:
-
User operation window
This is the period from when a system event is sent to its scheduled execution time. During the user operation window, you can perform the required operations yourself or wait for the system to automatically perform the operations. The duration of the window varies based on the event:
-
For scheduled O&M events, the window is typically 24 to 48 hours.
NoteFor system events in the Inquiring state, there is no time limit. The O&M task starts only after you grant authorization.
-
Unexpected O&M events caused by unplanned factors, such as sudden failures or non-compliant operations, usually do not have a user operation window.
-
For a system event that indicates a subscription instance will be stopped due to expiration, the window is 3 days.
-
For a system event that indicates a pay-as-you-go instance will be stopped due to an overdue payment, the window is less than 1 hour.
-
-
Event execution window
This is the period from when a task starts to when it is completed. The duration of the window varies based on the event:
-
For system events that notify you of a failure repair, the task is usually completed within 10 minutes.
-
Unexpected O&M events caused by unplanned factors, such as sudden failures or non-compliant operations, have only a short event execution window.
-
Operation guide
|
Operation |
Description and references |
|
Understand system events |
Before you begin, read this topic to understand system event names, severity levels, scenarios, limits, states, and naming formats. |
|
View system events |
You can view system events using the console or the Cloud Assistant CLI:
|
|
Respond to system events |
For some critical system events (such as events that affect the availability and performance of ECS resources), respond promptly in the console or using OpenAPI. Follow the recommended actions to avoid affecting your business operations.
|
|
Monitor system events |
To ensure the stability of services running on your ECS instances and to implement automated O&M, set up event notifications to monitor changes in the underlying environment. After you set up event notifications, the system pushes messages using your specified notification methods.
|
|
Modify system event settings |
Modify system event settings as needed:
|