ACS Pod custom events-Container Compute Service-阿里云帮助中心

This document describes pod events in Alibaba Cloud Container Service (ACS), which include custom events (pod-related, resource, and system maintenance) and scheduling and management events for GPU pods, including those with high-performance networking.

View custom events

Event Center

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Operations > Event Center.
Click the Events tab and filter events by criteria such as Level, Namespace, and Name. Event details can be seen in the list.
In the upper-right corner, click View Historical Data in Event Center to view historical events from Log Service (SLS).
This feature requires the alibaba-log-controller and kube-eventer add-ons. You can install them from the Add-ons page.
In the Simple Log Service (SLS) console, select the k8s-event Logstore, enter * and eventId.type: Warning in the search bar, and run the search to filter out the event records whose level is Warning.
- The following is an example of an OOMKilled event:
```
{
  "eventId": {
    "metadata": {},
    "involvedObject": {
      "kind": "Pod",
      "namespace": "default",
      "name": "helloworld-go-**********-*****",
      "uid": "*****",
      "apiVersion": "v1"
    },
    "reason": "OOMKilled",
    "message": "invoked",
    "source": {
      "component": "AcsService"
    },
    "firstTimestamp": "2025-**-**T02:33:49Z",
    "lastTimestamp": "2025-**-**T02:33:49Z",
    "count": 1,
    "type": "Warning",
    "eventTime": null,
    "reportingComponent": "",
    "reportingInstance": ""
  },
  "hostname": "",
  "level": "Warning",
  "pod_id": "*****",
  "pod_name": "helloworld-go-**********-*****",
  "clusterName": "*****",
  "clusterId": "*****7ec556cb4680a3a1e71201a*****"
}
```
- The eventId object contains the following key fields:
  - involvedObject: The resource object associated with the event. It includes the Kubernetes API version (apiVersion), resource type (kind), resource name (name), the resource's namespace (namespace), and the resource ID (uid).
  - message: The message of the event. For example, Out of memory.
  - reason: The reason for the event. For example, OOMKilled.
  - type: The type of the event. The value can be Normal or Warning.

Workloads

Log on to the ACS console. In the left navigation pane, click Clusters.
Take a Deployment as an example. On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Workloads > Deployments.
Click the name of your workload, and then click the Events tab to view the list of Pod events.
- An example of an OOMKilled event:
  In the left-side navigation pane, click Deployments, find the target application (for example, helloworld-go), and open its details page. Click the Events tab, and then select the Pod sub-tab. The event list shows the OOMKilled event record whose type is Warning.

General custom events

Applies to ACS pods of all compute types.

ACS pod-related events

Event name (reason)	Event type (type)	Event message (message)	Description
BestEffortDegraded	Warning	BestEffort instance will be degraded because the specified instance is out of stock	A BestEffort ACS pod is automatically downgraded to the default type because the requested BestEffort compute resources are out of stock.
BestEffortToBeReleased	Warning	BestEffort instance will be released	The BestEffort ACS pod is about to be released. Ensure your application can handle the termination.
StockExhaust	Warning	The stock of the specified zones will be used up	The stock of ACS compute resources in the current zone is nearly exhausted. We recommend switching to a different zone.
NoStock	Warning	Create ACS instance failed because the specified instance is out of stock	There is insufficient stock of ACS compute resources in the current zone. To increase the creation success rate, create the ACS pod across multiple zones and with multiple instance types.
StockClose	Warning	Create ACS instance failed because current zone closed or limited without living instance	Failed to create the ACS pod because the current zone is unavailable. Switch to a different zone.
FailedScheduling	Warning	Unknown error occurred	Failed to create the ACS pod due to an unknown error. Submit a ticket.
FailedScheduling	Warning	Schedule ACS instance failed	Failed to schedule the ACS pod. Retry the operation.
UnknownError	Warning	The ACS service is under heavy load while creating container, please wait and try again later	The ACS service is under heavy load. Wait and try to create the ACS pod again later.
UnknownError	Warning	An unknown error occurred	An unknown error occurred. Submit a ticket.
DiskCapacityQuotaFull	Warning	Your disk capacity quota is exceeded	The disk capacity has reached the quota limit. Go to Quota Center to request a quota increase.
AttachDiskFailed	Warning	Attach disk %s failed	Failed to attach the disk.
SystemFailureReboot	Warning	The specified instance is rebooting	The ACS pod is restarting.
StartingFailed	Warning	the instance starting failed: %s	The ACS pod failed to start.
Throttling	Warning	The request was denied due to system flow control, please wait and try again later	The request was denied due to system flow control. Wait and try the operation again later.
EphemeralStorageSizeExceededJobLimit	Warning	Ephemeral storage size: %s exceeded the limitation of job instance:%s	The ephemeral storage size exceeded the limit.
InplaceResizing	Normal	Starting to resize container resource for container: %s cpu request: %v -> %v, cpu limit %v -> %v, memory request: %v -> %v, memory limit %v -> %v, time: %v	In-place resizing has started for a container.
InplaceResizing	Warning	Failed to do the in-place update, err message: %v	In-place resizing failed.
InplaceResizedFinished	Normal	Pod vertical scaling cpu resource finished, consume: %v	The in-place resizing task is complete.
InplaceResizedFinished	Normal	scale task canceled because pod resource already updated %v	The in-place resizing task was canceled because the pod resources already meet the requirements.
InplaceResizedTimeoutFailed	Warning	Pod vertical scaling cpu resource failed, reason: timeout, time: %v	The in-place resizing task timed out.

Other resource events

Resource module	Event name (reason)	Event type (type)	Event message (message)	Description
vSwitch	ResourceInsufficient	Warning	The maximum number of IP addresses in the vSwitch %s is exceeded	The vSwitch has insufficient available IP addresses. Use a different vSwitch.
security group	QuotaFull	Warning	The maximum number of instances in the security group %s is exceeded	The security group has reached its instance quota. Use a different security group.
ENI	CreateENIFailed	Warning	%s	Failed to create the elastic network interface (ENI).
	AttachENIFailed	Warning	%s	Failed to attach the ENI.
	AttachENIConflict	Warning	%s	An ENI attachment conflict occurred.
	MissDefaultVpc	Warning	%s	The default Virtual Private Cloud (VPC) parameter is missing.
EIP	CreateEipFailed	Warning	%s	Failed to create the Elastic IP address (EIP).
	EipBandwidthPackageQuotaExceeded	Warning	%s	The EIP bandwidth plan quota was exceeded.
	EipNotFound	Warning	The specified EIP %s does not exist	The specified EIP does not exist.
	EipBoundOtherInstance	Warning	The specified EIP %s is already bound to another instance	The specified EIP is already bound to another instance.
SLS	AliyunSlsQuotaExceed	Warning	%s	The resource quota for Simple Log Service (SLS) has been exceeded. Submit a ticket to request a quota increase.
	AliyunSlsError	Warning	%s	An SLS-related error occurred.
	AliyunSlsProjectInvalid	Warning	%s	The specified SLS project is invalid.
	AliyunSlsConfigFormatError	Warning	sls config %s value %s is invalid	An SLS configuration parameter is invalid.

System O&M events

Event name (reason)	Example message	Description
SandboxNotReady	Critical error detected on Host OS	A proactive O&M restart was triggered.
SandboxNotReadyUnExpected	Unexpected error occurred on underlying infrastructure	An unexpected outage triggered a restart.
SandboxRebootCanceled	The reboot of current instance has been canceled	The restart was canceled.
SandboxRebootSucceeded	Current instance has been rebooted successfully.	The restart was successful.
ScheduledMaintenance	{"originReason":"Redeploy","originMessage":"Redeploy","planExecTime":"20**-03-12T09:00:00.000+08:00"}	A scheduled system O&M event. Important The `message` is in JSON format. The `planExecTime` field, which uses the RFC 3339 time format, indicates the scheduled execution time for the O&M event. To avoid potential impact on your business, you can recreate the pod before the scheduled time.
OOMKilled	Memory cgroup out of memory: Kill process 15848 (xsim_traffic_fl) score 337 or sacrifice child	An out-of-memory (OOM) error occurred inside the ACS pod.
PodOOMKilled	System OOM encountered, victim process: xxx, pid: xxx	An out-of-memory (OOM) error occurred within the ACS pod. This is not a kernel crash. The OOM can be triggered by ACS system components or container processes.
DiskFull	There is insufficient disk space for the current instance.	The disk space is full.
NfsError	NFS: state manager: bind conn to session failed on NFSv4 server 172.16.0.1 with error 121	An NFS error occurred.
RuntimeCrashed	Unexpected crash occurred on underlying infrastructure	The container runtime crashed unexpectedly.
RuntimeCrashed	Unexpected crash occurred on underlying infrastructure	The container runtime crashed unexpectedly.
RuntimeIssueDetected	Unexpected issue occurred on underlying infrastructure	An unexpected issue was detected in the underlying infrastructure.

GPU instance custom events

Supports custom events for ACS pods of the GPU and high-performance network GPU compute types.

Scheduling and management events

Event name (reason)	Type	Message or example	Description
Scheduled	Normal	Successfully assigned [namespace]/[Pod name]	The Pod is successfully scheduled and bound to a node.
GPUComputeClassScheduling	Normal	Waiting for ACS resource for scheduling	The GPU Pod is waiting for ACS to allocate resources. Delays can result from an insufficient GPU resource quota or a lack of available resources.
FailedScheduling	Warning	The Pod scheduling failed, [%s]	Pod scheduling failed. Check the resource quota, persistent volume configuration, or node affinity rules for conflicts.
PVCUnbound	Warning	The Pod scheduling failed, persistentvolumeclaim "[PVC name]" is being deleted/not found	Pod scheduling failed because the required persistent volume claim (PVC) was deleted or could not be found. Check the PVC status.
InsufficientResources	Warning	The Pod scheduling failed, [CPU/Memory/GPU] resources are insufficient on all nodes	All nodes have insufficient CPU, memory, or GPU resources. Scale out the nodes or reduce the resource requests.
AffinityConflict	Warning	The Pod scheduling failed, [X] Node(s) didn't match pod affinity/anti-affinity rules	No available nodes match the Pod's affinity or anti-affinity rules. Check the affinity configuration.
TopologySpreadConstraint	Warning	The Pod scheduling failed, GPU resources are unavailable or insufficient	GPU resources are unavailable or insufficient. Check the GPU resource quota.
GPUSharePreempted	Warning	GPU is preempted by <new-pod-name>	<new-pod-name> preempted the GPU resources of the current Pod. For more information, see What information is available when a GPU sharing Pod is preempted?
GPUSharePreempt	Warning	GPU is preempted from <old-pod-name>	The current Pod preempted GPU resources from <old-pod-name>. For more information, see What information is available when a GPU sharing Pod is preempted?
CPFSAttachFailed	Warning	CPFS filesystem ID is invalid or does not exist	The CPFS file system ID is invalid or does not exist.
CPFSAttachFailed	Warning	Waiting for CPFS mountpoints to become ready	The CPFS VSC failed to mount due to a timeout.
InvalidPovPV	Warning	Missing filesystem ID in PV configuration	The PV is missing the CPFS file system ID in its configuration.
DiskAttached	Normal	Attached disk d-xxxxxxxxx	The block storage attached successfully.
DiskAttached	Warning	error attach disk d-xxxxxxxxx: server returned error: The specified disk does not exist. (InvalidDiskId.NotFound)	Block storage attachment failed. The message details the reason for the failure.
ResourceDiskAttachFailed	Warning	Failed to attach resource disk	The resource disk, used for features like image caching, failed to attach. If this issue persists, submit a ticket.
PullImageFailed	Warning	Failed to pull image [image1] for [error message]	The image pull failed. The message includes the image name and the failure reason.
FailedCreate (PVC)	Warning	failed to create pvc: [pvc-name]	PVC creation failed. The message includes the name of the affected PVC.
FailedCreate (Pod)	Warning	failed to create pod: [pod-name]	Pod creation failed. The message includes the name of the affected Pod.
NodeBroken	Warning	The pod is proposed to be evicted at 20xx-xx-xx xx:xx:xx +0000 UTC, reason: xxx	A GPU-HPN node has failed. Evict the Pods from the faulty node immediately. Once all Pods are evicted, ACS automatically initiates the node repair and self-healing process.
GPUCardBroken	Warning	The pod is proposed to be evicted at 20xx-xx-xx xx:xx:xx +0000 UTC, reason: xxx	A GPU card has failed. Evict the Pods from the faulty node immediately. Once all Pods are evicted, ACS automatically initiates the node repair and self-healing process.
BestEffortToBeReleased	Warning	Best Effort Instance will be released	This BestEffort instance is about to expire. Verify that your services can handle its release.