ECS OpenAPI eventual consistency

Eventual consistency

When you use the OpenAPI to create, modify, or delete a resource, the changes may not be immediately reflected in subsequent read operations or other dependent operations. This is because data synchronization between system components takes time. The changes will eventually be reflected in the results of all related OpenAPI calls.

When you use the OpenAPI for resource orchestration or automation, you must account for eventual consistency in your process design. The term "resource" here covers all entities that ECS OpenAPI supports, such as instances, disks, Elastic Network Interfaces (ENIs), and security groups.

Typical scenarios

Data synchronization delays can cause unexpected results in the following scenarios:

1. Resource creation scenario (write-after-read)
   After a successful resource creation request, such as RunInstances, an immediate query (for example, DescribeInstances) or operation (for example, RunCommand or AttachDisk) on the new resource may fail. For a short time, the call may return a resource not found error, such as InvalidInstance.NotFound, or an empty list.
   

2. Resource release scenario (delete-after-read/delete-after-write)
   After a successful resource release request, such as DeleteInstances, an immediate query might still return information about the resource. An immediate operation on that resource ID, such as RunCommand, might succeed at the OpenAPI layer. However, the backend execution will typically fail because the resource has already been removed from the physical layer.
   

3. Resource property change scenario
   After a successful request to change a resource configuration, such as detaching a disk with DetachDisk, an immediate query of the instance details may still show the disk as attached for a short time.
   

Best practices

Use the following strategies to manage the effects of eventual consistency and ensure the stability of your automation flows:

• Strategy 1: Use an event-driven approach (Recommended)
  After you initiate a resource operation, avoid relying on polling. Instead, subscribe to resource status change events from Cloud Monitor or EventBridge. Execute subsequent steps only after you receive an event notification indicating that the resource creation is complete or the status change is complete. For more information, see Instance event notifications.
  

• Strategy 2: Use polling based on status queries
  After an OpenAPI operation, call a query API, such as DescribeInstances or DescribeDisks, to check the resource status. Use an exponential backoff policy to gradually increase the polling interval until the resource enters the expected state, such as Running or Stopped, or until a timeout occurs. To prevent an infinite loop, set a maximum wait time, such as 5 minutes.
  

• Strategy 3: Retry based on specific error codes
  If an operation returns a status conflict error code, treat it as a temporary fault. For example, you might receive an IncorrectInstanceStatus error if you try to roll back a disk immediately after stopping an instance. You can catch these specific error codes and retry the operation. When retrying, use an exponential backoff policy with a maximum retry window of 5 minutes.