ECS OpenAPI eventual consistency

Eventual consistency

When you use the OpenAPI to create, modify, or delete a resource, the changes may not be immediately reflected in subsequent read operations or other dependent operations. This is because data synchronization between system components takes time. The changes will eventually be reflected in the results of all related OpenAPI calls.

When the resource count changes, a resource count query can take time to become eventually consistent because data synchronization is delayed. Use NextToken-based pagination when you query resources to avoid result fluctuations caused by resource changes during the query.

When you use the OpenAPI for resource orchestration or automation, you must account for eventual consistency in your process design. The term "resource" here covers all entities that ECS OpenAPI supports, such as instances, disks, Elastic Network Interfaces (ENIs), and security groups.

Typical scenarios

Data synchronization delays can cause unexpected results in the following scenarios:

1. Resource creation scenario (write-after-read)
   After a successful resource creation request, such as RunInstances, an immediate query (for example, DescribeInstances) or operation (for example, RunCommand or AttachDisk) on the new resource may fail. For a short time, the call may return a resource not found error, such as InvalidInstance.NotFound, or an empty list.
   

2. Resource release scenario (delete-after-read/delete-after-write)
   After a successful resource release request, such as DeleteInstances, an immediate query might still return information about the resource. An immediate operation on that resource ID, such as RunCommand, might succeed at the OpenAPI layer. However, the backend execution will typically fail because the resource has already been removed from the physical layer.
   

3. Resource property change scenario
   After a successful request to change a resource configuration, such as detaching a disk with DetachDisk, an immediate query of the instance details may still show the disk as attached for a short time.
   

4. Resource count query scenario
   After a resource is created, released, or has its properties changed, the total returned by a resource query filtered by attributes or a query of all resources can differ from the actual resource count.
   

Best practices

Use the following strategies to manage the effects of eventual consistency and ensure the stability of your automation flows:

• Strategy 1: Use an event-driven approach (Recommended)
  After you initiate a resource operation, avoid relying on polling. Instead, subscribe to resource status change events from Cloud Monitor or EventBridge. Execute subsequent steps only after you receive an event notification indicating that the resource creation is complete or the status change is complete. For more information, see Instance event notifications.
  

• Strategy 2: Use polling based on status queries
  After an OpenAPI operation, call a query API, such as DescribeInstances or DescribeDisks, to check the resource status. Use an exponential backoff policy to gradually increase the polling interval until the resource enters the expected state, such as Running or Stopped, or until a timeout occurs. To prevent an infinite loop, set a maximum wait time, such as 5 minutes.
  

• Strategy 3: Retry based on specific error codes
  If an operation returns a status conflict error code, treat it as a temporary fault. For example, you might receive an IncorrectInstanceStatus error if you try to roll back a disk immediately after stopping an instance. You can catch these specific error codes and retry the operation. When retrying, use an exponential backoff policy with a maximum retry window of 5 minutes.
  

• Strategy 4: Use NextToken-based pagination and avoid strong dependencies on resource totals
  For resource query APIs, use NextToken-based pagination. Pagination returns results in sequence. If a resource change during the query does not affect resources that have not yet appeared on subsequent pages, the change does not affect those subsequent results, providing a more stable result set. Treat the value returned by an API that counts all resources as a reference only. When the resource total changes over a short period, do not use the value for business logic or reconciliation that requires strong consistency.