High disk usage and read-only indexes

Symptoms

• Write requests fail with: FORBIDDEN/12/index read-only / allow delete (api)

• Cluster health status is red. Run GET /_cat/nodes?v to check whether nodes have rejoined the cluster. Run GET /_cat/allocation?v to check shard allocation.

  Note

  Red cluster health means primary shards are unavailable. Data may be at risk.

• Kibana returns internal server error when creating ingest pipelines or enrolling Beats

• Cluster or Kibana monitoring shows disk usage approaching 100%

Root cause

Disk usage thresholds

Elasticsearch monitors disk usage continuously and enforces three watermark thresholds:

• 85% — low watermark: Elasticsearch stops allocating new shards to this node.

• 90% — high watermark: Elasticsearch relocates existing shards to nodes with more free disk space.

• 95% — flood stage: Elasticsearch sets the read_only_allow_delete attribute on all indexes, blocking all write operations.

In addition to disk watermark thresholds, the following conditions may also cause write failures or alerts. Investigate these causes together:

Shard count limit reached

By default, each data node in an Elasticsearch cluster supports up to 1,000 shards. You can verify the current limit by running GET /_cluster/settings?include_defaults=true&flat_settings=true and checking the cluster.max_shards_per_node parameter. When the shard count exceeds the limit, Logstash cannot write logs and new indexes cannot be created.

As a temporary workaround, run the following command to increase the shard limit:

PUT /_cluster/settings
{
  "transient": {
    "cluster": {
      "max_shards_per_node": 2000
    }
  }
}

For a long-term solution, clean up expired or unused indexes, or add more data nodes to the cluster.

System index lifecycle policies

If disk usage fluctuates frequently but no custom Index Lifecycle Management (ILM) policies are configured, the fluctuation may be caused by the default lifecycle policies of system indexes, such as .monitoring-es-* and .monitoring-kibana-*. These indexes are automatically created daily, and old indexes are automatically deleted. This is normal behavior.

Resource correlation impact

High disk usage may be accompanied by high JVM memory usage (for example, reaching 90%) and node anomalies, which may prevent Kibana from connecting. In this case, scale up both disk capacity and memory resources. If the cluster is unhealthy, enable forced changes during the configuration upgrade. Perform the operation during off-peak business hours to avoid service disruption caused by node restarts.

Quick fix (10–15 minutes)

1. Delete old or unused indexes to free disk space.

   Warning

   Deleted data cannot be restored. To preserve data, consider increasing storage capacity instead.

   curl -u <username>:<password> -XDELETE http://<host>:<port>/<index-name>

   • <host> is your cluster's internal or public endpoint. Configure the access allowlist before running this command.

   • If the cluster is unresponsive, trigger a forced restart and run this command during the restart.

2. Remove the read-only lock. Freeing disk space does not automatically lift the write block. Clear it by setting index.blocks.read_only_allow_delete to null:

   PUT /_all/_settings
   {
      "index.blocks.read_only_allow_delete": null
   }

3. Verify cluster health. If the status is still red, run GET /_cat/allocation?v to check for unassigned shards.

4. If unassigned shards remain, run GET /_cluster/allocation/explain to identify the cause. If the output shows exhausted allocation retries (as shown below), run POST /_cluster/reroute?retry_failed=true.

   "deciders": [
     {
       "decider": "max_retry",
       "decision": "NO",
       "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-02-14T00:22:25.939Z], failed_attempts[5], delayed=false, details[failed shard on node [xxx]: shard failure, reason [lucene commit failed], failure IOException[No space left on device], allocation_status[deciders_no]]]]"
     }
   ]

   Errors can persist after you scale up the disk or clean up data. Once disk usage has reached 100%, shard allocation fails and the failed-retry counter is exhausted. Even after you expand the disk or delete data, some shards can remain unassigned, and queries still fail with errors such as 503 search_phase_execution_exception.

   In this case, manually run POST /_cluster/reroute?retry_failed=true to retry the failed shard allocations. After the command returns, wait for shard recovery to finish. The cluster returns to normal service only after recovery completes.

5. If the cluster health status remains red after all recovery steps, contact Alibaba Cloud technical support.

FAQ

Why does the Elasticsearch console show normal cluster status but I still receive disk usage alerts?

A normal cluster status on the console only indicates that the cluster is currently available. However, disk usage may have already reached the alert threshold (for example, 91%). Although high disk usage has not immediately triggered read_only mode, it is approaching the critical watermark — at 95%, forced read-only mode is triggered, and write operations can be blocked at any time.

We recommend that you immediately check the actual disk usage on the Cluster Monitoring page, and take one of the following actions to eliminate the risk:

• Scale up the disk capacity.

• Clean up unused or expired indexes.

How do I temporarily adjust the disk watermark to restore cluster health?

Temporarily raise the low watermark only as an emergency measure: the cluster status is unhealthy because of disk watermark limits, shards are unassigned, and you cannot scale up immediately. In Kibana Dev Tools, call the cluster settings API to raise the low watermark, for example, to 88%, so that shards blocked by the watermark can be allocated again.

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "88%"
  },
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "88%"
  }
}

88% only relaxes the low watermark for the current cluster on a temporary basis. It does not change the 85%, 90%, and 95% default watermarks described in Root cause.

This is a temporary workaround. After the cluster status returns to green, upgrade the configuration through an in-place rolling change in the console or clean up data as soon as possible. Do not rely on the raised watermark long term.

How long does it take for disk space to be released after I delete an index?

After you delete an index as described in the first step of Quick fix (10–15 minutes), disk space usually starts to be released within a few minutes to half an hour, and the change is reflected in the monitoring metrics shortly afterwards.

If disk space is still not released after a long time, check whether other data is consuming the disk, such as remaining indexes, snapshots, or logs, or whether the refresh of the monitoring metrics is delayed.

Prevention

Enable disk usage monitoring and configure alerts to fire when usage exceeds 80%. Route alerts to your operations team so they can act before the flood stage is reached. For setup instructions, see Configure monitoring and alerting.