Storage resources

Storage resources are the hardware and software resources used to store and manage data, such as local disks on nodes, distributed file storage, databases, and caches. They provide durable, reliable, and efficient data storage that meets a system's access needs. The performance of storage resources directly affects the processing power of compute resources, and their reliability directly affects data accuracy. The following sections describe five storage resource risks and their corresponding fault tolerance policies.

Full local disk

A full local disk occurs when a node runs out of disk space and can no longer store or process data. This common problem is often caused by improper log configurations or insufficient disk capacity, and can lead to data loss or system crashes. Common fault tolerance policies include:

• Usage alerts: Monitor disk space usage on compute nodes and configure alerts. When disk space is nearly full, the system sends an alert to operations and maintenance (O&M) engineers for prompt action.

• Regular cleanup and archiving: Clean up and archive data that is no longer needed or is redundant. Free up disk space by regularly removing temporary files and deleting expired logs and backup files.

• Data compression: For data that requires long-term storage, use compression and storage optimization techniques to reduce the storage footprint.

• Asynchronous log writing: If a disk is full and an application writes service logs synchronously, threads can become blocked and the service may become unresponsive. Switch to asynchronous log writing to prevent a full disk from affecting online services.

High disk I/O load

High disk I/O load occurs when disk input/output (I/O) operations are too frequent or too heavy, which degrades disk performance and slows system response. Symptoms include slower read and write speeds, increased task running times, and reduced real-time capabilities. Excessive I/O load can cause disk failure, system crashes, or data loss. Common fault tolerance policies include:

• Data caching: Temporarily store data in memory to reduce frequent disk I/O access. Use a memory cache system or the operating system's file cache to improve data access speed and I/O efficiency.

• Distributed storage: Distribute data across the storage devices of multiple compute nodes by using a distributed storage system or data partitioning. This spreads the I/O load, balances access pressure, and leverages the I/O resources of multiple nodes to improve concurrency.

• Data compression: Compress and optimize data for storage to reduce the disk I/O load. Compressing data files and using more efficient storage formats reduces both the data volume and the number of read/write operations.

Cache penetration

Cache penetration occurs when many requests query for data that does not exist, causing every request to miss the cache and hit the backend data source. This increases the backend database load, slows response times, and can degrade overall system performance. Common fault tolerance policies include:

• Bloom filters: Use a Bloom filter to pre-screen and filter out requests for data that is known not to exist. A Bloom filter is a space-efficient data structure that quickly determines whether a requested key might exist, reducing pressure on subsequent queries.

• Cache null objects: For requests that return an empty result, store a null object in the cache with a short time-to-live (TTL). Subsequent identical requests hit the cached null object instead of querying the data source again, reducing the backend load.

• Rate limiting: Apply rate limiting to restrict sources that send frequent requests and prevent cache penetration from escalating.

• Asynchronous loading: For cache misses, load the data asynchronously and store it in the cache so that subsequent requests result in a cache hit.

• Prefetching: Use a cache prefetching mechanism to load hot data into the cache in advance when the system starts, reducing the likelihood of cache penetration.

Full database connection pool

A full database connection pool occurs when an application's connection pool reaches its configured maximum and cannot create new connections. This can cause response delays, blocked requests, and unresponsive services. Common fault tolerance policies include:

• Quota management: Configure the maximum connections for each application based on system load and demand. This prevents a single application with abnormal database access from exhausting the connection pool and affecting other services.

• Timeout and recycling: Set a connection timeout in the connection pool. Connections that remain idle beyond the specified period are automatically recycled and released, preventing long-held connections and improving connection availability.

• Connection reuse: Reuse connections whenever possible. After a request completes, release the connection back to the pool for other requests to use.

• Rate limiting: Upstream service invocations can increase the frequency of database access, pushing connections beyond the service's capacity. Apply rate limiting to the current service to ensure its availability.

Abnormal database instance

An abnormal database instance occurs when the database system encounters an issue that causes it to stop working or providing normal service. Possible causes include an abnormal end of the database service process, a node breakdown, or a network failure. Common fault tolerance policies include:

• Automatic restart: When a database instance terminates abnormally or crashes, set up an automatic restart and recovery mechanism. This restarts the database service and performs the necessary data restoration to ensure availability and data integrity.

• High Availability (HA) switchover: Ensure database redundancy and failover by using primary/standby, cluster, or distributed setups. When the primary instance becomes abnormal, the system automatically switches to a standby instance to maintain continuous availability.

• Data backup and restoration: Regularly back up data and store backups in a reliable location. If a database instance becomes abnormal, use the backup data for restoration to prevent data loss.

In addition to the five risks described above, other common storage resource risks include inaccessible mounted volumes, cloud disk read/write I/O hangs, non-existent dependency files, cache avalanches, configuration push errors, insufficient database I/O, insufficient database storage space, and lost database replicas. Address these risks with fault tolerance policies such as automatic detection and recovery, dependency degradation, failover, tiered caching, configuration validation and rollback, read/write splitting, sharding, and multiple replicas.