Cache acceleration

Feature description

• Release status: The EFC cache feature is currently in the invitational preview stage.

• Billing: This feature is free to use during the invitational preview.

• Data reliability: The cache feature does not guarantee high data availability. Cached data may be lost in certain situations, such as machine corruption, node replacement, or node decommissioning. You should promptly back up your important training data. Do not use the cache as the sole data storage.

Scenarios

Inference scenarios

During model loading, model files must be loaded quickly onto each GPU server. Model loading is expected to complete within 1 minute. As the number of GPU servers increases, the performance requirements for the storage system become higher. The distributed cache on local GPU disks or in memory decouples storage system performance from model loading speed. This allows for rapid model loading.

Training scenarios

In training scenarios, data is read repeatedly. A client-side cache improves read performance. This increases GPU utilization and boosts the overall training speed.

Technical advantages

• High-speed client-side cache

  EFC uses the memory and local disks of compute nodes to create single-node and distributed caches. This accelerates read and write operations for datasets and checkpoints and effectively reduces the latency of remote storage access.

• P2P model distribution

  EFC uses peer-to-peer (P2P) technology to support high-concurrency loading and distribution of large-scale models. It uses the high-speed network between GPU nodes to accelerate parallel reads of hot spot data. The aggregate throughput scales linearly with the number of compute nodes and supports scales from hundreds to thousands of nodes. The following figure illustrates how P2P works.

  

• Full-path hardware acceleration

  EFC supports Remote Direct Memory Access (RDMA) networks and VSC hardware-software integrated paths. It also features multi-link load balancing to automatically distribute requests across multiple backend servers. This ensures low latency and high throughput during high concurrency.

• POSIX compatibility and cloud-native integration

  It is designed for containers and is fully compatible with POSIX file interfaces. It provides container-level fencing, Quality of Service (QoS) capabilities, and application-oriented access control.

Limits

How to enable and use

• Platform for AI (PAI) (Lingjun resources)

  The EFC cache uses the local disks of Lingjun resources on PAI to create a distributed read cache. It accelerates data reads by reducing the reliance on remote storage, which improves application performance. For more information about how to enable and use EFC on the PAI platform, see Use EFC cache to accelerate tasks on the PAI platform.

• Lingjun bare metal

  For Lingjun bare metal scenarios, an activation process through the commercial console is not yet available. To request a trial, submit a ticket for consultation and activation.