Overview of elastic training on Kubernetes
Elastic training with Horovod
Horovod is an open source tool for distributed training that supports elastic training on Kubernetes. This document describes how to combine the distributed training feature of Horovod with the elastic scaling capabilities of Kubernetes to optimize the efficiency of model training. You can configure the distributed training policy of Horovod to distribute training jobs across multiple compute nodes. This improves training speed and resource utilization. This document also provides specific scenarios and best practices for using Horovod on Kubernetes to help you manage and schedule distributed training jobs more efficiently. For more information, see Elastic training with Horovod on Kubernetes.
Elastic training with spot instances
This document describes how to use Kubernetes spot instances for elastic training. Spot instances are compute resources that can be released on demand and are suitable for cost-sensitive training jobs that have a high tolerance for interruptions. You can configure an elastic training policy to automatically restart or reschedule training jobs when resources are preempted. This ensures the continuity and efficiency of the training procedure. This elastic training mechanism that uses spot instances can significantly reduce training costs and fully utilize idle resources. For more information, see Elastic training with spot instances.
Revocation awareness for ACS BestEffort pods on ACK
In Kubernetes, pods of the BestEffort class are typically used for jobs with low resource requirements. However, in environments with intense resource competition, these pods may be preempted or terminated with high priority. This document describes how to implement revocation awareness for BestEffort pods in ACK. This feature ensures that after a pod is preempted and its resources are released, the pod is automatically restarted or rescheduled. This mechanism improves system stability and resource utilization, and ensures that jobs are completed successfully. For more information, see Revocation awareness for ACS BestEffort pods on ACK.
Summary
Elastic training with Horovod: Combine Horovod and Kubernetes to achieve elastic scheduling and resource optimization for distributed training jobs.
Elastic training with spot instances: Use Kubernetes spot instances for elastic training to reduce costs and improve resource utilization.
Revocation awareness for BestEffort pods: Implement automatic recovery for BestEffort pods in ACK to ensure job continuity.
These features and practices help you more efficiently manage and optimize distributed training jobs in a Kubernetes environment. They also reduce operational costs and improve system stability and resource utilization.