Configure a shared NAS volume

Before you begin

Make sure you have:

• A running ACK cluster

• Permissions to create NAS file systems and manage ACK storage resources

• An Elastic Compute Service (ECS) instance in the same Virtual Private Cloud (VPC) as your ACK cluster, for mounting and populating the NAS file system

Usage notes

Note the following constraints:

• VPC co-location required. The NAS file system, its mount target, and your ACK cluster must all reside in the same VPC.

• Protocol type affects multi-writer performance. NFSv3 performs better when multiple ECS instances do not write to the same file concurrently. Extreme NAS supports NFSv3 only.

• PV capacity is used for matching only. The capacity values do not cap actual storage. Actual limits depend on the NAS file system specifications.

• Static PVs do not support `archiveOnDelete`. Setting Delete on a static PV does not actually delete PV or NAS data. Use Retain (default) unless using dynamically provisioned NAS volumes.

Background information

Configuring a shared storage volume for Arena job environments preserves data scientists' work (code and data) so that it is not lost when containers are deleted, and makes shared training data easy to access. In team development, allocating a shared storage pool lets data and code be shared across the team.

When submitting an Arena job, you can use the --data parameter to declare shared storage and a mount path. The shared storage is mounted at the specified directory, and subsequent jobs can reuse this data or code.

In Kubernetes, storage objects are described by Persistent Volumes (PVs) and Persistent Volume Claims (PVCs). As a cluster administrator, when assigning environments, create a separate PVC for each data scientist. For example, PVCs for user A and user B can mount the same NAS or CPFS backend, but must point to different subdirectories so their working environments remain isolated.

Step 1: Create a NAS instance

Create a General-purpose NAS file system in the same region and VPC as your ACK cluster.

See Create a General-purpose NAS file system using the console.

Use the following values:

Step 2: Mount the file system to an ECS instance

Mount the NAS file system to an ECS instance to later populate it with training data. This step uses the one-click mount feature. See Scenarios for mounting a file system for other methods.

mkdir /mnt/dir1
mkdir /mnt/dir2
touch /mnt/file1
echo 'some file content' > /mnt/file2
ls /mnt

Step 3: Create a PV and a PVC

Register the NAS file system as a PV in your ACK cluster, then create a PVC for pod access.

kubectl get pvc pvc-nas

NAME       STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-nas    Bound    pv-nas   5Gi        RWX                           1m

Step 4: Populate the PVC with data

A Kubernetes cluster accesses shared data through PVCs (which point to the NAS instance created in Step 1), so you only need to populate the NAS backing the PVC. The following procedure creates two Kubernetes Jobs that copy the PyTorch and TensorFlow MNIST datasets from example container images into the NAS at the specified directories. The example uses the PVC pvc-nas from Step 3.

1. Create a file named prepare-pytorch-mnist-data.yaml with the following content:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: prepare-pytorch-mnist-data
     namespace: default
   spec:
     completions: 1
     parallelism: 1
     backoffLimit: 3
     ttlSecondsAfterFinished: 3600
     template:
       spec:
         containers:
         - name: pytorch-mnist-example
           image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/pytorch-mnist-example:2.5.1-cuda12.4-cudnn9-runtime
           imagePullPolicy: Always
           command:
           - /bin/bash
           - -c
           args:
           - |
             set -eux
             mkdir -p /mnt/pytorch_data/MNIST/raw
             cp -r /data/MNIST/raw/* /mnt/pytorch_data/MNIST/raw || { echo "Copy failed"; exit 1; }
             echo "MNIST data prepared successfully at /mnt/pytorch_data/MNIST/raw."
           resources:
             requests:
               cpu: 100m
               memory: 128Mi
             limits:
               cpu: 100m
               memory: 128Mi
           volumeMounts:
           - name: training-data
             mountPath: /mnt
         volumes:
         - name: training-data
           persistentVolumeClaim:
             claimName: pvc-nas
         restartPolicy: Never

2. Create the Job to copy the PyTorch MNIST dataset to /pytorch_data on NAS:

   # Create the Job
   kubectl create -f prepare-pytorch-mnist-data.yaml
   
   # Wait for the Job to complete
   kubectl wait --for=condition=complete --namespace=default --timeout=300s job/prepare-pytorch-mnist-data

   When the Job finishes successfully, the output is:

   job.batch/prepare-pytorch-mnist-data condition met

3. Create a file named prepare-tensorflow-mnist-data.yaml with the following content:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: prepare-tensorflow-mnist-data
     namespace: default
   spec:
     completions: 1
     parallelism: 1
     backoffLimit: 3
     ttlSecondsAfterFinished: 3600
     template:
       spec:
         containers:
         - name: tensorflow-mnist-example
           image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/tensorflow-mnist-example:2.15.0-gpu
           imagePullPolicy: Always
           command:
           - /bin/bash
           - -c
           args:
           - |
             set -eux
             mkdir -p /mnt/tf_data
             cp -r /data/mnist.npz /mnt/tf_data/mnist.npz || { echo "Copy failed"; exit 1; }
             echo "MNIST data prepared successfully at /mnt/tf_data/mnist.npz."
           resources:
             requests:
               cpu: 100m
               memory: 128Mi
             limits:
               cpu: 100m
               memory: 128Mi
           volumeMounts:
           - name: training-data
             mountPath: /mnt
         volumes:
         - name: training-data
           persistentVolumeClaim:
             claimName: pvc-nas
         restartPolicy: Never

4. Create the Job to copy the TensorFlow MNIST dataset to /tf_data on NAS:

   # Create the Job
   kubectl create -f prepare-tensorflow-mnist-data.yaml
   
   # Wait for the Job to complete
   kubectl wait --for=condition=complete --namespace=default --timeout=300s job/prepare-tensorflow-mnist-data

   When the Job finishes successfully, the output is:

   job.batch/prepare-tensorflow-mnist-data condition met

This example uses the PVC named pvc-nas in the default namespace. If your PVC has a different name or is in a different namespace, adjust the manifests accordingly.

/tf_data contains the TensorFlow MNIST data and /pytorch_data contains the PyTorch MNIST data. Both are accessible to any pod that mounts pvc-nas.

Next steps

• Submit an Arena training job with --data to mount pvc-nas.

• To isolate user environments, create separate PVCs pointing to different subdirectories of the same NAS file system.

• For production workloads, consider enabling CNFS.