Accelerate data access for Job applications

Upload test data to OSS

1. Create a 2 GB test file such as this sample dataset.

2. Upload the file to your OSS bucket with ossutil.

Create the Dataset and JindoRuntime resources

This example uses JindoFS as the cache backend (JindoRuntime). Fluid represents data sources as two Custom Resources (CRs):

• Dataset — points to data in the external storage system.

• JindoRuntime — defines caching system configuration.

Fluid uses lazy loading: first access fetches from OSS to local cache; subsequent reads come from cache. To eliminate first-run latency, pre-warm the cache before submitting your Job.

1. Create a Secret to store the OSS credentials:

   kubectl create secret generic oss-access-key \
     --from-literal=fs.oss.accessKeyId=<access_key_id> \
     --from-literal=fs.oss.accessKeySecret=<access_key_secret>

2. Create a file named dataset.yaml:

   Parameter	Description
   mountPoint	OSS mount path in oss://<bucket_name>/<bucket_path> format. Set path to / for a single mount point.
   fs.oss.endpoint	OSS bucket endpoint. Use an internal endpoint (e.g., oss-cn-hangzhou-internal.aliyuncs.com) for same-region access; use a public endpoint (e.g., oss-cn-hangzhou.aliyuncs.com) otherwise.
   replicas	Number of cache worker pods. Determines total cache capacity.
   alibabacloud.com/burst-resource: eci_only	Disables virtual node scheduling on cache worker pods. Required — Fluid conflicts with virtual node scheduling (see Limitations).
   k8s.aliyun.com/eci-use-specs	ECI instance spec for each cache worker pod.
   k8s.aliyun.com/eci-image-cache	Enables instance image cache to speed up pod startup.
   tieredstore.levels.mediumtype	Cache medium. Valid values: MEM (memory), SSD, HDD. See Select a cache medium.
   tieredstore.levels.volumeType	Volume type. Use emptyDir for memory or system disk (prevents residual cache from affecting node availability). Use hostPath for data disks and set path to the mount point. Default: hostPath.
   tieredstore.levels.path	Cache path. Only one path supported.
   tieredstore.levels.quota	Maximum cache capacity per worker, for example 10Gi.
   tieredstore.levels.high / low	High and low watermarks for cache eviction.

   apiVersion: data.fluid.io/v1alpha1
   kind: Dataset
   metadata:
     name: demo-dataset
   spec:
     mounts:
       - mountPoint: oss://<bucket_name>/<bucket_path>
         name: demo
         path: /
         options:
           fs.oss.endpoint: oss-<region>.aliyuncs.com
         encryptOptions:
           - name: fs.oss.accessKeyId
             valueFrom:
               secretKeyRef:
                 name: oss-access-key
                 key: fs.oss.accessKeyId
           - name: fs.oss.accessKeySecret
             valueFrom:
               secretKeyRef:
                 name: oss-access-key
                 key: fs.oss.accessKeySecret
   ---
   apiVersion: data.fluid.io/v1alpha1
   kind: JindoRuntime
   metadata:
     name: demo-dataset
   spec:
     # Number of cache worker nodes
     replicas: 2
     worker:
       podMetadata:
         annotations:
           # Required: disable virtual node scheduling (conflicts with Fluid — see Limitations)
           alibabacloud.com/burst-resource: eci_only
           # ECI instance spec for the JindoFS cache worker pod
           k8s.aliyun.com/eci-use-specs: <eci_instance_spec>
           # Enable instance image cache to speed up pod startup
           k8s.aliyun.com/eci-image-cache: "true"
     tieredstore:
       levels:
         # 10 GiB of memory cache per worker node
         - mediumtype: MEM
           volumeType: emptyDir
           path: /dev/shm
           quota: 10Gi
           high: "0.99"
           low: "0.99"

   Key parameters:

3. Apply the manifest:

   kubectl create -f dataset.yaml

4. Wait one to two minutes for the caching system to deploy, then verify the Dataset status:

   If PHASE shows NotBound, the caching system is still initializing. Wait one to two minutes and recheck.

   kubectl get dataset demo-dataset

   Expected output:

   NAME           UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
   demo-dataset   1.16GiB          0.00B    20.00GiB         0.0%                Bound   2m58s

   PHASE: Bound confirms successful deployment. Other columns show OSS data size, cached amount, and total cache capacity.

Pre-warm the cache (optional)

Fluid uses lazy loading, so the first Job run fetches from OSS and can take tens of seconds for large datasets. Pre-warm the cache to eliminate first-run latency.

1. Create a file named dataload.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: DataLoad
   metadata:
     name: data-warmup
   spec:
     dataset:
       name: demo-dataset
       namespace: default
     loadMetadata: true

2. Start the pre-warm job:

   kubectl create -f dataload.yaml

   Wait until status shows Complete:

   NAME          DATASET        PHASE      AGE   DURATION
   data-warmup   demo-dataset   Complete   99s   58s

   Cache warm-up completed in about 58s.

Create a Job application

Use application containers or machine learning jobs to access accelerated data through JindoFS. This example creates a Job that reads OSS data.demo-dataset PVC read from JindoFS cache automatically — no code changes needed. The alibabacloud.com/fluid-sidecar-target: eci label triggers Fluid Webhook to inject the caching sidecar.

1. Create a file named job.yaml:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: demo-app
   spec:
     template:
       metadata:
         labels:
           alibabacloud.com/fluid-sidecar-target: eci
         annotations:
           # Required: disable virtual node scheduling (conflicts with Fluid — see Limitations)
           alibabacloud.com/burst-resource: eci_only
           # ECI instance spec for the application pod
           k8s.aliyun.com/eci-use-specs: ecs.g7.4xlarge
       spec:
         containers:
           - name: demo
             image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
             command:
               - /bin/bash
             args:
               - -c
               - du -sh /data && time cp -r /data/ /tmp
             volumeMounts:
               - mountPath: /data
                 name: demo
         restartPolicy: Never
         volumes:
           - name: demo
             persistentVolumeClaim:
               claimName: demo-dataset
     backoffLimit: 4

2. Submit the Job:

   kubectl create -f job.yaml

3. Check the Job logs after it completes:

   kubectl logs demo-app-jwktf -c demo

   Expected output:

   1.2G    /data
   
   real    0m0.992s
   user    0m0.004s
   sys     0m0.674s

   The real time for the file copy is 0m0.992s.

Clean up

Clean up resources to avoid unnecessary charges.

1. Delete the Job:

   kubectl delete job demo-app

2. Delete the Dataset. This also removes the associated caching system components:

   Important

   Cleanup takes about one minute. Wait until all caching pods are deleted before proceeding.

   kubectl delete dataset demo-dataset

3. Scale down the Fluid control plane:

   kubectl get deployments.apps -n fluid-system | awk 'NR>1 {print $1}' | xargs kubectl scale deployments -n fluid-system --replicas=0

   To re-enable data access, scale the control plane back up before creating new Dataset and JindoRuntime resources:

   kubectl scale -n fluid-system deployment dataset-controller --replicas=1
   kubectl scale -n fluid-system deployment fluid-webhook --replicas=1