Migrate BatchCompute jobs to a distributed Argo workflow cluster

Batch computing

Batch computing has five core components that form a linear dependency chain: a compute environment hosts the infrastructure, a job queue routes work across compute environments, a job definition specifies how a job runs, and a job (or array job) is the actual unit of work submitted and executed.

Argo Workflows

Argo Workflows has three core building blocks. A template defines the smallest unit of work. A workflow orchestrates one or more templates. A workflow template makes reusable workflow definitions available across multiple workflows. All tasks ultimately run in pods on an ACK serverless cluster.

Simple workflow

The following workflow creates a pod that uses the alpine image and runs the Shell command echo helloworld.

Modify this workflow to run custom Shell commands or execute commands in a custom image.

cat > helloworld.yaml << EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world-    # name of the workflow spec
spec:
  entrypoint: main         # invoke the main template
  templates:
    - name: main              # name of the template
      container:
        image: registry.cn-hangzhou.aliyuncs.com/acs/alpine:3.18-update
        command: [ "sh", "-c" ]
        args: [ "echo helloworld" ]
EOF
argo submit helloworld.yaml

Loops

Use loops to process large amounts of data in parallel — each iteration runs as an independent pod.

The following example packages a text file named pets.input and a script named print-pet.sh in a custom image. The loop creates five pods, passing job-index values 1 through 5 as input parameters. Each pod prints the pet listed on the corresponding row in pets.input. For more information about the source files, see the GitHub repository.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-
spec:
  entrypoint: loop-example
  templates:
  - name: loop-example
    steps:
    - - name: print-pet
        template: print-pet
        arguments:
          parameters:
          - name: job-index
            value: "{{item}}"
        withSequence:  # loop to run print-pet template with parameter job-index 1 ~ 5 respectively.
          start: "1"
          end: "5"
  - name: print-pet
    inputs:
      parameters:
      - name: job-index
    container:
      image: acr-multiple-clusters-registry.cn-hangzhou.cr.aliyuncs.com/ack-multiple-clusters/print-pet
      command: [/tmp/print-pet.sh]
      args: ["{{inputs.parameters.job-index}}"] # input parameter job-index as args of container

For more information, see Argo Workflows - Loops.

DAGs (MapReduce)

In batch computing systems, you must specify job dependencies using job IDs — but a job ID is only available after the job is submitted. The typical workaround is a script like this:

// Job B depends on Job A. Job B starts only after Job A completes.
batch submit JobA | get job-id
batch submit JobB --dependency job-id (JobA)

As the number of jobs grows, these dependency scripts become hard to maintain.

Argo Workflows lets you declare dependencies by task name directly in the workflow definition:

• Task B and Task C depend on Task A.

• Task D depends on Task B and Task C.

# The following workflow executes a diamond workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-diamond-
spec:
  entrypoint: diamond
  templates:
  - name: diamond
    dag:
      tasks:
      - name: A
        template: echo
        arguments:
          parameters: [{name: message, value: A}]
      - name: B
        depends: "A"
        template: echo
        arguments:
          parameters: [{name: message, value: B}]
      - name: C
        depends: "A"
        template: echo
        arguments:
          parameters: [{name: message, value: C}]
      - name: D
        depends: "B && C"
        template: echo
        arguments:
          parameters: [{name: message, value: D}]
  - name: echo
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:3.7
      command: [echo, "{{inputs.parameters.message}}"]

For a sample MapReduce workflow that creates shards and aggregates results, see map-reduce.

Step 1: Assess and plan

Before converting any jobs, audit your existing batch workloads across these dimensions:

• Job inventory: List all jobs, including their dependencies, resource requests (CPU, memory, GPU), and runtime parameters.

• Prioritization: Identify business-critical jobs and high-complexity jobs separately. Start your migration with a simple, non-critical job to build confidence before tackling complex workflows.

• Feature gaps: Map each batch computing feature to its Argo Workflows equivalent using the feature mappings table above. Skip compute environment design and job priority configuration — Kubernetes clusters for distributed Argo workflows use serverless elastic container instances and require no queue management.

Step 2: Create a Kubernetes cluster for distributed Argo workflows

See Create a workflow cluster.

Step 3: Convert job definitions

Convert batch jobs to Argo workflows based on the feature mappings. Use the Argo Workflows SDK to automate workflow creation and system integration where needed.

Step 4: Prepare storage

Make sure the cluster can access the data that your workflows need. Mount OSS buckets, NAS file systems, CPFS file systems, or disks to the cluster. For instructions, see Use volumes.

Step 5: Verify workflows

Run both systems in parallel before decommissioning the batch system:

1. Keep existing batch jobs inactive but available for comparison.

2. Trigger the same workload on both systems for the same input data.

3. Write outputs to separate paths to avoid conflicts.

4. Compare results — data accuracy, output structure, and resource usage — to confirm parity.

5. After confirming correctness, decommission the batch jobs.

Keep the parallel-run period short. Extended dual maintenance increases operational overhead and the risk of divergence.

Step 6: Enable monitoring and logging

Enable observability for the cluster to check workflow status and logs. Use preemptible instances to reduce compute costs further.