Orchestrate genetic computation workflows with Argo Workflows

Background

Genetic computation workflows

A genetic computation workflow is a process that organizes interconnected computing tasks and data processing steps in a defined order to achieve a specific analysis goal in genomics research. These workflows typically include complex steps such as data preprocessing, sequence alignment, variant detection, gene expression analysis, and phylogenetic tree construction.

Argo Workflows for genetic computation workflows

Argo Workflows is an open-source, Kubernetes-native workflow engine designed for containerized environments that orchestrates complex workflows with flexibility and efficiency. The advantages of Argo Workflows are particularly significant in genetic computation scenarios.

• Containerization and environmental consistency: Genetic analysis involves numerous software tools and dependency libraries. By encapsulating each analysis step into a Docker container, Argo Workflows ensures cross-platform consistency and reproducibility, reducing reproducibility issues across different environments.

• Flexible orchestration capabilities: Workflows in genomics research often involve multiple steps, conditional branches, and parallel processing requirements. Argo Workflows supports complex logic and conditional controls, making it simple and intuitive to customize workflows as needed.

Although open-source Argo Workflows offers significant advantages in orchestrating genetic computation workflows, users face several practical challenges.

• Large-scale operational challenges: Faced with a massive number of tasks, users with a research background may lack deep experience in cluster operations and maintenance. This makes it difficult to implement effective cluster optimization and maintenance strategies.

• Complex workflow orchestration challenges: Scientific experiments often involve vast parameter spaces and numerous process steps, often involving thousands of jobs. Open-source workflow engines struggle to support this scale.

• Resource optimization and elastic scaling difficulties: Genetic data analysis is typically computationally intensive. Users want to intelligently schedule resources based on workload to achieve high resource utilization and support automatic scaling of computing power on demand. Open-source solutions often fall short in meeting these needs.

To address challenges in genetic computation, such as large-scale operations, complex workflow orchestration, and resource optimization with elastic scaling, the Alibaba Cloud ACK One team has introduced the distributed workflow Argo cluster.

Distributed workflow Argo cluster

The Alibaba Cloud distributed workflow Argo cluster is based on the open-source Argo Workflows project. It adopts a serverless model and runs workflows on Elastic Container Instance (ECI). By optimizing Kubernetes cluster parameters, the solution achieves efficient elastic scheduling for large-scale workflows and optimizes costs by using spot instances of ECI. This solution supports various execution strategies such as concurrency, loops, and retries, making it suitable for typical genetic computation processes and highly complex workflow orchestration tasks.

Orchestrate a genetic computation workflow

This example uses a classic Burrows-Wheeler Aligner (BWA) sequence alignment pipeline to demonstrate how to edit and run a genetic computation workflow using Argo Workflows.

1. Create a distributed workflow Argo cluster.

2. Enable Argo Server to access the workflow cluster.

3. Mount an Alibaba Cloud OSS storage volume, allowing the workflow to operate on files in OSS as if they were local files. For more information, see Use storage volumes.

4. Use the following YAML to create a workflow. For more information, see Create a workflow.

   The process consists of three main stages:

   • bwaprepare: The data preparation stage. Downloads and decompresses the fastq and reference files, and builds an index for the reference genome.

   • bwamap: The sequence alignment stage. Aligns the sequencing data with the reference genome and generates alignment results. This stage processes multiple files in parallel.

   • bwaindex: The indexing stage. Generates, sorts, and indexes the BAM file produced by aligning raw sequencing data with the reference genome, and provides a view of the alignment results.

   Sample YAML

   apiVersion: argoproj.io/v1alpha1
   kind: Workflow
   metadata:
     generateName: bwa-oss-
   spec:
     entrypoint: bwa-oss
     arguments: 
       parameters:
       - name: fastqFolder     # The path where the downloaded files are saved.
         value: /gene
       - name: reference       # The reference genome file.
         value: https://ags-public.oss-cn-beijing.aliyuncs.com/alignment/subset_assembly.fa.gz
       - name: fastq1          # The raw sequencing data.
         value: https://ags-public.oss-cn-beijing.aliyuncs.com/alignment/SRR1976948_1.fastq.gz
       - name: fastq2
         value: https://ags-public.oss-cn-beijing.aliyuncs.com/alignment/SRR1976948_2.fastq.gz
     volumes:                  # Mounts the remote storage path.
     - name: ossdir
       persistentVolumeClaim:
         claimName: pvc-oss
     templates:
     - name: bwaprepare      # Data preparation stage: downloads and decompresses the fastq and reference files, and builds an index for the reference genome.
       container:
         image: registry.cn-beijing.aliyuncs.com/geno/alltools:v0.2
         imagePullPolicy: Always
         command: [sh,-c]
         args:
         - mkdir -p /bwa{{workflow.parameters.fastqFolder}}; cd /bwa{{workflow.parameters.fastqFolder}}; rm -rf SRR1976948*;
           wget {{workflow.parameters.reference}};
           wget {{workflow.parameters.fastq1}};
           wget {{workflow.parameters.fastq2}};
           gzip -d subset_assembly.fa.gz;
           gunzip -c SRR1976948_1.fastq.gz | head -800000 > SRR1976948.1;
           gunzip -c SRR1976948_2.fastq.gz | head -800000 > SRR1976948.2;
           bwa index subset_assembly.fa;
         volumeMounts:
         - name: ossdir
           mountPath: /bwa
       retryStrategy:         # The retry mechanism.
         limit: 3
     - name: bwamap           # Sequence alignment stage: aligns the sequencing data with the reference genome and generates alignment results.
       inputs:
         parameters:
         - name: object
       container:
         image: registry.cn-beijing.aliyuncs.com/geno/alltools:v0.2
         imagePullPolicy: Always
         command:
         - sh
         - -c
         args:
         - cd /bwa{{workflow.parameters.fastqFolder}};
           bwa aln subset_assembly.fa {{inputs.parameters.object}} > {{inputs.parameters.object}}.untrimmed.sai;
         volumeMounts:
         - name: ossdir
           mountPath: /bwa
       retryStrategy:
         limit: 3
     - name: bwaindex       # Indexing stage: generates, sorts, and indexes the BAM file produced by aligning raw sequencing data with the reference genome, and provides a view of the alignment results.
       container:
         args:
         - cd /bwa{{workflow.parameters.fastqFolder}};
           bwa sampe subset_assembly.fa SRR1976948.1.untrimmed.sai SRR1976948.2.untrimmed.sai SRR1976948.1 SRR1976948.2 > SRR1976948.untrimmed.sam;
           samtools import subset_assembly.fa SRR1976948.untrimmed.sam SRR1976948.untrimmed.sam.bam;
           samtools sort SRR1976948.untrimmed.sam.bam -o SRR1976948.untrimmed.sam.bam.sorted.bam;
           samtools index SRR1976948.untrimmed.sam.bam.sorted.bam;
           samtools tview SRR1976948.untrimmed.sam.bam.sorted.bam subset_assembly.fa -p k99_13588:1000 -d T;
         command:
         - sh
         - -c
         image: registry.cn-beijing.aliyuncs.com/geno/alltools:v0.2
         imagePullPolicy: Always
         volumeMounts:
         - mountPath: /bwa/
           name: ossdir
       retryStrategy:
         limit: 3
     - name: bwa-oss           # Orchestrates the workflow stages.
       dag:
         tasks:
         - name: bwaprepare          # Runs the data preparation task first.
           template: bwaprepare
         - name: bwamap         # Performs the initial alignment step.
           template: bwamap
           dependencies: [bwaprepare]        # Depends on the data preparation stage.
           arguments:
             parameters:
             - name: object
               value: "{{item}}"
           withItems: ["SRR1976948.1","SRR1976948.2"]      # Processes multiple files in parallel.
         - name: bwaindex               # Generates the final alignment file and indexes it.
           template: bwaindex
           dependencies: [bwamap]           # Depends on the initial alignment stage.

5. After the workflow runs, you can view the task's Directed Acyclic Graph (DAG) and the run results in the Workflow Console (Argo).

   

   You can also log on to the OSS console to find the generated alignment result files in the corresponding OSS folder.

   The process generates 17 files in total. The generated files include the input FASTQ files (SRR1976948_1.fastq.gz and SRR1976948_2.fastq.gz), intermediate BWA alignment files (.sai and .sam), the final BAM result files (.bam, .sorted.bam, and .bai), and the reference genome subset_assembly.fa with its index files (.amb, .ann, .bwt, .fai, .pac, and .sa).

Related documents

• For more information about the distributed workflow Argo cluster, see Overview of the distributed workflow Argo cluster.

• To learn more about Argo Workflows, see the open-source Argo Workflows documentation.