Traditional systems for batch and flow orchestration often struggle with complex scenarios, such as batch data processing, machine learning pipelines, infrastructure automation, and CI/CD. These systems cannot handle increasing complexity or support automated scaling. To simplify batch task orchestration, Alibaba Cloud provides a component that is compatible with the cloud-native workflow engine Argo Workflows.
Introduction to Argo Workflows
Argo Workflows is a powerful, cloud-native workflow engine designed to define, manage, and schedule complex workflows in Kubernetes. A workflow consists of multiple tasks with dependencies, and this flexibility simplifies task configuration.
Scenarios
Argo Workflows supports scenarios such as batch data processing, machine learning pipelines, infrastructure automation, and CI/CD. It is widely used in industries such as autonomous driving, scientific computing, quantitative finance, and digital media.
Batch data processing: Common scenarios include large-scale high-precision map processing, quantitative finance backtesting and simulation, parallel audio and video processing, and animation rendering.
Scientific computing: Common scenarios include complex scientific computing simulation, drug discovery and training, gene sequencing, mutation comparison and detection, and energy exploration.
Simulation: Common scenarios include autonomous driving algorithm simulation, molecular dynamics simulation, astronomical data simulation, and financial modeling.
Machine learning pipeline: Common scenarios include machine learning data pre-processing, distributed training, Large Language Model (LLM) parameter tuning, and model evaluation and deployment.
Infrastructure automation: Common scenarios include automated cloud resource management, resource backup and recovery, node pool migration, and cluster migration and upgrades.
CI/CD: Common scenarios include parallel CI pipelines, multi-stage build and testing, cross-cloud application deployment, and approval flow integration.
Advantages of Argo Workflows
Cloud-native: Designed specifically for Kubernetes. Each task is a pod, which takes full advantage of the lightweight and flexible nature of containers.
Lightweight and scalable: Argo Workflows is lightweight and has no extra overhead or limitations compared to traditional virtual machines (VMs). Using the scheduling capabilities of Kubernetes, you can run thousands of tasks in parallel to improve processing efficiency.
Flexible orchestration: The flexible combination of Directed Acyclic Graphs (DAGs) and steps supports custom workflow logic of any complexity. Powerful retry and caching mechanisms improve the success rate of workflow execution.
Rich ecosystem: Supports the orchestration of various task types, such as Spark, Ray, and TensorFlow Job. Combined with event-driven features, you can build a fully automated task processing platform.
Use Argo Workflows
ACK Argo Workflows is compatible with the community version of Argo Workflows and includes enhancements over the open source version. You can seamlessly migrate existing Argo workflows without modification. ACK Argo Workflows provides the following advantages over the open source version:
High elasticity, automatic scaling, and optimized computing costs.
High reliability with multi-zone load balancing and dependable scheduling.
Enhanced control plane with significant improvements in scale, performance, efficiency, stability, and observability.
Enhanced Object Storage Service (OSS) storage management that supports large file uploads, artifact garbage collection (GC), and streaming.
Support from container service technical experts to help your business team optimize workflows, improve performance, and reduce costs.
ACK Argo Workflows provides two usage options to meet different user needs:
Serverless Argo Workflows: If you want a fully managed service to focus on business flow orchestration and require large-scale, high-performance workloads, you can build a separate workflow cluster. For more information, see Serverless Argo Workflows.
Argo Workflows component on ACK: If you already have an ACK cluster and want to use its resources, you can use the Argo Workflows component to orchestrate your business workflows. This topic describes how to use the Argo Workflows component on an ACK cluster.
After you install the Argo Workflows component, you must enable batch task orchestration. You can then submit and manage workflows using the Alibaba Cloud Argo command-line interface (CLI) or the Argo console.
The general process for different roles is as follows.
Process | Description |
1. Preparations |
|
2. Set up the environment |
For more information, see Enable batch task orchestration. |
3. Manage workflows | (Data engineer) After orchestrating parallel tasks, you can submit and manage them using the Argo CLI or the Argo console.
|
(Cluster administrator)
|
Billing
The batch task orchestration feature is free of charge. However, in addition to standard ACK billing, the Argo Server automatically creates a pay-as-you-go Classic Load Balancer (CLB) instance when you use this feature. You are charged by CLB for this instance. For more information, see CLB billing overview.
Contact us
If you have any product suggestions or questions, you can contact us by joining the DingTalk group (ID: 35688562).