Agentic RL training on ACK with VeRL

Agentic RL

Agentic RL uses reinforcement learning to train autonomous agents to complete complex tasks through multi-turn interactions. Compared with traditional reinforcement learning from human feedback (RLHF), agentic RL involves longer interaction trajectories, more complex action spaces, and sparser reward signals.

Architecture

This solution involves the following core components:

• VeRL: An open-source LLM reinforcement learning training framework that supports RL training for scenarios such as multi-turn dialogue and tool use.

• SWE-bench: A benchmark for evaluating the ability of AI systems to solve real-world GitHub issues. The AI system must read the issue description, generate a code fix (patch), and verify the fix by using the project's test suite.

• Harbor-framework: A framework that provides a sandbox execution environment and agent orchestration capabilities, and supports the creation of isolated execution environments in Kubernetes.

The built-in AgentLoops in VeRL, such as SingleTurnAgentLoop and ToolAgentLoop, do not support distributed execution, token-level data capture, third-party agent integration, or sandbox isolation. This solution uses RemoteAgentLoop + harbor-framework to address these limitations:

• Isolated execution: The agent runs in a sandbox, which prevents contamination of the training environment and allows the execution of untrusted code.

• Token-level tracing: The ProxyServer captures the token_ids and logprobs of all LLM calls, providing the complete data required for RL training.

• Massive parallelism: Run thousands of trials simultaneously in K8s clusters to accelerate training convergence.

The system architecture is shown in the following figure and consists of two parts: a VeRL Ray cluster (including RemoteAgentLoop, the vLLM Rollout inference service, and ProxyServerActor) and an Agent execution environment (the Harbor-framework sandbox).

The system supports two running modes. This topic covers only the local mode, where REMOTE_AGENT_USE_LOCAL_TRIAL is set to true. In this mode, RemoteAgentLoop directly creates a sandbox pod in the Kubernetes cluster and runs the agent, without requiring an additional Harbor Server. This mode is suitable for single-cluster training scenarios.

The ProxyServer forwards LLM requests from the agent to the vLLM inference service. The ProxyServer also transparently records token-level data, such as token_ids and logprobs, for RL training.

Training pipeline

The end-to-end agentic RL training consists of five stages:

1. Data preparation: Load the SWE-bench Verified dataset, which contains 500 issues, and split it into a training set of 400 issues and a validation set of 100 issues.

2. Rollout sampling: RemoteAgentLoop submits tasks to the harbor-framework sandbox. The SWE-agent reads issues, generates patches, and runs tests in an isolated environment. The ProxyServer synchronously records the completion_token_ids and completion_logprobs for all LLM calls.

3. Reward calculation: The system calculates a reward signal based on the task execution result. A successful test pass receives a reward of +1.0, and a test failure receives a reward of 0.0. Optional additional rewards can be configured, such as +0.1 for a patch with fewer lines of code or +0.05 for a faster test pass.

4. PPO/GRPO policy update: The GRPO (Group-wise Relative Policy Optimization) algorithm performs policy updates. It calculates the advantage and PPO clip loss, and then performs backpropagation to update the model parameters.

5. Evaluation and iteration: Every 100 steps, the system evaluates the issue resolution rate on the validation set, saves a model checkpoint, and iterates the process until convergence.

Step 1: Create model and dataset PVCs

Create Persistent Volume Claims (PVCs) for the model files and datasets, respectively. The following steps in this article use verl-models (mounted to /var/model) and verl-dataset (mounted to /var/model-dataset) by default. Replace them based on your actual configuration.

1. Select a storage type for the PVCs.

   Select an OSS volume or a Cloud Disk volume based on your use case:

   Storage type	Use cases	Notes
   OSS volume	Multi-availability zone clusters or shared reading across multiple nodes	Not limited by availability zones or Cloud Disk types
   Cloud Disk volume	High-performance read and write operations on a single node	Must be in the same availability zone as the GPU node. Otherwise, the volume cannot be mounted. The supported cloud disk types are restricted by the ECS instance type. For example, the ecs.gn5 family does not support ESSD cloud disks (cloud_essd), but supports SSD cloud disks (cloud_ssd) and Ultra Disks (cloud_efficiency). On the Instance Type Family page, check the cloud disk types that are supported by your GPU instance type.

2. Plan PVC capacity.

   • Model PVC: We recommend that you reserve 50 GiB or more. Qwen2.5-7B requires approximately 15 GiB.

   • Dataset PVC: We recommend that you reserve 100 GiB or more.

3. Create the verl-models and verl-dataset PVCs.

   Based on the selected storage type, see the following topics to create PVCs:

   • Select a client for an OSS volume.

   • Use a static cloud disk volume or Use a dynamic cloud disk volume.

4. Verify that the PVCs are ready.

   kubectl get pvc verl-models verl-dataset

   If the STATUS column in the expected output is Bound, the PVC is created successfully and you can proceed to the next step.

   NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   AGE
   verl-models    Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   50Gi       RWX            1m
   verl-dataset   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   100Gi      RWX            1m

Step 2: Deploy RayCluster

Create a Ray cluster that includes the VeRL framework in your ACK cluster.

1. Obtain the KubeConfig file of a cluster and use kubectl to connect to the cluster or click Manage Clusters Using Workbench on the ACK console.

2. Create the rbac.yaml file.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: rayclustertest
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: rayclustertest
     namespace: default
   rules:
   - apiGroups: [""]
     resources: ["pods"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   - apiGroups: ["batch"]
     resources: ["jobs"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   - apiGroups: ["apps.kruise.io"]
     resources: ["sandboxes", "sandboxsets", "sandboxclaims"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: rayclustertest
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: rayclustertest
   subjects:
     - kind: ServiceAccount
       name: rayclustertest
       namespace: default

3. Run the following command to apply the configuration.

   kubectl apply -f rbac.yaml

4. Create a raycluster.yaml file.

   apiVersion: ray.io/v1
   kind: RayCluster
   metadata:
     name: raycluster-verl
     namespace: default
   spec:
     headGroupSpec:
       rayStartParams:
         dashboard-host: 0.0.0.0
       serviceType: ClusterIP
       template:
         spec:
           serviceAccountName: rayclustertest
           imagePullSecrets:
           - name: acr-registry
           containers:
           - name: ray-head
             image: <Replace this with the path to your VeRL image>
             imagePullPolicy: IfNotPresent
             env:
             - name: RAY_GRAFANA_HOST
               value: DISABLED
             resources:
               limits:
                 cpu: "50"
                 memory: 200Gi
                 nvidia.com/gpu: "8"
               requests:
                 cpu: "50"
                 memory: 200Gi
                 nvidia.com/gpu: "8"
             securityContext:
               runAsUser: 0
             # Replace /var/model and /var/model-dataset with the actual paths.
             volumeMounts:
             - mountPath: /var/model
               name: model
             - mountPath: /var/model-dataset
               name: model-dataset
           # Replace verl-models and verl-dataset with the names of your PVCs.
           volumes:
           - name: model
             persistentVolumeClaim:
               claimName: verl-models
           - name: model-dataset
             persistentVolumeClaim:
               claimName: verl-dataset

   Modify the following parameters based on your environment:

   • image: The address of the VeRL training image. You can build the image yourself based on the VeRL open-source repository or use the pre-built image provided by the team. The region identifier in the image address, such as ap-southeast-1, must match the region where the cluster is located. If you need to build the image yourself, you can refer to the following Dockerfile:

     FROM verlai/verl:vllm017.latest
     
     WORKDIR /home/verl
     ENV PYTHONPATH="/home/verl:$PYTHONPATH"
     
     COPY . .
     
     RUN git clone https://github.com/alibaba/harbor.git
     RUN git clone https://github.com/alibaba/verl-recipe.git
     
     ENV VERSION=v0.20.2
     RUN curl -L "https://github.com/google/go-containerregistry/releases/download/${VERSION}/go-containerregistry_Linux_x86_64.tar.gz" -o crane.tar.gz \
         && tar -xzf crane.tar.gz && mv crane /usr/local/bin/
     
     ENV BUILDKIT_VERSION=v0.13.2
     RUN curl -sL "https://github.com/moby/buildkit/releases/download/${BUILDKIT_VERSION}/buildkit-${BUILDKIT_VERSION}.linux-amd64.tar.gz" \
         | tar -xz -C /usr/local bin/buildctl
     
     RUN pip install kubernetes oauthlib
     RUN pip install -e /home/verl/harbor
     
     RUN apt update && apt install -y openssh-server vim
     RUN apt remove python3-blinker -y; pip install -U "ray[data,train,tune,serve]"; pip install -e .

   • imagePullSecrets: If you use a private image repository, you must first create a Secret that contains the repository credentials, and then replace acr-registry with the name of the Secret. For example: kubectl create secret docker-registry acr-registry --docker-server=<repository-address> --docker-username=<username> --docker-password=<password>. If you use a public image, you can delete the imagePullSecrets configuration.

   • claimName: The name of the PVC for the model and dataset. Set this to the name of the PVC that you created. The model PVC is mounted to /var/model, and the dataset PVC is mounted to /var/model-dataset. Paths in subsequent steps are based on this mount configuration.

   • resources: Adjust the number of CPUs, memory, and GPUs based on the actual specifications of your GPU node. This example uses the configuration for an 8-card GPU node. If your node has a different number of GPUs, see the resource adaptation instructions below.

   If your GPU node specifications are different from the example (for example, a single node has only 1 to 2 GPUs), in addition to adjusting the resources configuration of the RayCluster, you also need to adjust the following training parameters in Step 4:

   • trainer.n_gpus_per_node: Set to the actual number of GPUs per node.

   • actor_rollout_ref.rollout.tensor_model_parallel_size: Cannot exceed the number of GPUs. Set to 1 in a single-GPU environment.

   • actor_rollout_ref.actor.ulysses_sequence_parallel_size: The value cannot exceed the number of GPUs. In a single-GPU environment, set the value to 1.

   • actor_rollout_ref.rollout.gpu_memory_utilization: If you have a small number of GPUs, reduce this value to a level such as 0.4 to 0.6 to prevent insufficient GPU memory.

   • If you have limited GPU resources, you can reduce actor_rollout_ref.rollout.n, data.max_prompt_length, and data.max_response_length to decrease VRAM usage.

5. Run the following command to apply the configuration.

   kubectl apply -f raycluster.yaml

6. Verify the deployment status of the RayCluster.

   kubectl get rayclusters

   The expected output is as follows. A STATUS of ready indicates that the deployment is successful.

   NAME              DESIRED WORKERS   AVAILABLE WORKERS   CPUS   MEMORY   GPUS   STATUS   AGE
   raycluster-verl                                         50     200Gi    8      ready    44s

7. Confirm that the Ray head pod is ready.

   kubectl get pods -l ray.io/node-type=head

   The expected STATUS is Running. If a Pod remains in the ContainerCreating state for an extended period, run kubectl describe pod <pod-name> to view the event details. Common causes include PVC mount failures or image pull failures. For more information, see Common Issues.

Step 3: Prepare dataset

Run the following commands in the Ray head pod to download the SWE-bench dataset by using harbor-framework.

1. Log on to the Ray head pod.

   kubectl exec -it $(kubectl get pods -l ray.io/node-type=head -o name) -- bash

2. Install dependencies and download the dataset.

   harbor datasets download swebench-verified --export

3. Verify the integrity of the dataset.

   ls swebench-verified/ | wc -l

   The expected output is 500, which is the total number of issues in the SWE-bench Verified dataset. If the number is less than 500, run the download command again.

4. Download the base model from ModelScope. This example uses Qwen2.5-7B-Instruct (approximately 15 GiB). Replace /var/model/Qwen2.5-7B-Instruct with the actual mount directory of the PVC verl-models.

   pip install modelscope
   modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir /var/model/Qwen2.5-7B-Instruct

   After the download is complete, verify the model files.

   ls /var/model/Qwen2.5-7B-Instruct/config.json && echo "Model downloaded successfully"

   Note

   The download time depends on your network environment and typically takes 10 to 30 minutes. When you use other models, replace the model name and path, and also update the actor_rollout_ref.model.path parameter in the training command.

For more information about how to build a custom dataset, see the Harbor-framework Dataset Building Documentation.

Step 4: Deploy BuildKit service (optional)

SWE-bench training requires building a separate Docker image for each Instance. The training code automatically checks if {registry}/swe-bench/{instance_id}:latest exists:

• If the image exists, it is used directly.

• If the image does not exist, the system automatically builds and pushes the image by using the specified BuildKit address.

You can build your own BuildKit service in the cluster to ensure stable and fast image builds. To deploy the service, perform the following steps:

1. Create a buildkit.yaml file.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: buildkitd
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: buildkitd
     template:
       metadata:
         labels:
           app: buildkitd
       spec:
         containers:
         - name: buildkitd
           image: moby/buildkit:latest
           args:
           - --addr
           - tcp://0.0.0.0:1234
           securityContext:
             privileged: true
           ports:
           - containerPort: 1234
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: buildkitd
     namespace: default
   spec:
     selector:
       app: buildkitd
     ports:
     - port: 1234
       targetPort: 1234

2. Run the following command to apply the configuration.

   kubectl apply -f buildkit.yaml

3. Verify the status of the BuildKit service.

   kubectl get pods -l app=buildkitd

   In the expected output, the Pod status is Running.

4. Verify that the BuildKit service port is accessible.

   kubectl run buildkit-test --rm -it --restart=Never --image=busybox -- nc -zv buildkitd 1234

   The expected output contains open or succeeded, which indicates that the port connection is successful. If the connection fails, check the Service configuration and the Pod status.

Step 5: Configure and start training

In this step, you start the training by using RemoteAgentLoop. RemoteAgentLoop provides a unified AgentLoop interface that is compatible with various agents, which lets you integrate them into the RL training process without modifying the agent source code.

1. Log on to the Ray head pod.

   kubectl exec -it $(kubectl get pods -l ray.io/node-type=head -o name) -- bash

2. Modify the key parameters in the training script recipe/agentic/agentic-qwen2.5-7b.sh.

   • actor_rollout_ref.model.path: Set to the actual path of your model in the PVC.

   • trainer.n_gpus_per_node: Set this to the actual number of GPUs per node. For example, set the value to 1 for a single GPU.

   • actor_rollout_ref.rollout.tensor_model_parallel_size: The value cannot exceed the number of GPUs. For a single-GPU environment, set the value to 1.

   • actor_rollout_ref.actor.ulysses_sequence_parallel_size: Cannot exceed the number of GPUs. Set this parameter to 1 in a single-card environment.

   • actor_rollout_ref.rollout.gpu_memory_utilization: If you have a small number of GPUs, reduce the value to 0.4 to 0.6 to avoid insufficient GPU memory.

   • actor_rollout_ref.rollout.n: The number of REPLICAS for the SandboxSet is equal to the value of actor_rollout_ref.rollout.n in the training script. For example, if you set this parameter to 4, each Issue runs four sampling trajectories in parallel. If you have limited GPU resources, lower this value (for example, to 1).

   • actor_rollout_ref.actor.ppo_max_token_len_per_gpu and actor_rollout_ref.ref.log_prob_max_token_len_per_gpu: Decrease these values if the GPU memory is insufficient.

   • trainer.default_local_dir: The storage path for model checkpoints. Ensure that you have sufficient disk space.

   For more information about the parameters, see Agentic Recipe.

3. Start the training.

   bash recipe/agentic/agentic-qwen2.5-7b.sh

4. After the training starts, open another terminal and run the following command to quickly check the training status.

   kubectl logs $(kubectl get pods -l ray.io/node-type=head -o name) --tail=20

   When the training starts normally, the logs show information about model and dataset loading. If an error occurs, see FAQ for troubleshooting. For more verification methods, such as checking the sandbox status, using the Ray Dashboard, and monitoring training logs, see Verify the results.

Sandbox resource status

The system automatically creates SandboxSet and Sandbox resources for each SWE-bench instance.

kubectl get sandboxsets

Expected output:

NAME                    REPLICAS   AVAILABLE   AGE
astropy-astropy-12907   1          1           54s
astropy-astropy-13033   1          1           53s
astropy-astropy-13236   1          1           54s
astropy-astropy-13398   1          1           53s
astropy-astropy-13453   1          1           53s

kubectl get sandboxes

Expected output:

NAME                          STATUS    AGE
astropy-astropy-12907-dkrd9   Running   60s
astropy-astropy-12907-gczlt   Running   62s
astropy-astropy-12907-lgx6m   Running   62s
astropy-astropy-13033-2fvtq   Running   60s
astropy-astropy-13033-8mt4z   Running   61s
astropy-astropy-13033-t8lqs   Running   61s

Each SandboxSet manages a group of Sandbox instances, and each Sandbox corresponds to an independent agent execution environment.

Monitor training status

Use the Ray Dashboard to monitor the status and GPU utilization of each worker.

kubectl port-forward svc/raycluster-verl-head-svc 8265:8265

Open http://localhost:8265 in your browser to view the Ray Dashboard. On the dashboard, you can observe the following:

• Resource usage of each worker node.

• GPU memory and utilization.

• GPU memory and utilization.

Training logs

View the training logs in the Ray head pod to confirm that rollout, reward calculation, and policy updates are running as expected.

kubectl logs $(kubectl get pods -l ray.io/node-type=head -o name) --tail=100

After each Rollout is complete, the log outputs reward statistics, such as average reward, maximum reward, and resolution rate. If reward > 0 entries persistently appear in the log, this indicates that the model is learning to solve the Issue.

Common error patterns: If the following keywords appear in the logs, it indicates a training issue:

• CUDA out of memory: Insufficient video memory. For more information, see Training OOM (out of video memory).

• ConnectionRefused or TimeoutError: The Agent cannot connect to the ProxyServer. Check the LOCAL_IP configuration.

• FileNotFoundError: The model or dataset path is configured incorrectly. Check the PVC mount path and the path parameters in the training command.

• The reward for all Rollouts is consistently 0.0: Check the Harbor sandbox environment variable configuration and ensure that the Sandbox Pod is running properly.

Automatic image builds

For SWE-bench instances that have not yet been built, the system automatically calls BuildKit to build the corresponding Docker image and pushes it to the specified registry. To view the status of build jobs, run the following command:

kubectl get jobs -l type=image-build

Subsequent training steps can reuse the already built images.

PVC mount failure (InvalidInstanceType.NotSupportDiskCategory)

Symptom: A Pod is stuck in the ContainerCreating state, and the kubectl describe pod command returns a FailedAttachVolume error, indicating InvalidInstanceType.NotSupportDiskCategory.

Cause: The ECS instance type of the GPU node does not support the cloud disk type associated with the PVC. For example, the ecs.gn5 series does not support ESSD cloud disks.

Solution: Replace the cloud disk type with one that is compatible with the GPU instance and recreate the PV and PVC. For example, for the ecs.gn5 series, you can change the cloud disk type from cloud_essd to cloud_ssd.

PVC mount failure (ResourcesNotInSameZone)

Symptom: The Pod is stuck in the ContainerCreating state, and the kubectl describe pod command shows a ResourcesNotInSameZone error.

Cause: The Cloud Disk and the GPU node are not in the same availability zone.

Solution: Run kubectl get nodes -o custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone' to check the availability zone of the node, and then recreate the Cloud Disk in the same availability zone.

Sandbox pod startup failure

Symptom: The kubectl get sandboxes command shows that the Pod status is Pending or ImagePullBackOff.

Troubleshooting:

1. Check if the image exists. Confirm that {registry}/swe-bench/{instance_id}:latest has been pushed to the image repository.

   kubectl describe pod <sandbox-pod-name>

2. Check whether the imagePullSecret is correctly configured.

   kubectl get secret acr-registry -o yaml

3. If the image is not built, check whether the BuildKit service is running as expected.

Agent to ProxyServer connection failure

Symptom: A connection timeout error appears in the training logs.

Troubleshooting:

1. Confirm that LOCAL_IP is set correctly and that the Sandbox Pod where the Agent is located can access this IP.

   kubectl exec <sandbox-pod> -- curl http://<LOCAL_IP>:<proxy_port>/health

2. Check whether the network policy allows the sandbox pod to access the Ray head node.

Training OOM (out-of-memory)

Symptom: A CUDA out of memory error occurs during training.

Solution:

• Decrease ppo_max_token_len_per_gpu (default: 12288).

• Enable parameter and optimizer offload (fsdp_config.param_offload=true, fsdp_config.optimizer_offload=true).

• Decrease the value of rollout.n to reduce the number of parallel sampling trajectories.

• Increase the Pod's memory resource request. When FSDP offload is enabled, model weights are offloaded to system memory. The Qwen2.5-7B model requires about 20 to 25 GiB of RAM, so we recommend that you set the Pod memory to 55 Gi or more.

Incompatible CUDA architecture (no kernel image)

Symptom: The following error is reported when training starts: CUDA error: no kernel image is available for execution on the device.

Cause: The Compute Capability of the GPU is lower than the minimum supported version compiled in PyTorch in the image. This typically occurs when you use older GPU architectures, such as P100 (sm_60) or V100 (sm_70).

Solution: Switch to GPU nodes with a Compute Capability of 8.0 or higher, such as the ecs.gn7i series (A10) or the ecs.gn7e series (A100). For more information about the supported instance types, see Prerequisites.

Dataset download or image build timeout

Symptom: The harbor datasets download command hangs, the git clone command times out, or BuildKit fails to download dependencies when building the SWE-bench image.

Cause:

• The pod cannot access the internet: The cluster's VPC lacks a NAT gateway or an SNAT rule.

• The cluster is in a Chinese mainland region: Access to GitHub and Hugging Face may be restricted in some regions.

Solution:

• Confirm that the VPC is configured with a NAT gateway and SNAT rules. You can run curl -I https://github.com in a Pod to verify network connectivity.

• If you cannot access GitHub through a proxy in a Chinese mainland region, we recommend that you migrate to the China (Hong Kong) region or another region outside the Chinese mainland.

• If a pip installation times out, you can use the Alibaba Cloud mirror source to accelerate it: pip install <package> -i https://mirrors.aliyun.com/pypi/simple/.

RayCluster pod image pull failure (ImagePullBackOff)

Symptom: The RayCluster Pod status is ImagePullBackOff or ErrImagePull, and the kubectl describe pod command shows 401 Unauthorized or manifest unknown.

Troubleshooting:

1. Verify that the imagePullSecret is created and the credentials are correct:kubectl get secret <secret-name> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

2. Ensure that the imagePullSecrets name in the RayCluster YAML matches the Secret name.

3. Confirm that the region identifier in the image address matches the docker-server region in the Secret. For example, if the image is from registry-ap-southeast-1.ack.aliyuncs.com/..., the docker-server in the Secret should also be registry-ap-southeast-1.ack.aliyuncs.com.

Pod scheduling failure (insufficient cpu/memory)

Symptom: A Pod remains in the Pending state for a long time, and kubectl describe pod shows Insufficient cpu or Insufficient memory.

Cause: The resource requests in the RayCluster YAML file exceed the allocatable resources of the node. A portion of the total node resources is occupied by system components, such as kubelet and the OS. Therefore, the allocatable resources are usually less than the total resources.

Solution:

1. View the allocatable resources of a node:kubectl describe node <node-name> | grep -A 6 Allocatable

2. Adjust resources.requests in the RayCluster YAML to ensure that it does not exceed the node's Allocatable value. For example, an 8 vCPU node typically has an allocatable CPU of about 7.9, so the CPU request should be set to 7 or lower.