Evaluate agents on ACK with the Harbor CLI

Install the Harbor CLI

Clone the repository from GitHub and install it using pip:

git clone https://github.com/alibaba/harbor.git
cd harbor
pip install -e .
harbor --version

Configure kubeconfig

The Harbor ACK environment uses kubeconfig to connect to the cluster. Run the following command to verify the connection:

kubectl cluster-info

To specify a context or a kubeconfig file, pass the corresponding parameters to the harbor run command:

--ek context=my-ack-context
# or
--ek kubeconfig=/path/to/kubeconfig

Create an evaluation namespace

Use a dedicated namespace to store all evaluation-related resources, such as trial pods, Secrets, and PVs/PVCs:

kubectl create namespace harbor-eval

Other tools (optional)

• crane: Used to verify images pushed to Container Registry (ACR), as described in Build and push images in bulk.

• ossutil: Used to view evaluation result files in OSS from the command line, as described in View the evaluation result files.

Deploy BuildKit in the cluster

Because clusters typically lack a Docker daemon, deploy BuildKit to serve as a remote build backend:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkitd
  namespace: harbor-eval
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      labels:
        app: buildkitd
    spec:
      containers:
        - name: buildkitd
          image: moby/buildkit:v0.15.0
          args:
            - --addr
            - tcp://0.0.0.0:1234
            - --config
            - /etc/buildkit/buildkitd.toml
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "2"
              memory: 4Gi
          volumeMounts:
            - name: buildkitd-config
              mountPath: /etc/buildkit/buildkitd.toml
              subPath: buildkitd.toml
            - name: registry-auth
              mountPath: /registry-auth/
      volumes:
        - name: buildkitd-config
          configMap:
            name: buildkitd-config
        - name: registry-auth
          secret:
            secretName: buildkitd-registry-auth
---
apiVersion: v1
kind: Service
metadata:
  name: buildkit-service
  namespace: harbor-eval
spec:
  type: ClusterIP
  selector:
    app: buildkitd
  ports:
    - port: 1234
      targetPort: 1234

Configure buildkitd authentication and registry mirror

Because BuildKit v0.15.0 does not support registry authentication through CLI parameters, you must configure it in a configuration file. Create a buildkitd.toml ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: buildkitd-config
  namespace: harbor-eval
data:
  buildkitd.toml: |
    [registry."docker.io"]
      mirrors = ["mirror.gcr.io"]

    [registry."<YOUR-ACR-ADDRESS>"]
      config = "/registry-auth/"

In this example, the Docker Hub image source is set to mirror.gcr.io. The Alibaba Cloud registry mirror does not proxy third-party repositories such as swebench, and pulling from them returns an insufficient_scope error.

Create an ACR authentication Secret (in Docker config format):

kubectl create secret generic buildkitd-registry-auth \
  --namespace=harbor-eval \
  --from-file=config.json=$HOME/.docker/config.json

Build and push images in bulk

The harbor admin upload-images command scans all tasks in the dataset, builds the Dockerfile of each task, and pushes the images to ACR:

harbor admin upload-images \
  -t /path/to/dataset \
  -r <YOUR-ACR-ADDRESS> \
  --remote-buildkit tcp://buildkit-service:1234 \
  --update-config \
  --sanitize-image-names \
  --skip-unchanged \
  -n 4

Command execution flow:

1. Reads the Dockerfile for each task.

2. Builds the image on the remote BuildKit service by using buildctl.

3. Pushes the image to ACR.

4. Writes docker_image, built_content_hash, and image_sha256 back to task.toml.

With -n 4 parallelism, building 500 tasks takes approximately 2 to 3 hours. Run the command with nohup ... & disown to prevent the process from being terminated if your terminal disconnects.

After the build is complete, verify the image:

crane digest <YOUR-ACR-ADDRESS>/<image-name>:<tag>

docker_image field in task.toml

After upload-images runs, the following fields are added to the [environment] section of task.toml:

[environment]
docker_image = "registry.cn-hangzhou.aliyuncs.com/<ns>/harbor/<task-name>@sha256:..."
built_content_hash = "abc123..."
image_sha256 = "sha256:..."

When harbor run starts, the ACK environment reads docker_image first and pulls the image directly. This takes seconds, whereas building the image in the cluster takes minutes.

Configure ACR pull credentials

The trial pod requires authentication to pull pre-built images from ACR. Create a docker-registry Secret:

kubectl create secret docker-registry acr-pull-secret \
  --namespace=harbor-eval \
  --docker-server=registry.cn-hangzhou.aliyuncs.com \
  --docker-username=<username> \
  --docker-password=<password>

Run evaluation tasks

The following command starts an evaluation in an ACK cluster. Harbor creates a trial pod for each task in the dataset and runs the evaluation. Replace -a and -m with the agent and model you use.

harbor run \
  -p ./my-dataset \
  --env ack \
  -n 1 \
  --ek namespace=harbor-eval \
  --ek registry=registry.cn-hangzhou.aliyuncs.com/<ns>/harbor \
  --ek image_pull_secret=acr-pull-secret \
  -a qwen-coder \
  -m qwen3.7-max \
  --no-delete \
  -y

Common parameters:

Schedule pods to specific nodes

Use a node selector to schedule trial pods to specific instance types and override resource configurations:

harbor run \
  -p ./my-dataset \
  --env ack \
  -n 1 \
  --ek namespace=harbor-eval \
  --ek image_pull_secret=acr-pull-secret \
  --ek 'node_selector={"node.kubernetes.io/instance-type":"ecs.gn7i-c8g1.2xlarge"}' \
  --override-cpus 4 \
  --override-memory-mb 8192 \
  --override-gpus 1 \
  -a qwen-coder \
  -m qwen3.7-max \
  -y

Filter tasks

Use the following parameters to filter which tasks to run:

-i "task-prefix-*"   # Only run tasks with matching names
-x "skip-*"          # Exclude tasks with matching names
-l 10                # Run a maximum of 10 tasks

Pass environment variables to the agent

Use the --ae parameter to pass environment variables, such as an API Key, to the trial pod for the agent to use when it calls the model:

--ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
--ae DASHSCOPE_API_KEY=$DASHSCOPE_API_KEY

Use a YAML configuration file

To manage numerous parameters, use a YAML configuration file:

# eval-config.yaml
dataset: ./my-dataset          # Or a registry reference

agent: qwen-coder
model: qwen3.7-max

environment:
  type: ack
  kwargs:
    namespace: harbor-eval
    registry: registry.cn-hangzhou.aliyuncs.com/<ns>/harbor
    image_pull_secret: acr-pull-secret
    service_account: harbor-eval-sa

n_concurrent_trials: 1

agent_env:
  - name: ANTHROPIC_API_KEY
    value: "${ANTHROPIC_API_KEY}"

harbor run -c eval-config.yaml -y

Use ACS sandbox mode (optional)

Enable sandbox mode to run evaluations on Alibaba Cloud Serverless Container (ACS) or to use OpenKruise SandboxSet to pre-warm pods and accelerate trial startup.

The SandboxClaim mode relies on the OpenKruise CRD (agents.kruise.io/v1alpha1). Before you enable this mode, ensure that OpenKruise is installed in the cluster and that the ServiceAccount has the required CRD permissions (for more information, see Configure RBAC).

Core parameters:

Example of running on ACS:

harbor run \
  -p ./dataset \
  --env ack \
  -n 4 \
  --ek namespace=harbor-eval \
  --ek image_pull_secret=acr-pull-secret \
  --ek use_sandbox_claim=true \
  --ek sandboxset_replicas=4 \
  --ek 'sandbox_annotations={"k8s.aliyun.com/product-on-demand":"acs"}' \
  -a qwen-coder \
  -m qwen3.7-max \
  -y

To schedule standard trial pods to virtual-kubelet nodes without sandbox mode, use the following parameters:

--ek 'node_selector={"type":"virtual-kubelet"}' \
--ek 'tolerations=[{"key":"virtual-kubelet.io/provider","operator":"Exists"}]'

Configure RBAC

The Job pod needs permissions to perform operations such as creating trial pods. Grant these permissions through a ServiceAccount:

# harbor-eval-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: harbor-eval-sa
  namespace: harbor-eval
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: harbor-eval-role
  namespace: harbor-eval
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "pods/log"]
    verbs: ["get", "list", "create", "delete", "watch"]
  - apiGroups: [""]
    resources: ["configmaps", "secrets"]
    verbs: ["get", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: harbor-eval-rolebinding
  namespace: harbor-eval
subjects:
  - kind: ServiceAccount
    name: harbor-eval-sa
    namespace: harbor-eval
roleRef:
  kind: Role
  name: harbor-eval-role
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f harbor-eval-rbac.yaml

When using sandbox mode, you must also add permissions to the preceding Role for sandboxsets, sandboxclaims, and sandboxes to allow get/list/create/delete/watch operations.

Check the status of trial pods

# View all trial pods
kubectl get pods -n harbor-eval -l app=sandbox

# View the logs of a specific pod
kubectl logs -n harbor-eval <pod-name> --tail=100

Check the Job status

When orchestrating evaluations with a Kubernetes Job, check the execution status and logs of the Job:

kubectl get jobs -n harbor-eval
kubectl describe job harbor-eval-job -n harbor-eval
kubectl logs -l job-name=harbor-eval-job -n harbor-eval --tail=200

View the evaluation result files

The result directory structure for each trial is as follows:

harbor-eval/
└── <task-name>/
    ├── agent/          # Agent execution logs
    ├── verifier/       # Verifier validation results
    ├── artifacts/      # Task artifacts
    └── results.json    # Summary results

When orchestrating a Kubernetes Job, use -o /mnt/oss/results to specify the output path. The results are written directly back to OSS through OSS CSI FUSE, so no additional upload is required. After the evaluation is complete, use ossutil to view the output:

ossutil ls oss://my-harbor-bucket/results/harbor-eval/

Pod is stuck in Pending state

Check the pod's events:

kubectl describe pod <pod-name> -n harbor-eval

Common causes include:

• Insufficient node resources: Adjust --override-cpus or --override-memory-mb.

• Image pull failed: Check image_pull_secret and the ACR address configuration.

• Node selector mismatch: Confirm that the labels in the node_selector exist on the cluster nodes.

Image pull failure

# Confirm that the target image exists in ACR
docker manifest inspect <acr-address>/<image>:<tag>

# Confirm that the Secret and the pod are in the same namespace
kubectl get secret acr-pull-secret -n harbor-eval

Harbor CLI cannot connect to the cluster

# Verify the kubeconfig
kubectl cluster-info

# Confirm RBAC permissions from within a pod
kubectl auth can-i create pods -n harbor-eval --as=system:serviceaccount:harbor-eval:harbor-eval-sa

When running in a cluster, the VPC private IP address that the kubeconfig file points to may not be accessible from the pod CIDR. Change the API Server address to the in-cluster Service endpoint: https://kubernetes.default.svc:443.

Evaluation task times out

Task timeout is controlled by the following fields in task.toml:

[agent]
timeout_sec = 300

[verifier]
timeout_sec = 60

For global adjustments, use the top-level CLI parameters (not the --ek parameter):

--agent-timeout-multiplier 2.0

The docker_image field has no effect

Make sure the Harbor CLI version is 0.6.5 or later. Older versions do not read the docker_image field.

harbor --version