Mount OSS storage for an Agent Sandbox by using Agent Identity

Background information

The traditional AccessKey approach mounts a single long-term key pair to all sandboxes. Permissions cannot be scoped to individual instances. Agent Identity uses the ack-agent-identity add-on to assign each sandbox an independent identity instead. Each sandbox dynamically obtains short-lived STS credentials, enabling sandbox-level fine-grained permission isolation without storing any long-term credentials inside the sandbox.

Agent Identity uses a two-layer permission model:

• Layer 1: Resource Access Management (RAM) role permissions (maximum permission boundary). Created in the RAM console, this layer covers the full range of buckets that all sandboxes may access. An administrator configures this layer once.

• Layer 2: CredentialProvider policy (effective permissions). Defined in the cluster through a CredentialProvider custom resource (CR), this policy must be a subset of Layer 1. Template functions dynamically narrow the policy per sandbox based on the declared bucket and subpath.

The STS credentials that each sandbox ultimately obtains represent the intersection of the RAM role permissions and the CredentialProvider policy. The effective scope is further limited to the bucket and subpath declared by that sandbox.

Prerequisites

• The basic environment for Agent Sandbox is set up. For more information, see Create an Agent Sandbox.

• The following add-ons are installed or upgraded and configured on the cluster Add-ons page:

  • ack-agent-identity v0.4.0 or later, with the agentTokenDelegation option selected.

  • ack-agent-sandbox-controller v0.5.22-release.1 or later, with the identityProvider option selected.

    The add-on runtime and sandbox configuration depend on the csi-agent, csi-plugin, and agent-runtime images. Configure image caches in advance to avoid prolonged scale-out times.

  • ack-sandbox-manager v0.6.8 or later, with the identityProvider option selected.

• RRSA OIDC is enabled in the Security and Auditing section on the Basic Information tab of the Cluster Information page. Record the provider URL and provider ARN for later use when configuring the RAM role trust policy. For more information, see Step 1: Enable RRSA for your cluster.

For other usage limits, see Usage limits later in this topic.

Network allowlist configuration

Sandboxes need network access to both the Credential Provider (for STS credentials) and the OSS endpoint (for data). Configure both a TrafficPolicy for pod-level traffic and a security group for Elastic Compute Service (ECS) network interface-level traffic.

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: sandbox-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: sandbox
  ingress:
    rules:
      # Allow traffic from Sandbox Gateway, Manager, and Controller
      - action: allow
        from:
          - service:
              name: sandbox-gateway
              namespace: sandbox-system
          - service:
              name: sandbox-manager
              namespace: sandbox-system
          - service:
              name: sandbox-controller-manager-webhook-service
              namespace: sandbox-system
      # Deny all other sources
      - action: deny
        from:
          - cidr: 0.0.0.0/0
  egress:
    rules:
      # Deny access to the cloud product metadata service
      - action: deny
        to:
          - cidr: 100.100.100.200/32
      # Allow DNS resolution for the Credential Provider service domain
      - action: allow
        to:
          - service:
              namespace: kube-system
              name: kube-dns
      # Allow access to the Credential Provider service to obtain STS tokens
      - action: allow
        to:
          - service:
              namespace: ack-agent-identity
              name: credential-provider
      # Deny access to private networks (prevent sandbox from probing the internal network)
      - action: deny
        to:
          - cidr: 10.0.0.0/8
      - action: deny
        to:
          - cidr: 172.16.0.0/12
      - action: deny
        to:
          - cidr: 192.168.0.0/16
      # Allow access to the Internet (for OSS endpoints, etc.)
      - action: allow
        to:
          - cidr: 0.0.0.0/0

telnet credential-provider.ack-agent-identity.svc 8443

Step 1: Create a RAM role and configure the trust policy

The Agent Identity add-on relies on RAM Roles for Service Accounts (RRSA) to assume a RAM role and obtain STS temporary credentials. The RAM role permissions configured in this step define the maximum permission boundary. The actual permissions that each sandbox obtains are further narrowed by the CredentialProvider policy in Step 2. Configure this role to cover the full range of buckets that all sandboxes may access.

1. On the Basic Information tab of the Cluster Information page, enable RRSA OIDC in the Security and Auditing section.

   Hover over the Enabled status next to RRSA OIDC to view the provider URL and ARN.

2. Go to the RAM console - Create Role page, select Principal Type as IdP Type, and replace <oidc_issuer_url> with the provider URL and <oidc_provider_arn> with the provider ARN in the following template:

   Important

   In large-scale scenarios, to avoid frequent STS token rotation, change the Max Session Duration on the Roles details page to 1 hour or more after the role is created.

   If the current RAM user does not have the permissions to create roles or policies, ask the Alibaba Cloud account administrator to grant the AliyunRAMFullAccess permission or similar permissions to the current user. For more information, see Grant permissions to a RAM user.

   {
     "Statement": [
       {
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "oidc:aud": "sts.aliyuncs.com",
             "oidc:iss": "<oidc_issuer_url>",
             "oidc:sub": [
               "system:serviceaccount:ack-agent-identity:credential-provider"
             ]
           }
         },
         "Effect": "Allow",
         "Principal": {
           "Federated": [
             "<oidc_provider_arn>"
           ]
         }
       }
     ],
     "Version": "1"
   }

   The value ack-agent-identity:credential-provider in the oidc:sub field is the ServiceAccount used by the ack-agent-identity add-on and cannot be customized.

3. Create a custom permission policy for the RAM role. Select a read-only or read/write policy based on business requirements, replace <YOUR-BUCKET-NAME> with the actual bucket name, and use wildcards or list multiple resources to cover multiple buckets.

   Read-only policy

   {
     "Statement": [
       {
         "Action": [
           "oss:GetObject",
           "oss:ListObjects"
         ],
         "Effect": "Allow",
         "Resource": [
           "acs:oss:*:*:<YOUR-BUCKET-NAME>",
           "acs:oss:*:*:<YOUR-BUCKET-NAME>/*"
         ]
       }
     ],
     "Version": "1"
   }

   Read/write policy

   {
     "Statement": [
       {
         "Action": [
           "oss:GetObject",
           "oss:PutObject",
           "oss:DeleteObject",
           "oss:AbortMultipartUpload",
           "oss:ListMultipartUploads",
           "oss:ListObjects"
         ],
         "Effect": "Allow",
         "Resource": [
           "acs:oss:*:*:<YOUR-BUCKET-NAME>",
           "acs:oss:*:*:<YOUR-BUCKET-NAME>/*"
         ]
       }
     ],
     "Version": "1"
   }

4. Go to the RAM console - Roles page. In the Actions column of the RAM role list, click Attach Policy for the target role to attach the permission policy created in the previous step to the RAM role.

Step 2: Create Agent Identity CRs

Create four CRs (AgentIdentity, CredentialProvider, AgentRole, and AgentRoleBinding) to complete the authorization binding between sandbox identities and credentials.

1. Save the following YAML as agent-identity.yaml to define the agent identity:

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: AgentIdentity
   metadata:
     name: my-storage-agent            # Agent identity name
     namespace: <YOUR-NAMESPACE>
   spec:
     description: "Agent identity for OSS storage mounting"

2. Save the following YAML as credential-provider.yaml to define the STS credential permission scope for each sandbox. The CredentialProvider policy must be a subset of the RAM role permissions in Step 1. Permissions that exceed the RAM role scope do not take effect. Create multiple CredentialProviders as permission templates to match different use cases. The following examples provide read-only and read/write templates.

   Replace <YOUR-RAM-ROLE-NAME> with the RAM role name created in Step 1:

   Read-only CredentialProvider

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: CredentialProvider
   metadata:
     name: oss-ro                # Read-only CredentialProvider
     namespace: <YOUR-NAMESPACE>
   spec:
     type: RAM
     ram:
       source:
         provider: RRSA
         rrsa:
           roleName: <YOUR-RAM-ROLE-NAME>
           tokenValidity: 1h
           policy: |
             {
               "Statement": [
                 {
                   "Action": [
                     "oss:GetObject"
                   ],
                   "Effect": "Allow",
                   "Resource": {{ build_policy_oss_resource() }}
                 },
                 {
                   "Action": [
                     "oss:ListObjects"
                   ],
                   "Effect": "Allow",
                   "Resource": {{ build_policy_oss_resource(limit_sub_path=false) }},
                   "Condition": {
                     "StringLike": {
                       "oss:Prefix": {{ build_policy_oss_prefix_condition() }}
                     }
                   }
                 }
               ],
               "Version": "1"
             }

   ListObjects is a bucket-level operation. Use the oss:Prefix condition to restrict listing to objects under the specified subpath. Otherwise, the sandbox can list the entire bucket contents.

   Read/write CredentialProvider

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: CredentialProvider
   metadata:
     name: oss-rw                      # Read/write CredentialProvider
     namespace: <YOUR-NAMESPACE>
   spec:
     type: RAM
     ram:
       source:
         provider: RRSA
         rrsa:
           roleName: <YOUR-RAM-ROLE-NAME>
           tokenValidity: 1h
           policy: |
             {
               "Statement": [
                 {
                   "Action": [
                     "oss:GetObject",
                     "oss:PutObject",
                     "oss:DeleteObject",
                     "oss:AbortMultipartUpload",
                     "oss:ListMultipartUploads"
                   ],
                   "Effect": "Allow",
                   "Resource": {{ build_policy_oss_resource() }}
                 },
                 {
                   "Action": [
                     "oss:ListObjects"
                   ],
                   "Effect": "Allow",
                   "Resource": {{ build_policy_oss_resource(limit_sub_path=false) }},
                   "Condition": {
                     "StringLike": {
                       "oss:Prefix": {{ build_policy_oss_prefix_condition() }}
                     }
                   }
                 }
               ],
               "Version": "1"
             }

   The template functions used in the CredentialProvider policy are automatically populated by the sandbox-controller based on the mount configuration of each sandbox instance. For more information, see CredentialProvider policy template functions in this topic.

3. Save the following YAML as agent-role.yaml. Use an AgentRole to reference all storage-related CredentialProviders, and bind the role to the agent identity through an AgentRoleBinding:

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: AgentRole
   metadata:
     name: oss-storage-role
     namespace: <YOUR-NAMESPACE>
   spec:
     rules:
     - effect: Allow
       action: "GetResourceCredential"
       resource: "CredentialProvider/oss-ro"     # Reference the read-only CredentialProvider
     - effect: Allow
       action: "GetResourceCredential"
       resource: "CredentialProvider/oss-rw"           # Reference the read/write CredentialProvider
   ---
   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: AgentRoleBinding
   metadata:
     name: my-agent-oss-binding
     namespace: <YOUR-NAMESPACE>
   spec:
     agentRoleRef:
       apiGroup: agentidentity.alibabacloud.com
       kind: AgentRole
       name: oss-storage-role
     subjects:
     - authorizationType: "Agent"
       agentAuthorizationConfiguration:
         agentName: my-storage-agent                   # Must match the AgentIdentity name

4. Apply all the preceding YAML files in sequence:

   kubectl apply -f agent-identity.yaml
   kubectl apply -f credential-provider.yaml
   kubectl apply -f agent-role.yaml

Step 3: Configure the SandboxSet and create a PV

Enable Container Storage Interface (CSI) and agent-runtime capabilities in the SandboxSet, and create a PersistentVolume object that declares authType: agent-identity.

1. Save the following YAML as sandboxset.yaml (note that dnsPolicy must be set to ClusterFirst), and run kubectl apply -f sandboxset.yaml to apply it:

   apiVersion: agents.kruise.io/v1alpha1
   kind: SandboxSet
   metadata:
     name: code-interpreter-ossfs-agent-identity
     namespace: default
   spec:
     replicas: 3
     runtimes:
     - name: csi                        # Enable CSI mount capability
     - name: agent-runtime              # Inject envd and other environment management tools
     template:
       metadata:
         annotations:
           network.alibabacloud.com/wait-clusterip-ready: "*"
         labels:
           alibabacloud.com/acs: "true"
           alibabacloud.com/compute-class: agent-sandbox
           alibabacloud.com/compute-qos: default
       spec:
         automountServiceAccountToken: false
         dnsPolicy: ClusterFirst        # Must be set to ClusterFirst
         containers:
         - image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/code-interpreter:v1.6
           imagePullPolicy: IfNotPresent
           name: sandbox
           resources:
             requests:
               cpu: "1"
               memory: 1Gi
             limits:
               cpu: "1"
               memory: 1Gi
         terminationGracePeriodSeconds: 30

   Enable the network.alibabacloud.com/wait-clusterip-ready annotation to ensure that the sandbox can correctly resolve Credential Provider and storage service addresses when performing dynamic storage mounting.

2. Save the following YAML as oss-pv.yaml, replace the bucket name, region, and endpoint with actual values, and run kubectl apply -f oss-pv.yaml to create the resource.

   In Agent Identity mode, the PV does not require nodePublishSecretRef, and no Secret needs to be created.

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     labels:
       alicloud-pvname: oss-pv-sandbox-system
     name: oss-pv-sandbox-system
   spec:
     accessModes:
     - ReadWriteMany
     capacity:
       storage: 50Gi
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeAttributes:
         authType: agent-identity                             # Fixed value. Declares Agent Identity authentication.
         bucket: <YOUR-BUCKET-NAME>                           # Replace with the actual bucket name.
         url: https://oss-cn-hangzhou-internal.aliyuncs.com   # Replace with the actual endpoint. HTTPS internal endpoint recommended.
         path: /
         otherOpts: "-o sigv4 -o region=cn-hangzhou -o umask=022 -o allow_other"  # Use signature version 4. Replace region as needed.
     volumeHandle: oss-pv-sandbox-system                      # Must match the PV name.
     persistentVolumeReclaimPolicy: Retain
     storageClassName: test
     volumeMode: Filesystem

   Important

   The volumeAttributes of a PV cannot be modified after creation. To change the authentication method, delete and recreate the PV. This operation causes OSS access failures for sandboxes that have not been upgraded. Proceed with caution.

   The following table describes key fields:

   Parameter	Description
   authType	Set to agent-identity to use Agent Identity authentication.
   bucket	The name of the OSS bucket to mount.
   url	The OSS access endpoint. Internal network format: https://oss-{region}-internal.aliyuncs.com. HTTPS endpoints are recommended.
   otherOpts	ossfs mount options. When using signature version 4, include -o sigv4 and -o region=<YOUR-REGION>.
   path	The mount point path relative to the bucket root directory. Default value: /.

Step 4: Mount the storage volume

When specifying mount configurations for a sandbox, use attributes.credentialProviderName to specify the CredentialProvider and security.agents.kruise.io/agent-name to associate the AgentIdentity created in Step 2. ACS supports the following two methods to trigger mounting:

import json
from e2b_code_interpreter import Sandbox

sbx = Sandbox.create(
    template="code-interpreter-ossfs-agent-identity",
    timeout=600,
    metadata={
        "e2b.agents.kruise.io/csi-volume-config": json.dumps([
            {
                "pvName": "oss-pv-sandbox-system",
                "mountPath": "/data-oss",
                "subPath": "user-a-data",
                "attributes": {
                    "credentialProviderName": "oss-ro"
                }
            }
        ]),
        "security.agents.kruise.io/agent-name": "my-storage-agent"
    }
)
print(f"sandbox id: {sbx.sandbox_id}")

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxClaim
metadata:
  name: code-interpreter-claim
  namespace: default
spec:
  templateName: code-interpreter-ossfs-agent-identity     # Name of the associated SandboxSet
  replicas: 1
  claimTimeout: 5m
  ttlAfterCompleted: 15m
  annotations:
    # Must exactly match the AgentIdentity name
    security.agents.kruise.io/agent-name: my-storage-agent
  dynamicVolumesMount:
  # Company-wide read-only subdirectory 1
  - pvName: oss-pv-sandbox-system
    mountPath: "/office-skill-readonly"
    subPath: "office-skill-readonly"
    readOnly: true
    attributes:
      credentialProviderName: "oss-ro"
  # Company-wide read-only subdirectory 2
  - pvName: oss-pv-sandbox-system
    mountPath: "/bu-office-skill-sub"
    subPath: "bu-office-skill-sub-readonly"
    readOnly: true
    attributes:
      credentialProviderName: "oss-ro"
  # Per-user read/write subdirectory
  - pvName: oss-pv-sandbox-system
    mountPath: "/user-owner-dir-rw"
    subPath: "user-a-owner-dir-rw"
    attributes:
      credentialProviderName: "oss-rw"

The following table describes mount fields:

Step 5: Verify the mount result

1. After the sandbox is created, its status changes to Running, which indicates that the claim and mount process is complete. Query the allocated sandboxes:

   kubectl get sandbox -n default -l agents.kruise.io/claim-name=code-interpreter-claim

   Expected output:

   NAME                                             STATUS    AGE   CLAIMED
   code-interpreter-ossfs-agent-identity-6vh94      Running   22h   true

2. Replace <POD_NAME> with the pod name of the allocated sandbox, and enter the container to verify that the mount directory can list and read/write files:

   kubectl exec -it <POD_NAME> -- ls /data-oss
   kubectl exec -it <POD_NAME> -- sh -c "echo 'hello agent identity' > /data-oss/test.txt && cat /data-oss/test.txt"

   If ls lists files in the OSS bucket subdirectory and write operations are denied (the Mount through E2B SDK example configures read-only permissions), the Agent Identity-authenticated OSS storage volume mount is working correctly.

Usage limits

• Runtime storage mounting is supported through E2B-based Create API calls, the hibernation/wakeup feature, and in-place image upgrades.

• Sandboxes must use dnsPolicy: ClusterFirst to resolve the Credential Provider service domain name.

• When configuring network policies or traffic policies, make sure to allow outbound traffic to the OSS endpoint, CoreDNS, and the Credential Provider service.

• In concurrent single-file write scenarios, the overwrite-upload behavior of OSS may cause data overwriting. Ensure data consistency at the application level.

• When the readdir operation is performed on a large number of files for the first time, ossfs loads all metadata at once, which may cause the process to run out of memory (OOM). Mount a bucket subdirectory instead of the root directory.

• For large-scale deployments, request a quota increase for the AssumeRoleWithOIDC API in the Quota Center to prevent credential acquisition failures.

• CSI dynamic mounting relies on privileged containers and host path (hostPath: /var/run/csi) permissions, which break the standard container security boundary. Follow these best practices: enable dynamic mounting only when required, use read-only mounting by default for shared storage volumes, use TrafficPolicy and security groups for compensating controls, and use HTTPS endpoints for all OSS mounts.

CredentialProvider policy template functions

The policy field of CredentialProvider supports the following dedicated template functions. The sandbox-controller automatically populates these functions based on the mount configuration of each sandbox instance. This eliminates the need to create a separate CredentialProvider for each subpath.

[
  "acs:oss:*:*:bucket-a/sub-path-a",
  "acs:oss:*:*:bucket-a/sub-path-a/*",
  "acs:oss:*:*:bucket-b/sub-path-b",
  "acs:oss:*:*:bucket-b/sub-path-b/*"
]

[
  "acs:oss:*:*:bucket-a",
  "acs:oss:*:*:bucket-b"
]

[
  "sub/path/a/*",
  "sub/path/b/*"
]

FAQ

Agent Identity-based OSS mounting operates in three layers: the sandbox accesses CredentialProvider over the network, CredentialProvider issues an STS token, and the STS token is used to access OSS. Each layer has distinct symptoms and troubleshooting methods. Troubleshoot layer by layer in the following order.

# Check the token issuance status
kubectl get pod <POD_NAME> -n <namespace> -o jsonpath='{.metadata.annotations.security\.agents\.kruise\.io/token-status}'

# Check csi-agent-sidecar logs to locate specific ossfs errors
kubectl -n <namespace> logs <POD_NAME> -c csi-agent-sidecar

# Verify connectivity to CredentialProvider from inside the sandbox
telnet credential-provider.ack-agent-identity.svc 8443

"ossfs exited with error" err="signal: terminated"

Security Token refresh failed

Invalid Credentials(host=xxxx message=xxx.)