Use API Key credentials stored in KMS Secrets Manager by configuring a CredentialProvider

Feature introduction

In AI Agent scenarios, Agents often need to call APIs of third-party services that require API Keys, such as large language model (LLM) services like OpenAI and Tongyi Qianwen. Embedding a real API Key directly in application code, images, or environment variables poses risks of credential leakage and difficulty in rotation.

CredentialProvider is a custom resource (CR) that defines a credential source, provided by ack-agent-identity. When the type is APIKey and the source is KMS, it reads the value of a specified credential from Alibaba Cloud KMS Secrets Manager as the API Key and provides the following capabilities:

• The real API Key is hosted in KMS Secrets Manager and is centrally managed, rotated, and audited by KMS. Neither applications nor business Pods access plaintext keys or connect to KMS directly. Instead, the ack-agent-identity add-on retrieves the API Key on their behalf through OIDC.

• The AgentRole / AgentRoleBinding authorization mechanism precisely controls which CredentialProviders each Agent identity can obtain credentials from.

• secretName supports template variables, allowing different KMS credentials to be referenced dynamically based on context such as the Agent identity. One configuration therefore serves multiple identities, each receiving its own credential.

Compared with the Kubernetes source, the KMS source moves key hosting from in-cluster Secrets to cloud-side KMS Secrets Manager. It is suitable for scenarios that require centralized key hosting, rotation, and auditing.

In addition to APIKey, CredentialProvider also supports types such as RAM. This topic only covers type: APIKey with a dedicated KMS instance OIDC (ACK) source. For other types and sources, see the corresponding documentation.

Resource objects involved

Setting up KMS credential delivery typically involves the following CRs, which are created in sequence in the steps below:

Applicability

• The cluster runs Kubernetes 1.28 or later.

• On the Add-ons page of the cluster, the version of the ack-agent-identity add-on is 0.5.0 or later.

• The CoreDNS add-on is installed in the cluster.

• A dedicated KMS instance is purchased and enabled, and its image version is dkms-4.0.0 or later. If the version is earlier than dkms-4.0.0, upgrade it first. For more information, see Upgrade the image version of a KMS instance.

• RRSA OIDC is enabled on the Basic Information tab of the Cluster Information page of the cluster. OIDC (ACK) authentication depends on this feature.

• If the cluster and the dedicated KMS instance are in the same region but different Virtual Private Clouds (VPCs), VPC sharing (binding) is configured on the KMS side for the cluster VPC. Otherwise, the cluster cannot access KMS through the instance endpoint. For more information, see Configure Multi-VPC access to KMS instances within the same region.

• If the cluster and the dedicated KMS instance are in different regions, cross-region access to KMS is configured. For more information, see Configure cross-region access to KMS instances for applications.

Prepare access to the dedicated KMS instance

Before you create resources in the cluster, complete the OIDC (ACK) access configuration of the dedicated instance on the KMS side to obtain the Application Access Point, instance endpoint, and instance CA certificate that the add-on requires to access KMS.

An Application Access Point (AAP) is the access identity used by KMS for cloud-native access. It binds an authentication method (in this topic, OIDC (ACK), which trusts the JSON Web Token (JWT) of a specified cluster ServiceAccount) with a set of RBAC permissions. The add-on exchanges the ServiceAccount token of the cluster for temporary credentials with KMS, and then accesses the credentials in the instance with the identity and permissions of that AAP.

1. Log on to the Key Management Service console. In the left-side navigation pane, choose Application Access > Cloud-native Access, and then switch to Container > ACK.

2. In the cluster list, find the target cluster and click Configure ACK Access on the right side. Set Authentication Method to OIDC (ACK) and configure the following parameters to generate the Application Access Point:

   For the complete console operations of OIDC (ACK) access, including enabling RRSA OIDC and creating an AAP, see Quick access from ACK.

   Parameter	Value
   Namespace	Enter the fixed value ack-agent-identity.
   ServiceAccount	Enter the fixed value credential-provider.
   Scope	Select the specified KMS instance, that is, this dedicated instance.
   RBAC Permissions	Select CryptoServiceSecretUser, which allows reading credentials in the instance.
   Accessible Resources	Select the credentials and keys (for credential encryption and decryption) that the application needs to access.

   You must enter the preceding fixed values for Namespace and ServiceAccount. This is because the identity proven to KMS is the ack-agent-identity add-on (whose ServiceAccount is credential-provider in the ack-agent-identity namespace), not the ServiceAccount of the business Pod. The business Pod never connects to KMS directly.

3. After the configuration is complete, record the following information for creating the ExternalSecretService later:

   • Application Access Point (AAP): You can use its short name (name, such as my-aap) or the full ARN (such as acs:kms:cn-hangzhou:123456789012:applicationaccesspoint/my-aap).

   • Instance endpoint: the gateway domain name of the dedicated instance, in the format kst-hzz00example.cryptoservice.kms.aliyuncs.com.

   • Instance CA certificate: the CA certificate (PEM, public information) of the dedicated instance gateway, used to verify the TLS certificate of the instance gateway.

4. Create or confirm the credential that stores the API Key in KMS Secrets Manager, and record its credential name (secretName).

Configure KMS credential delivery

The following in-cluster resources (AgentIdentity, ExternalSecretService, CredentialProvider, AgentRole, and AgentRoleBinding) must be created in the same namespace.

1. Use ACS with kubectl.

2. Save the following content as kms-external-secret-service.yaml and run the kubectl apply -f kms-external-secret-service.yaml command to create the ExternalSecretService, which carries the access information of the dedicated KMS instance. Replace endpoint, aap.name, and caBundle with the actual values recorded in the previous step.

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: ExternalSecretService
   metadata:
     name: kms-dedicated-instance
     namespace: <YOUR-NAMESPACE>
   spec:
     provider: KMS
     kms:
       endpoint: kst-hzz00example.cryptoservice.kms.aliyuncs.com   # Endpoint of the dedicated instance gateway
       auth:
         type: AAP                                                 # AAP OIDC authentication
         aap:
           name: my-aap                                            # Short name of the AAP
       # CA certificate of the dedicated instance gateway. Required. Accepts base64-encoded PEM (such as `base64 -w0 ca.pem`) or raw PEM
       caBundle: REPLACE_WITH_BASE64_ENCODED_PEM_OF_YOUR_DEDICATED_KMS_INSTANCE_CA

   When an ExternalSecretService is created, it synchronously validates that caBundle is a valid PEM and probes the network connectivity of the endpoint. If caBundle is invalid or the instance endpoint is unreachable, the apply request is rejected. Make sure that the cluster can connect to the dedicated KMS instance (see the VPC sharing instructions in Applicability).

3. Save the following content as agent-identity.yaml and run the kubectl apply -f agent-identity.yaml command to create an AgentIdentity that defines the Agent identity.

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: AgentIdentity
   metadata:
     name: my-agent          # The Agent identity name, referenced in later authorization
     namespace: <YOUR-NAMESPACE>
   spec:
     description: "Sample AI Agent identity"

4. Save the following content as credential-provider-kms.yaml and run the kubectl apply -f credential-provider-kms.yaml command to create a CredentialProvider that references the preceding ExternalSecretService and specifies the KMS credential to read.

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: CredentialProvider
   metadata:
     name: llm-api-key
     namespace: <YOUR-NAMESPACE>
   spec:
     type: APIKey
     apiKey:
       source:
         provider: KMS
         kms:
           secretName: my-kms-secret-name        # Credential name in KMS Secrets Manager
           serviceRef:
             name: kms-dedicated-instance        # Name of the ExternalSecretService created in the previous step
           # versionStage: ACSCurrent            # Optional. Version stage of the credential. Default: ACSCurrent
           # versionId: ""                       # Optional. Specified version ID of the credential
           # maxCacheValidity: 15m               # Optional. Maximum cache validity of the credential value. Default: 15m

   secretName supports template variables that dynamically reference different KMS credentials based on context. For more information, see Template variables in secretName.

5. Save the following content as agent-role-kms.yaml and run the kubectl apply -f agent-role-kms.yaml command to create an AgentRole and an AgentRoleBinding that authorize the Agent identity to obtain credentials from this CredentialProvider.

   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: AgentRole
   metadata:
     name: get-llm-key
     namespace: <YOUR-NAMESPACE>
   spec:
     rules:
     - effect: Allow
       action: "GetResourceCredential"
       resource: "CredentialProvider/llm-api-key"   # Name of the CredentialProvider created in the previous step
   ---
   apiVersion: agentidentity.alibabacloud.com/v1alpha1
   kind: AgentRoleBinding
   metadata:
     name: my-agent-get-llm-key
     namespace: <YOUR-NAMESPACE>
   spec:
     agentRoleRef:
       apiGroup: agentidentity.alibabacloud.com
       kind: AgentRole
       name: get-llm-key
     subjects:
     - authorizationType: "Agent"
       agentAuthorizationConfiguration:
         agentName: my-agent                        # Must match the AgentIdentity name

6. Confirm that the resources are ready. Available of the ExternalSecretService indicates that the spec is valid and the network probe of the endpoint succeeded. Available of the CredentialProvider indicates that its serviceRef has been resolved to an existing ExternalSecretService.

   kubectl get externalsecretservice kms-dedicated-instance -n <YOUR-NAMESPACE>
   kubectl get credentialprovider llm-api-key -n <YOUR-NAMESPACE>

   Expected output:

   NAME                     PROVIDER   AVAILABLE   AGE
   kms-dedicated-instance   KMS        True        30s
   
   NAME          AVAILABLE   AGE
   llm-api-key   True        30s

   Available=True of the CredentialProvider only indicates that serviceRef has been resolved. It does not mean that the credential has been retrieved from KMS successfully. Issues such as insufficient AAP permissions, mismatched instance CA, nonexistent credentials, or disconnected VPC networks surface only when credentials are actually retrieved. For more information, see Status and troubleshooting.

Consume credentials

A created and authorized API Key credential is typically consumed through credential injection for Agent Sandbox egress traffic: reference this CredentialProvider in a tokenTransformation rule of a SecurityProfile. The Sandbox application initiates requests with a placeholder token, and the egress gateway automatically replaces it with the real API Key obtained from KMS when forwarding. For the complete end-to-end configuration (SecurityProfile, SandboxSet, SandboxClaim, and verification), see Configure credential injection for Agent Sandbox.

Template variables in secretName

In addition to a fixed credential name, secretName can also embed ${ack:agent-identity/...} template variables. When obtaining credentials, the system renders the actual credential name based on the identity context of the current request, thereby reading different KMS credentials per identity. For example, llm-api-key-${ack:agent-identity/agent-name} resolves to llm-api-key-my-agent based on the Agent name.

The following template variables are supported:

The following example demonstrates how to dynamically reference different KMS credentials per tenant: the API Keys of different tenants are stored in KMS credentials named llm-api-key-<tenant-id>, and the CredentialProvider resolves the corresponding credential name through ${ack:agent-identity/metadata/tenant-id} when obtaining credentials.

The metadata values come from the custom metadata written when the identity token was issued. In Agent Sandbox scenarios, they can be passed in through the security.agents.kruise.io/<key> annotation of a SandboxClaim (the part after removing the prefix is the metadata key):

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxClaim
metadata:
  name: my-claim
  namespace: <YOUR-NAMESPACE>
spec:
  templateName: my-sandbox-set
  replicas: 1
  annotations:
    # After removing the prefix, the following annotation can be referenced in templates via ${ack:agent-identity/metadata/tenant-id}
    security.agents.kruise.io/tenant-id: acme
  labels:
    security.agents.kruise.io/agent-name: my-agent

The corresponding CredentialProvider:

apiVersion: agentidentity.alibabacloud.com/v1alpha1
kind: CredentialProvider
metadata:
  name: llm-api-key
  namespace: <YOUR-NAMESPACE>
spec:
  type: APIKey
  apiKey:
    source:
      provider: KMS
      kms:
        secretName: 'llm-api-key-${ack:agent-identity/metadata/tenant-id}'   # Resolves to llm-api-key-acme
        serviceRef:
          name: kms-dedicated-instance

With the preceding configuration, when the tenant-id in the identity metadata of the request is acme, secretName resolves to llm-api-key-acme, reading the KMS credential dedicated to that tenant.

If a template variable cannot be resolved in the current request context (for example, the referenced metadata key does not exist), the credential retrieval request fails with an explicit error message instead of silently returning an empty value.

Field descriptions

spec:
  provider: KMS                      # Required. Type of the external credential service. Cannot be changed after creation
  kms:
    endpoint: <instance-endpoint>    # Required. Access domain name of the KMS instance
    auth:
      type: AAP                      # Required. Authentication method. In this topic, set it to AAP (OIDC)
      aap:
        arn: <aap-arn>               # Either arn or name. Full ARN of the AAP
        # name: <aap-name>           # Either arn or name. Short name of the AAP
    caBundle: <ca-pem-or-base64>     # Required. CA certificate of the dedicated instance gateway

spec:
  type: APIKey                       # Required. Credential type. Cannot be changed after creation
  apiKey:
    source:
      provider: KMS                  # Required. Credential source
      kms:
        secretName: <secret-name>    # Required. KMS credential name. Supports template variables
        serviceRef:
          name: <ess-name>           # Required. Name of the ExternalSecretService in the same namespace
        versionStage: ACSCurrent     # Optional. Version stage of the credential. Default: ACSCurrent
        versionId: <version-id>      # Optional. Specified version ID of the credential
        maxCacheValidity: 15m        # Optional. Maximum cache validity of the credential value. Default: 15m

Status and troubleshooting

The running status of both CRs is recorded in their respective status.conditions, which can be viewed with kubectl describe.

Common Conditions and Reasons of the ExternalSecretService:

Common Conditions and Reasons of the CredentialProvider (KMS source):

The reconciliation phase of the CredentialProvider only verifies whether serviceRef exists. It does not verify whether the credential can be retrieved from KMS successfully. Therefore, errors related to KMS access are not reflected in the CredentialProvider status. They surface only when credentials are actually retrieved. Common runtime errors and troubleshooting directions:

Limits

FAQ

Related documentation

• Configure credential injection for Agent Sandbox

• Quick access from ACK

• Upgrade the image version of a KMS instance

• Configure Multi-VPC access to KMS instances within the same region