Confidential inference service

1. Prepare the model (optional)

Use ModelScope to download the Qwen2.5-3B-Instruct model. Python 3.9 or later is required.

pip3 install modelscope importlib-metadata
modelscope download --model Qwen/Qwen2.5-3B-Instruct

The model is downloaded to the ~/.cache/modelscope/hub/models/Qwen/Qwen2.5-3B-Instruct/ directory.

2. Encrypt the model

Two encryption methods are available. This example uses Sam.

• Gocryptfs: AES-256-GCM encryption compatible with the open source Gocryptfs standard.

• Sam: An Alibaba Cloud trusted AI model encryption format that prevents unauthorized use and license tampering.

3. Upload the model

Various storage backends are supported for encrypted models. EAS mounts the decrypted model to the service instance during deployment. For more information, see Storage mounts.

This example uses Object Storage Service (OSS). Follow the instructions in Console quick start to create a bucket and a directory named qwen-encrypted, for example, oss://examplebucket/qwen-encrypted/. For large model files, use ossbrowser to upload the encrypted model to the directory.

An uploaded Sam-encrypted model is shown below. With Gocryptfs, filenames are also encrypted and appear as garbled text.

4. Set up the attestation service and upload the key

The remote attestation service hosts the decryption key and releases it only after EAS passes trust verification.

Deploy the self-managed Trustee remote attestation service on ACK Serverless with KMS as the key storage backend.

1. Create a KMS instance as the key storage backend.

   1. Go to the KMS console. In the left-side navigation pane, choose Resource > Instances. On the Software Key Management tab, create and enable an instance. When you enable the instance, select the same Virtual Private Cloud (VPC) as the ACK cluster. For more information, see Purchase and enable a KMS instance.

      Activation takes about 10 minutes.

   2. After the instance is enabled, in the left-side navigation pane, choose Resource > Keys. On the Keys page, create a customer master key (CMK) for the instance. For more information, see Step 1: Create a software key.

   3. In the left-side navigation pane, choose Application Access > Multi-Cloud Access (formerly AAP). On the Application Access tab, create an application access point for the instance. For Scope, select the KMS instance that you created. For more configuration details, see Method 1: Quick creation.

      After the application access point is created, your browser automatically downloads a ClientKey*.zip file. This file contains:

      • Credential (ClientKeyContent): The default filename is clientKey_****.json.

      • Credential Password (ClientKeyPassword): The default filename is clientKey_****_Password.txt.

   4. Go to the Resource > Instances page and click the name of the KMS instance. On the Details page, click Instance CA Certificate next to Download to export the public key certificate file PrivateKmsCA_***.pem.

2. Create an ACK Serverless cluster and install the csi-provisioner component.

   1. Go to the Create cluster page to create an ACK Serverless cluster. For more information, see Create a cluster. Key parameters:

      1. Cluster configurations: Configure the following parameters and click Next: Component configurations.

         Parameter	Description
         VPC	Select Use Existing and select the Configure SNAT for VPC checkbox. This is required to pull the Trustee image.
         vSwitch	Ensure at least two vSwitches exist in the VPC. Otherwise, a public-facing ALB cannot be exposed.

      2. Component configurations: Configure the following parameters and click Next: Confirm configuration.

         Parameter	Description
         Service Discovery	Select CoreDNS.
         Ingress	Select ALB Ingress. For the ALB cloud-native gateway instance source, select New, and select two vSwitches.

      3. Confirm configuration: Confirm the configuration information and terms of service, and then click Create Kubernetes Cluster.

   2. After the cluster is created, install the csi-provisioner (Managed) component. For more information, see Manage components.

3. Deploy the Trustee remote attestation service in the ACK cluster.

   1. Connect to the cluster over the public network or an internal network. For more information, see Connect to a cluster.

   2. Upload the KMS credential (clientKey_****.json), password (clientKey_****_Password.txt), and CA certificate (PrivateKmsCA_***.pem) to the ACK Serverless cluster environment. Then run the following command to deploy Trustee with KMS as the key backend.

      # Install the plugin
      helm plugin install https://github.com/AliyunContainerService/helm-acr
      
      helm repo add trustee acr://trustee-chart.cn-hangzhou.cr.aliyuncs.com/trustee/trustee
      helm repo update
      
      export DEPLOY_RELEASE_NAME=trustee
      export DEPLOY_NAMESPACE=default
      export TRUSTEE_CHART_VERSION=1.7.6
      
      # Set the region of the ACK cluster, for example, cn-hangzhou
      export REGION_ID=cn-hangzhou
      
      # Information about the KMS instance that you exported
      # Replace with your KMS instance ID
      export KMS_INSTANCE_ID=kst-hzz66a0*******e16pckc
      # Replace with the path to your KMS instance application identity credential
      export KMS_CLIENT_KEY_FILE=/path/to/clientKey_KAAP.***.json
      # Replace with the path to your KMS instance credential password
      export KMS_PASSWORD_FILE=/path/to/clientKey_KAAP.***_Password.txt
      # Replace with the path to your KMS instance CA certificate
      export KMS_CERT_FILE=/path/to/PrivateKmsCA_kst-***.pem
      
      helm install ${DEPLOY_RELEASE_NAME} trustee/trustee \
        --version ${TRUSTEE_CHART_VERSION} \
        --set regionId=${REGION_ID} \
        --set kbs.aliyunKms.enabled=true \
        --set kbs.aliyunKms.kmsIntanceId=${KMS_INSTANCE_ID} \
        --set-file kbs.aliyunKms.clientKey=${KMS_CLIENT_KEY_FILE} \
        --set-file kbs.aliyunKms.password=${KMS_PASSWORD_FILE} \
        --set-file kbs.aliyunKms.certPem=${KMS_CERT_FILE} \
        --namespace ${DEPLOY_NAMESPACE}

      Note

      helm plugin install... may be slow. If it fails, run helm plugin uninstall cm-push and retry.

      Example output:

      NAME: trustee
      LAST DEPLOYED: Tue Feb 25 18:55:33 2025
      NAMESPACE: default
      STATUS: deployed
      REVISION: 1
      TEST SUITE: None

   3. Get the Trustee service endpoint.

      export TRUSTEE_URL=http://$(kubectl get AlbConfig alb-$DEPLOY_RELEASE_NAME -o jsonpath='{.status.loadBalancer.dnsname}')/api
      echo ${TRUSTEE_URL}

      Example output: http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/api.

   4. Test the connectivity of the Trustee service.

      cat << EOF | curl -k -X POST ${TRUSTEE_URL}/kbs/v0/auth -H 'Content-Type: application/json' -d @-
      {
          "version":"0.4.0",
          "tee": "tdx",
          "extra-params": "foo"
      }
      EOF

      A successful connection returns output similar to:

      {"nonce":"PIDUjUxQdBMIXz***********IEysXFfUKgSwk=","extra-params":""}

4. Configure a network whitelist for Trustee.

   Note

   This allows the EAS deployment environment to access Trustee.

   1. Go to the ALB console, create an access control policy group, and add the IP addresses or address ranges that require access to Trustee. For more information, see Access control. Add the following address ranges:

      • The public IP address of the VPC where the EAS service is deployed.

      • The egress IP address of the inference client.

   2. Run the following command to get the ALB instance ID used by the Trustee instance.

      kubectl get ing --namespace ${DEPLOY_NAMESPACE} frontend-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | cut -d'.' -f1 | sed 's/[^a-zA-Z0-9-]//g'

      Expected output:

      alb-llcdzbw0qivhk0****

   3. In the left-side navigation pane of the ALB console, choose ALB > Instances. Find the ALB instance from the previous step in your cluster's region and click its ID to go to the details page. At the bottom of the page, in the Instance Information section, turn off Configuration Read-only Mode.

   4. Switch to the Listener tab. In the access control column of the target listener, enable access control and set the whitelist to the access control policy group that you created.

5. Create a secret to store the model decryption key.

   Trustee stores the model decryption key in KMS. The key is released only after attestation succeeds.

   Go to the KMS console. In the left-side navigation pane, choose Resource > Secrets. On the Generic Secrets tab, click Create Secret. Key parameters are described below:

   • Secret name: A custom name used to index the key, for example, model-decryption-key.

   • Secret value: Enter the key that you used to encrypt the model. For example, 0Bn4Q1wwY9fN3P. Use your actual key.

   • Encryption key: Select the customer master key (CMK) that you created.

6. Log on to the Trustee management UI to view key access history.

   1. Get the address of the Trustee frontend management UI.

      kubectl get AlbConfig alb-$DEPLOY_RELEASE_NAME -o jsonpath='{.status.loadBalancer.dnsname}'

      Access the address in a browser.

   2. Get the private key for logging on to the Trustee frontend management UI.

      kubectl get secret kbs-auth-keypair -o jsonpath="{.data.private\.key}" | base64 -d

      Paste the key into the UI to log on.

   3. On the Trustee Gateway Management Platform, click Audit log in the left-side navigation pane. Then, click the Resource audit interface to view the history of model decryption key access.