Store API keys with RDS MySQL

Background

ack-sandbox-manager supports two API key storage backends controlled by the keyStorage.mode parameter: secret (default) and mysql. The Secret backend works for development, testing, and scenarios with a small number of keys. For multi-tenant or high-volume production environments, switch to RDS MySQL to remove the single-Secret size constraint.

Select a backend based on the total number of API keys in the cluster:

• Up to 500: Secret is sufficient.

• 501–1,000: Plan a migration to MySQL.

• More than 1,000: Use MySQL to avoid write failures as the Secret approaches its 1 MiB size limit.

Prerequisites

• Agent Sandbox multi-tenant management is configured with ack-sandbox-manager (v0.6.7 or later) installed and API key authentication enabled. For more information, see Multi-tenancy management.

Create and configure an RDS MySQL instance

API key storage is a lightweight Online Transaction Processing (OLTP) workload with low data volume and hash-based point queries. Expand the following section for sizing recommendations before purchasing an RDS MySQL instance.

1. Go to the RDS purchase page and create an RDS MySQL instance. Select the same region and VPC as the Container Compute Service cluster and refer to the sizing recommendations above.

2. Create a database and an account. On the Databases and Accounts pages of the RDS instance, complete the following steps. For more information, see Create an account.

   1. Create a database, for example e2b.

   2. Create a standard database account and record the username and password.

   3. Grant Read/Write (DDL + DML) permissions on the target database to this account.

3. Configure the IP allowlist.

   Control plane add-ons such as sandbox-manager send traffic from the control plane vSwitches. Add the control plane vSwitch Classless Inter-Domain Routing (CIDR) blocks to the RDS allowlist.

   1. Log on to the Container Compute Service console and navigate to the Basic Information page of the target cluster.

   2. In the Network section, click Edit next to Control Plane vSwitch and copy the CIDR blocks of all selected vSwitches.

   3. Log on to the RDS console and navigate to the Whitelist and SecGroup page of the target instance.

   4. Edit the default allowlist group and add the control plane vSwitch IPv4 CIDR blocks.

4. On the database connection page of the RDS instance, record the Internal Endpoint and Internal Port, for example:

   rm-******************.mysql.rds.aliyuncs.com:3306

5. Manually initialize the table schema. Run the mysql-schema.sql script provided by the OpenKruise community:

   mysql -h <RDS_internal_endpoint> \
     -P <port> \
     -u <database_account> \
     -p \
     <database_name> < mysql-schema.sql

   This script creates the teams and team_api_keys tables and initializes the built-in admin team. The admin API key hash does not need to be migrated. ack-sandbox-manager automatically computes and writes it to the team_api_keys table based on the current hashPepper at startup.

Configure the MySQL storage backend

1. Log on to the Container Compute Service console and navigate to the target cluster.

2. On the Add-ons page, find ack-sandbox-manager and click Configuration.

3. Configure the following parameters:

   Parameter	Value or description
   keyStorage.mode	Set to mysql.
   keyStorage.mysql.dsn	MySQL Data Source Name (DSN) that specifies the account, password, endpoint, port, and connection parameters for the target database. See the format below.
   keyStorage.mysql.hashPepper	Long-term secret used to compute API key hashes.

4. Enter the DSN in the following format:

   <account>:<password>@tcp(<RDS_internal_endpoint>:<port>)/<database_name>?charset=utf8mb4&parseTime=true&loc=Local

   Example:

   e2b_user:********@tcp(rm-******************.mysql.rds.aliyuncs.com:3306)/e2b?charset=utf8mb4&parseTime=true&loc=Local

   Generate a 32-byte (256-bit) random string as the hashPepper:

   openssl rand -hex 32

   The output is a 64-character hexadecimal string. Use it directly as the value of keyStorage.mysql.hashPepper.

   Important

   Store the hashPepper securely for the long term. Do not set it to the same value as the database password. Changing the hashPepper invalidates all existing tenant API keys, which must then be reissued.

5. Save the configuration. ack-sandbox-manager automatically restarts and connects to RDS MySQL.

   When keyStorage.mode is set to mysql, the add-on fails to start if the DSN or hashPepper is not configured.

Verify the configuration

1. Confirm that the ack-sandbox-manager pods are running:

   kubectl get pods -n sandbox-system -l component=sandbox-manager

   Expected output:

   NAME                             READY   STATUS    RESTARTS   AGE
   sandbox-manager-79b7449778-xxx   2/2     Running   0          78m
   sandbox-manager-79b7449778-yyy   2/2     Running   0          79m
   sandbox-manager-79b7449778-zzz   2/2     Running   0          78m

2. Create, query, and delete a temporary API key to verify that the MySQL backend can read and write data. Replace the following variables with actual values before running the script:

   • BASE_URL: External endpoint of ack-sandbox-gateway.

   • ADMIN_KEY: Admin API key configured in ack-sandbox-manager.

   The following script requires jq (a command-line JSON processor) to parse responses. Verify that jq is installed locally before running the script.

   BASE_URL="https://api.your.domain.com"
   ADMIN_KEY="your-admin-key"
   
   TEST_KEY_ID=$(
     curl -fsS -X POST \
       -H "X-API-KEY: ${ADMIN_KEY}" \
       -H "Content-Type: application/json" \
       -d '{"name":"mysql-storage-verification"}' \
       "${BASE_URL}/api-keys" |
       jq -r '.id'
   )
   echo "Created: ${TEST_KEY_ID}"
   
   curl -fsS \
     -H "X-API-KEY: ${ADMIN_KEY}" \
     "${BASE_URL}/api-keys" |
     jq -e --arg id "${TEST_KEY_ID}" \
       'any(.[]; .id == $id)' >/dev/null && echo "Query OK"
   
   curl -fsS -X DELETE \
     -H "X-API-KEY: ${ADMIN_KEY}" \
     "${BASE_URL}/api-keys/${TEST_KEY_ID}" && echo "Deleted: ${TEST_KEY_ID}"

   Expected output:

   Created: 4f2ab49a-aced-4830-b5da-5d382xxxxxxx
   Query OK
   Deleted: 4f2ab49a-aced-4830-b5da-5d382xxxxxxx

Migrate existing API keys (optional)

After switching to the MySQL backend, use the migrate_secret_keys_to_mysql.py script provided by the OpenKruise community to generate the migration SQL for API keys stored in the original Secret backend. The migration script:

• Reads the e2b-key-store Secret by using kubectl.

• Computes the HMAC-SHA256 hash of each API key using the same value as keyStorage.mysql.hashPepper.

• Generates data definition language (DDL) statements for table creation and UPSERT statements for teams and API keys.

• Preserves the original teams, key names, creation metadata, and quota configurations.

The script requires only the Python 3 standard library and kubectl.

1. Validate the Secret data:

   python3 migrate_secret_keys_to_mysql.py \
     --namespace sandbox-system \
     --dry-run

   This command parses all entries in the Secret, validates fields such as id, name, and key, checks team metadata consistency, and verifies key uniqueness. Any validation failure aborts the process to prevent generating incomplete SQL.

2. Set the hashPepper through hidden input (the value must exactly match the configured keyStorage.mysql.hashPepper) and generate the migration SQL:

   read -rsp "Enter hashPepper: " E2B_KEY_HASH_PEPPER
   echo
   export E2B_KEY_HASH_PEPPER
   
   python3 migrate_secret_keys_to_mysql.py \
     --namespace sandbox-system \
     --output e2b_key_migration.sql

3. Import the migration SQL into RDS MySQL:

   mysql -h <RDS_internal_endpoint> \
     -P <port> \
     -u <database_account> \
     -p \
     <database_name> < e2b_key_migration.sql

4. Clean up environment variables and migration files:

   unset E2B_KEY_HASH_PEPPER
   rm -f e2b_key_migration.sql