Use the Queryable Encryption feature in MongoDB 7.0

Background information

The transparent data encryption (TDE) and disk encryption features provided by ApsaraDB for MongoDB are encryption at rest solutions. They serve the following purposes:

• Data protection: Protects data on disks from unauthorized access. Even if a malicious user has physical access to the HDDs or SSDs on which data is stored, the user cannot access the unencrypted data.

• Leak prevention: If a storage device is stolen or lost, encryption ensures that sensitive data cannot be accessed by unauthorized users.

• Compliance requirements: Multiple industry standards and regulations require enterprises to encrypt sensitive data, such as personal user data and financial information. Encryption at rest helps enterprises meet these regulatory requirements.

With encryption at rest, data read into memory remains in plaintext. To fully protect your data, we recommend that you implement additional security measures such as network encryption (SSL/TLS), database access control, auditing, and monitoring. To prevent security risks from internal O&M access to Elastic Compute Service (ECS) instances that host your database services, Alibaba Cloud provides customer authorization and mandatory auditing.

If you require additional encryption beyond encryption at rest, you can use the Queryable Encryption feature officially released in MongoDB 7.0.

Introduction

Queryable Encryption was previewed in MongoDB 6.0 and officially released in MongoDB 7.0.

Queryable Encryption keeps data encrypted until it reaches the client. Queries are sent to the server along with the encryption key managed by Key Management Service (KMS). The server queries and returns data in ciphertext. After the data is returned to the client, it is decrypted by using the key and displayed in plaintext.

Queryable Encryption provides the following capabilities:

• Encrypts sensitive data from the client side. Only the client can obtain the encryption key.

• Encrypts data throughout the entire lifecycle, including transmission, storage, usage, auditing, and backup.

• Supports expressive queries on encrypted data, including equality, range, prefix, suffix, and substring queries.

• Strengthens data privacy. Only authorized users with access to the server application and encryption keys can view data in plaintext.

• Simplifies application development. Developers can use the built-in encryption capabilities of the database to ensure security and compliance.

• Reduces security concerns when storing sensitive data in ApsaraDB for MongoDB.

MongoDB Community Edition does not support automatic encryption, which differentiates it from Enterprise Edition (Atlas).

For more information about driver versions and encrypted database versions, see Queryable Encryption Compatibility.

Limits

• Diagnostic commands and query logs on an encrypted collection are redacted or hidden, which limits problem analysis:

  • Commands applicable to an encrypted collection, such as aggregate, count, find, insert, update, and delete, are not recorded in slow query logs and profilers.

  • The results of diagnostic commands such as collStats, currentOp, top, or $planCacheStats are redacted and some fields are hidden.

• Contention among encrypted fields may increase write latency. The default contention factor is 8.

• Metadata collections that exceed 1 GB in size must be manually compacted. For more information, see Metadata Collection Compaction.

• The encryptedFieldsMap object cannot be changed, including the query fields in the object.

• Queryable Encryption is supported only by replica set or sharded cluster instances.

• Encrypted data on secondary nodes cannot be read.

• Documents cannot be updated in a batch by running the updateMany or bulkWrite command, and parameters in the findAndModify command are limited.

• The upsert operation is not supported. When upsert is triggered, encrypted fields are not inserted.

• Client-Side Field Level Encryption (CSFLE) and Queryable Encryption cannot be enabled on the same collection. A collection with CSFLE enabled or an unencrypted collection cannot be converted to a Queryable Encryption collection.

• Queryable Encryption can be enabled only on new empty collections.

• A collection that contains encrypted fields cannot be renamed, and the fields cannot be renamed by running the $rename command.

• If jsonSchema is specified when an encrypted collection is created, the encrypt keyword cannot be included.

• Views, time series collections, and capped collections are not supported.

• TTL indexes or unique indexes are not supported.

• jsonSchema cannot be closed.

• A collection must be deleted by using a MongoClient with Queryable Encryption enabled. Otherwise, metadata remains on the server.

• Queryable Encryption does not support collation. Collation blocks normal sorting for encrypted fields.

• The _id field cannot be specified as an encrypted field.

• Only a limited number of commands and operators are supported. For more information, see Supported Operations for Queryable Encryption.

Preparations

This guide uses an ECS instance as the client to demonstrate the procedure. If your test environment already has the necessary dependencies, you can skip the corresponding steps. This guide uses the Node.js driver for the demonstration because mongosh supports only automatic encryption, while MongoDB Community Edition supports only explicit encryption.

1. Install Node.js and npm.

   curl -fsSL https://rpm.nodesource.com/setup_lts.x | sudo bash -
   sudo yum install nodejs
   node -v
   npm -v

2. Install the official Node.js driver for MongoDB.

   mkdir node_quickstart
   cd node_quickstart
   npm init -y
   npm install mongodb@6.6

3. Install the libmongocrypt library.

   vi /etc/yum.repos.d/libmongocrypt.repo
   // Add the following content to the file:
   [libmongocrypt]
   name=libmongocrypt repository
   baseurl=https://libmongocrypt.s3.amazonaws.com/yum/redhat/8/libmongocrypt/1.8/x86_64
   gpgcheck=1
   enabled=1
   gpgkey=https://pgp.mongodb.com/libmongocrypt.asc
   // Install the library
   sudo yum install -y libmongocrypt

4. Install the mongodb-client-encryption package on which the Node.js driver depends.

   sudo yum groupinstall 'Development Tools'
   npm install mongodb-client-encryption

5. Install mongosh and configure the MONGODB_URI environment variable.

   wget https://repo.mongodb.org/yum/redhat/8/mongodb-org/7.0/x86_64/RPMS/mongodb-mongosh-2.2.5.x86_64.rpm
   yum install -y ./mongodb-mongosh-2.2.5.x86_64.rpm
   export MONGODB_URI="mongodb://root:xxxxxx@dds-2zef23cef14b4f142.mongodb.pre.rds.aliyuncs.com:3717,dds-2zef23cef14b4f141.mongodb.pre.rds.aliyuncs.com:3717/admin?replicaSet=mgset-855706"
   // Test the connection
   mongosh ${MONGODB_URI}

6. Obtain the automatically encrypted shared library.

   Select the client that corresponds to your machine and distribution version in Download Center, and select the crypt_shared package. For more information, see MongoDB Enterprise Server Download.

   // Decompress the local directory to obtain the lib/mongo_crypt_v1.so file.
   tar -xzvf mongo_crypt_shared_v1-linux-x86_64-enterprise-rhel80-7.0.9.tgz

Procedure

Enter the Node.js REPL environment and then perform the following steps:

node -i -e "const MongoClient = require('mongodb').MongoClient; const ClientEncryption = require('mongodb').ClientEncryption;"

1. Create a customer master key (CMK).

   Note

   The following example shows the sample configurations of a local KMS provider. We recommend that you do not use the configurations in the production environment.

   Create a 96-byte CMK and store the CMK in the customer-master-key.txt file of the local file system.

   const fs = require("fs");
   const crypto = require("crypto");
   try {
     fs.writeFileSync("customer-master-key.txt", crypto.randomBytes(96));
   } catch (err) {
     console.error(err);
   }

   This example generates the CMK using a Node.js crypto call. Alternatively, you can generate the 96-byte CMK in a shell using /dev/urandom:

   echo $(head -c 96 /dev/urandom | base64 | tr -d '\n')

2. Initialize variables.

   // KMS provider name should be one of the following: "aws", "gcp", "azure", "kmip" or "local"
   const kmsProviderName = "local";
   const uri = process.env.MONGODB_URI;
   const keyVaultDatabaseName = "encryption";
   const keyVaultCollectionName = "__keyVault";
   const keyVaultNamespace = "encryption.__keyVault";
   const encryptedDatabaseName = "medicalRecords";
   const encryptedCollectionName = "patients";

   In the preceding sample code, the following variables are initialized:

   • kmsProviderName: the name of the KMS provider. In this example, local is used.

   • uri: the MongoDB URI. The MongoDB URI can be specified by the MONGODB_URI environment variable. You can also specify the MongoDB URI.

   • keyVaultDatabaseName: the name of the database that stores data encryption keys (DEKs).

   • keyVaultCollectionName: the name of the collection that stores DEKs. The collection must be different from a regular collection.

   • keyVaultNamespace: equal to the keyVaultDatabaseName or keyVaultCollectionName variable.

   • encryptedDatabaseName: the name of the database that stores encrypted data.

   • encryptedCollectionName: the name of the collection that stores encrypted data.

3. Create a unique index on the collection that stores DEKs.

   const keyVaultClient = new MongoClient(uri);
   await keyVaultClient.connect();
   const keyVaultDB = keyVaultClient.db(keyVaultDatabaseName);
   // Delete the database with the same name as the database that stores DEKs to prevent excess data.
   await keyVaultDB.dropDatabase();
   const keyVaultColl = keyVaultDB.collection(keyVaultCollectionName);
   await keyVaultColl.createIndex(
     { keyAltNames: 1 },
     {
       unique: true,
       partialFilterExpression: { keyAltNames: { $exists: true } },
     }
   );
   // double check
   await keyVaultColl.indexes();

4. Create an encrypted collection.

   1. Obtain the created CMK and specify the KMS provider.

      const localMasterKey = fs.readFileSync("./customer-master-key.txt");
      kmsProviders = {local: {key: localMasterKey}};

   2. Create a DEK.

      Note

      Before you perform this step, make sure that the user specified in the uri variable has the dbAdmin permission on the encryption._keyVault and medicalRecords databases.

      const clientEnc = new ClientEncryption(keyVaultClient, {
        keyVaultNamespace: keyVaultNamespace,
        kmsProviders: kmsProviders,
      });
      const dek1 = await clientEnc.createDataKey(kmsProviderName, {
        keyAltNames: ["dataKey1"],
      });
      const dek2 = await clientEnc.createDataKey(kmsProviderName, {
        keyAltNames: ["dataKey2"],
      });

   3. Specify the fields to be encrypted and configure the created DEK.

      const encryptedFieldsMap = {
        [`${encryptedDatabaseName}.${encryptedCollectionName}`]: {
          fields: [
            {
              keyId: dek1,
              path: "patientId",
              bsonType: "int",
              queries: { queryType: "equality" },
            },
            {
              keyId: dek2,
              path: "medications",
              bsonType: "array",
            },
          ],
        },
      };

   4. Specify the automatically encrypted shared library and create a MongoClient.

      const extraOptions = {cryptSharedLibPath: "/root/lib/mongo_crypt_v1.so"};
      const encClient = new MongoClient(uri, {
        autoEncryption: {
          keyVaultNamespace,
          kmsProviders,
          extraOptions,
          encryptedFieldsMap,
        },
      });
      await encClient.connect();

   5. Create an encrypted collection.

      const newEncDB = encClient.db(encryptedDatabaseName);
      await newEncDB.dropDatabase();
      await newEncDB.createCollection(encryptedCollectionName);

5. Create a MongoClient that is used to encrypt read and write operations.

   1. Specify the collection that stores the created DEK.

      const eDB = "encryption";
      const eKV = "__keyVault";
      const keyVaultNamespace = `${eDB}.${eKV}`;
      const secretDB = "medicalRecords";
      const secretCollection = "patients";

   2. Specify the created CMK.

      Important

      Do not use the local key file in the production environment.

      const fs = require("fs");
      const path = "./customer-master-key.txt";
      const localMasterKey = fs.readFileSync(path);
      const kmsProviders = {
        local: {
          key: localMasterKey,
        },
      };

   3. Obtain the created DEK.

      Note

      The DEK name must be the same as the name of the DEK created in the secondary substep of Step 4.

      const uri = process.env.MONGODB_URI;;
      const unencryptedClient = new MongoClient(uri);
      await unencryptedClient.connect();
      const keyVaultClient = unencryptedClient.db(eDB).collection(eKV);
      const dek1 = await keyVaultClient.findOne({ keyAltNames: "dataKey1" });
      const dek2 = await keyVaultClient.findOne({ keyAltNames: "dataKey2" });

   4. Specify the automatically encrypted shared library and create a MongoClient.

      const extraOptions = {
        cryptSharedLibPath: "/root/lib/mongo_crypt_v1.so",
      };
      const encryptedClient = new MongoClient(uri, {
        autoEncryption: {
          kmsProviders: kmsProviders,
          keyVaultNamespace: keyVaultNamespace,
          bypassQueryAnalysis: true,
          keyVaultClient: unencryptedClient,
          extraOptions: extraOptions,
        },
      });
      await encryptedClient.connect();

   5. Create a ClientEncryption object.

      const encryption = new ClientEncryption(unencryptedClient, {
        keyVaultNamespace,
        kmsProviders,
      });

6. Insert a document that contains encrypted fields into the created encrypted collection.

   const patientId = 12345678;
   const medications = ["Atorvastatin", "Levothyroxine"];
   const indexedInsertPayload = await encryption.encrypt(patientId, {
     algorithm: "Indexed",
     keyId: dek1._id,
     contentionFactor: 1,
   });
   const unindexedInsertPayload = await encryption.encrypt(medications, {
     algorithm: "Unindexed",
     keyId: dek2._id,
   });
   const encryptedColl = encryptedClient.db(secretDB).collection(secretCollection);
   await encryptedColl.insertOne({
     firstName: "Jon",
     patientId: indexedInsertPayload,
     medications: unindexedInsertPayload,
   });

7. Perform a field-level query on the created encrypted collection.

   const findPayload = await encryption.encrypt(patientId, {
     algorithm: "Indexed",
     keyId: dek1._id,
     queryType: "equality",
     contentionFactor: 1,
   });
   console.log(await encryptedColl.findOne({ patientId: findPayload }));

   The following example shows the returned document.

   > console.log(await encryptedColl.findOne({ patientId: findPayload }));
   {
     _id: new ObjectId('6645b56f58abf955ebd95caf'),
     firstName: 'Jon',
     patientId: 12345678,
     medications: [ 'Atorvastatin', 'Levothyroxine' ],
     __safeContent__: [
       Binary.createFromBase64('IrPf972hlhDvnasQH6rIAW6BqERo0ZEgC6C0/zNQiIY=', 0)
     ]
   }

8. Use the client that contains encrypted options to access encrypted fields. Otherwise, the encrypted fields cannot be accessed.

   Use the unencryptedClient client that is unencrypted for the field-level query.

   console.log(await unencryptedClient.db(secretDB).collection(secretCollection).findOne());

   The following example shows the returned document. The sensitive fields appear as binary data.

   > console.log(await unencryptedClient.db(secretDB).collection(secretCollection).findOne());
   {
     _id: new ObjectId('6645b56f58abf955ebd95caf'),
     firstName: 'Jon',
     patientId: Binary.createFromBase64('DtmNrEDyTEBDidZxWkbGU/MQdYNxwmnqYj5tSr9uhHbwWj8bsSD3TWlZ8aMMvw6FY00cmdc1QLLoEX3NwlKRhz0zax9LcQhN3vKUf4eq3hAfBYWkyOQxsiwbPsU0AiXnMV+qM6J2p2JZGLrDvxfbTY+obBmRqdvlgJ51dKmYopvDNToWBXDQkqAis9/3vaGWE0+dqxAfqsgBboGoRGjRkSALoI7/M1CIhvmpds5LR7/232uI4f5QDbk0JVfjnI0Doov6b0GrAXe9', 6),
     medications: Binary.createFromBase64('EKqAnwBkWUBon0Qf9sZHVIkEphUdfDK/aqYPs5M1Xc58CkojwX0kvC+KjwYyEozia41F5cnD9NFBwnVuDJUaqjTLc1YwG1DEIUZdcYCMf3JiureqA0voYP3gZxPyFmf/h1DS80Jz+g', 6),
     __safeContent__: [
       Binary.createFromBase64('IrPf972hlhDvnasQH6rIAW6BqERo0ZEgC6CO/zNQiIY=', 0)
     ]
   }

   You can also use mongosh to externally access the encrypted fields. This simulates access to the created encrypted collection without the client key.

   // In a separate terminal session, connect directly to the MongoDB URI with mongosh.
   mongosh ${MONGODB_URI}
   db.getSiblingDB("medicalRecords").patients.findOne()

   The following example shows the returned document.

   mgset-855706 [primary] admin> db.getSiblingDB("medicalRecords").patients.findOne()
   {
     _id: ObjectId('6645b56f58abf955ebd95caf'),
     firstName: 'Jon',
     patientId: Binary.createFromBase64('DtmNrEDyTEBYDidZxWkbGU/MQdYNxwmnqYj5tSr9uhHbwWj8bsSD3TWlZ8aMMvw6fYO0cmdc1QLoEX3Nw1KRhz0zax9LcQhN3vKUf4eq3hAfBYWkyQxsiwbPsU0AiXnMV+qM6JZGLrDvxfbTY+obBmRqdvlJ51dKmYopvDNTowBXdQkqAis9/3vaGWEO+dqxAfqsgBboGoRGjRkSALoI7/M1CIhvmpds5LR7/232uI4f5QDbk0JVfjnIODoov6b0GrAXe9', 6),
     medications: Binary.createFromBase64('EKqAnwBkWUBon0Qf9sZHVIkEphUdfdk/aqYPs5M1Xc58CkojX0kvC+KjwYyEozia41F5cnD9NFBwnVuDJUaqjTLc1YwG1DEIUZdcYMf3liuregA0qvoYP3qZxPyFmf/h1IJz+g', 6),
     __safeContent__: [
       Binary.createFromBase64('IrPf972hlhDvnasQH6rIAW6BqERo0ZEgC6CO/zNQiIY=', 0)
     ]
   }

References

• use CSFLE with node.js

• Node.js Driver: Download & install

• QE: basic

• QE:explicit encryption

• QE: shared library

• QE: libmongocrypt

• QE: limitations

• QE: supported operations