Use event-driven scaling with a MongoDB event source

ScaledJob vs. ScaledObject

Use ScaledJob when each event maps to a discrete, long-running task that runs to completion in its own pod. ack-keda schedules one Job per event: it initializes, processes the event, and terminates. This isolation prevents slow tasks from blocking others and enables precise concurrency control with maxReplicaCount.

Use ScaledObject when events drive continuous throughput and your workload is best handled by a pool of long-running Deployment replicas.

The tutorial below uses ScaledJob for video transcoding. When a record with "state":"waiting" is inserted into MongoDB, ack-keda creates a Job to process the task and sets the record to "state":"finished" on completion. Finished Jobs are cleaned up automatically.

Create example namespaces

This example uses the mongodb namespace for the database and mongodb-test for autoscaling configurations.

kubectl create ns mongodb
kubectl create ns mongodb-test

Deploy MongoDB

If you already have a MongoDB service, skip this step.

1. Create mongoDB.yaml.

   Important

   This MongoDB service is for demonstration only without high availability. Do not use it in production.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mongodb
     namespace: mongodb
   spec:
     replicas: 1
     selector:
       matchLabels:
         name: mongodb
     template:
       metadata:
         labels:
           name: mongodb
       spec:
         containers:
         - name: mongodb
           image: registry-cn-shanghai.ack.aliyuncs.com/acs/mongo:v5.0.0
           imagePullPolicy: IfNotPresent
           ports:
           - containerPort: 27017
             name: mongodb
             protocol: TCP
   ---
   kind: Service
   apiVersion: v1
   metadata:
     name: mongodb-svc
     namespace: mongodb
   spec:
     type: ClusterIP
     ports:
     - name: mongodb
       port: 27017
       targetPort: 27017
       protocol: TCP
     selector:
       name: mongodb

2. Deploy MongoDB.

   kubectl apply -f mongoDB.yaml

Initialize the MongoDB database

1. Get the MongoDB pod name.

   MONGO_POD_NAME=$(kubectl get pods -n mongodb -l name=mongodb -o jsonpath='{.items[0].metadata.name}')
   echo "MongoDB pod name: $MONGO_POD_NAME"

2. In the test database, create user test_user and collection test_collection.

   # Create user
   kubectl exec -n mongodb ${MONGO_POD_NAME} -- mongo --eval 'db.createUser({ user:"test_user",pwd:"test_password",roles:[{ role:"readWrite", db: "test"}]})'
   
   # Authenticate user
   kubectl exec -n mongodb ${MONGO_POD_NAME} -- mongo --eval 'db.auth("test_user","test_password")'
   
   # Create collection
   kubectl exec -n mongodb ${MONGO_POD_NAME} -- mongo test --eval 'db.createCollection("test_collection")'

Configure TriggerAuthentication and ScaledJob

ack-keda uses TriggerAuthentication to securely manage event source credentials. ScaledJob defines scaling rules, polling intervals, and the Job template.

The following file combines all three resources—Secret, TriggerAuthentication, and ScaledJob.

1. Create keda-mongodb.yaml. The secretTargetRef in TriggerAuthentication reads the connection string from the Secret for MongoDB authentication. The query in ScaledJob triggers Job creation when documents in test_collection match {"type":"mp4","state":"waiting"}.

   apiVersion: v1
   kind: Secret
   metadata:
     name: mongodb-secret
     namespace: mongodb-test
   type: Opaque
   data:
     # Base64-encoded value of:
     # mongodb://test_user:test_password@mongodb-svc.mongodb.svc.cluster.local:27017/test
     connect: bW9uZ29kYjovL3Rlc3RfdXNlcjp0ZXN0X3Bhc3N3b3JkQG1vbmdvZGItc3ZjLm1vbmdvZGIuc3ZjLmNsdXN0ZXIubG9jYWw6MjcwMTcvdGVzdA==
   ---
   apiVersion: keda.sh/v1alpha1
   kind: TriggerAuthentication
   metadata:
     name: mongodb-trigger
     namespace: mongodb-test
   spec:
     secretTargetRef:
       - parameter: connectionString
         name: mongodb-secret
         key: connect
   ---
   apiVersion: keda.sh/v1alpha1
   kind: ScaledJob
   metadata:
     name: mongodb-job
     namespace: mongodb-test
   spec:
     jobTargetRef:
       template:
         spec:
           containers:
             - name: mongo-update
               image: registry-cn-shanghai.ack.aliyuncs.com/acs/mongo-update:v6
               args:
                 - --dataBase=test
                 - --collection=test_collection
                 - --operation=updateMany
                 - --update={"$set":{"state":"finished"}}
               env:
                 - name: MONGODB_CONNECTION_STRING
                   value: mongodb://test_user:test_password@mongodb-svc.mongodb.svc.cluster.local:27017/test
               imagePullPolicy: IfNotPresent
           restartPolicy: Never
       backoffLimit: 1
     pollingInterval: 15         # Check MongoDB every 15 seconds
     maxReplicaCount: 5          # Run at most 5 concurrent Jobs
     successfulJobsHistoryLimit: 0  # Delete completed Jobs immediately
     failedJobsHistoryLimit: 10     # Keep the last 10 failed Jobs for debugging
     triggers:
       - type: mongodb
         metadata:
           dbName: test
           collection: test_collection
           query: '{"type":"mp4","state":"waiting"}'  # Launch a Job for each matching document
           queryValue: "1"
         authenticationRef:
           name: mongodb-trigger

2. Deploy the configuration.

   kubectl apply -f keda-mongodb.yaml