Opcounters and Repl Opcounters metrics

Why are secondary Repl Opcounters higher than primary Opcounters?

Operations that affect multiple documents, such as a batch insert or a multi-document update or delete, are counted as a single operation on the primary node. When an operation that affects multiple documents is replicated to secondary nodes, the replication process is document-based. As a result, the Repl Opcounters value on a secondary node may be higher than the Opcounters value on the primary node.

The following example shows this behavior:

1. Before the update, check the opcounters.

   • The Opcounters update counter on the primary node is 13.

     > db.serverStatus().opcounters.update
     NumberLong(13)

   • The Repl Opcounters update counter on the secondary node is 11.

     > db.serverStatus().opcountersRepl.update
     NumberLong(11)

2. Execute a batch update on the primary node. The returned modifiedCount field shows that this batch update modified four documents.

   > db.coll.updateMany({x:2},{$set:{x:3}})
   { "acknowledged" : true, "matchedCount" : 4, "modifiedCount" : 4 }

3. After the update, check the opcounters again.

   • The Opcounters update counter on the primary node is now 14.

     > db.serverStatus().opcounters.update
     NumberLong(14)

   • The Repl Opcounters update counter on the secondary node is now 15.

     > db.serverStatus().opcountersRepl.update
     NumberLong(15)

Why does Repl Opcounters show inserts during updates?

This may occur because your update operation specifies the {upsert:true} option. If the document that you want to update does not exist, MongoDB converts the operation to an insert operation. If the oplog records an insert operation, replication also applies an insert operation on the secondary node. Therefore, Repl Opcounters records the corresponding insert operation.

The following example shows this behavior:

1. Before the update, check the opcounters.

   • On the primary node, the Opcounters show update: 33 and insert: 1516.

     > db.serverStatus().opcounters
     {
       "insert" : NumberLong(1516),
       "query" : NumberLong(70),
       "update" : NumberLong(33),
       "delete" : NumberLong(1043),
       "getmore" : NumberLong(2662),
       "command" : NumberLong(4000)
     }

   • On the secondary node, the Repl Opcounters show update: 24 and insert: 1539.

     > db.serverStatus().opcountersRepl
     {
       "insert" : NumberLong(1539),
       "query" : NumberLong(0),
       "update" : NumberLong(24),
       "delete" : NumberLong(6),
       "getmore" : NumberLong(0),
       "command" : NumberLong(26)
     }

2. On the primary node, run an update operation with the {upsert:true} option. The returned upsertedId field shows that one document was inserted.

   > db.coll.updateOne({x:"a"}, {$set:{x:"b"}}, {upsert:true})
   {
     "acknowledged" : true,
     "matchedCount" : 0,
     "modifiedCount" : 0,
     "upsertedId" : ObjectId("64bf72b829907f52b4b363ea")
   }

3. After the update, check the opcounters again.

   • On the primary node, the Opcounters update counter is now 34 and the insert counter is still 1516.

     > db.serverStatus().opcounters
     {
       "insert" : NumberLong(1516),
       "query" : NumberLong(70),
       "update" : NumberLong(34),   // Note the change in this counter.
       "delete" : NumberLong(1043),
       "getmore" : NumberLong(2706),
       "command" : NumberLong(4286)
     }

   • On the secondary node, the Repl Opcounters insert counter is now 1540 and the update counter is still 24.

     > db.serverStatus().opcountersRepl
     {
       "insert" : NumberLong(1540), // Note the change in this counter.
       "query" : NumberLong(0),
       "update" : NumberLong(24),
       "delete" : NumberLong(6),
       "getmore" : NumberLong(0),
       "command" : NumberLong(26)
     }

Why is the primary update count higher than the secondary?

This may be because your application logic includes repetitive update operations. If a repeated update operation does not actually change any data, it is not replicated to the secondary nodes. Therefore, the number of update operations recorded on the secondary nodes is lower.

The following example shows this behavior:

1. Before the update, check the opcounters.

   • The update count in the Opcounters on the primary node is 34.

     > db.serverStatus().opcounters
     {
       "insert" : NumberLong(1516),
       "query" : NumberLong(70),
       "update" : NumberLong(34),
       "delete" : NumberLong(1043),
       "getmore" : NumberLong(2760),
       "command" : NumberLong(4609)
     }

   • The update count in the Repl Opcounters on the secondary node is 24.

     > db.serverStatus().opcountersRepl
     {
       "insert" : NumberLong(1540),
       "query" : NumberLong(0),
       "update" : NumberLong(24),
       "delete" : NumberLong(6),
       "getmore" : NumberLong(0),
       "command" : NumberLong(26)
     }

2. Execute an update operation on the primary node. The returned information shows that no actual modification occurred.

   > db.coll.updateMany({x:"ab"},{$set:{x:"cd"}})
   { "acknowledged" : true, "matchedCount" : 0, "modifiedCount" : 0 }

3. After the update, check the opcounters again.

   • The update count in the Opcounters on the primary node is now 35.

     > db.serverStatus().opcounters
     {
       "insert" : NumberLong(1516),
       "query" : NumberLong(70),
       "update" : NumberLong(35),    // Note the change in this counter.
       "delete" : NumberLong(1043),
       "getmore" : NumberLong(2778),
       "command" : NumberLong(4729)
     }

   • The update count in the Repl Opcounters on the secondary node is still 24.

     > db.serverStatus().opcountersRepl
     {
       "insert" : NumberLong(1540),
       "query" : NumberLong(0),
       "update" : NumberLong(24),
       "delete" : NumberLong(6),
       "getmore" : NumberLong(0),
       "command" : NumberLong(26)
     }

Why are Opcounters active on an idle database?

Even with no user traffic, the monitoring data still reports a small number of operations. These consist of the following:

• Basic internal operations that keep the MongoDB database running, such as replica set heartbeats, primary/secondary synchronization, and session refreshes.

• Routine operations from the ApsaraDB for MongoDB management components, such as health checks, monitoring, and event listening.

Why do Opcounters metrics differ from oplog aggregations?

In the oplog, you can use the op field to distinguish between operation types. Common values include:

kCommand: "c"
kInsert: "i"
kUpdate: "u"
kDelete: "d"
kNoop: "n"

Transactions are recorded in the oplog with an operation type of op: "c". The individual insert, update, and delete operations within the transaction are stored internally in the o.applyOps field. If you aggregate results based only on the top-level op field of the oplog, your results will not match the Opcounters metrics in scenarios that involve transactions. For more information about the oplog format, see Analysis of oplog fields.

To count the specific operation types within a transaction, connect to your MongoDB instance by using the mongo shell and run the following command.

use local
db.oplog.rs.aggregate([{$match:{"op":"c","ts":{"$gte": Timestamp(1733849400,0)},"o.applyOps":{$exists:true},"o.applyOps.0.op":"u"}},{$count:"count"}])

The following describes the parameters in this command:

• Timestamp(1733849400,0): A lower bound for the ts timestamp in the oplog query. Replace this value with your desired query time. You can read this timestamp from local.oplog.rs or use a standard UNIX timestamp.

• "o.applyOps.0.op":"u": Specifies that the first operation in the transaction oplog is an update. You can replace this with other operation types. For example, if the first operation is an insert, use "o.applyOps.0.op":"i".

• {$count:"count"}: Counts only the number of matching oplog entries. You can use other aggregation operators for other types of analysis.

To view transaction-related monitoring metrics, go to the Monitoring Information page in the ApsaraDB for MongoDB console and view the Transaction Operands metric. For more information about this metric, see Monitoring items and metrics.

The following example shows this behavior:

1. Before the update, check the Opcounters.

   > db.serverStatus().opcounters
   {
     "insert" : NumberLong(4), 
     "query" : NumberLong(6723),
     "update" : NumberLong(110489),
     "delete" : NumberLong(3065),
     "getmore" : NumberLong(222670),
     "command" : NumberLong(1917525)
   }

2. On the primary node, execute a write operation within a transaction.

   // Start a session.
   session = db.getMongo().startSession( { readPreference: { mode: "primary" } } );
   coll1 = session.getDatabase("mydb1").foo;
   coll2 = session.getDatabase("mydb2").bar;
   // Start a transaction.
   session.startTransaction( { readConcern: { level: "local" }, writeConcern: { w: "majority" } } );
   // Execute two insert operations in the transaction.
   try {
      coll1.insertOne( { abc: 1 } );
      coll2.insertOne( { xyz: 999 } );
   } catch (error) {
      // Abort the transaction if an error occurs.
      session.abortTransaction();
      throw error;
   }
   // Commit the transaction.
   session.commitTransaction();
   session.endSession();

3. After the update, check the Opcounters again. The insert count in the Opcounters has changed from 4 to 6.

   > db.serverStatus().opcounters
   {
     "insert" : NumberLong(6),   // Note the change in this counter.
     "query" : NumberLong(6728),
     "update" : NumberLong(110532),
     "delete" : NumberLong(3067),
     "getmore" : NumberLong(222823),
     "command" : NumberLong(1918887)
   }