Learning MongoDB - Monitoring, Maintenance & Troubleshooting

#Introduction

A healthy database is invisible — until suddenly the application slows down, memory spikes, and everyone panics. The difference between a panicked team and a calm team isn't luck, but visibility: they know what's measured, what tools are used, and what procedures to run when the numbers deviate.

Episode 19 equips you with observation lenses. The roadmap: first the health monitoring tools — db.serverStatus, db.currentOp, mongostat, mongotop, second the Prometheus + Grafana pipeline, third operational maintenance — compact, reIndex, log rotation, WiredTiger tuning, and fourth troubleshooting common problems. Let's get started.

#Monitoring MongoDB Health

#db.serverStatus

db.serverStatus() is a single window into server condition — uptime, connections, operations, memory, and storage engine metrics in one call:

db.serverStatus()

The metrics most often observed:

• connections — the number of active connections; nearing the limit signals connection pool exhaustion.
• mem.resident and mem.virtual — process memory usage.
• opcounters — the number of inserts, queries, updates, and deletes served; useful for load trends.
• uptime — how long the server has been running since the last restart.

#db.currentOp

When the server feels slow, the first question is "what operations are running?". The answer is in db.currentOp():

db.currentOp()

The result shows active operations, how long they've been running (secs_running), what query is being executed, and its execution plan. This is the main tool for finding "stubborn" queries holding resources hostage. To stop a truly stuck operation:

db.killOp(12345)

db.killOp accepts the opid you see in the db.currentOp() output. Use it carefully — killing an operation mid-transaction can trigger a rollback.

#mongostat and mongotop

For continuous observation, these two built-in utilities are invaluable:

mongostat --uri "mongodb://localhost:27017" --discover --rowcount 20

mongostat displays per-second metrics: read/write operations, connection counts, and memory usage. It runs continuously and can be used to see load spikes in real time.

mongotop --uri "mongodb://localhost:27017"

mongotop shows how much time the server spends on each collection — helping identify which collections are busiest and need index attention.

#Prometheus Exporter and Grafana

For production-level observation that is stored and alerted on, use percona/mongodb_exporter together with Prometheus and Grafana:

services:
  mongo-exporter:
    image: percona/mongodb_exporter:0.40.0
    command:
      - --mongodb.uri=mongodb://monitor:pass@mongo:27017
      - --collector.diagnosticdata
      - --collector.replicasetstatus
    ports:
      - "9216:9216"

The exporter exposes MongoDB metrics in the Prometheus format on port 9216. Prometheus scrapes it periodically, and Grafana displays visual dashboards with alerts. Metrics typically monitored: operations per second, queue, replication lag, WiredTiger memory cache, and connections. This pattern is the foundation of MongoDB observability in modern production.

#Operational Maintenance

#Compact and Repair

Over time, WiredTiger storage can develop fragmentation — unused empty space between documents. Compact rearranges data to reclaim space:

db.runCommand({ compact: "orders" })

Compact consumes resources and blocks operations on that collection — do it during low load (maintenance window). For more aggressive repair, db.repairDatabase() repairs and compresses the entire database, but it's a heavy operation that stops the service — rarely used, and restoring from a backup is preferable.

#Rebuild Index and Log Rotation

If an index is corrupt or its version changed, rebuilding the index restores its integrity:

db.orders.reIndex()

reIndex() drops and recreates all of a collection's indexes — useful after a restore or when an index is suspected of being corrupt, but do it in a maintenance window because it consumes resources.

For log rotation, MongoDB provides a command that rotates the active log without a restart:

db.adminCommand({ logRotate: 1 })

This pattern is scheduled with cron to prevent logs from growing out of control, then old logs are archived or deleted according to retention.

#WiredTiger Cache Tuning

WiredTiger stores the data "working set" in memory cache. The default cache size is roughly 50% of RAM minus 1GB. Tuning can be done through configuration:

storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 4

A cache that's too small causes a lot of disk reads (high IO); a cache that's too large can crowd other processes on the host. Tuning requires observing cache pressure metrics — part of the art of MongoDB administration.

#Troubleshooting Common Problems

#High CPU and Slow Queries

Symptoms: CPU spikes, slow application. The most common cause: missing index forcing a COLLSCAN on a large collection. Diagnose with the profiler (episode 13), then create the right index following the ESR rule (episode 12). The second solution: rewriting non-selective queries.

#High Memory and Cache Pressure

Symptoms: memory nearly full, swap active. WiredTiger is indeed designed to use as much cache as possible — not immediately a problem. Measure cache pressure (the percentage of cache used for new pages) and evictions per second. If pages keep getting evicted before they're used, the working set exceeds the cache — time to add RAM or trim rarely accessed data.

#Replication Lag

Symptoms: a secondary falls far behind the primary; data read from the secondary is stale. Common causes: an oplog too small so the secondary loses track, or writes on the primary outpacing the secondary's replication ability. Check rs.status().members[].optime to see how large the lag is. Solutions: increase the oplog size, review the write workload, or evaluate the network bandwidth between nodes.

#Connection Pool Exhaustion

Symptoms: the application errors with "Too many connections" or "connection pool exhausted". Each mongod connection needs memory; thousands of idle connections eat resources. Solutions: adjust the connection pool size in the driver, limit the server's maximum connections, and eliminate unnecessary idle connections.

#Conclusion

In episode 19 you equipped yourself with visibility: checking health with db.serverStatus, db.currentOp, mongostat, and mongotop; building production-level observability with percona/mongodb_exporter, Prometheus, and Grafana; performing maintenance like compact, reIndex, log rotation, and WiredTiger cache tuning; and handling the four most common problems — slow queries due to missing indexes, memory pressure, replication lag, and connection pool exhaustion.

Key takeaways:

• db.serverStatus and db.currentOp are the first diagnostic tools when problems occur.
• Prometheus + Grafana turn monitoring into permanent dashboards and alerts.
• Compact and reIndex restore space and integrity in a maintenance window.
• Slow queries usually = missing index; cache pressure = working set exceeds RAM.
• Always have a backup before heavy maintenance and know your normal baseline.

In the next episode, episode 20, we tie everything learned into one whole: Production-Grade E-Commerce Document Database Case Study. You'll design the schema for users, a product catalog, orders, and analytics; build a deployment with replica sets, TLS, automatic backups, and monitoring; and review a production readiness checklist as the series finale. See you in episode 20 — the closing episode!