Learning Redis - Troubleshooting & Operational Maintenance

#Introduction

Every production system will have problems — what separates a professional engineer is preparedness. Episode 19 equips you with troubleshooting of the most common Redis problems and the routine maintenance that prevents them from recurring.

We'll dissect the four classic complaints — high latency spikes, OOM command not allowed, connection exhaustion, and replication disconnects — then schedule backups, upgrade without downtime, and prepare disaster recovery procedures. This is the most sought-after material when you're on pager duty.

#Troubleshooting Common Problems

#High Latency Spikes

A sudden latency spike almost always stems from background operations that use fork:

• RDB snapshots (periodic save) perform a fork that copies the memory page table.
• Long AOF rewrites.
• BGSAVE/BGREWRITEAOF commands triggered manually during high load.

The first way to confirm: check LATENCY LATEST for fork events:

redis-cli LATENCY LATEST
redis-cli INFO stats | grep latest_fork_usec

LATENCY LATEST shows the fork event if present — latest_fork_usec reveals how long the fork lasted. Mitigation: schedule snapshots during quiet hours, raise the save interval, and use activedefrag carefully to avoid clashes.

#Memory Full: OOM Command Not Allowed

The error OOM command not allowed when used memory > 'maxmemory' means maxmemory has been reached and the noeviction policy is active (or eviction failed to lower memory):

redis-cli INFO memory | grep -E "used_memory|maxmemory"
redis-cli CONFIG GET maxmemory-policy

The steps: make sure the policy isn't noeviction for a cache workload, raise maxmemory if the server still has spare RAM, and audit memory-hungry keys with MEMORY USAGE. For the long term, consider sharding (episode 13).

#Client Connection Exhaustion

The error max number of clients reached indicates maxclients has been reached — often because of a connection leak in the application (episode 16) or a pool that's too small:

redis-cli INFO clients
redis-cli CONFIG GET maxclients

connected_clients that keeps rising even with flat traffic = leak. Fixing the application side (use a pool, close connections) is better than raising maxclients, which just adds memory and CPU load.

#Replication Disconnect & Full Resync

Replicas that disconnect repeatedly trigger repeated full resyncs — draining the master's bandwidth and memory:

redis-cli INFO replication
redis-cli INFO stats | grep total_sync

master_link_status:up indicates health. total_sync increasing rapidly indicates repeated resyncs — usually a timeout or a repl backlog buffer that's too small. Raise repl-backlog-size and check the network between master and replica.

#Operational Maintenance

#Scheduling Backups

Snapshots and rewrites should be scheduled, not left to automatic triggers that can clash with busy hours:

redis-cli BGSAVE
redis-cli BGREWRITEAOF

BGSAVE creates an RDB snapshot in the background; BGREWRITEAOF rewrites the AOF to keep it compact. Schedule both outside peak hours and not at the same time — stacked forks are a recipe for latency spikes. In production, back up .rdb files to external storage daily as a safety net.

#Rolling Upgrade without Downtime

To upgrade Redis in a Sentinel or Cluster architecture, do it node by node so the service never fully goes down:

1. upgrade replicas (not serving traffic)
2. move traffic / promote replica → new master
3. upgrade the old master (now a replica) and restore the topology

With Sentinel: upgrade replicas first, then run SENTINEL FAILOVER to promote an already-upgraded replica, and upgrade the remaining nodes. In a Cluster, follow the same order — always upgrade the node not serving traffic first.

#Disaster Recovery

#Restoring from an RDB Snapshot

When an instance is destroyed and must be rebuilt from backup:

redis-cli --rdb /tmp/backup.rdb
redis-cli SHUTDOWN

The steps: stop the server, place the .rdb file with the name matching dbfilename in the dir directory, then start. For AOF, just place the AOF file (Redis 7 uses a multi-part format in the appenddirname directory) — make sure the AOF restore doesn't run alongside a stale snapshot. Always verify DBSIZE after restoring.

#Cluster Recovery When a Majority of Masters Are Down

When many masters in a cluster die at once, cluster-require-full-coverage (default yes) makes the cluster refuse all queries because some slots have no owner:

redis-cli -p 7000 CLUSTER INFO
redis-cli -p 7000 CLUSTER NODES

The strategy: promote the surviving replicas (CLUSTER FAILOVER), bring the dead nodes back up, and let resharding restore distribution. If replicas aren't enough, weigh the risk of cluster-require-full-coverage no — the availability vs. data consistency trade-off — and add nodes promptly. Recovery procedures must be written, tested, and simulated before a real incident.

#Summary

Episode 19 equipped you with the operational side of Redis: handling high latency spikes from fork, OOM errors, connection exhaustion, and replication disconnects; scheduling BGSAVE/BGREWRITEAOF; rolling upgrades; and RDB restore and cluster recovery procedures.

Key takeaways:

• Latency spikes almost always come from fork — check LATENCY LATEST and latest_fork_usec.
• OOM command not allowed means maxmemory is reached with the noeviction policy.
• max number of clients reached is usually an application connection leak, not the server.
• Repeated replication disconnects signal a too-small repl backlog or poor network.
• Schedule BGSAVE/BGREWRITEAOF outside peak hours and not simultaneously.
• Rolling upgrade: always upgrade the node not serving traffic first.
• RDB restore: stop the server, place the .rdb, start, verify DBSIZE.
• DR procedures must be tested with regular failover drills.

In the next episode, episode 20 — the final episode of this series — we assemble everything into a production-grade Redis architecture case study: a caching layer with Cache-Aside, session store, rate limiting, real-time features, leaderboard, all on a 6-node cluster with TLS, ACL, and monitoring. Let's design the complete architecture!