Learn Apache Flink - Operational Readiness & Runbooks

#Introduction

All the techniques in this series come down to one practical question: when a production job fails in the middle of the night, do you know what to do? Episode 19 answers with runbooks — documented procedures that turn incident response into reflex, not improvisation.

We'll write runbooks for job failures, respond to checkpoint failures and job crashes, prepare backups and disaster recovery, and close with chaos testing to deliberately probe system resilience. The end goal is simple: failures no longer surprise your team.

#Writing Runbooks

#Anatomy of a Good Runbook

A runbook is a step-by-step document for handling a specific situation. Every good runbook contains:

• Symptoms — what the team sees when the incident happens.
• Common causes — a list of the most likely hypotheses.
• Recovery procedure — the exact commands to run.
• Rollback — how to return to a stable state.

./bin/flink list -a
tail -n 100 $FLINK_HOME/log/taskexecutor.log
./bin/flink run -d -s /backups/savepoint-latest target/app.jar

./bin/flink list -a shows the job status, tail -n 100 reads the latest logs, and ./bin/flink run -d -s restores from a savepoint. A good runbook writes exact steps like these — complete with the expected output examples.

#Written to Be Read Quickly

Store runbooks somewhere accessible from anywhere during an incident. Short formats are better than complete ones: bullet points, commands, and expected outputs. A runbook that's too long simply won't be read when the system is having problems.

#Incident Response: Checkpoint Failure and Job Crash

#Checkpoint Failure

A repeatedly failing checkpoint is an early warning. Common causes: storage can't hold the state, state size grows quickly, or checkpoint time exceeds the timeout. The right response:

1. Check the logs for the specific snapshot error.
2. Inspect the storage capacity where checkpoints are stored.
3. If state is ballooning, apply TTL (episode 15) and check for hot keys.

execution.checkpointing.interval: 5min
execution.checkpointing.min-pause: 2min
execution.checkpointing.tolerable-failed-checkpoints: 3

execution.checkpointing.tolerable-failed-checkpoints tolerates several checkpoint failures before the job fails outright — giving you time to handle the cause without panicking.

#Job Crash and Restart

When a job fails, the restart strategy determines automatic recovery:

restart-strategy.type: exponential-delay
restart-strategy.exponential-delay.initial-backoff: 10s
restart-strategy.exponential-delay.max-backoff: 1min
restart-strategy.exponential-delay.backoff-multiplier: 2.0
restart-strategy.exponential-delay.reset-backoff-threshold: 1h

exponential-delay lengthens the pause between restart attempts — preventing downstream systems from being flooded with retries while the problem persists. If the job keeps crashing, don't just restart: find the root cause in the logs first.

#Backups, Cluster Snapshots, and Disaster Recovery

#Backing Up Savepoints

Savepoints are artifacts that must be backed up. Send them to object storage separate from the cluster:

./bin/flink savepoint <jobId> s3://flink-backup/savepoints

./bin/flink savepoint with an s3:// target stores the savepoint directly to a durable location. Set a policy: create routine savepoints before major upgrades and keep the last few versions.

#Disaster Recovery Plan

Disaster recovery isn't just about backups, but a tested recovery sequence:

• Rebuild the cluster in a new region or environment.
• Restore the configuration from version control (git).
• Restore state from the latest savepoint.
• Verify data is flowing and metrics are normal.

Without practice, a DR plan is just a document. Test it periodically so the sequence is committed to the team's memory.

#Chaos Testing

#Breaking Things on Purpose

Chaos testing deliberately breaks components to test system resilience:

./bin/taskmanager.sh stop
./bin/flink list -a

./bin/taskmanager.sh stop forcibly kills one TaskManager. A healthy job will restart its subtasks in the remaining slots and recover from checkpoints — if it doesn't, there's a gap to fix.

#Valuable Chaos Scenarios

Focus on the scenarios with the biggest impact, not the most dramatic ones:

• One TaskManager dies suddenly.
• Network between TaskManagers cuts out briefly.
• Checkpoint storage is full or slow.
• Kafka goes down for a few minutes.

Record the results of each experiment into the runbook: what happened, how long recovery took, and what was fixed. Chaos testing turns theoretical knowledge about fault tolerance into empirical evidence.

#Conclusion

Episode 19 closed the operational side: writing runbooks that can be executed during an earthquake, responding calmly to checkpoint failures and job crashes, preparing tested backups and disaster recovery, and validating resilience through chaos testing.

The key takeaways:

• A runbook contains symptoms, causes, procedures, and rollback — written short.
• Checkpoint failures often come from storage and state size; check before panicking.
• Savepoints are backed up to object storage as the foundation of disaster recovery.
• A DR plan must be tested, not just written.
• Chaos testing turns fault tolerance claims into evidence.

In the next episode, episode 20, we'll discuss real-time analytics & use cases — building monitoring and alerting dashboards, implementing fraud detection, real-time recommendation, and IoT processing, and designing an end-to-end streaming pipeline with data enrichment and stream joins. All the theory you've learned will come together in one real architecture.