Learn Apache Flink - Savepoints, Upgrades & Migration

#Introduction

Changing a running job is the most tense moment for a data engineer: one mistake and state is lost, aggregations are wrong, or data is duplicated. Episode 16 removes that tension. Savepoints are your tool for upgrading, migrating, and rolling back jobs without losing state.

We'll dissect the savepoint lifecycle, compatibility between Flink versions, how to perform job upgrades with code changes, rollback strategies, and the practice of testing restores in staging. By the end of this episode, you can replace a production job version with confidence.

#The Savepoint Lifecycle

#Creating, Listing, and Deleting

Savepoints are created manually and can be managed via the CLI:

./bin/flink savepoint <jobId> /tmp/savepoints
./bin/flink list -s
./bin/flink savepoint -d /tmp/savepoints/savepoint-<id>

./bin/flink savepoint takes a snapshot of state, ./bin/flink list -s lists all savepoints the cluster recognizes, and ./bin/flink savepoint -d deletes those no longer needed. Store savepoints in a durable location — usually object storage in production.

#Savepoint vs Checkpoint

A savepoint is a deliberate snapshot made for upgrades and migrations, while a checkpoint is an automatic snapshot for recovery after failure. Because savepoints are made manually and stored long-term, they become reliable rollback points — and artifacts that must be well maintained.

#Compatibility and State Migration

#The Rules Between Versions

Flink guarantees savepoint compatibility for versions still within the support range. Important principles:

• Upgrade one minor version at a time, not in big jumps.
• State structure changes (adding fields) need planning.
• Always verify the restore in staging before touching production.

state.savepoints.dir: s3://flink-state/savepoints

state.savepoints.dir sets the default directory so all jobs use a consistent, durable location.

#Migrating State Between Shapes

When code changes — for example adding a field to ValueState — Flink attempts automatic migration. If the change is incompatible (for example changing a data type), the restore fails with an exception. The solution: use state types that tolerate change (MapState), or perform a two-stage migration with an intermediate job.

#Job Upgrades with Restore

#Keeping Operator Identity with uid

For state to connect correctly after code changes, give every stateful operator a stable uid:

source
    .map(new ParseOrder()).uid("parse-order")
    .keyBy(Order::getUserId)
    .process(new Aggregator()).uid("aggregate")
    .sinkTo(sink);

.uid("aggregate") gives the operator a permanent name. Without a uid, Flink relies on operator order, which can change after a refactor — state can get mixed up. uids are a mandatory habit for every job that might be upgraded.

#The Correct Upgrade Flow

./bin/flink savepoint <jobId> /tmp/savepoints
./bin/flink run -d -s /tmp/savepoints/savepoint-<id> target/app-v2.jar

The steps: create a savepoint from the old version, then run the new JAR with the -s flag pointing at that savepoint. After verifying the v2 job runs correctly, then cancel the v1 job. This order ensures there's no processing downtime.

#Rollback and Recovery Drills

#Rollback Strategies

If an upgrade fails, restore from the last healthy savepoint:

./bin/flink run -d -s /tmp/savepoints/savepoint-<healthy> target/app-v1.jar

This practice only works if a savepoint was taken before the change. So make "savepoint before upgrade" a mandatory rule, not an option.

#Recovery Drills in Staging

Don't wait for an incident to test recovery. Run scheduled drills in staging:

• Forcefully stop a TaskManager and make sure the job recovers via checkpoint.
• Perform an upgrade from a savepoint and verify state isn't lost.
• Simulate a rollback and compare the results with the baseline.

./bin/taskmanager.sh stop
./bin/flink list -a

./bin/taskmanager.sh stop shuts down one TaskManager to test recovery. If the job restarts automatically and state is fully restored, your fault tolerance mechanism works.

#Testing Restores in Staging

#Verifying State Before Production

Every upgrade must be tested in staging with this scenario: restore from a savepoint created in production, then compare metrics (for example total aggregates) between before and after. A difference in numbers means the state migration has a problem — find it before touching production.

#Migration Checklist

• A savepoint is created from a healthy production job.
• The restore succeeds in staging with the new JAR.
• All stateful operators have a stable uid.
• Aggregation results are consistent with the baseline.
• Rollback is tested and the old savepoint is still stored.

#Conclusion

Episode 16 made upgrades a calm procedure: understanding the savepoint lifecycle, respecting version compatibility, keeping operator identity with uid, applying tested upgrade and rollback flows, and running recovery drills in staging.

The key takeaways:

• Savepoints are made manually for upgrades and migrations; checkpoints are for recovery.
• Upgrade one minor version at a time and test the restore in staging first.
• A stable uid connects state correctly after code changes.
• Always take a savepoint before an upgrade as a rollback point.
• Scheduled recovery drills prepare the team for real incidents.

In the next episode, episode 17, we'll discuss advanced streaming patterns — stateful joins, stream-stream joins, and temporal joins, hopping windows and sessionization, iterative streaming with feedback loops, and hybrid batch and stream processing. This is where Flink's advanced techniques come together.