Learn Apache Flink - Error Handling & Debugging Stream Jobs

#Introduction

Episode 6 made sure your jobs can recover from system failures. Episode 7 turns to the most human side: wrong code. Data in the real world is dirty — missing fields, changing formats, broken JSON. The question isn't whether errors will appear, but how you respond to them without losing data or stopping the job.

We'll learn to catch exceptions in operators and sources, separate corrupt data into a side output, read logs and job status, use savepoints for controlled recovery, and debugging techniques with a local cluster. By the end of this episode, you'll have a complete toolkit for debugging any streaming job.

#Handling Exceptions in Operators and Sources

#Catching Errors with Side Outputs

The healthiest way to handle corrupt data is to catch it in-process rather than letting the job die. Use a ProcessFunction with try-catch and route failed data to a side output:

OutputTag<String> badTag = new OutputTag<String>("bad-rows") {};
 
DataStream<Event> valid = lines
    .process(new ProcessFunction<String, Event>() {
        @Override
        public void processElement(
                String line, Context ctx, Collector<Event> out) throws Exception {
            try {
                out.collect(parseJson(line));
            } catch (Exception e) {
                ctx.output(badTag, line);
            }
        }
    });
 
DataStream<String> badRows = valid.getSideOutput(badTag);

ctx.output sends the problematic data to a side output instead of dropping it. From there you can write it to a dead-letter topic or a log for later inspection — data isn't lost, and the job keeps running.

#The Decision: Fail Fast vs Skip

There are two philosophies: fail fast (let the job error so it's known quickly) and skip bad records (keep going, record what's wrong). For production pipelines, the wisest combination: route failed parses to a side output, then count them with a custom metric so the team knows when the number rises.

#Monitoring Job Status and Logs

#Logging with SLF4J

Flink uses SLF4J as its logging interface. Get used to writing contextual logs in operators:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
private static final Logger LOG = LoggerFactory.getLogger(ParseEventFunction.class);
 
LOG.info("Memproses event {}", event.getId());
LOG.warn("Event tanpa timestamp diterima: {}", event);

LOG.info uses placeholder formatting so log messages stay concise. For errors you can anticipate, use LOG.warn; for unexpected conditions that could fail the job, use LOG.error.

#Reading Cluster Logs

Every component's logs live in the log/ folder:

tail -f $FLINK_HOME/log/taskexecutor.log

The tail -f command follows the log in real-time. When debugging, look for the stack trace containing your job class name — that's where the error originated, not just the generic message on the last line.

#Savepoints for Recovery

#Creating a Manual Savepoint

Checkpoints are automatic and temporary; savepoints are created manually and stored for the long term. Create one when you want to stop a job without losing state:

./bin/flink savepoint <jobId> /tmp/flink-savepoints
./bin/flink run -d -s /tmp/flink-savepoints/savepoint-<id> target/app.jar

./bin/flink savepoint takes a snapshot of the current state, and the -s flag on flink run -d restores a job from that savepoint. This is also the main way to upgrade a job without losing state — we'll dive deeper in episode 16.

#Choosing Between Checkpoint and Savepoint

Savepoints are better for deliberate decisions (version upgrades, moving clusters), while checkpoints are for unexpected events.

#Debugging with a Local Cluster

#Running in Detached Mode

When developing, run the job in detached mode then observe logs and the dashboard without blocking the terminal:

$FLINK_HOME/bin/flink run -d -p 2 target/debug-job.jar
$FLINK_HOME/bin/flink list -a

flink run -d -p 2 submits the job with parallelism 2. In the web dashboard, the Task and Backpressure tabs give visual hints about which subtasks are slow or failing.

#Common Debugging Patterns

Some patterns that often save time:

• Start with env.fromElements containing small, deterministic data.
• Print operator output stage by stage with .print() before adding the next operator.
• Write a unit test for the parse function before wiring it into the pipeline.
• Use EXPLAIN in Flink SQL to see the query execution plan.

small data → one operator → print → add operator → observe output

#Conclusion

Episode 7 trained you to face failures calmly: catching exceptions in operators and sources via side outputs, writing informative logs, monitoring job status, using savepoints for controlled recovery, and debugging incrementally with a local cluster.

The key takeaways:

• Route corrupt data to a side output instead of silently dropping it.
• Logging with SLF4J and placeholders keeps messages concise and easy to search.
• tail -f on the task executor log reveals the stack trace where the error originates.
• Savepoints are created manually for upgrades and rollbacks; checkpoints are automatic for recovery.
• Debug iteratively: small data, one operator, then add complexity.

In the next episode, episode 8, we'll discuss source & sink integrations — connecting Kafka, Kinesis, RabbitMQ, and file sources, writing to Kafka, databases, object storage, and Elasticsearch, and understanding the connector ecosystem and the JSON, Avro, Protobuf, and CSV data formats. This is the bridge between Flink and all the systems around you.