Learn Apache Flink - Data Governance & Observability

#Introduction

A running job doesn't mean a healthy job. Episode 13 shifts your mindset from "successfully submitted" to "observable": is throughput dropping, is backpressure piling up, is latency widening, and where's the bottleneck? This is the essence of observability for streaming.

We'll expose Flink metrics to Prometheus and Grafana, read logs and job metrics, monitor backpressure and task health, track audit and lineage for governance, and close with tracing for latency analysis. After this episode, you can answer "what's happening in my pipeline?" without guessing.

#Metrics with Prometheus and Grafana

#Flink Metric Reporters

Flink exposes metrics through reporters. Enable the Prometheus reporter in config.yaml:

metrics.reporters: prom
metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
metrics.reporter.prom.port: 9250

metrics.reporter.prom.factory.class determines the reporter implementation, and port 9250 becomes the Prometheus scrape endpoint. After a cluster restart, jobs and components automatically expose their metrics.

#Scraping Metrics

curl -s http://localhost:9250/metrics | grep flink_job

The curl -s http://localhost:9250/metrics command pulls all metrics in Prometheus format. Filter with grep flink_job to see job-specific metrics. From here, Grafana can display dashboards, and Alertmanager can trigger alarms when numbers look suspicious.

#Logging, Job Metrics, and Task Health

#Custom Metrics in Operators

Besides built-in metrics, create custom metrics for business signals:

import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.Gauge;
 
Counter eventsProcessed = getRuntimeContext()
    .getMetricGroup()
    .addGroup("custom")
    .counter("events_processed");
eventsProcessed.inc();
 
Gauge<Long> backlog = getRuntimeContext()
    .getMetricGroup()
    .addGroup("custom")
    .gauge("backlog", () -> queue.size());

addGroup(...).counter(...) creates a metric named custom.events_processed. Metrics like this are your eyes on the dashboard — you can see volume per operator, not just job status.

#Task Health

The web dashboard shows each task's health: RUNNING, FAILING, RESTARTING statuses, with restart history per operator. Watch the Task Metrics tab for CPU, heap usage, and record throughput per subtask. A spike in task restarts is an early signal of problematic code.

#Backpressure Monitoring

#Why Backpressure Happens

Backpressure occurs when a downstream operator is slower than its upstream operator, so buffers pile up. Flink handles this gracefully by slowing down the upstream — but if it persists, latency rises and throughput drops.

curl -s http://localhost:8081/jobs/overview | python3 -m json.tool

The curl -s http://localhost:8081/jobs command returns the job list; the Backpressure tab on the dashboard marks each operator with OK, LOW, or HIGH status. An operator marked HIGH is a tuning candidate — most likely because of large state, expensive queries, or low parallelism.

#Handling Backpressure

The standard steps: raise the parallelism of the slow operator, reduce work per record, check state usage (a full RocksDB is often the culprit), and make sure the sink isn't the bottleneck.

#Audit Events, Lineage, and Data Quality

#Governance for Streaming

Streaming data needs governance too. Flink SQL supports recording operations and configuration through the SQL Gateway; audit logs leave a trail of who ran which query. Lineage can be drawn from EXPLAIN — understanding where data comes from and where it flows.

#Data Quality Checks

Data quality is maintained with validation in the pipeline: reject records that fail the schema, count the corrupt data ratio with custom metrics (as in episode 7), and route anomalies to a side output for the data team to review. Good governance is a combination of lineage, schema definitions, and measurement.

metrics → log → backpressure → tracing

#Tracing and Latency Analysis

#OpenTelemetry for End-to-end Visibility

Tracing connects events in Flink with upstream and downstream systems. Enable the OpenTelemetry tracing reporter:

traces.reporters: otel
traces.reporter.otel.factory.class: org.apache.flink.tracing.opentelemetry.OpenTelemetryTraceReporterFactory

With tracing enabled, each record carries span context that can be tracked across services. In an observability dashboard (for example Grafana Tempo or Jaeger), you can measure end-to-end latency and find which operator contributes the largest delay.

#Conclusion

Episode 13 made your system transparent: Prometheus metrics displayed in Grafana, logs and job metrics for task health, backpressure monitoring to find bottlenecks, and audit, lineage, and tracing for governance and latency analysis.

The key takeaways:

• The Prometheus reporter on port 9250 makes all Flink metrics scrapable.
• Custom metrics with counters and gauges give per-operator business visibility.
• HIGH backpressure status indicates an operator that needs tuning.
• Audit, lineage, and data quality checks keep streaming governance in place.
• OpenTelemetry tracing connects end-to-end latency across services.

In the next episode, episode 14, we'll discuss scaling & resource management — managing parallelism and task slot sizes, autoscaling practices for Flink on Kubernetes, controlling backpressure and throughput, and optimizing the RocksDB versus in-memory state backends. You'll learn to build pipelines that grow with the load.