Learn Apache Flink - Advanced Streaming Patterns

#Introduction

The previous episodes built a strong set of basic skills. Episode 17 combines them into advanced patterns that recur in the real world: joining two streams, building user sessions, processing data iteratively, and uniting batch and stream in one pipeline.

We'll cover stateful joins in three forms — window join, interval join, and temporal join — then hopping windows and sessionization, iterative streaming with feedback loops, and hybrid batch and stream processing. This is the episode that makes you "level up" as a stream engineer.

#Stream Joins

#Window Join

A window join matches elements from two streams that fall into the same window:

import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
 
orders.join(payments)
    .where(Order::getOrderId)
    .equalTo(Payment::getOrderId)
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .apply((order, payment) -> Tuple2.of(order, payment))
    .print();

.where(...).equalTo(...) sets the join key, and .window limits matches to the same window. This is the simplest pattern for combining events from two sources.

#Interval Join

An interval join matches elements based on relative time, without a rigid window:

import org.apache.flink.streaming.api.functions.co.ProcessJoinFunction;
import org.apache.flink.streaming.api.windowing.time.Time;
 
orders.keyBy(Order::getUserId)
    .intervalJoin(actions.keyBy(Action::getUserId))
    .between(Time.minutes(-30), Time.minutes(30))
    .process(new ProcessJoinFunction<Order, Action, JoinedRow>() {
        @Override
        public void processElement(
                Order order, Action action, Context ctx, Collector<JoinedRow> out) {
            out.collect(new JoinedRow(order, action));
        }
    });

.between limits matches to a time window relative to the event. An interval join stores both streams' state for the time span — understand its state cost when joining very large streams.

#Temporal Join

A temporal join matches an event stream with a dimension table that changes over time:

SELECT o.order_id, o.amount, c.country
FROM orders o
JOIN customer_catalog FOR SYSTEM_TIME AS OF o.event_ts AS c
ON o.customer_id = c.id;

FOR SYSTEM_TIME AS OF o.event_ts takes the dimension version valid at the time the event occurred — not the latest version. This is the accurate answer for enriching events with changing data.

#Hopping Windows and Sessionization

#Hopping Windows in SQL

A hopping (sliding) window is expressed with the HOP function:

SELECT user_id, SUM(amount) AS total
FROM orders
GROUP BY user_id, HOP(event_ts, INTERVAL '5' MINUTE, INTERVAL '10' MINUTE);

HOP(event_ts, INTERVAL '5' MINUTE, INTERVAL '10' MINUTE) opens a 10-minute window every 5 minutes — each event falls into two windows at once.

#Sessionization

Sessionization groups user activity into sessions based on a time gap:

SELECT user_id,
       SESSION_START(event_ts, INTERVAL '30' MINUTE) AS mulai,
       SESSION_END(event_ts, INTERVAL '30' MINUTE) AS selesai,
       COUNT(*) AS aktivitas
FROM clicks
GROUP BY user_id, SESSION(event_ts, INTERVAL '30' MINUTE);

SESSION_START and SESSION_END give the boundaries of each session. This pattern is very common for product analytics: how long users stay, how much activity per session, and when sessions happen most often.

#Iterative Streaming and Feedback Loops

#iterate for Repetitive Computation

Some algorithms need repetition until convergence — for example, computing PageRank. Flink provides the iterate operator:

import org.apache.flink.streaming.api.datastream.IterativeStream;
 
IterativeStream<Long> iteration = numbers.iterate();
DataStream<Long> proses = iteration.map(step);
DataStream<Long> selesai = proses.filter(konvergen);
iteration.closeWith(proses.filter(belumKonvergen));
selesai.print();

iteration.closeWith sends results that haven't converged back to the start of the iteration, while the konvergen filter emits the final results. Remember: each round adds latency, so use this only for computations that truly need feedback.

#Feedback Loop Limitations

Feedback loops hinder checkpointing and make scheduling more complex. Consider alternatives: stateful processing with MapState, or a loop outside Flink. Understand that pipeline simplicity is often more valuable than algorithmic perfection.

#Hybrid Batch + Stream

#One Model for Two Kinds of Data

Flink offers batch and stream processing in a single programming model. The old DataSet API is now deprecated — the direction is the Table API, which works well for both bounded and unbounded data.

import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.table.api.TableEnvironment;
 
EnvironmentSettings settings = EnvironmentSettings.inBatchMode();
TableEnvironment tableEnv = TableEnvironment.create(settings);
 
tableEnv.executeSql("SELECT user_id, SUM(amount) FROM orders_hist GROUP BY user_id").print();

inBatchMode() runs the query with batch semantics (full results when the query finishes). The same code can run in streaming mode — only the context changes. This is the power of unifying the batch and stream models in Flink.

#Conclusion

Episode 17 expanded your repertoire: window joins, interval joins, and temporal joins for combining data; hopping windows and sessionization for time-based analytics; iterative streaming for repetitive computation; and hybrid batch and stream via the Table API.

The key takeaways:

• A window join limits matches to a window; an interval join to a relative time window.
• A temporal join takes the dimension version valid when the event occurred.
• SESSION groups user activity into sessions based on a time gap.
• Feedback loops enable iterative computation at the cost of latency and complexity.
• The Table API runs the same code for batch and stream.

In the next episode, episode 18, we'll discuss custom connectors & extensions — building custom sources and sinks, extending Flink with custom serializers and codecs, understanding operator lifecycle and checkpoint hooks, and contributing to the connector ecosystem. You'll learn to build Flink bridges to systems that don't have an official connector yet.