Learn Apache Flink - History, Background & Why Choose Flink

#Introduction

Episode 0 made sure your environment is ready. Now it's time to understand why Flink exists. Episode 1 answers three big questions: where does Apache Flink come from, what is its position in the big data ecosystem, and why do you — as a data or backend engineer building real-time systems — need it.

Many people start using Flink because of tutorials, but understanding its background is far more valuable. By knowing the problems it solves, you'll find it easier to decide when to use Flink, when not to, and how to position it in your architecture. Let's start from the beginning of the story.

#The Evolution from Batch to Real-time

#The Batch Processing Era

The data processing world began with batch. In the Hadoop era (early 2010s), flowing data was stored first, then processed periodically — daily, hourly, or per minute. MapReduce and Spark Batch models work in cycles: read, process, write, done. This approach is reliable and simple, but it has high latency: decisions are made based on data that might already be minutes to hours old.

#The Emergence of Real-time Needs

When businesses began to demand immediate responses — fraud detection at the moment a transaction happens, recommendations when a user interacts, live infrastructure monitoring — the batch paradigm started to feel awkward. Data must be processed immediately as it arrives, without waiting for the next batch. This is where stream processing was born, and Apache Flink is one of its pioneers.

batch → micro-batch → stream processing → stateful stream processing

This line describes the leap we want to understand: from processing data periodically to processing data as a never-ending flow, while still keeping state for computations that depend on history.

#The History of Apache Flink

#From the Stratosphere Research Project

Apache Flink was born from the Stratosphere research project developed at TU Berlin around 2010-2014. Unlike other projects born from industry needs, Stratosphere was rooted in academic research on distributed query processing and dataflow — and from this came the innovation that is Flink's DNA to this day: a true streaming engine based on pipelined execution, not micro-batch.

In 2014, the project was donated to the Apache Software Foundation as an incubator and officially renamed Apache Flink. The name Flink itself, in German, means agile or nimble — describing a fast and flexible approach.

#Important Milestones

Flink's journey from incubator to one of the most established stream processing engines can be summarized in the following table:

One innovation that set Flink apart from the start is pipelined streaming — data flows between operators without having to wait for a batch to finish. This concept is what allows Flink to deliver millisecond latency, something micro-batch engines struggle to achieve.

./bin/flink --version

The ./bin/flink --version command shows the Flink version installed in your environment. Routine checks like this are important because Flink releases new versions regularly, and you need to know where your version stands.

#Comparison with Alternatives

#Apache Spark Streaming

Spark Streaming processes streams as micro-batches — the stream is cut into small batches and then processed with Spark's batch engine. Its strengths: Spark's large ecosystem and familiar APIs. Its weaknesses: higher latency than true streaming, and the per-batch processing model makes some patterns like event time less natural.

#Kafka Streams

Kafka Streams is a lightweight library for stateful processing directly on top of Kafka, running inside a regular JVM application without a separate cluster. It's a great fit for simple pipelines within a single Kafka ecosystem. However, for complex pipelines with many joins and varied windows, Kafka Streams is more limited than Flink, which has cross-system sources and sinks.

#Apache Beam

Apache Beam is an abstraction layer — you write a pipeline once with a standardized model, then run it on many runners such as Flink, Spark, and Google Dataflow. Beam is not an engine; it's a portable API. Flink itself is one of the best runners for Beam because of its complete support for event time and state.

Flink        → true streaming engine, stateful, low latency
Spark Stream → micro-batch, large ecosystem, medium latency
Kafka Streams → lightweight library, tied to the Kafka ecosystem
Beam         → portable API, needs a runner like Flink

#When to Choose Flink

Choose Flink if you need true streaming with millisecond latency, strong state management with exactly-once, strict event time semantics, CEP, and broad integration with Kafka, Kinesis, databases, and object storage. Choose an alternative if your needs are simple, tightly coupled to a single system, or you only need batch processing.

#Ideal Apache Flink Use Cases

Here are the application patterns that are Flink's sweet spot:

• Stateful stream processing: computing running metrics (rolling windows) that need to remember previous conditions.
• Event-driven analytics: automatic reactions to events, such as raising an alert when a threshold is crossed.
• Complex event processing (CEP): detecting sequences of events, like fraud patterns or failure cascades.
• Real-time data pipelines: streaming ETL from Kafka to a warehouse or object storage with transformations in between.
• Hybrid batch and stream: Flink can process historical (bounded) data and live (unbounded) data in a single model.

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 
public class StreamingApp {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.fromElements("order", "refund", "order")
           .map(String::toUpperCase)
           .print();
        env.execute("contoh-streaming");
    }
}

The code above is just a skeleton — we'll dissect map and all the DataStream transformations in depth in episode 4. For now, just understand that env.execute marks the point where the whole pipeline is compiled and executed.

#Conclusion

Episode 1 gave you the context: Flink was born from the Stratosphere research project at TU Berlin, entered Apache in 2014, and is now one of the most established stream processing engines with stable version 2.3.0 as of 2026. It wins on true streaming needs with millisecond latency, state management, and event time semantics — against Spark Streaming's micro-batch approach, Kafka Streams' lightweight but limited nature, and Beam, which is only an abstraction layer.

The key takeaways:

• Flink is rooted in the Stratosphere academic research and became a top-level Apache project in 2015.
• Its key innovation is pipelined streaming: data flows between operators without waiting for a batch.
• Spark Streaming uses micro-batch, Kafka Streams is tied to the Kafka ecosystem, Beam needs a runner.
• Flink excels at stateful stream processing, event-driven analytics, CEP, and real-time pipelines.
• Flink can handle bounded and unbounded data in a single programming model.

In the next episode, episode 2, we'll discuss the core concepts and architecture of Apache Flink — the stream model and DataStream API, the concepts of event time, processing time, and ingestion time, stateful computation with state backends and checkpointing, and the execution architecture with JobManager, TaskManager, slots, and parallelism. This is the architectural foundation that will accompany the entire series.