Learn Apache Flink - Core Concepts & Apache Flink Architecture

#Introduction

Episode 1 answered why Flink exists. Episode 2 builds the conceptual bridge: before writing code, you must understand how Flink views the world. We'll dissect three concepts: what a stream and its API layers are, how Flink understands time, and how stateful computation and its execution architecture work.

These concepts aren't just theory — every design decision in the coming episodes (windowing, checkpointing, tuning) is rooted in this chapter. If you master episode 2, the other technical parts will feel like variations on a single story. Let's get started.

#Streams and API Layers

#The Unbounded Stream Model

In Flink, all data is viewed as a stream. There are two kinds: a bounded stream (limited data, like a file that is already complete) and an unbounded stream (data that keeps flowing endlessly, like Kafka logs). This concept matters because Flink treats batch as a special case of streaming.

#DataStream API, Table API, and SQL

Flink provides several API layers with different levels of abstraction:

• DataStream API: an imperative API based on Java/Scala for full control over state, time, and transformations.
• Table API: a declarative, SQL-like API with pipelines based on dynamic tables.
• Flink SQL: a full SQL language for streaming queries, the most concise and closest to analytical needs.
• DataSet API: the old batch API, now deprecated and steered toward the Table API.

These layers can be mixed in a single application. For example, you can create a source with the DataStream API, then analyze it with Flink SQL.

Flink SQL and Table API
        ↑
DataStream API (V1 and V2)
        ↑
Runtime and State Management

#The Basic Pipeline Flow

Every Flink application follows the same pattern: source → transformation → sink. The source supplies data, transformations change it, and the sink writes it to its destination. We'll fill in the details of each stage throughout the series.

#Time Concepts: Event, Processing, and Ingestion

#Three Definitions of Time

Flink distinguishes three kinds of time that often confuse beginners:

• Event time: the time when the event actually occurred at the source (a timestamp inside the data). This is the most accurate.
• Ingestion time: the time when the event entered the Flink system.
• Processing time: the time when the event is processed by an operator, i.e. the local machine time.

This difference is crucial. Imagine logs sent with a 5-minute delay: event time points to the moment the event actually happened, while processing time points to when the data is processed. For accurate analytics, almost always use event time.

#Watermark as a Progress Marker

Because events can arrive late, Flink uses watermarks to mark "how far event time is safe to process". We'll dissect watermarks in depth in episode 5. For now, think of it as a clock that moves with the data, not the system clock.

{"event": "order_created", "ts": "2026-08-10T10:15:30Z", "amount": 120000}

The line above is an example event in JSON format with a ts column as the event time. Columns like this are what Flink will use for windowing in episodes 5 and 9.

#Stateful Computation

#Why State Is Needed

Many analytics can't be done event-by-event: computing total transactions per user, detecting sequential patterns, or computing running metrics all require remembering something from the past. This memory is what's called state.

#State Backends and Checkpointing

State is stored in a state backend — some in memory (HashMap) and some on disk (RocksDB). To keep state from being lost when a process dies, Flink creates checkpoints periodically — snapshots of state and source consumption positions. With checkpoints, Flink can provide exactly-once guarantees: every event is processed exactly once, even in the event of a failure.

#Savepoint vs Checkpoint

Besides automatic checkpoints, there's the savepoint — a snapshot created manually and used for upgrades, migrations, or controlled recovery. We'll dissect the differences in episodes 7 and 16.

execution.checkpointing.interval: 1min
execution.checkpointing.mode: exactly-once
state.backend.type: rocksdb

The configuration snippet above enables a checkpoint every minute with exactly-once mode and the RocksDB state backend. We'll discuss the details of each line in episode 6.

#Execution Architecture

#JobManager: The Brain of the Cluster

The JobManager is the coordinator process: it receives jobs, manages checkpoints, schedules tasks, and recovers from failures. It stores metadata but not the processed result data. There are two modes: session mode (one cluster shared by many jobs) and application mode (one cluster per application).

#TaskManager and Task Slots

The TaskManager is the worker: it runs tasks from the job and stores state and buffers. Each TaskManager has a number of task slots — units of resources (memory and CPU) for running subtasks. The number of slots determines how many subtasks can run simultaneously in one process.

#Parallelism and Subtasks

Every operator has parallelism: how many parallel subtasks run it. One subtask uses one slot. For example, an operator with parallelism 4 runs 4 subtasks in 4 slots. This relationship between resources and parallelism is key to performance tuning in episodes 14 and 15.

JobManager (1)
  ├── checkpoint coordinator
  └── scheduler
TaskManager (2 processes)
  ├── slot 1 → subtask map [0]
  ├── slot 2 → subtask map [1]
  ├── slot 3 → subtask window [0]
  └── slot 4 → subtask window [1]

The structure above illustrates a job with two operators (map and window), each with parallelism 2, running on two TaskManagers with four slots in total. Understand that parallelism determines how many subtasks there are, and slots determine the capacity to run them.

#Conclusion

Episode 2 built Flink's way of thinking: all data is a stream, managed through the DataStream API, the Table API, or SQL; time is understood through event time, processing time, and ingestion time; stateful computation runs on top of state backends with checkpointing and savepoints; and execution is orchestrated by the JobManager and TaskManagers with slots and parallelism.

The key takeaways:

• Batch is a special case of streaming: Flink views all data as streams.
• Use event time for accurate analytics, and understand Flink's three definitions of time.
• State enables stateful computation; checkpointing guarantees exactly-once.
• Savepoints are used for controlled upgrades and migrations, unlike automatic checkpoints.
• The JobManager coordinates, TaskManagers execute, slots are units of resources, and parallelism determines the number of subtasks.

In the next episode, episode 3, we'll discuss installing and running Flink jobs — the Flink directory structure, configuration in config.yaml and flink-conf.yaml, running applications with flink run, understanding the job lifecycle, and reading the web dashboard. We'll put into practice what you just learned in episode 2.