Learn Apache Flink - Prerequisite Skills & Environment Setup

#Introduction

Welcome to the Learn Apache Flink series! This series will take you to mastery of Apache Flink — an open-source framework for stateful computation over unbounded and bounded data streams — from the fundamentals of concepts all the way to production readiness. There are 23 episodes in total, organized into six learning phases.

But before writing your first Flink job, there are some foundational skills and software you must have. Why do these prerequisites matter? Because Flink isn't just an ETL tool. It's a distributed stream processing engine that runs Java or Scala on the JVM, stores state in memory or on disk, and coordinates across processes. Without a basic understanding of these concepts, terms like parallelism, checkpoint, and watermark will feel like a black box.

Episode 0 is your roadmap: we'll make sure the foundational skills are in place, set up JDK and Docker, install the stable version of Apache Flink, and perform the first verification with a local cluster. Once this episode is done, the whole series can be followed comfortably.

#Foundational Skills You Must Have

#Data Processing: Batch vs Stream

Flink is built to process streams — data that keeps flowing without end. First, distinguish these two basic models:

• Batch processing: data is complete and stored, processed periodically, for example daily reports.
• Stream processing: data arrives continuously, processed immediately as it arrives, for example transaction anomaly detection.
• Streaming as a generalization: an important line of thinking — batch is actually a stream that is bounded and finishes.

Flink can handle both with the same model. You must understand this difference before moving on, because almost every Flink job design is rooted in how you look at data.

#Distributed Systems Concepts

Flink is a distributed system, so master the following core concepts:

• Partitioning: splitting data into several parts so it can be processed in parallel.
• Fault tolerance: the ability of a system to keep producing correct results when a process dies.
• Replication: copying data or state to several places for safety.
• Exactly-once semantics: the guarantee that every event is processed exactly once, neither fewer times nor doubled.

These terms will keep coming up throughout the series, especially when we discuss checkpointing in episode 6.

#JVM, Java, and CLI Tools

Flink code runs on the Java Virtual Machine. You should be familiar with Java (or Scala) syntax, and the concepts of classes, generics, and lambdas. For tools, make sure you're comfortable with the terminal: tar, wget, java, and environment variables like PATH. Maven or Gradle will also be used starting in episode 3 to build projects.

java -version
javac -version
which java

If javac produces output, the Java Development Kit is installed. The java -version command above is the first check you should get used to.

#Software to Prepare

#JDK 17 or 11

Apache Flink 2.x requires JDK 17, while Flink 1.20 (LTS) still supports JDK 11. To follow this series with the latest version, install JDK 17:

sudo apt update
sudo apt install -y openjdk-17-jdk
java -version

Make sure the output shows 17.0.x or newer. You can use another distribution manager such as sdkman or Homebrew on macOS — what matters is that the version matches.

#Docker and Docker Compose

Starting in episode 8, you'll connect Flink to Kafka and other systems. Docker is very helpful for running those dependencies locally:

docker --version
docker compose version

Set up Docker now so things go smoothly later. We'll use docker compose to spin up Kafka and a Kinesis emulator in episode 8.

#IDE, Git, and Streaming Data Sources

Install IntelliJ IDEA Community or VS Code with the Java extension, plus Git for version control:

git --version

Optional but useful: Apache Kafka or Kinesis as streaming data sources for integration experiments. If you don't have one yet, Flink has a built-in DataGen connector that can generate synthetic data — we'll use it starting in episode 4.

#Installing Apache Flink

#Download and Extract

At the time this series was written, Apache Flink 2.3.0 is the latest stable release. Download the binary from an Apache mirror:

wget https://archive.apache.org/dist/flink/flink-2.3.0/flink-2.3.0-bin-scala_2.12.tgz
tar -xzf flink-2.3.0-bin-scala_2.12.tgz
cd flink-2.3.0
ls

The flink-2.3.0 folder structure contains bin/ for CLI scripts, conf/ for configuration, examples/ for sample jobs, and lib/ for dependencies. We'll dissect this structure more deeply in episode 3.

#Running a Local Cluster

Flink supports standalone mode for learning. Start one JobManager and one TaskManager with the built-in scripts:

./bin/start-cluster.sh
./bin/flink list

The ./bin/start-cluster.sh command turns on the cluster on your local machine, and ./bin/flink list shows the list of running jobs — empty for now. Also pay attention to the logs that appear in the log/ folder.

#Environment Verification

#Running a Sample Job

Flink ships with the WindowWordCount example program. Run it with flink run:

./bin/flink run examples/streaming/WindowWordCount.jar --input /etc/hostname
./bin/flink list

When it's done, flink list will be empty again because the job has finished. You've just run your first Flink job — the flink run operation will be your companion throughout the series.

#Checking the Web Dashboard

Open a browser and access the Flink dashboard:

http://localhost:8081

There you can see cluster status, the list of jobs, and metrics. This dashboard is important for monitoring, which we'll discuss in episodes 7 and 13.

#Final Verification

Repeat all the verification steps at once to make sure everything is ready:

• java -version shows JDK 17.
• docker --version and docker compose version work normally.
• The flink-2.3.0 folder is extracted and the cluster can be started.
• The WindowWordCount job runs successfully and finishes.
• The dashboard at http://localhost:8081 is accessible.

./bin/stop-cluster.sh

#Summary of Skills You Must Have

Here's a summary of the prerequisites you've prepared in episode 0:

• Batch vs stream concepts and streaming as a generalization.
• Distributed systems concepts: partitioning, fault tolerance, replication.
• JDK 17 installed and validated via java -version.
• Docker and Docker Compose ready for integration in the coming episodes.
• Apache Flink 2.3.0 installed, the local cluster can be started, and the sample job runs successfully.

If anything is missing, stop here and complete it before continuing. A strong foundation will make the next 22 episodes feel much lighter.

#Conclusion

In this episode 0 you've laid the foundation for the whole series: understanding data processing and distributed systems concepts, setting up JDK 17 and Docker, installing Apache Flink 2.3.0, running a local standalone cluster, and verifying the installation with the WindowWordCount sample job and the web dashboard.

The key takeaways:

• Flink is a distributed stream processing engine that runs on the JVM, so master Java and JVM concepts.
• Distinguish batch and stream, and understand that batch is just a bounded stream.
• Flink 2.x requires JDK 17, while Flink 1.20 LTS supports JDK 11.
• A local standalone cluster is started with ./bin/start-cluster.sh and the web dashboard is at http://localhost:8081.
• First verification: run WindowWordCount with flink run and check the results in the dashboard.

In the next episode, episode 1, we'll discuss the history, background, and why choose Flink — from the evolution of stream processing from batch to real-time, the birth of Flink from the Stratosphere research project, to its comparison with Apache Spark Streaming, Kafka Streams, and Apache Beam. Make sure your environment is ready, because the Learn Apache Flink journey has only just begun!