Learn Apache Flink - Installation & Running Flink Jobs

#Introduction

Episode 2 gave you the concepts. Episode 3 turns concepts into practice: we'll operate a real Flink cluster, understand the directory structure and configuration files, run jobs with the CLI, then read their lifecycle from the web dashboard. Everything you do here will be used every day when working with Flink.

The ability to operate a cluster isn't a supplementary skill — it's a core skill. Engineers who can read logs and understand job status will be far more effective than those who can only write code. This episode is the first step toward that level.

#Installing a Standalone Cluster

#Preparing the Folder and Environment

Make sure JDK 17 is installed and the Flink folder is extracted as in episode 0. Create environment variables to make things easier:

export FLINK_HOME=/home/kalian/flink-2.3.0
export PATH=$FLINK_HOME/bin:$PATH
flink --version

With PATH updated, the flink command can be called from anywhere. The flink --version command should show Version: 2.3.0.

#Starting and Stopping the Cluster

Start the standalone cluster, which consists of one JobManager and one TaskManager:

$FLINK_HOME/bin/start-cluster.sh
jps

The jps command (Java Process Status) shows the running JVM processes. You'll see StandaloneSessionClusterEntrypoint (JobManager) and TaskManagerRunner (TaskManager). These two processes are the heart of your cluster.

#Flink Directory Structure

The Flink installation folder has a standard structure that's important to understand:

Understanding this structure helps you deploy connectors: to add a Kafka connector in standalone mode, you place its JAR in lib/ and then restart the cluster.

flink-2.3.0/
  ├── bin/        → flink, start-cluster.sh, sql-client.sh
  ├── conf/       → config.yaml, log4j.properties
  ├── lib/        → flink-dist, flink-table JAR
  ├── examples/   → streaming, batch, table
  ├── log/        → .log and .out per component
  └── plugins/    → s3-fs, cloud-fs, etc.

#Configuration: config.yaml

#From flink-conf.yaml to config.yaml

Before Flink 1.19, the main configuration file was called flink-conf.yaml. Since 1.19, this file was restructured into config.yaml with kebab-case keys. For this series, which is based on 2.3, we use config.yaml — but you'll often find old documentation that still writes flink-conf.yaml.

#Commonly Used Configuration Keys

Some keys you must know from the start:

jobmanager.rpc.address: localhost
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 2048m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 2
rest.port: 8081

• taskmanager.numberOfTaskSlots determines the slots per TaskManager.
• parallelism.default is the default job parallelism without explicit settings.
• rest.port determines the REST API port and the web dashboard.

Remember that every change requires a cluster restart. The parallelism.default configuration will be touched often when you experiment with parallelism.

#Checking Logs

Logs are the source of truth when debugging:

ls $FLINK_HOME/log/
tail -f $FLINK_HOME/log/*standalonesession*.log

Files ending in .log contain detailed component logs, while .out files contain console output. Get used to tail -f on the JobManager log to watch activity in real-time.

#Running Applications with flink run

#Attached and Detached Modes

The flink run command has two important modes:

• Attached (default): the CLI waits until the job finishes or is cancelled.
• Detached (-d): the CLI returns immediately after the job is submitted, suitable for production.

$FLINK_HOME/bin/flink run -d examples/streaming/WindowWordCount.jar \
  --input /etc/passwd --output /tmp/wordcount-result

The flink run -d command submits the job to the cluster and immediately returns control to the terminal. You'll use the -d flag throughout the series.

#Other Important Flags

Some flags commonly used with flink run:

$FLINK_HOME/bin/flink run -c com.example.WordCount target/wordcount.jar

• -c specifies the main class when a JAR contains many classes.
• -p sets the job parallelism.
• -s <path> restores a job from a savepoint (covered in episode 16).

#Job Lifecycle and the Web Dashboard

#Job Status and Listing

Every job goes through a lifecycle: CREATED → RUNNING → FINISHED or FAILED, with intermediate states like RESTARTING and CANCELLING. Monitor them with the CLI:

$FLINK_HOME/bin/flink list -a
$FLINK_HOME/bin/flink cancel <jobId>

The flink list -a command shows all jobs including finished ones, along with their job IDs. flink cancel <jobId> stops a running job.

#Reading the Web Dashboard

Open http://localhost:8081. The dashboard shows:

• Overview: the number of TaskManagers and available slots.
• Running Jobs: the list of active jobs with status and duration.
• Job Details: job graph, parallelism, and per-operator metrics.

From the Job Details tab you can see the DAG graph of the pipeline — a direct visualization of the source, transformation, and sink concepts from episode 2. For now, just make sure the dashboard shows one TaskManager with slots matching the configuration.

start-cluster.sh → flink run -d job.jar → check flink list → watch the dashboard → cancel/stop

#Conclusion

Episode 3 trained you to operate a Flink cluster: understanding the directory structure and the role of each folder, configuring config.yaml (the successor to flink-conf.yaml), running jobs with flink run in attached and detached modes, monitoring status via the CLI, and reading the job graph from the web dashboard on port 8081.

The key takeaways:

• A standalone cluster is started with start-cluster.sh and verified with jps.
• The main configuration has been named config.yaml since Flink 1.19, replacing flink-conf.yaml.
• Use flink run -d to submit a job without waiting, and flink list -a to see all jobs.
• Logs live in log/ and are the primary source for debugging.
• The web dashboard at http://localhost:8081 shows the job graph, parallelism, and per-operator metrics.

In the next episode, episode 4, we'll discuss the DataStream API and core transformations — creating Flink jobs with Java, using basic transformations like map, flatMap, filter, keyBy, window, and reduce, and understanding stream partitioning and parallelism. This is the most fundamental episode for writing your own streaming pipelines.