Learn Apache Flink - Job Configuration & Deployment

#Introduction

Throughout this series you've run jobs on a local cluster. Episode 11 answers a bigger question: how do you take a job from your laptop to production? The answer spans three layers: packaging code into a runnable artifact, choosing a deployment mode, and allocating the right resources.

We'll build an uber JAR with dependency shading, compare the standalone, YARN, and Kubernetes modes, tune JobManager and TaskManager resources, and close with job lifecycle management practices. After this episode, you can explain to your team where your jobs run and why.

#Creating a Job JAR Package

#Dependency Shading with Maven

Flink jobs depend on external libraries. To keep the cluster from needing to know every dependency, bundle them all into a single uber JAR using the Maven Shade Plugin:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <transformers>
      <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
    </transformers>
  </configuration>
</plugin>

ServicesResourceTransformer is important for connectors that use a service loader (like Flink SQL). Without this transformer, service metadata is lost and connectors aren't detected.

#Building and Shipping the JAR

mvn clean package -DskipTests
ls target/*.jar

The mvn clean package command produces a single JAR containing your code and all dependencies. Note the name and location of this JAR — it's the only artifact you ship to the cluster.

#Deployment Modes

#Standalone

Standalone mode is the simplest: the cluster is started manually and jobs are submitted via the CLI. Suitable for development and small workloads. Its downside: no automatic resource management — you manage when to add or remove TaskManagers yourself.

#YARN

In Hadoop environments, Flink integrates with YARN. There are two forms: session (one cluster shared by many jobs) and application (a cluster is created per job and then removed).

./bin/yarn-session.sh -d
./bin/flink run -d target/app.jar

./bin/yarn-session.sh -d starts a YARN session cluster in the background, after which flink run -d submits a job to that session. Application mode is more recommended for production because isolation is better.

#Kubernetes

Flink on Kubernetes can run directly (native) or via the Flink Kubernetes Operator. With the operator, the entire job lifecycle is declared as a resource:

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: order-pipeline
spec:
  image: registry.example.com/flink-job:1.0
  flinkVersion: v2_3
  jobManager:
    replicas: 1
    resource:
      memory: "2048m"
  taskManager:
    replicas: 2
    resource:
      memory: "4096m"

FlinkDeployment declares the image, Flink version, and JobManager and TaskManager resources. The operator handles roll-out, restart, and scaling — this mode is the main direction of modern Flink deployment.

#Resource Configuration

#JobManager and TaskManager

Each process's resources are configured in config.yaml:

jobmanager.memory.process.size: 2048m
taskmanager.memory.process.size: 4096m
taskmanager.numberOfTaskSlots: 4
parallelism.default: 2

taskmanager.numberOfTaskSlots determines the maximum subtasks running per TaskManager. parallelism.default sets the default job parallelism without explicit settings. A rule of thumb: total slots must cover the whole job parallelism, and don't add slots per TaskManager beyond the number of available CPUs.

#Calculating Slot Requirements

The number of slots needed = the job's parallelism. If the job has operators with parallelism 8, make sure the cluster has at least 8 slots. Extra slots are fine; a shortage of slots makes the job wait.

#Managing the Job Lifecycle

#CLI for Daily Operations

./bin/flink list -a
./bin/flink cancel <jobId>
./bin/flink stop --savepointPath /tmp/sp <jobId>

flink list -a shows all jobs, flink cancel forcefully stops a job, and flink stop --savepointPath gracefully stops a job while creating a savepoint.

#Web UI for Observation

The web dashboard on port 8081 shows the job lifecycle: RUNNING, FINISHED, FAILED, and CANCELLED statuses, complete with history. The Job Manager and Task Manager tabs show resource usage — important when deciding whether parallelism needs to be raised.

build JAR → choose a deployment mode → set resources → submit → monitor

#Conclusion

Episode 11 closed the job shipping cycle: packaging code into an uber JAR with shading, choosing between standalone, YARN, and Kubernetes (native or operator), configuring JobManager and TaskManager resources, and managing the job lifecycle with the CLI and web UI.

The key takeaways:

• An uber JAR with ServicesResourceTransformer guarantees connectors are detected.
• Standalone for learning, YARN for the Hadoop ecosystem, Kubernetes for modern production.
• FlinkDeployment declares image, version, and resources declaratively.
• Total slots must cover the job's parallelism; don't add slots beyond CPU count.
• flink list, flink cancel, and flink stop cover the daily lifecycle operations.

In the next episode, episode 12, we'll discuss security, authentication & authorization — securing cluster communication with TLS, Kerberos authentication and RBAC, securing access to the web dashboard and REST API, and protecting source and sink credentials. Security isn't an add-on feature; it's a production prerequisite.