Learn Apache Flink - DataStream API & Core Transformations

#Introduction

The first three episodes built your foundation: concepts, architecture, and cluster operations. Now it's time for the most exciting part — writing real pipelines with the DataStream API. Episode 4 equips you with the basic transformations used in almost every Flink job: map, flatMap, filter, keyBy, window, and reduce.

Transformations are where business logic lives. Once you master the source → transformation → sink pattern and understand how data moves between operators, you'll be able to build the majority of real-world streaming pipelines. Let's get started.

#Creating a Flink Job with Java

#Maven Project Structure

Create a Maven project with the Flink dependencies. The core dependencies are flink-streaming-java and flink-clients:

mvn archetype:generate \
  -DarchetypeGroupId=org.apache.maven.archetypes \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DgroupId=com.example -DartifactId=flink-basic -Dversion=1.0.0

Add the dependencies to pom.xml with versions aligned with your Flink cluster, for example 2.3.0. Make sure the main class is specified in pom.xml so flink run can find it.

#The Job Skeleton

All DataStream applications follow the same skeleton:

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
 
public class JobPertama {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
 
        env.fromElements("budi", "siti", "andi")
           .map(String::toUpperCase)
           .print();
 
        env.execute("job-pertama");
    }
}

env.fromElements creates a stream from static data (useful for experiments), and env.execute marks the end of the pipeline. This job is ready to be built with Maven and run with flink run -d.

#Basic Transformations

#map

map transforms one element into one other element. It's a one-to-one transformation:

env.fromElements("budi", "siti", "andi")
   .map(kata -> kata.length())
   .print();

The code above converts each name into the length of its characters. Applying map is useful for data normalization, format conversion, and field extraction.

#flatMap

flatMap transforms one element into zero, one, or many elements. It's the most flexible transformation:

env.fromElements("budi makan nasi", "siti minum kopi")
   .flatMap((kalimat, out) -> {
       for (String kata : kalimat.split(" ")) {
           out.collect(kata);
       }
   })
   .print();

Note that flatMap uses a Collector to emit results. The flatMap operator is very useful for splitting rows, normalizing events, and selectively dropping elements.

#filter

filter keeps the elements that satisfy a condition:

env.fromElements(1, 2, 3, 4, 5, 6)
   .filter(angka -> angka % 2 == 0)
   .print();

This code only lets even numbers through. Use filter to trim data early in the pipeline — filtering earlier means less data for downstream operators to process.

#keyBy and Partitioning

#Splitting a Stream by Key

keyBy divides the stream into several logical partitions based on a key. All elements with the same key are sent to the same subtask. This is a prerequisite for keyed stateful operations like aggregation:

env.fromElements("budi:jakarta", "siti:bandung", "andi:jakarta")
   .map(baris -> baris.split(":"))
   .map(parts -> new String[] { parts[0], parts[1] })
   .keyBy(data -> data[1])
   .print();

After keyBy, all data for the same city is guaranteed to be held by the same subtask. Understand keyBy as Flink's way of grouping data for consistent parallel computation.

#Other Partitioning Types

Besides keyBy, there's more explicit partitioning:

• rebalance(): distributes data round-robin to all subtasks.
• broadcast(): sends a copy of the data to all subtasks.
• rescale(): efficient redistribution between neighboring subtasks.

These partitioning strategies are used for specific needs such as sharing dynamic configuration (broadcast) or breaking up load skew (rebalance).

#Window and Reduce

#window for Time-based Aggregation

The window operator collects elements within a certain time window for aggregation. Example of a 10-second tumbling window:

import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
 
env.fromData(Tuple2.of("jakarta", 100L), Tuple2.of("bandung", 50L), Tuple2.of("jakarta", 75L))
   .keyBy(t -> t.f0)
   .window(TumblingEventTimeWindows.of(Time.minutes(1)))
   .reduce((a, b) -> Tuple2.of(a.f0, a.f1 + b.f1))
   .print();

reduce aggregates elements in the window incrementally — very efficient because it doesn't store all elements. The full windowing details (tumbling, sliding, session) will be covered in episode 5.

#Parallelism and Chaining

#Setting Parallelism

Every operator can have different parallelism, set per-job or per-operator:

env.setParallelism(4);
env.fromElements("a", "b", "c")
   .map(kata -> kata.toUpperCase()).setParallelism(2)
   .print();

env.setParallelism(4) sets a default of 4 for the whole job, and setParallelism(2) overrides the map operator's parallelism to 2. Remember that parallelism runs within the limits of available slots — if slots run out, the job waits.

#Operator Chaining

Flink automatically merges adjacent stateless operators into one task to reduce overhead. This chaining can be adjusted with disableChaining() or startNewChain() — we'll discuss it as a tuning strategy in episode 15.

source → map → filter → keyBy → window → reduce → sink

The graph above illustrates the pipeline you learned in this episode: elements flow from the source, get transformed, partitioned by key, aggregated in a window, then sent to the sink.

#Conclusion

Episode 4 equipped you with the core DataStream API transformations: map for one-to-one transformation, flatMap for one-to-many, filter for filtering, keyBy for logically partitioning the stream, plus window and reduce for time-based and key-based aggregation. You also understood the parallelism and chaining that determine how a pipeline is executed.

The key takeaways:

• Every job follows the source → transformation → sink pattern with env.execute at the end.
• flatMap is the most flexible transformation because it can emit zero to many elements.
• keyBy groups data per key so stateful computation and aggregation run consistently.
• reduce aggregates elements incrementally without storing the whole window contents.
• Parallelism can be set globally or per-operator, and is limited by the number of available slots.

In the next episode, episode 5, we'll discuss event time, watermarks, and windowing — understanding the difference between event time and processing time, generating watermarks and handling lateness, getting to know the tumbling, sliding, and session window types, and configuring window triggers and allowed lateness. This is the foundation for accurate time-based analytics.