Learn Pentaho - Big Data Integration

#Introduction

Data no longer lives only in relational databases. Large volumes now reside in Hadoop, cloud data lakes, and streaming platforms. Episode 17 covers big data integration — how Pentaho works within that ecosystem.

You'll learn integration with Hadoop and Spark, understand columnar file formats like Parquet, Avro, and ORC, apply near real-time ingestion patterns, and see Pentaho's position in the modern data lakehouse architecture.

#Pentaho and the Hadoop Ecosystem

PDI talks to the Hadoop ecosystem through the Pentaho Big Data plugin. Its main capabilities:

• Reading and writing to HDFS (Hadoop Distributed File System) as a source and target.
• Interacting with Hadoop services: Hive, HBase, and Spark.
• Running transformations as cluster jobs.

The two most commonly used patterns:

• ELT with Hive/Spark: PDI manages orchestration, while heavy transformations execute on the big data engine. PDI doesn't pull big data into itself.
• Writing formatted files: PDI writes data into efficient columnar formats for analytical engines to read.

An important principle: don't pull millions of rows from a cluster into the PDI machine. Let the heavy load run where the data lives.

#Understanding File Formats: Parquet, Avro, and ORC

CSV is simple but wasteful: no schema, no compression, and slow to read. For big data, three modern formats you must know:

All three store the schema along with the data, so there's no separate file to keep in sync. In PDI, writing to these formats is usually done through the Pentaho Big Data plugin or by using the Spark execution engine for conversion.

#Choosing a Format for Your Work

A selection rule of thumb: if the workload is column scans for analytics (averages, date filters), Parquet or ORC are far more efficient because only the needed columns are read from disk. If the workload is an event pipeline that reads whole rows one by one, Avro is more natural because it's row-oriented and supports schema evolution. Many modern lakehouses adopt Parquet as the de facto standard — so the skill of reading and writing Parquet is one you'll almost always use.

Here's a look at the configuration to prepare when working with Hadoop in PDI — a properties file pointing to your cluster:

pentaho.hadoop.configurations.path=/home/kalian/hadoop-config
pentaho.authentication.default.application.id=guest

The values above are placeholders — adapt them to your cluster. For a lab without a real cluster, many teams use a MiniCluster or standalone Spark to practice.

#HDFS as Source and Target

In PDI, the steps working with HDFS follow the same pattern as regular file steps: Hadoop File Input and Hadoop File Output replace Text file input/output when the target is a distributed file system. What you should understand early is the difference between namenode and datanode: the namenode stores metadata and file addresses, while datanodes store the actual data. A wrong namenode address is the most common cause of confusing HDFS connection failures for beginners — so check this value first when a connection fails without an obvious reason. To verify access from the command line, try hdfs dfs -ls / before relying on the PDI connector.

For those who've never touched the Hadoop ecosystem, don't worry about the many terms. For use through PDI, you only need to recognize four concepts: HDFS as the place for files, Hive as the SQL table interface, HBase as column storage, and Spark as the compute engine. PDI connects to all of them, and the skill of choosing the right file format (the previous section) will be used more often than understanding the internal details of each component.

#Streaming Ingestion and Near Real-Time ETL

A daily batch cycle isn't always enough. For near real-time needs, the patterns used:

• File-based ingestion: monitor a folder where other systems drop files, then process immediately when a file appears. In PDI, this pattern is done via a periodically running job with the File exists or Get file names step.
• Message queue: consume events from Kafka or MQ as a source. PDI has connectors for messaging systems, and this pattern is combined with frequent small batch processing.
• Short periodic execution: run the job every 5 minutes (instead of daily), processing data that arrived since the last run.

This isn't true streaming like Kafka Streams, but micro-batching — and it's enough for most operational needs. An example of running an ingestion job at short intervals via cron:

*/5 * * * * /home/kalian/lab/pdi-ce/kitchen.sh -file=/home/kalian/lab/jobs/ingest_orders.kjb -level=Basic

Note the pattern above: cron triggers Kitchen every five minutes. With an idempotent design (episode 7), micro-batches are safe to rerun without duplicating data.

Also note how to avoid execution overlap: if the ingestion process takes longer than the schedule interval, two runs can happen simultaneously and interfere with each other. The simple solution — a Locked job entry or checking for a still-running Kitchen process before starting a new run. This is an operational problem that often appears in micro-batching and is rarely covered in tutorials.

#Data Lakehouse and Pentaho's Position

A data lakehouse combines the flexibility of a data lake (holds any format, low cost) with the reliability of a data warehouse (schema, transactions, quality). In this architecture:

• Pentaho can act as an orchestrator and transformer: preparing data from various sources, writing to lakehouse tables in columnar format.
• Its position is often as a data mover: moving data between layers — landing zone, curated zone, up to the analysis-ready layer.
• Combined with engines like Spark, heavy transformations run in the lakehouse, not on the PDI machine.

For you, the most important understanding isn't the product catalog, but the pattern: from any source, data passes raw → processed → analytics-ready, and Pentaho is responsible for maintaining quality and schedules at every layer transition.

#Conclusion

In episode 17 you understood big data integration: Pentaho's integration with Hadoop, Spark, and cloud data lakes; the differences between the Parquet, Avro, and ORC formats; streaming ingestion and near real-time patterns; and Pentaho's position in the data lakehouse architecture.

The key takeaways:

• Don't pull big data into PDI — let heavy transformations run where the data lives.
• Parquet, Avro, and ORC beat CSV for large volumes; the schema is stored with the data.
• Micro-batching with short schedules covers most near real-time needs.
• Healthy ETL patterns still apply in big data; what changes is the engine and format.

In episode 18, we enter the analytics world: analytics & machine learning workflow — preparing data pipelines for analysis, integrating R, Python, and Weka for machine learning, building predictive data flows, and visualizing results and model outputs.