Learn Pentaho - Jobs & Orchestration

#Introduction

In episode 4 you built a single transformation that processes one data flow. In the real world, ETL work is never that simple: data must be pulled from several sources in sequence, loaded after other processes finish, and reported to the team by email. This is where jobs come in.

A job is an orchestrator. It doesn't process rows of data; it manages when and in what order transformations and other steps run, including what happens when one step fails. This episode will teach you how to design reliable jobs.

#Job vs Transformation: Division of Roles

Before building a job, recall the division of roles covered in episode 2:

• Transformation processes data: reads, transforms, and writes a row stream.
• Job manages the flow: runs transformations, shell scripts, file transfers, sends emails, and makes decisions based on results.

A job always starts from the START step and runs other job entries sequentially or conditionally. The difference from transformations: hops in a job don't carry data, only success or failure status.

Create a new job in Spoon via File > New > Job. You'll see a canvas with a Job entries palette that differs from the transformation step palette.

#Popular Job Entries You Should Know

Some job entries are the backbone of orchestration. Get to know them now:

• START: the starting point of execution. Every job begins here.
• Transformation: runs a .ktr file. This is the most used entry.
• Job: calls another job in a nested manner, useful for splitting up large orchestrations.
• Shell: runs operating system commands or scripts, for example moving files with mv or calling external programs.
• FTP / SFTP: downloads or uploads files to remote servers — common for cross-system data exchange.
• Email: sends notifications by email after a process finishes or fails.
• Copy/Move files: moves already-processed files to an archive folder — an important habit for tidy pipelines.
• Success / Failure: marks the final result of the job and stops execution as appropriate.

The most common chain: START to the first Transformation, then to the next transformation, then to Email at the end. Let's build it.

#Assembling Your First Job

Follow these steps to assemble a simple job: pull data from CSV, load it into the database, archive the source file, then send a notification.

1. Add START on the canvas.
2. Add the Transformation job entry and point it to the csv_to_db.ktr file you created in episode 4.
3. Add the Move files job entry to move the processed CSV to the processed/ folder.
4. Add the Email job entry with a subject like ETL complete and the team's email address.
5. Connect with hops: START to Transformation, then to Move files, then to Email.

It will look like the following scheme:

START -> csv_to_db.ktr -> Move files -> Email

Now save it as etl_daily.kjb in the jobs folder. Before running it fully, learn how flow control works in the next section.

#Flow Control: Success, Failure, and Evaluation

A job's strength lies in its ability to make decisions based on results. There are several patterns you must master:

• Success hop: the default — the next step only runs if the previous step succeeded.
• Failure hop: a hop that runs precisely when a step fails, for example directing to an Email with a "process failed" subject or to Failure.
• Evaluation: the Simple evaluation job entry checks a condition (for example a variable value, a query result, or a row count) then routes the flow to a true or false path.

The classic pattern most often used in production:

1. Run the transformation.
2. If successful, move the file and send a success email.
3. If it fails, send a warning email and stop the job with a failure status.

To create a failure hop, click and hold the line from an entry then select Fail in the hop options. Note that failure hop icons are red, success hops are green, and evaluation hops are blue — these colors help you read a job design quickly.

#Running a Job with Kitchen

Just as Pan runs transformations, Kitchen is the command-line job runner. This is the command you'll use millions of times in real-world work:

kitchen.sh -file=/home/kalian/lab/jobs/etl_daily.kjb

Add useful options: -level=Detailed for more detailed logs, -param:NAMA=value to pass parameters, and -logfile=/path/etl_daily.log to write logs to a file. Logging to a file matters because when a job runs automatically in cron, there's no screen to read the output.

#Scheduling with Cron

A reliable job must run automatically. On Linux, the simplest way is cron. Open the crontab with crontab -e then add the following line to run the job every day at 02:00:

0 2 * * * /home/kalian/lab/pdi-ce/kitchen.sh -file=/home/kalian/lab/jobs/etl_daily.kjb -level=Basic -logfile=/var/log/etl_daily.log

For more centralized scheduling needs — schedule dashboards, retries, notifications — episode 10 will introduce the Pentaho Server with its much more complete Scheduler. For now, cron is enough to understand the concept of automation.

#Conclusion

In episode 5 you designed your first job: getting to know popular job entries like START, Transformation, Shell, FTP, and Email; managing success, failure, and evaluation flow control; running jobs with Kitchen; and scheduling them with cron.

The key takeaways:

• Jobs orchestrate steps; hops in a job carry success/failure status, not data.
• START is required, and the "success to the next step, failure to a warning email" pattern is industry standard.
• Kitchen is the Pan of jobs — always write logs to a file when running automatically.
• Cron is simple and effective for scheduling; the Pentaho Server offers more complete management.

In episode 6, we dive into data quality: data quality & transformation patterns — validation, lookup, cleansing, normalization, handling missing values, deduplication, aggregation, and healthy transformation patterns.