Learn Pentaho - Monitoring & Observability

#Introduction

An unmonitored data platform is a time bomb. Jobs can fail silently, performance degrades slowly, and no one knows until users complain. This episode covers monitoring and observability — how you know the platform is healthy, find problems early, and understand their causes.

You'll learn to monitor job executions, read and capture logs, measure resources and throughput, and integrate Pentaho into external monitoring tools when needed.

#Monitoring Job and Transformation Executions

There are several levels where job execution is monitored:

• In Spoon: during development, the Execution results tab shows the history and status of every run.
• On the Pentaho Server: Schedules in PUC shows each job's schedule, status, and last execution log.
• On the command line: the logfiles produced by Pan and Kitchen become the trail for scripts or external tools to monitor.

For jobs running in cron or CI/CD, execution results should always be written to a logfile and, when they fail, send a notification. The pattern you already built in episode 5 — a failure hop to email — is the basis of reactive monitoring.

Get into the habit of writing logs to a file in a consistent location:

kitchen.sh -file=etl_daily.kjb -level=Basic -logfile=/var/log/pentaho/etl_daily.log

With a stored logfile, monitoring scripts or external tools can read execution results programmatically. While a job is running, monitor it live with tail -f /var/log/pentaho/etl_daily.log to see progress without waiting for it to finish.

#Structured Logging with Database Logs

Besides text logfiles, PDI can write logs to a database via Logging configuration — every job/transformation records executions, steps, and errors into special tables. For jobs running on the Pentaho Server, these log tables are managed automatically and can be queried directly for operational reporting, for example:

SELECT COUNT(*) AS gagal
FROM r_job
WHERE status = 'STOPPED'
  AND log_date > CURRENT_DATE - INTERVAL '7 days';

With queryable log tables, teams don't have to guess pipeline health from memory — the answer is in the database and can be put into a monitoring dashboard.

#Log Analysis and Error Capture

Logs are the source of truth during investigation. Things you should watch:

• Log levels: Error, Minimal, Basic, Detailed, Debug, Rowlevel — choose per need. Production is usually Basic; Debug only while investigating.
• Common error patterns: ORA-... or SQL error for database problems, File not found for file problems, and Unknown type for wayward metadata.
• Time correlation: compare the error time with other events — for example a database crash or a configuration change.

To capture errors automatically, the scenario is usually: grep the logfile for error lines, then trigger a notification or open a ticket. An example of searching for error patterns in a logfile:

grep -i "error\|exception\|failed" /var/log/pentaho/etl_daily.log

You can see the suspicious parts of the log with the command above. For a more structured investigation, open the full logfile in an editor and trace from the first strange message.

#Resource and Throughput Metrics

Besides logs, you need to monitor quantitative metrics:

• Server resources: CPU, RAM, and disk of the machine running PDI or the Pentaho Server. A Java server running out of memory is a leading cause of mysterious failures.
• ETL throughput: rows per second and the duration of each job. Gradual degradation signals growing data or the need for optimization (the topic of episodes 14 and 15).
• Error count: error trends between days — if rising, something changed in the data source.

For PDI, the Step Metrics tab in Spoon already shows each step's throughput. For the long term, you need to record the duration and status of every run — in logs, a spreadsheet, or a metrics database.

#Key Metrics to Monitor

So monitoring doesn't drown you in numbers, focus on the five metrics that tell the most:

• Execution duration: daily trend — rising means data is growing or there's a new bottleneck.
• Rows read vs written: a big gap means many rows are discarded — validation or filters too aggressive.
• Error count: zero is normal; a sudden rise is an alarm.
• Server CPU and memory: a persistent rise signals heap tuning is needed (episode 14).
• Scheduler slots: jobs waiting in queue indicate an overloaded server.

Record these metrics in one place — a spreadsheet, database, or monitoring tool — every time a job runs, so day-to-day comparison becomes easy.

#Integration with External Monitoring Tools

For organizations that already have a monitoring infrastructure, Pentaho must join it. Common integration patterns:

• Log collection: route PDI and Pentaho Server logs to a log aggregator like the Elastic Stack or Loki, so log search is centralized.
• Metrics export: export job duration and status as metrics to Prometheus (via an exporter or script) for visualization in Grafana.
• Alerting: build alerts based on conditions — for example a job failing twice in a row or duration exceeding the normal threshold.
• Health check API: use the Pentaho REST API to check server status from a monitoring system.

Here's a look at a scenario: a script reads the logfile, extracts job duration, then publishes metrics queryable by a monitoring tool:

grep "Finished job entry" /var/log/pentaho/etl_daily.log | tail -1

The result can become a throughput metric or alerting input. This is just a simple bridge; for full scale, build a proper metrics pipeline in episodes 17-18.

#Conclusion

In episode 13 you equipped yourself to monitor the platform: monitoring job executions at various levels, analyzing logs and capturing errors, monitoring resources and throughput, and integrating Pentaho with external monitoring tools.

The key takeaways:

• Log to a file with the right level in production, then monitor the results programmatically.
• Monitoring tells you "what's wrong", observability helps you understand "why".
• Server resources and job throughput are core metrics to measure continuously.
• Integration with log aggregators and alerting brings the platform into the company's operational ecosystem.

In episode 14, we push performance: deployment & scalability — comparing standalone and clustered architectures, scaling transformations with parallel execution, tuning the resources of Spoon, Carte, and the Pentaho Server, and considering load balancing and failover.