Learn Pentaho - Metadata, Repository, & Version Control

#Introduction

So far you've built transformations that work. Episode 9 addresses the question that arises as soon as a project grows: how do you keep all connections, configurations, and files consistent and traceable? This is about metadata, repository, and version control — the foundation of team collaboration and long-term project continuity.

You'll learn to manage connections as shared objects, choose between the Pentaho repository and file-based storage, apply Git to .ktr/.kjb files, and use variable substitution to separate configuration between environments.

#Connection Metadata and Shared Objects

In PDI, metadata is "data about data" — the definitions of connections, fields, and data sources used in many places. Two key concepts:

• Database connections: connections defined once and reused by every transformation in your workspace. Changing one connection applies immediately to all files that use it.
• Shared objects: objects that can be shared between transformations — database connections, step connections, value mappers, and others. Stored in the Shared Objects folder on the tree panel.

The recommended practice: save database connections as shared objects at the Transformation or Job level, rather than rewriting them in every file. This prevents hard-to-trace differences in connection parameters between files.

You can also store connections as .properties files in the project folder — some teams prefer this approach because it goes along with version control.

#Pentaho Repository vs File-Based Storage

There are two ways to store and manage PDI objects:

For local development and learning, file-based plus Git is the healthiest choice — simple, portable, and free of server dependencies. The Pentaho repository is more attractive when you work in large teams needing access control and direct integration with the Pentaho Server.

#Version Control for PDI Projects with Git

.ktr and .kjb files are actually XML — so they can be version-controlled like regular code. This changes how teams work: reviewing changes, rolling back, and having a history of who changed what.

Start by making sure your project lives in a Git repository, then get used to committing with messages that explain the change:

git init
git add transformations jobs resources
git commit -m "feat: add order staging transformation from CSV"

Add a .gitignore file so temporary files and personal environments aren't committed:

*.log
kettle.properties
.tmp/

You can view the change history with git log and compare two file versions with git diff. Because the files are XML, diffs can be a bit noisy — but small changes like a wrong connection or SQL are easy to recognize.

#Managing Environments and Variable Substitution

A classic problem in data projects: the same code must run in different environments (development, staging, production) with different connections and paths. PDI's solution is variable substitution — configuration values are stored in variables, not hardcoded in files.

The levels where variables are determined, from most specific:

• Parameters: passed at runtime, for example -param:ENV=dev.
• Variables in a job entry: set inside a job, for example the result of a Set variables step.
• Variables in a properties file: global in kettle.properties.
• System default values.

Example: instead of writing the database host in every file, use the variables DB_HOST, DB_PORT, DB_NAME, and DB_PASSWORD. In Spoon, just fill the host, port, and database name fields with the variable names — PDI will replace them with their values at execution time, following the substitution syntax recognized by the PDI engine. When running from the command line, provide the values:

pan.sh -file=etl_order.ktr -param:DB_HOST=localhost -param:DB_NAME=lab

With this pattern, one identical .ktr file can run in both development and production — only the parameters differ. This is the foundation of environment management, covered more deeply in episodes 14 and 19.

#Structuring a Maintainable Project Layout

A good folder structure is metadata in its own right. A standard widely used by PDI teams:

project-name/
├── transformations/
│   ├── staging/
│   ├── dimension/
│   └── facts/
├── jobs/
│   ├── daily/
│   └── monthly/
├── resources/
│   ├── sql/
│   └── files/
├── scripts/
└── .gitignore

Separating by function (staging, dimension, facts) and frequency (daily, monthly) means the team immediately knows where to find things. Combine this structure with variable substitution and Git, and your project is ready to be worked on by many people.

#Conclusion

In episode 9 you organized your project assets: managing connections as shared objects, understanding the difference between the Pentaho repository and file-based storage, applying Git to PDI files, and using variable substitution to separate configuration between environments.

The key takeaways:

• Save connections once as shared objects to stay consistent across all files.
• File-based plus Git suits development; the Pentaho repository is for centralized control and large-scale collaboration.
• .ktr and .kjb are XML — treat them like code with commits and reviews.
• Variable substitution separates configuration from logic; never hardcode credentials in files.

In episode 10, we set the stage for production: Pentaho Server & web console — setting up the server, uploading jobs and transformations, scheduling execution via the BI Server, and managing user access and roles.