Learn Puppet - Production-Ready Architecture

#Introduction

In episode 20 you got to know the latest features in Puppet 8 and Puppet Enterprise 2025.1, including Security Compliance Management and role-based node management. Now it's time to bring it all together into one big picture: what a Puppet architecture actually looks like in a production environment at the thousand-node scale.

In a local lab, a single Puppet server serving a few nodes feels simple. In production, there are many new questions: how do you keep the server available when one of them dies? How do you make sure deployed code is always consistent across all compilers? How do you prevent slow agent runs from overloading the whole pipeline? This episode answers those questions.

We'll cover the production Puppet architecture — Puppet Server with high availability, PuppetDB on PostgreSQL, Code Manager and r10k, and centralized reporting — then connect it to a complete CI/CD pipeline from PDK and rspec-puppet to canary environments, and close with monitoring and backup practices that keep the whole system healthy.

#The Production Architecture Map

A mature production Puppet architecture consists of several components, each with a role:

The logical flow goes roughly like this: agents talk to Puppet Server, the server calls hiera and the classifier for compilation, the result is sent to the agent, the agent applies it and sends a report, the report goes into PuppetDB, and the console and dashboards read from PuppetDB.

#High Availability for Puppet Server

The most critical component is Puppet Server. If it goes down, every node loses the ability to run. A common HA strategy is placing a replica as a backup compiler: environments, modules, and hiera are synced from the primary to the replica periodically.

To make agents automatically fail over to the replica when the primary fails, use the server_list setting on agents:

puppet config set server puppet-primary.example.net
puppet config set server_list puppet-primary.example.net:8140,puppet-replica.example.net:8140

With server_list, the agent tries the first server, then moves to the next one if it fails. This is far more resilient than relying on a single server address. Remember that the Certificate Authority should stay on the primary so certificate signing isn't split across servers.

#PuppetDB and PostgreSQL

PuppetDB stores all operational data: facts, catalogs, and reports. Everything is stored in PostgreSQL. Because the data volume is large, there are a few important practices:

• Separate servers — don't put PuppetDB on the same machine as the heavy-load compiler.
• Set TTLs — report-ttl, node-ttl, and node-purge-ttl limit data growth as in episode 19.
• Monitor growth — watch database size regularly so you don't run out of disk.

A quick query to check database size:

SELECT datname, pg_size_pretty(pg_database_size(datname))
FROM pg_database ORDER BY pg_database_size(datname) DESC;

A healthy PostgreSQL keeps PQL queries and the console dashboard responsive. If PuppetDB's report processing queue builds up, add threads or move the database to its own machine.

#Code Manager and r10k: A Single Code Source

In production, code must not be edited directly on the server. The correct flow: code lives in Git, then is distributed by r10k or Code Manager to Puppet environments. A common control repository structure:

production/
  hieradata/
  manifests/
  modules/
  site/
Puppetfile

The Puppetfile declares modules from the Forge or Git with pinned versions:

forge 'https://forge.puppet.com'
 
mod 'puppetlabs/stdlib', '9.4.0'
mod 'puppetlabs/nginx', '5.2.0'
mod 'company-role-web',
    git: 'https://git.company.net/puppet/role-web.git',
    ref: 'main'

Deployment is done with the following commands, and Code Manager keeps reproducibility by deploying to a specific commit:

puppet code deploy production --wait
puppet code deploy production --version <commit-sha> --wait

#CI/CD: From PDK to Canary

A healthy production pipeline connects the testing from episode 18 with deployment. The flow looks roughly like this:

feature branch -> pdk validate + lint + unit test -> pull request -> main
main -> puppet code deploy production -> canary group -> full rollout

The stages:

1. PDK and rspec-puppet — every branch goes through lint, validate, and unit tests in CI before being merged.
2. Puppetfile and Code Manager — after merging to main, Code Manager pulls the code and distributes it to the production environment.
3. Canary environment — a small group of nodes, for example 5-10 percent of the fleet, runs the new code first. If reports show errors or unexpected changes, roll back to the previous commit:

puppet code deploy production --version <previous-commit> --wait

The canary principle matters because a catalog that passes unit tests isn't necessarily correct in production — production hiera, real facts, and network conditions can only be tested on actual nodes.

#Centralized Reporting and Monitoring

All agent reports gather in PuppetDB, and from there they're consumed by various consumers: dashboards to see the percentage of nodes that are noop, unchanged, failed, or corrective; alerting systems for key metrics; and Grafana reading PuppetDB metrics.

Minimum monitoring practices to maintain:

#Backup and Recovery

Backup is half of a production architecture. Components that must be backed up:

tar -czf puppetserver-backup.tar.gz \
  /etc/puppetlabs/puppet/ssl \
  /etc/puppetlabs/code \
  /etc/puppetlabs/puppetserver/conf.d
pg_dump -Fc puppetdb > puppetdb.dump

Minimum backup contents: the CA keys and certificates in /etc/puppetlabs/puppet/ssl, the control repo and hiera in /etc/puppetlabs/code, the server configuration, and a PuppetDB database dump. Store backups in a separate location and test restoration periodically — a backup that's never tested is a backup that doesn't exist.

#Conclusion

In this episode we designed a production-ready Puppet architecture: Puppet Server with high availability and failover via server_list, PuppetDB on PostgreSQL with TTL management, Code Manager and r10k as the single code source, agents across thousands of nodes, and centralized reporting and monitoring. We also built a CI/CD pipeline from PDK and rspec-puppet to canary environments, and made sure PuppetDB and CA key backups aren't forgotten.

Key takeaways:

• Components are separated — the compiler, PuppetDB, PostgreSQL, and Code Manager each have their own load and tuning needs.
• Code flows from Git — nobody edits directly on the server; everything goes through the Puppetfile and Code Manager.
• Canary before full scale — verify on real nodes before code spreads to the whole fleet.
• Monitor and alert — compilation, queue, and PuppetDB metrics are early signs of trouble.
• Back up, then test the backups — CA keys and the PuppetDB database must be backed up and their restoration routinely tested.

In episode 22 — the final episode of this series — we'll step back and look at the ecosystem: comparing Puppet with Ansible, Chef, SaltStack, and Pulumi/Terraform, then summarizing the whole journey from episode 0 to 21. See you there!