Learn Puppet - Performance & Troubleshooting

#Introduction

In episode 18 you built a testing pyramid for Puppet code: lint, syntax validation, unit testing with rspec-puppet, and acceptance testing with Beaker in CI. Tested code is a good foundation — but in production, correct code isn't enough. It also has to run fast and be easy to fix when something goes wrong.

A Puppet infrastructure serving hundreds to thousands of nodes faces new challenges: increasingly slow catalog compilation, delayed reports, a bloated PuppetDB, and a stream of mysterious agent-side errors. Without the ability to analyze and fix, these grow into hard-to-trace service disruptions.

This episode covers the operational side of Puppet: how to read the timing of each agent run stage, analyze catalog compilation and report lag, tune Puppet Server on the JVM side, and troubleshooting techniques for the most common problems like certificate mismatch, catalog compilation errors, resource conflicts, and wrong facts.

#The Anatomy of One Agent Run

Before analyzing performance, we need to understand what happens when puppet agent -t runs. A normal agent run goes through this sequence:

The time of each stage can be seen in the agent's verbose output:

puppet agent -t --evaltrace

Info: Applying configuration version '1739000000'
Info: Evaluated state of File[/etc/nginx/nginx.conf] (0.03 seconds)
Info: Evaluated state of Service[nginx] (0.10 seconds)

If --evaltrace shows one resource taking far longer than others, that's the point to investigate — for example, an exec waiting on a network timeout or a package being downloaded.

#Analyzing Catalog Compilation

Catalog compilation is the most CPU-expensive stage. Every agent run triggers a compile process on Puppet Server, and every compile needs a JRuby instance from the pool. When the number of agents spikes at the same time — for example, all nodes running at the same minute — the queue piles up and agents start waiting.

The main server health indicator is at the Trapperkeeper status endpoint:

curl -s https://localhost:8140/status/v1/services \
  --cert /etc/puppetlabs/puppet/ssl/certs/puppet-server.example.net.pem \
  --key /etc/puppetlabs/puppet/ssl/private_keys/puppet-server.example.net.pem \
  --cacert /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem | jq

In addition, Puppet Server exposes metrics that can be connected to Prometheus and Grafana. The most important metrics to watch: average compile time, number of queued requests, and compilation error rate.

One of the most common causes of queue buildup is all agents running at the same time. Spread the run schedule with splay:

[agent]
server = puppet.example.net
runinterval = 30m
splay = true
splaylimit = 5m

With splay = true and splaylimit = 5m, each agent randomly delays its run by up to 5 minutes after the interval, spreading the server load instead of piling it up at one point.

#Report Lag and PuppetDB Load

Every agent run produces a report sent to the server, then forwarded to PuppetDB for storage and querying. If report volume is high, PuppetDB can become the bottleneck. The main symptoms: reports appear in the console with a delay, and the puppetdb or pe-puppetdb process uses very large amounts of memory.

The most common cause of report lag: historical data piling up without limits. PuppetDB provides TTL settings to clean up old data:

[global]
# Delete reports older than 14 days
report-ttl = 14d
# Delete nodes that haven't reported in 30 days
node-ttl = 30d
# Purge nodes and related data 7 days after node-ttl
node-purge-ttl = 7d

After changing the config, restart PuppetDB. To monitor the volume of stored data, use a simple PQL query:

puppet query 'reports[catalog_format, end_time, start_time, noop] {
  latest_report = true and certname = "web01.example.net"
}'

#Tuning Puppet Server on the JVM Side

Puppet Server runs on the JVM and JRuby. The two setting groups with the most impact: JVM heap size and JRuby pool size.

JVM heap size is set in the server's sysconfig file:

JAVA_ARGS="-Xms2g -Xmx4g -XX:MaxMetaspaceSize=512m"

A common rule of thumb: provide around 4-8 GB of heap for a server serving thousands of nodes, and make sure MaxMetaspaceSize is enough to avoid Metaspace errors.

The JRuby pool is configured in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf:

jruby-puppet:
  max-active-instances: 4
  max-requests-per-instance: 100000
  max-queued-requests: 0
  max-retry-delay: 1800

• max-active-instances determines how many requests can be compiled in parallel. Default 4; raising it means using more CPU and memory.
• max-requests-per-instance forces JRuby to reset after a number of requests to prevent memory leaks.
• max-queued-requests limits the queue; if exceeded, the server returns 503 so agents don't pile up in the queue.

#Common Troubleshooting

#Certificate Mismatch

One of the most frequent errors early in setup:

Exiting; no certificate found and waitforcert is disabled

Or on the server side:

Could not find certificate request for node web01.example.net

Common causes: the agent's hostname differs from what was registered, or an old certificate is still stored. The fix: make sure the agent's certname is consistent, then check the certificate status:

sudo puppet ssl clean             # on the agent, removes old SSL
sudo puppet cert list             # on the server, lists certificate requests
sudo puppet cert sign web01.example.net

#Catalog Compilation Error

Compilation errors usually show up in the server logs and agent reports. The most common causes are references to resources that don't exist or hiera data errors. The best way to see the details is to ask the agent to show the full trace:

sudo puppet agent --test --debug --trace

The --trace flag shows the call stack down to the location of the problematic code, while --debug surfaces details like the hiera data used and evaluation decisions. The combination of both is the main troubleshooting weapon.

#Resource Conflict

A duplicate declaration error happens when two resources with the same type and title are declared twice:

Error: Duplicate declaration: File[/etc/nginx/conf.d/app.conf] is already declared

Typical causes: a class included twice through different paths, or two resources using the same title. The fix: make sure every resource has a unique title, or use contain, require, and virtual resource patterns from episode 8 to control evaluation.

#Wrong Facts

Sometimes the catalog chooses the wrong branch because facts are off — for example an old OS version stored in the cache. Facts are cached by Facter. To refresh them:

sudo rm -rf /opt/puppetlabs/facter/cache
sudo facter -p os.release.major

If the wrong facts come from a custom fact or facts delegated from outside, check that the fact is actually produced and isn't being overwritten by hiera data.

#Quick Diagnostic Checklist

When production is having trouble, follow this order:

puppet agent --test --debug --trace
puppet agent --test --noop --evaltrace
puppet config print server certname environment
puppet lookup --node web01.example.net nginx::worker_processes
puppet query 'nodes[certname] { latest_report_status = "failed" }'

From the resulting output, you can decide whether the problem is in the network, the certificates, the compilation, or the data — then move to the right fix. The last query finds all nodes whose latest report failed, so fixes can be aimed at the nodes that actually have problems.

#Conclusion

In this episode we covered the operational side of Puppet: the anatomy of an agent run and how to measure each stage, catalog compilation and report lag analysis, limiting PuppetDB load through TTLs and run schedule spreading, tuning Puppet Server on the JVM and JRuby pool side, and troubleshooting the most common problems like certificate mismatch, catalog compilation errors, resource conflicts, and wrong facts with puppet agent --debug --trace.

Key takeaways:

• Measure before optimizing — use --evaltrace, the status endpoint, and PuppetDB metrics to find the real bottleneck.
• Spread agent load with splay and splaylimit so there's no simultaneous run spike.
• Limit data growth in PuppetDB via report-ttl, node-ttl, and node-purge-ttl.
• Tune the JVM gradually — raise heap and max-active-instances little by little while monitoring memory.
• --debug --trace is your best friend when facing unclear agent errors.

In the next episode, episode 20, we'll cover the latest stable features — what's new in Puppet 8 open source and Puppet Enterprise 2025.1, including Security Compliance Management and role-based node management. See you in the next episode!