Learn Chef - Performance & Troubleshooting

#Introduction

In episode 18 you tested cookbooks with Test Kitchen and made sure they work in a controlled environment. But in the real world, a cookbook does not run once — it runs continuously on hundreds of nodes, on a schedule, and sometimes fails in ways that never appear in CI. Episode 19 equips you with the two skills most used in the field: analyzing run performance and troubleshooting when something does not behave as expected.

#Anatomy of a Chef Client Run

Every chef-client run passes through phases that can be imagined as a track:

1. Ohai collects node facts (OS, memory, network).
2. Chef downloads cookbooks and evaluates the run_list.
3. Recipes are compiled into a collection of resources.
4. Resources are converged — applied to the node.
5. Reports are sent to the server or handlers.

Most of the time is spent in the compile and converge phases. If a run slows down, the first question is always: which phase is eating the time? The answer is in the log.

#Analyzing Runs with Debug Logs

The default chef-client log only shows a summary. To dissect every step, run with debug level:

chef-client -l debug

The debug output shows every evaluated resource, its idempotency decision, and the reasoning behind the action taken. Combined with an interval, it becomes useful for scheduled runs:

chef-client -l debug -i 3600

The -i parameter sets the interval in seconds — a value of 3600 means chef-client runs every hour. In this mode chef-client stays in memory; it is more efficient than starting a new process each time, because cookbooks, Ohai facts, and the server connection are cached.

#Why-Run Mode: Trying Without Changing

Before a real run, -W (why-run) shows what chef-client would do without actually applying anything:

chef-client -W --runlist 'recipe[webserver::default]'

Why-run is the best tool for testing a new cookbook on an already-running node. If a resource is non-idempotent, why-run will show it — the resource always "wants to change" even though the state is already correct. That occurrence is the first red flag in a performance audit.

#Scheduling: systemd Timer vs Cron

Chef-client provides the chef-client.service and chef-client.timer units that can be enabled directly on Linux systems:

systemctl enable --now chef-client.timer
systemctl list-timers | grep chef

0 */2 * * * /usr/bin/chef-client -j /etc/chef/run_list.json > /dev/null

The systemd timer wins because it can trigger a service exactly on time, catch up on missed runs, and has centralized logging via journald. For simple needs, cron works too. A common production schedule: a 15–30 minute interval for application nodes, and longer pauses for batch nodes. More important than the exact numbers is consistency — a run that never happens is configuration that never gets converged.

#Caching for Performance

Chef-client performs several kinds of caching that you can control:

• Ohai facts are cached to avoid rescanning devices every run.
• Cookbooks are cached per version on the node, so they are not re-downloaded if unchanged.
• Packages are cached by the package manager (apt/dnf) as long as the resource uses the same version.

The remote_file resource also supports the use_conditional_get and use_etag properties so unchanged files are not re-downloaded:

remote_file '/opt/tools/tool.tar.gz' do
  source 'https://cdn.example.com/tool.tar.gz'
  owner 'root'
  mode '0644'
  use_conditional_get true
end

Avoid disabling caching without a reason — a node that re-converges every run with a full download is a waste of bandwidth and time.

#Common Troubleshooting

Field experience shows five problems dominating the rest. Let's discuss them one by one.

#Cookbook Not Uploaded

Symptom: the run fails with a cookbook not found message. Cause: the cookbook on the server is older than the version on the workstation. The fix:

knife cookbook upload webserver
knife cookbook list

Get into the habit of uploading after every change you are about to release, and do not forget to check the cookbook version on the server with knife cookbook show webserver.

#Wrong Run List

Symptom: the node does not get the expected configuration, even though the recipe seems to be written correctly. Cause: the run_list points at the wrong or incomplete recipe. Check what the node actually runs:

knife node show web-01 -a run_list

Fix it by updating the node's run_list. Always verify the run_list after bootstrap, because an error here silently makes a node "look healthy" when it is actually not managed.

#Authentication Error

Symptom: Failed to authenticate or an SSL error. Cause: the client key is missing or has been rotated, or the server fingerprint changed. Check the credentials in /etc/chef/client.pem, validate the connection with knife ssl check, and make sure the correct certificate authority exists in /etc/chef/trusted_certs. If a node was rebuilt, its client key must be regenerated and re-registered with the Infra Server.

#Non-Idempotent Resource

Symptom: why-run always shows a change, or the run always flags "updated" even though the state is already correct. Cause: the resource action or properties do not take advantage of built-in idempotency. A classic example — writing a file whose content gets overwritten without any check. The fix is to use a declarative resource:

template '/etc/nginx/nginx.conf' do
  source 'nginx.conf.erb'
  mode '0644'
  notifies :reload, 'service[nginx]'
end

The template resource only rewrites a file if its content changed. Use execute with a not_if or only_if property for commands that should only run under certain conditions, so they are not re-executed needlessly.

#Dependency Conflict

Symptom: the run fails because cookbook versions clash, usually when metadata.rb demands depends constraints that are too strict. Solution: use looser version constraints, for example requiring only a minimum version, and better still — move to Policyfile, which locks the entire dependency tree in a lockfile. Conflicts spreading across many cookbooks are a sign to invest in Policyfile from the start.

#Conclusion

Episode 19 closes the operational side with the ability to read runs precisely: debug logs as a slow-motion camera, why-run as a risk-free test, systemd timer and cron as schedule conductors, caching as a cost saver, and the troubleshooting catalog as a first-aid kit.

Key takeaways:

• Analysis starts with the phase — determine first whether the slowness is in Ohai, compile, or converge.
• Why-run is a safety net — run chef-client -W before a real run on an existing node.
• Scheduling needs consistency — a chef-client that is never run converges nothing.
• Most failures follow patterns — upload, run list, authentication, idempotency, and dependencies; recognize the symptoms, apply the fixes.

In episode 20 we will look at the latest stable features: Chef Infra Client 19 built on Habitat, standard licensing, Ruby 3.4, the oci? cloud helper, and the Infra Server 15 updates with Valkey and PostgreSQL 14. See you there!