Learn Chef - Production-Ready Architecture

#Introduction

In episode 20 you saw the new foundations of Infra Client 19 and Infra Server 15.10. Now is the time to bring together all the components you have learned over 20 episodes into a single production-ready whole. Episode 21 is not about new features, but about architecture: how Workstation, Infra Server, nodes, Policyfile, Automate, and the CI/CD pipeline sit together, and how that architecture is kept alive through monitoring, backup, and disaster recovery.

#The Production Architecture Map

A healthy Chef architecture has three layers:

• Workstation — where code is written, tested, and released by engineers.
• Infra Server — the central registry of cookbooks, node data, and policies; in production it runs high-availability with failover.
• Nodes — the machines that run chef-client periodically and pull policies from the server.

The flow: engineers work on cookbooks in the Workstation, upload them to the Infra Server, nodes pull policies through runs, and run results are sent back to Automate for visibility. This model is pull-based (discussed in episode 1): the node initiates communication, so the architecture works behind NAT without needing inbound connections to nodes.

#Workstation and Infra Server HA

The Workstation is a development machine — it needs no high specs, but it must keep its configuration clean. The client.pem credentials for the Infra Server are stored in ~/.chef and must never be committed. For multi-engineer management, align conventions: one shared cookbook repository, review via pull requests, and releases through Policyfile.

Infra Server HA means running several server nodes (usually three) with active-passive or multi-primary components, plus a load balancer in front. With chef-server-ctl, HA configuration is managed through a shared configuration file:

chef-server-ctl status
chef-server-ctl ring-list

The goal is not just fault tolerance — HA gives you a maintenance window: one server node can be upgraded while the others keep serving. That is what production-grade architecture means: the failure of a component is not the failure of the service.

#Policyfile in Production

In episode 10 you were introduced to the Policyfile. In a production architecture, the Policyfile is the key to deterministic releases — it locks the run_list and the entire dependency tree in a lockfile, so dev, staging, and production nodes run exactly the same policy:

name 'webserver'
run_list 'webserver::default', 'compliance::hardening'
 
cookbook 'webserver', path: 'cookbooks/webserver'
cookbook 'compliance', path: 'cookbooks/compliance'

Releasing to environments uses policy groups, not mutating cookbooks on the server:

chef update policyfile/Policyfile.rb
chef push production policyfile/Policyfile.lock.json

Every push produces a recorded lockfile version — a clear audit trail of what code runs in which environment. If a regression appears, rollback is simply pushing the previous lockfile version. That is the power of the Policyfile: a release can be undone with one command, instead of guessing at cookbook versions.

#Automate for Compliance and Visibility

Chef Automate (episode 12) is the layer that turns run data into decisions. In a production architecture it plays a dual role: compliance scanner and visibility dashboard. Nodes send run results and InSpec scan results to Automate through the data collection service; Automate displays node status, compliance with profiles, and alarms when runs fail.

{
  "name": "web-01",
  "chef_environment": "production",
  "run_list": ["recipe[webserver::default]"]
}

To have nodes send data to Automate, the data collector configuration is installed on the node — usually through attributes set by a cookbook. The result: one dashboard that answers "are my nodes compliant with policy?" without having to log into each one.

#Bootstrap Flow

New nodes enter the fleet through bootstrap: a single command from the Workstation (or CI) that installs chef-client, registers the node with the Infra Server, and runs the first run:

knife bootstrap 203.0.113.10 \
  -N web-01 \
  -E production \
  -r 'recipe[webserver::default]'

In production, bootstrap is rarely done manually. It is triggered automatically by the pipeline: when a cloud instance is created (episode 14), user-data runs the bootstrap script; when a new image is born, chef-client is already baked into it. What matters is consistency: every node is registered with a unique name, the correct environment, and a run_list that does not deviate from policy.

#CI/CD Pipeline: Test Kitchen to InSpec to Automate

A production-grade pipeline formalizes the flow you already know from episode 18:

• A commit triggers Test Kitchen against the cookbook changes.
• Passing means the cookbook is pushed to the staging environment via the Policyfile.
• InSpec scans staging to make sure the compliance profile is satisfied.
• Passing means a release to production, and the results are monitored in Automate.

steps:
  - name: kitchen test
    command: kitchen test
  - name: push policy to staging
    command: chef push staging policyfile/Policyfile.lock.json
  - name: inspec scan staging
    command: inspec exec profiles/hardening -t ssh://staging
  - name: promote to production
    command: chef push production policyfile/Policyfile.lock.json

Every gate in the pipeline is an auditable decision. No cookbook reaches production without passing Kitchen, without an InSpec scan, and without a recorded release trail.

#Monitoring Cookbook Runs and Backup DR

A chef-client run that completes on its own is not proof of health. Monitor the last run status of every node — failed, slow, or never appearing at all:

• Automate shows the run history per node and the compliance profile.
• Alerts fire when a node does not report within a certain interval — a sign of a failed run or a dead node.
• Export run metrics (duration, number of updates) to an external monitoring system for long-term trends.

Backup and disaster recovery are layered three deep:

• Infra Server: a full backup with chef-server-ctl backup (episode 9), stored off-site, and its restoration tested regularly.
• Data bags and secrets: encrypted backups, preferably through a vault — never in a repository.
• Node state: nodes can be rebuilt from policy; node data lives on the server and is covered by the server backup.

Regular recovery drills are part of the architecture — a backup that is never tested is hope, not a plan.

#Conclusion

Episode 21 assembles the production architecture from the components you have learned one by one: the Workstation as the source of change, the HA Infra Server as the center of truth, nodes as convergence points, the Policyfile as the key to determinism, Automate as the eyes and the watcher, bootstrap as the entrance, the pipeline as the gate, and backup as the final safety net.

Key takeaways:

• Architecture is three layers — Workstation, Infra Server, and nodes — with a pull-based model that works behind NAT.
• The Policyfile makes releases reversible — every environment is locked to a lockfile version that can be re-pushed.
• Automate makes compliance data — not a manual per-node activity.
• Bootstrap and the pipeline remove manual work — nodes are born from code, not from hands.
• A backup without recovery drills is not a plan — DR must be tested regularly.

In episode 22 — the final episode — we will look outward and forward: the alternative ecosystem and final reflection — comparing Chef with Ansible, Puppet, SaltStack, and also Terraform and Pulumi, then summarizing the whole journey from episode 0 to 21. See you there!