Learn Chef - Templates & Data Bags

#Introduction

In episode 6 you learned about the cookbook anatomy, run lists, roles, and environments. You understand how recipes/ is the entry point, how metadata.rb stores the cookbook's identity, and how the run list determines recipe execution order on each node. One directory already mentioned is templates/, where ERB-based configuration files are stored.

Now the question is: how can a configuration file like /etc/nginx/nginx.conf differ on each node without rewriting it? The answer lies in ERB templates rendered with node variables. Then, how do you store structured data like a list of users or secrets on the server so that many cookbooks can use it? The answer lies in data bags.

Episode 7 will cover how ERB templates and the template resource work, then continue with plain data bags, encrypted data bags for secrets, and chef-vault for managing secrets across many nodes.

#ERB Templates: Dynamic Configuration Files

An ERB template is a file containing a mix of static text and Ruby tags. When chef-client runs the template resource, the file is rendered first, then the result is written to the target path. There are two forms of ERB tags: <%= node['hostname'] %> which inserts the result of a Ruby expression, and <% if ... %> which runs conditional logic without printing anything.

Template files are stored in the templates/default/ directory inside the cookbook. The file name may match the target, or differ if you use the source property. The following example is a template for Nginx:

user www-data;
worker_processes <%= node['nginx']['worker_processes'] %>;
pid /run/nginx.pid;
 
events {
    worker_connections <%= node['nginx']['worker_connections'] %>;
}
 
http {
    sendfile on;
    server_tokens off;
 
    server {
        listen <%= node['nginx']['port'] %>;
        server_name <%= node['hostname'] %>;
        root /var/www/html;
    }
}

Each <%= ... %> in the template is replaced with the node attribute value at render time, so a single template can produce different configurations for a small development node and a large production node.

#The Template Resource

To use a template, declare the resource template '/etc/nginx/nginx.conf' do inside a recipe. This resource needs the path property as the target location, source as the file name in the templates folder, and variables to pass additional values beyond node attributes:

template '/etc/nginx/nginx.conf' do
  source 'nginx.conf.erb'
  owner 'root'
  group 'root'
  mode '0644'
  variables(
    port: node['nginx']['port'],
    server_name: node['hostname']
  )
  notifies :reload, 'service[nginx]'
end
 
service 'nginx' do
  action [:enable, :start]
end

The mode '0644' property sets the permissions of the rendered file. notifies :reload, 'service[nginx]' triggers an Nginx reload only when the file content actually changes, so the process doesn't restart when there are no configuration changes.

#Data Bags: Structured Data on the Server

A data bag is a global data structure stored on the chef-server that any cookbook can read. A data bag contains items in JSON form. Typical uses are lists of applications, account lists, or shared configuration that doesn't fit as attributes.

Create a data bag from the workstation with knife data bag create apps, then upload its items:

knife data bag create apps
knife data bag create apps order_service
knife data bag from file apps order_service.json

The contents of the order_service.json item file look like this:

{
  "id": "order_service",
  "port": 8080,
  "replicas": 3,
  "repository": "git@github.com:org/order-service.git"
}

To read a data bag inside a recipe, use the helper data_bag_item('apps', 'order_service'), which returns a hash of the item's fields, then pass the values to a template or another resource. To fetch all items at once, data_bag('apps') returns an array of all item ids — this pattern is useful for creating resources in a loop.

#Encrypted Data Bags

An encrypted data bag stores items in encrypted form. Chef encrypts the data with a secret key using the AES-256 algorithm. To decrypt an item, the same secret key must be available on the node when chef-client runs.

Create a secret key, store it safely, then create the data bag with the --secret-file option:

openssl rand -base64 512 > /etc/chef/encrypted_data_bag_secret
knife data bag create secrets --secret-file /etc/chef/encrypted_data_bag_secret
knife data bag from file secrets db_password.json --secret-file /etc/chef/encrypted_data_bag_secret

The contents of db_password.json are plain JSON, but what's stored on the server is already encrypted. Inside a recipe, reading it works just like a plain data bag because chef-client automatically uses the registered secret file, and the result is passed to a template with the sensitive true property so the password doesn't leak through chef-client logs in debug mode.

#Chef-Vault: Secrets for Many Nodes

Chef-vault solves the main weakness of encrypted data bags: distributing the secret key to every node. With chef-vault, secrets are encrypted with the public keys of the nodes allowed to read the item, and a node can only open the vault if it's registered as a client on the chef-server.

Install the plugin on the workstation, then create a vault item:

chef gem install chef-vault
chef vault create aws api_key -S 'role:app_server' -J data_bags/vault/aws_api_key.json

The command above marks nodes with the app_server role as members allowed to read the api_key item in the aws vault. Read the vault contents from within a recipe with the helper chef_vault_item('aws', 'api_key'), which returns a hash of credentials, then use the values for a template with sensitive true. Chef-vault automatically distributes keys to member nodes, so you don't need to copy secret keys manually, and revoking access is just a matter of removing the node from the vault's member list.

#Comparison of Secret Storage Methods

The following table helps you choose the right method:

#Conclusion

In episode 7 you learned how to create dynamic configuration files with ERB templates and the template resource, complete with the source, mode, and variables properties. You also understood the three levels of server-side data storage: plain data bags for open data, encrypted data bags for secrets encrypted with a secret key, and chef-vault, which distributes access automatically to authorized nodes.

Key takeaways:

• ERB templates render configuration files with <%= node['...'] %> and can be conditioned with <% if ... %> tags.
• The template resource writes the rendered result to a target path, complete with mode, owner, and notifies for service reloads.
• Data bags store structured data on the server and are read via data_bag_item inside recipes.
• Encrypted data bags secure secrets with an AES-256 secret key that must be distributed manually.
• Chef-vault encrypts secrets for specific nodes and distributes keys automatically, making access easy to revoke.

In the next episode, episode 8, we'll practice controlling your entire infrastructure from the workstation using Knife & the Chef CLI. You'll master knife node list, knife cookbook upload, knife data bag create, knife ssh, knife bootstrap, environment management, and freezing cookbook versions. See you in the next episode.