Learn Ansible - Production Deployment Checklist & Best Practices

#Introduction

After episode 29, where we covered scaling Ansible for enterprise — centralized architecture with AAP/AWX, GitOps repositories, inventory/playbook/role separation, and performance optimization for thousands of hosts — you now have a picture of how Ansible runs at organizational scale. But there's a more fundamental question that's often forgotten: how do we know the automation is truly ready to run in production without causing a catastrophe?

The answer isn't in a single magic feature, but in process and discipline. A well-written playbook that's run without testing, without code review, and without documentation is still dangerous — like a pilot who flies the plane smoothly but forgets to run the pre-flight checklist.

Episode 30 is the final episode of the Learn Ansible series. After 29 episodes building the foundation from zero — starting with agentless architecture, playbooks, roles, all the way to enterprise scaling — it's time to summarize all those lessons into one complete package: the production deployment checklist and best practices. In this episode we'll cover the pre-production gateway, operational best practices, common mistakes to avoid, operational monitoring, disaster recovery, and continuous improvement. This isn't just theory: this is how a healthy infrastructure team works in the real world.

#Main Discussion

#The Pre-Production Checklist: The Gateway to Production

The best analogy for a pre-production checklist is the pilot's checklist before takeoff. A pilot doesn't argue with the checklist; they run it sequentially, step by step, because they know one missed item could mean the difference between a normal flight and a crash. The Ansible playbooks you write can also touch hundreds of servers at once — and one wrong line of YAML can take down a service entirely.

Here's the pre-production gateway you must pass every time you deploy a change to a production environment:

#Recommended Production Project Structure

A tidy directory structure is the first documentation people will read. A healthy production project usually looks like this:

ansible-prod/
├── ansible.cfg                  # konfigurasi global (forks, callback, ssh)
├── requirements.yml             # dependensi collection (episode 13)
├── inventory/
│   ├── production.yml           # inventory production
│   └── staging.yml              # inventory staging
├── group_vars/
│   ├── production/
│   │   ├── all.yml              # variabel bersama environment production
│   │   └── webservers.yml       # variabel spesifik group
│   └── staging/
├── host_vars/                   # variabel per-host
├── playbooks/
│   ├── site.yml                 # playbook utama / entry point
│   ├── deploy-app.yml           # playbook deploy & rollback
│   └── maintenance.yml
├── roles/
│   ├── nginx/
│   ├── postgresql/
│   └── hardened/                # security hardening (episode 27)
├── vault/
│   ├── production.vault.yml     # secret terenkripsi per environment
│   └── staging.vault.yml
├── docs/
│   ├── README.md                # cara menjalankan project
│   └── runbooks/                # prosedur insiden & recovery
├── .github/workflows/
│   ├── lint-test.yml            # CI: lint + Molecule
│   └── deploy.yml               # CD: deploy ke staging/production
└── .ansible-lint                # konfigurasi ansible-lint

Notice the strict separation between inventory/, group_vars/, playbooks/, and roles/. This is the separation of concerns principle we covered in episode 29 — data separated from logic, so one role can be reused across many environments just by swapping inventory and variables.

#The --check + --diff Workflow: Dry-Run Before Touching Production

The two flags that most often save infrastructure are --check and --diff. --check (dry-run) simulates execution without making real changes, while --diff shows the text changes that would be made to configuration files. Their combination gives you a "change map" before actually executing.

ansible-playbook -i inventory/production.yml playbooks/site.yml \
  --check --diff --limit webservers --tags config

TASK [nginx : Salin virtual host] ******************************
--- before: /etc/nginx/conf.d/app.example.com.conf
+++ after: /etc/nginx/conf.d/app.example.com.conf
@@ -5,7 +5,7 @@
  server {
      listen 80;
-     server_name app.example.com;
+     server_name app.example.com www.app.example.com;
      root /var/www/html;
-     index index.html;
+     index index.php;
  }
 
PLAY RECAP *****************************************************
webserver-01 : ok=12    changed=1    unreachable=0    failed=0
webserver-02 : ok=12    changed=1    unreachable=0    failed=0

#Operational Best Practices

After passing the pre-production gateway, you'll run and maintain this automation every day. Here are operational principles that keep automation healthy long-term.

#Version Everything

All Ansible assets must be in Git: playbooks, roles, inventory (without secrets), ansible.cfg, requirements, even runbook documentation. This isn't just a backup — it's an audit trail (episode 19). If production breaks, you can answer the question: what change caused this? who wrote it? when? Without Git, that question can only be answered by guessing.

git checkout -b fix/nginx-reload-handler
# ... perbaiki playbook ...
ansible-lint .                      # 1. lint
molecule test -s production         # 2. test
git add playbooks/ roles/
git commit -m "fix: pindahkan reload handler ke role nginx"
git push -u origin fix/nginx-reload-handler
# 3. buka PR → code review → merge → pipeline deploy

#Use Roles & Collections for Reusability

Don't rewrite the same logic in every playbook. Wrap reusable functionality into roles (episode 12) and use collections from Galaxy (episode 13) for common things. A mature nginx role can be used in dev, staging, and production just by changing variables — not by copy-pasting 200 lines of YAML. This saves time and, more importantly, reduces the bug surface.

#Implement Proper Error Handling

Recall the lessons from episode 10: use block/rescue/always to handle failures gracefully, failed_when to set the right failure conditions, and ignore_errors only with a clear reason. A playbook that gives up midway without a clear message is an operator's worst enemy while handling an incident at night.

#Use Tags for Selective Execution

Large playbooks that run everything every time are wasteful and risky. Use tags to break playbooks into units that can be executed selectively — for example, only updating configuration without restarting, or only pulling new artifacts without touching system configuration.

- name: Deploy web server production
  hosts: webservers
  become: true
  vars:
    app_version: "1.4.2"
    app_domain: "app.example.com"
  tasks:
    - name: Install Nginx
      ansible.builtin.apt:
        name: nginx
        state: present
      tags: install
 
    - name: Salin konfigurasi virtual host
      ansible.builtin.template:
        src: vhost.conf.j2
        dest: "/etc/nginx/conf.d/{{ app_domain }}.conf"
        mode: "0644"
      notify: reload nginx
      tags: config
 
    - name: Deploy artefak aplikasi
      ansible.builtin.copy:
        src: "build/app-{{ app_version }}.tar.gz"
        dest: "/var/www/{{ app_domain }}/"
      tags: deploy
 
  handlers:
    - name: reload nginx
      ansible.builtin.systemd_service:
        name: nginx
        state: reloaded

Now you can run different layers of changes independently:

ansible-playbook -i inventory/production.yml playbooks/webserver.yml --tags install

ansible-playbook -i inventory/production.yml playbooks/webserver.yml --tags install,config

ansible-playbook -i inventory/production.yml playbooks/webserver.yml --skip-tags install

#Implement Idempotency Everywhere

Idempotency has been Ansible's DNA since episode 1, and in production it's no longer a nice-to-have but an absolute requirement. An idempotent playbook can be rerun without side effects, so it's safe to be triggered by cron, retried after a failure, or run twice by a CI pipeline. If you find a task that's always changed every run, that's a signal something isn't declarative — for example, using shell for something that should use a dedicated module, or writing a random timestamp into a config file.

#Use Variables Effectively (the DRY Principle)

Don't Repeat Yourself. Don't hardcode the same value in ten places. Store values that change between environments in group_vars/ and host_vars/ (episode 7), then make playbooks pure logic that reads variables:

domain: "app.example.com"
nginx_worker_processes: "auto"
nginx_ssl_cert: "/etc/ssl/certs/app.example.com.crt"
backend_ports: [8080, 8081, 8082]
monitoring_enabled: true

The playbook then just references {{ domain }} and {{ nginx_worker_processes }} without needing to know their values — an environment change is done in one variable file, not by editing the playbook.

#Document Variables & Dependencies

A variable without documentation is a puzzle. In every role, provide a defaults/main.yml with short comments, and write a concise README explaining required variables, role dependencies, and usage examples. Runbooks for important operations (deploy, rollback, backup restore) should be written as if they'll be read by someone who joined last week — because one day they will.

#Common Pitfalls to Avoid

After writing many playbooks in production, these error patterns appear over and over. Recognize their signs so you can avoid them early:

One of the most common cases is using shell for package installation. Notice the following diff — converting it to a dedicated module makes the playbook idempotent and readable:

- name: Install Nginx (CARA LAMA - salah)
  ansible.builtin.shell: apt-get update && apt-get install -y nginx
- name: Install Nginx (CARA BENAR)
  ansible.builtin.apt:
    name: nginx
    state: present

#Operational Monitoring of Ansible

Automation that isn't observed is just hope. In production, you must know the answers to these questions: how long do playbooks take? how many succeed and fail? what actually changed? who ran what and when?

Here are the key metrics you need to monitor:

To build an analyzable trail, enable callbacks that emit structured output:

[defaults]
stdout_callback = json
callbacks_enabled = timer, profile_tasks

With stdout_callback = json, playbook output can be parsed by other tooling. Timer records total duration per task, and profile_tasks shows which tasks are slowest — valuable data for performance optimization. At enterprise scale, AWX/AAP (episode 20) already provides job history, RBAC, and a REST API that can be integrated into observability dashboards (episode 25) to trigger alerts if a playbook starts failing repeatedly.

#Disaster Recovery: Ready When Everything Fails

Accept the reality: someday something will fail. A playbook will delete the wrong file, a release will break, or a production server will be compromised. The question isn't whether it happens, but how fast you can recover.

#Backup Strategy for Ansible Infrastructure

What needs to be backed up is the source of truth of your automation:

• Git repository — playbooks, roles, inventory, and documentation. A Git remote (GitHub/GitLab) plus an offsite backup is the primary backup.
• Vault keys — without the vault password, all your secrets become useless. Store it in a team password manager / secret store, not just on one person's laptop.
• ansible.cfg and requirements — global config and the version-pinned collection list (for reproducibility).
• Stateful data — although Ansible isn't a backup tool, make sure application data (databases, uploaded files) is backed up by dedicated tools and its restore mechanism is tested.

#Rollback Procedures

The most reliable rollback pattern is based on deploying the same way as rolling back — use tags to mark the relevant tasks, and make the target version a variable that can be overridden at run time:

- name: Deploy / rollback aplikasi
  hosts: appservers
  become: true
  vars:
    target_version: "1.4.2"
  tasks:
    - name: Unduh artefak versi target
      ansible.builtin.get_url:
        url: "https://artifacts.internal/app-{{ target_version }}.tar.gz"
        dest: "/opt/app/app-{{ target_version }}.tar.gz"
      tags: [deploy, rollback]
 
    - name: Arahkan symlink ke versi target
      ansible.builtin.file:
        src: "/opt/app/releases/app-{{ target_version }}"
        dest: /opt/app/current
        state: link
      notify: restart app
      tags: [deploy, rollback]
 
  handlers:
    - name: restart app
      ansible.builtin.systemd_service:
        name: app
        state: restarted

When version 1.4.3 has problems, a rollback is just one command — without changing a single line of the playbook:

ansible-playbook -i inventory/production.yml playbooks/deploy-app.yml \
  --tags rollback -e target_version=1.4.2

#Testing Recovery Procedures

Even the best recovery plan means nothing if it's never tested. Schedule routine drills: restore a database backup on a staging host, roll back a release, rebuild a control node from scratch from the Git repository. Record the results in a runbook and fix procedures that fail. Teams that regularly practice recovery are far calmer and faster when a real incident happens.

#Continuous Improvement

Automation isn't a one-shot project; it's a living system that must be maintained. Here are habits that keep automation quality rising over time:

• Periodic refactoring — review old playbooks and roles; replace deprecated patterns, remove duplication, and move logic into roles. Ansible actively deprecates old syntax (e.g., with_items → loop), and ansible-lint will help detect it.
• Performance optimization — use data from the profile_tasks callback (episode 15) to find bottlenecks; disable unneeded fact gathering, leverage fact caching, and tune forks appropriately.
• Security updates — update ansible-core and collections on a schedule, monitor security advisories, and run security hardening periodically (episode 27).
• Contribute to the community — if you find a bug or build a great role, contribute it to Ansible Galaxy or a community collection. It's the best way to learn and give back at the same time.
• Follow Ansible releases — read the changelog with every release, follow the official blog, and watch community discussions so you don't miss new features and best practices.

#Conclusion

Congratulations — you've completed the long journey from episode 0 to episode 30! Let's briefly recap the big map we've traveled together:

From just understanding ansible -m ping in episode 0, you now have the ability to automate Linux and Windows servers, networks, multi-vendor clouds, Kubernetes, and databases — with quality maintained by linting, testing, and CI/CD pipelines, and executed safely at enterprise scale. That's a rare capability with very high value in the DevOps and SRE job market.

Remember the three core principles that have always guided us: idempotency, declarative thinking, and automation over documentation — write what you do, and make machines do what you write.

The journey doesn't stop here. After completing this series, here are the next steps you can take to master Ansible even deeper:

1. Build a real project — automate your personal server or home server setup, then develop it to production-grade with this episode's checklist.
2. Learn the Red Hat ecosystem officially — explore Ansible Automation Platform (AAP), ansible-navigator, and Red Hat certification (EX294/RHCE) to validate your skills.
3. Contribute to the community — open issues or submit PRs to community collections, publish public roles on Ansible Galaxy, or share your experience in writing.
4. Expand into complementary tooling — learn Terraform for provisioning and an observability stack for monitoring, so your tooling combination is more complete.

Automation is an investment that keeps paying compound interest: every hour you spend writing good automation will save hundreds of hours in the future. Keep automating, keep practicing, and make infrastructure something you can rebuild with a single command. See you on the next journey — and happy building as a true infrastructure engineer!