Learn Ansible - Error Handling & Resilience Strategies

#Introduction

After episode 9, where we covered giving logic to playbooks through conditionals and loops, in this episode we'll cover the other equally important side: what happens when something goes wrong.

In production, failure isn't a question of "whether" but "when". Imagine you run a deployment playbook against 100 servers, and server number 17 fails halfway through because the package repository is unreachable. With Ansible's default behavior, that failure stops the entire playbook — servers 18 to 100 are never touched, and the system state becomes inconsistent: some servers are already updated, others aren't. For an engineer, this kind of scenario is a nightmare, and this is where building resilience matters.

In this episode, we'll cover three keywords for controlling task status — ignore_errors, failed_when, and changed_when — then continue to the block, rescue, and always structure, which is similar to try/catch/finally in programming languages. With this combination, you can write playbooks that know how to survive: ignoring non-fatal errors, defining for yourself what "failed" and "changed" mean, performing rollback when errors occur, and cleaning up temporary resources no matter the outcome.

#Main Discussion

#Error Handling Strategies

Before diving into the complex block structure, let's master three basic keywords that control how Ansible judges a task's result. All three work at the task level and are the first tools you should have for error handling.

#ignore_errors: true: Keep Going Even When a Task Fails

Ansible's default behavior is fail fast: the moment a task FAILED, the playbook stops for that host. The ignore_errors: true keyword tells Ansible to treat that failure as no big deal and continue with the next task.

When is ignore_errors genuinely appropriate? One case is for best-effort tasks — tasks that can fail without changing the end result. Classic examples: removing cache files or sending notifications:

- name: Hapus cache aplikasi (best-effort)
  ansible.builtin.file:
    path: /var/cache/app
    state: absent
  ignore_errors: true
 
- name: Kirim notifikasi ke Slack (gagal tidak fatal)
  ansible.builtin.uri:
    url: https://hooks.slack.com/services/xxx
    method: POST
    body_format: json
    body: { "text": "Deploy selesai" }
  ignore_errors: true

Even if the second task fails (for example, because the URL is unreachable), the playbook continues to the next task. This prevents one small failure from stopping an entire deployment that actually already succeeded.

#failed_when: Define "Failed" Yourself

failed_when takes over Ansible's failure judgment. Instead of judging by the module's return code, you define explicit conditions for when a task should be considered failed. This is very useful when using the command or shell modules, because those modules sometimes return a non-zero return code for things that are actually normal, or vice versa.

Real example: the apt-get update command might return rc != 0 only because one repo is down, even though the other repos were refreshed successfully. We can evaluate its output more carefully:

- name: Update cache apt dengan evaluasi khusus
  ansible.builtin.command: apt-get update
  register: apt_result
  failed_when:
    - apt_result.rc != 0
    - "'The repository is not signed' not in apt_result.stderr"

The combination above reads: the task is considered failed if the return code is non-zero and the error message isn't about the unsigned repo that can actually be ignored. Pattern matching on stdout or stderr using regex is also common, for example:

- name: Verifikasi service berjalan
  ansible.builtin.shell: systemctl is-active app
  register: svc_status
  failed_when: svc_status.stdout != "active"

failed_when is also an important bridge to the until we covered in episode 9 — both work together with registered task results.

#changed_when: Control the Changed Status

The command and shell modules always return a CHANGED status, because Ansible has no way to know whether the command actually changed something. Yet the CHANGED status is very important: this status is what triggers handlers (which we covered in episode 6) and determines whether a run is considered "touching" the system.

changed_when lets you set when a task reports CHANGED status:

- name: Restart service hanya jika konfigurasi benar-benar berubah
  ansible.builtin.shell: |
    systemctl restart app
    systemctl status app
  register: restart_result
  changed_when: "'inactive' in restart_result.stdout"

With changed_when, a task that makes no changes is reported as OK (not CHANGED), making the playbook execution honest: running it a second time is expected to produce "nothing to do", which is the essence of idempotency.

Here's a comparison of the three keywords:

#The block, rescue, and always Structure

The three keywords above handle task status judgment individually. But what if you want to run a group of tasks, and when one of them fails, run certain fallback tasks, then make sure there's a cleanup that always runs? This is where the block, rescue, and always structure comes in.

#block: Grouping Tasks & Applying Shared Attributes

block groups several tasks into one unit. There are two main reasons to use it:

1. Handling errors as one unit — combined with rescue and always (discussed below).
2. Applying shared attributes — when, become, and tags placed at the block level automatically apply to all tasks inside. This avoids repeating the same condition over and over.

An example of the second point: without a block, we write the same condition three times:

- name: Tanpa block - kondisi diulang tiap task
  hosts: all
  tasks:
    - name: Install package
      ansible.builtin.apt:
        name: nginx
        state: present
      when: ansible_facts['os_family'] == "Debian"
 
    - name: Copy konfigurasi
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      when: ansible_facts['os_family'] == "Debian"
 
    - name: Start service
      ansible.builtin.systemd_service:
        name: nginx
        state: started
      when: ansible_facts['os_family'] == "Debian"

With block, the condition is written once and applies to all tasks inside:

- name: Dengan block - kondisi cukup sekali
  hosts: all
  tasks:
    - name: Setup nginx untuk Debian
      when: ansible_facts['os_family'] == "Debian"
      block:
        - name: Install package
          ansible.builtin.apt:
            name: nginx
            state: present
 
        - name: Copy konfigurasi
          ansible.builtin.template:
            src: nginx.conf.j2
            dest: /etc/nginx/nginx.conf
 
        - name: Start service
          ansible.builtin.systemd_service:
            name: nginx
            state: started

#rescue: Exception Handling (Fallback on Error)

When one of the tasks inside a block fails, Ansible stops the remaining tasks inside that block and immediately jumps to the rescue section. All tasks inside rescue are executed as failure handling — usually containing fallbacks, error logs, or rollbacks.

#always: Cleanup That Always Runs

The always section runs without exception: whether the block succeeds, fails and enters rescue, or even throws an error — always still runs. This is the right place for cleanup: removing temporary files, stopping services, or restoring state.

Let's look at a complete example combining all three: application installation with automatic rollback and a cleanup that always runs.

- name: Deploy aplikasi dengan rollback otomatis
  hosts: app-servers
  become: true
  vars:
    app_version: "2.4.0"
    backup_path: "/opt/app_backups/{{ app_version }}"
 
  tasks:
    - name: Deploy aplikasi versi baru
      block:
        - name: Unduh artefak aplikasi
          ansible.builtin.get_url:
            url: "https://artifacts.internal/app-{{ app_version }}.tar.gz"
            dest: "/tmp/app-{{ app_version }}.tar.gz"
 
        - name: Backup versi lama
          ansible.builtin.copy:
            src: /opt/app/
            dest: "{{ backup_path }}"
            remote_src: true
 
        - name: Ekstrak versi baru
          ansible.builtin.unarchive:
            src: "/tmp/app-{{ app_version }}.tar.gz"
            dest: /opt/app/
            remote_src: true
 
      rescue:
        - name: Rollback ke versi lama
          ansible.builtin.copy:
            src: "{{ backup_path }}/"
            dest: /opt/app/
            remote_src: true
 
        - name: Laporkan kegagalan deployment
          ansible.builtin.debug:
            msg: "Deployment versi {{ app_version }} gagal, dilakukan rollback."
 
      always:
        - name: Restart service aplikasi
          ansible.builtin.systemd_service:
            name: app
            state: restarted
 
        - name: Bersihkan artefak sementara
          ansible.builtin.file:
            path: "/tmp/app-{{ app_version }}.tar.gz"
            state: absent

The execution flow in the example above:

1. block runs: download artifact → backup old version → extract new version.
2. If all succeed → rescue is skipped, always still runs (restart service + clean up temporary files).
3. If one task fails (e.g., the artifact isn't available) → rescue runs to rollback to the old version and log the failure message, then always still runs.

Notice that in the failure case, the service is still restarted in the always section, but because rescue already restored the old version to /opt/app/, that restart actually ensures the service runs again with a stable version. This is the power of this structure: the system state always returns to consistency.

Here's the output you'll see when rescue is actually active. Notice the output changes from the ok/changed pattern to failed on the failing task, then rescue executes, and always still runs:

TASK [Unduh artefak aplikasi] *************************************************
fatal: [app-01]: FAILED! => {"changed": false, "msg": "HTTP Error 404: Not Found"}
 
TASK [Backup versi lama] *****************************************************
skipping: [app-01]
 
TASK [Ekstrak versi baru] ****************************************************
skipping: [app-01]
 
TASK [Rollback ke versi lama] ************************************************
changed: [app-01]
 
TASK [Laporkan kegagalan deployment] *****************************************
ok: [app-01] => {
    "msg": "Deployment versi 2.4.0 gagal, dilakukan rollback."
}
 
TASK [Restart service aplikasi] **********************************************
changed: [app-01]
 
TASK [Bersihkan artefak sementara] *******************************************
ok: [app-01]

#Common Pitfalls

1. Overusing ignore_errors — masking the real problems

ignore_errors is a double-edged sword. Using it too much makes the playbook "look green" when actually many tasks fail silently. A playbook that's always ok while the service isn't running is the most dangerous trap in production. Use ignore_errors only for genuinely non-critical best-effort tasks, and always include debug or logging so ignored failures remain visible.

2. Misusing failed_when without evaluating rc

failed_when completely replaces Ansible's default judgment. If you only write failed_when: "'error' in result.stdout" without including an rc check, a task returning a non-zero return code — say, from a total crash — is actually considered successful as long as the word "error" doesn't appear in stdout. Always consider combining the rc condition with output content, like the apt-get update example above.

3. A wrong changed_when disables handler triggers

Because handlers are triggered by the changed status, writing changed_when: false on a task that actually changes configuration will prevent the handler (e.g., service restart) from running — and the config change never really gets applied. Make sure changed_when reflects the real change condition.

4. Placing block without rescue/always while intending to handle errors

A block without rescue and always is just plain grouping. If your goal is error handling, make sure the complete block → rescue → always structure is installed. A block alone provides no try/catch/finally behavior.

5. when relying on a task result that was ignore_errors

When a task fails and is ignore_errors'd, its result is still registered with the failed: true attribute. If the next task uses when: result is succeeded or when: result.failed, you need to realize that failed is still true. A common safe pattern is:

- name: Cek kesehatan service
  ansible.builtin.command: systemctl is-active app
  register: health
  ignore_errors: true
 
- name: Restart jika service tidak aktif
  ansible.builtin.systemd_service:
    name: app
    state: restarted
  when: health.rc != 0

#Conclusion

In episode 10, we built a resilience foundation for Ansible playbooks. We learned three status-controlling keywords — ignore_errors to ignore non-fatal failures, failed_when to define "failed" ourselves (e.g., by evaluating regex output from command), and changed_when to control the changed status and handler triggers. Then we dissected the block, rescue, and always structure — Ansible's try/catch/finally equivalent — complete with a deployment example that performs automatic rollback on failure and always runs cleanup.

With these capabilities, your playbooks no longer stop dead in the middle of failure: they can survive, judge conditions intelligently, recover, and always leave the system in a consistent state. However, a long and resilient playbook simultaneously becomes increasingly hard to maintain if all tasks are written in one giant file.

In episode 11, we'll cover Modularization with Includes & Imports — the technique of splitting large playbooks into well-organized small files, and understanding the fundamental difference between dynamic include_tasks and static import_tasks. Keep your enthusiasm up!