Learn Ansible - Asynchronous Actions & Polling

#Introduction

After episode 15, where we covered performance tuning — forks, SSH pipelining, fact caching, and execution strategies — you might think playbooks are fast enough. True, for the most part. But there's a category of tasks that tuning can't speed up at all, because the time needed is intrinsic: OS upgrades across hundreds of packages, multi-million-row database migrations, backing up large volumes, or running data migration scripts on production servers. Such tasks can run for 20 minutes, an hour, or even longer.

The problem is that Ansible's default mechanism is synchronous: the control node opens an SSH connection, runs the module, then waits with the connection kept open until the module finishes and returns a result. A connection hung that long is fragile. SSH timeout, idle timeout from a load balancer or firewall, or mere network jitter can cut the connection mid-way. The result: the playbook reports failure — even though on the server side, the running process may still be running normally. This is the most feared scenario: an ambiguous status, and a server that's "unclear about what it's doing".

In episode 16, we'll cover Asynchronous Actions & Polling — Ansible's mechanism for releasing long tasks from the SSH connection. You'll learn the async and poll parameters, the fire-and-forget mode with poll: 0, and how to periodically check job status with the ansible.builtin.async_status module. This material is one of the most frequently asked skills in the working world, because everyone has encountered a database migration or OS upgrade that has to run overnight.

#Main Discussion

#Why Long Tasks Fail: Anatomy of SSH Timeout

To understand why long tasks fail, we have to understand how Ansible runs a task synchronously. The flow goes like this:

1. The control node opens an SSH connection to the managed node.
2. The Python module is sent and executed (remember episode 2: the module is sent via SSH, executed, then the JSON result is returned).
3. The control node blocks and waits — the SSH connection stays open — until the remote Python process finishes and writes JSON to stdout.
4. The connection closes.

The problem is in step 3. Every network component between the control node and the managed node has a timeout: the OpenSSH server has ServerAliveInterval, firewalls/load balancers have idle timeouts (typically 5-30 minutes), and an SSH connection with no traffic for a long time will be cut. When that happens, Ansible can no longer read the module's result:

TASK [Upgrade semua paket sistem] *****************************************
fatal: [db01]: UNREACHABLE! => {"changed": false, "msg": "Timeout (12s) waiting for privilege escalation prompt", "unreachable": true}
	to retry, use: --limit @/home/arman/playbook.retry
 
PLAY RECAP *********************************************************************
db01                      : ok=1    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

An UNREACHABLE status means Ansible lost the connection — but the apt-get process or database migration on db01 most likely keeps running in the background. You don't know whether it finished, is running, or is stuck. Re-running the playbook can cause a race: two upgrade processes running simultaneously on the same server.

SSH tuning like adding ServerAliveInterval does help keep connections alive:

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=600s -o ServerAliveInterval=30 -o ServerAliveCountMax=3

...but this only delays the problem, not solves it. If a task runs for 2 hours, no realistic SSH setting will keep a single connection open that long — especially when there's a network device in the middle that cuts idle connections. The correct solution is changing the execution model: don't hold the task inside one SSH connection, but run it in the background and monitor it separately. That's what async does.

#The Async & Poll Concept

Ansible's async mechanism works by flipping the execution logic:

1. Ansible runs the module on the managed node, but asks it to run as a background process.
2. The module immediately returns a job ID (ansible_job_id) along with the result file location (in the async_dir directory, default ~/.ansible_async on the remote).
3. The SSH connection closes — no connection is left hanging.
4. The control node (if poll > 0) periodically opens new connections to ask the job's status from the ansible.builtin.async_status module using that job ID, until the job finishes.

The two key parameters are async and poll:

• async: <seconds> is the maximum time limit (timeout) allowed for that task. This isn't a "target" duration, but a hard limit: if the job hasn't finished after async seconds, Ansible considers it failed due to timeout.
• poll: <seconds> is the interval the control node uses to check job status. If poll: 0, the mode changes to fire-and-forget (we'll discuss that in a moment).

Notice the difference from the default:

If you set async without mentioning poll, Ansible uses the default poll = 10 seconds. So the easiest way to remember: async limits the duration, poll sets how often you peek.

#Example 1: Long-Running Task with Async & Poll

The most common scenario is an OS upgrade. Run it with async: 3600 (maximum limit of 1 hour) and poll: 30 (check status every 30 seconds):

---
- name: OS upgrade dengan async & poll
  hosts: db01
  become: true
 
  tasks:
    - name: Upgrade semua paket (sampai 1 jam)
      ansible.builtin.apt:
        upgrade: dist
        update_cache: true
      async: 3600
      poll: 30

When the playbook runs, you'll see polling lines like this — notice the connection opens and closes repeatedly, rather than hanging:

TASK [Upgrade semua paket (sampai 1 jam)] *********************************
ASYNC POLL ON db01: job_id=178183079391.21821
ASYNC POLL ON db01: job_id=178183079391.21821
ASYNC POLL ON db01: job_id=178183079391.21821
ASYNC RESULT ON db01: {"changed": true, "cmd": ["apt-get", "dist-upgrade", ...], "finished": 1, "rc": 0, ...}
 
PLAY RECAP *********************************************************************
db01                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

When finished: 1 and rc: 0 appear, the task is considered successful. If the duration passes async: 3600, the job will be reported as failed with an async task timed out message. Because of that, always set the async value with a wide margin over the task's normal estimated duration — you don't want an upgrade that usually takes 40 minutes to fail just because you hastily set async: 1800.

#Example 2: Fire-and-Forget with poll: 0

Now the more interesting part: fire-and-forget. With poll: 0, the playbook doesn't wait at all — the task runs in the background, the job ID is immediately returned, and the playbook continues to the next task. This is useful when:

• The long task doesn't block other tasks (e.g., a backup that's allowed to run alongside an app deployment).
• You want to run many long tasks in parallel across several hosts.
• The task triggers a server reboot — there's no point in the playbook waiting on a connection that's certainly going to die.

Example playbook: kick off a large backup, then the playbook continues with other work while the backup runs:

---
- name: Jalankan backup besar tanpa memblokir playbook
  hosts: db01
  become: true
 
  tasks:
    - name: Mulai backup database (tidak menunggu)
      ansible.builtin.shell:
        cmd: pg_dumpall > /backups/db-$(date +%Y%m%d).sql
      async: 7200
      poll: 0
      register: backup_job
 
    - name: Lanjutkan pekerjaan lain yang tidak terkait backup
      ansible.builtin.debug:
        msg: "Backup berjalan di background, job_id={{ backup_job.ansible_job_id }}"

Because of register: backup_job, this task's result stores the ansible_job_id that can be used later. Its status won't contain the backup's final result — that's normal for fire-and-forget mode.

#Checking Status with ansible.builtin.async_status

A fire-and-forget job doesn't just vanish. It runs on the managed node, and its status can be checked anytime using the ansible.builtin.async_status module with the jid (job ID) parameter:

---
- name: Kick-off backup lalu pantau sampai selesai
  hosts: db01
  become: true
 
  tasks:
    - name: Mulai backup database (fire-and-forget)
      ansible.builtin.shell:
        cmd: pg_dumpall > /backups/db-$(date +%Y%m%d).sql
      async: 7200
      poll: 0
      register: backup_job
 
    - name: Tunggu sampai backup selesai
      ansible.builtin.async_status:
        jid: "{{ backup_job.ansible_job_id }}"
      register: backup_result
      until: backup_result.finished
      retries: 120
      delay: 15

Let's break down this checking part:

• async_status asks the managed node for the job's status based on jid.
• The backup_result.finished variable will be 1 when the job finishes (whether success or failure).
• The until + retries + delay combination (we learned it in episode 9 on control flow) makes this task re-check every 15 seconds, up to 120 times — a total waiting window of 30 minutes.

The async_status result when the job finishes looks like this:

Output async_status

{
    "changed": false,
    "cmd": "pg_dumpall > /backups/db-20260802.sql",
    "finished": 1,
    "job": "875623478192.28173",
    "rc": 0,
    "results_file": "/root/.ansible_async/875623478192.28173",
    "started": 1,
    "stdout": "...",
    "stderr": ""
}

Notice the finished: 1 key — this is the marker used as the until condition. After the job finishes, you can read rc and stdout to determine success or failure, e.g., with the next task using failed_when: backup_result.rc != 0.

The "kick-off with poll: 0 → monitor with async_status" pattern is a very flexible technique. One favorite production use is parallel reboot: all hosts are kicked to reboot at once, then the playbook waits for each to come back online with wait_for_connection:

---
- name: Reboot semua host secara paralel
  hosts: all
  become: true
  serial: 50%
 
  tasks:
    - name: Kick-off reboot (fire-and-forget)
      ansible.builtin.reboot:
        reboot_timeout: 600
      async: 900
      poll: 0
      register: reboot_job
 
    - name: Tunggu host kembali online
      ansible.builtin.wait_for_connection:
        delay: 30
        timeout: 300

#Async Parameter Reference Table

#Async Limitations & Risks

Async isn't a magic solution — there are limits and risks you must understand before using it:

1. Not all modules support async. Modules that work via connection delegation (like wait_for, win_* based on WinRM, or certain network modules) generally don't support async mode. As a rule of thumb, the command, shell, script, apt, dnf, and yum modules work well with async; before using async on other modules, check their documentation.

2. Fire-and-forget = loss of control. With poll: 0, if the control node dies or the playbook is cancelled, the job on the managed node keeps running — but you lose the way to check it (unless you saved the job ID and check manually via async_status). Make sure there's another mechanism (logs, monitoring) to ensure that job truly finishes.

3. async is a timeout, not a planned duration. A async value that's too small makes a still-healthy job be considered failed. Always give a wide margin (e.g., 2× the normal estimate).

4. Job result files can disappear. Job status is stored in the managed node's async_dir (~/.ansible_async). If the directory is cleaned, the process reboots (result files don't survive a reboot), or the job ID has been cleaned up, async_status will report job not found. For very long tasks, consider writing your own log to a server file as an additional source of truth.

5. Async can't be used in handlers. Handler modules don't support the async parameter — handlers run after the play/notify, and async jobs inside them won't wait properly. If you need a long service restart after a change, do it as a regular task, not a handler.

#Common Async Mistakes

A summary of the traps most often encountered in the field:

1. Using async without a clear reason. Normal tasks lasting < 5 minutes don't need async. Async adds complexity and polling cycles; use it only for genuinely long tasks or when you want to run things in parallel.

2. async set too small. A job exceeding the async value is considered failed even though it ran normally. Give a wide margin.

3. Accessing a fire-and-forget job's result directly from register. As explained above, the final result only exists after checking with async_status.

4. Forgetting until when waiting on async_status. Without until: ... finished, the async_status task only checks once — almost certainly still finished: 0 because the job hasn't finished.

5. Using async on modules that don't support it. An error appears at runtime, and fixing it usually takes time because it's not always clear from the error message.

6. Putting async on a handler. Handlers don't support async; move it to a regular task.

#Conclusion

In episode 16, we covered how to handle long-running tasks — one of the most real problems in infrastructure operations. You now understand why synchronous tasks are vulnerable to SSH timeouts and produce ambiguous statuses, and how the async mechanism changes the execution model: the job runs in the background, the SSH connection closes, and status is monitored via async_status. We also practiced async: 3600 + poll: 30 for OS upgrades, the fire-and-forget poll: 0 mode for backups, the async_status + until + retries pattern, and parallel reboots with wait_for_connection. Finally, we discussed async's limitations: modules that don't support it, fire-and-forget risks, and the fact that async can't be used in handlers.

Key points to take home:

• async = maximum time limit; poll = check interval; poll: 0 = fire-and-forget.
• The job ID (ansible_job_id) is the key to checking a job's status anytime.
• Use async_status + until: finished to wait for a fire-and-forget job.
• Async solves the connection problem, not a replacement for good playbook logic.

Starting this episode, we're slowly "reaching into" Ansible: from just consuming built-in modules, to understanding how extensions work. In episode 17, we'll step further into its foundations — Custom Modules & Custom Filters (Python Extension) — where we'll write Python filter plugins to manipulate data and create custom modules using the AnsibleModule library. This is where you truly stop being limited by what's available, and start building what you need. Keep your enthusiasm up!