Learn Ansible - Modern Execution Environments (AAP / AWX, ansible-navigator & ansible-builder)

#Introduction

After episode 19, where we covered integrating Ansible into CI/CD pipelines and its collaboration with Terraform, you now have a picture of how automation runs at the organization level. However, there's one fundamental problem that often only gets noticed when the team reaches dozens of engineers: different environments between machines.

Have you ever heard the phrase "It works on my laptop"? That's one of the most famous curses in the software engineering world. In the Ansible world, that phrase usually sounds like this: "This playbook runs on my machine, why does it error on yours?" The cause is often simple: a different Python version, a different ansible-core version, collections that aren't installed, or conflicting Python libraries — known as dependency hell.

Episode 20 will bring you out of that problem permanently. We'll cover the Containerized Execution Environments (EE) concept, then learn two Red Hat Ansible mainstay tools: ansible-builder to build a container image containing the entire Ansible environment, and ansible-navigator to run automation consistently inside the container. Finally, we'll get to know AWX / Ansible Automation Platform (AAP) — the enterprise platform that turns Ansible from a CLI tool into a centralized system with a web UI, REST API, RBAC, and audit logging.

#Main Discussion

#The "Works on My Machine" Problem on the Control Node

Let's honestly look at what happens in a team managing infrastructure with Ansible without environment standardization.

Imagine three engineers named Adi, Budi, and Cici:

• Adi installed Ansible a year ago and never upgraded. His version is ansible-core 2.14.
• Budi just joined and installed the latest ansible-core 2.19.
• Cici works on a team whose playbooks depend on a specific version of the community.postgresql collection, which turns out to be incompatible with the version Adi has.

When they work on the same playbook, the results can differ. Adi gets deprecation errors, Budi gets errors because behavior changed, and Cici has to debug a bug that can't be reproduced. On top of that, each engineer uses a different operating system — macOS, Ubuntu, Windows WSL — which adds unexpected variables.

This problem can be summarized in three words: inconsistency, non-reproducibility, and vulnerability. A playbook only tested on one machine is a time bomb in production.

The traditional approach to solving it is a virtualenv per project. But virtualenv only solves Python isolation; it doesn't solve: system package versions, SSH configuration, collection availability, or libraries with native dependency limitations (e.g., psycopg2 which needs a compiler during installation).

The modern solution used in industry is containerization of the entire environment. This isn't a new concept — Docker already taught it for applications — and now the same paradigm is applied to Ansible.

#Containerized Execution Environments (EE): Basic Concept

An Execution Environment (EE) is a container image containing everything needed to run Ansible automation: ansible-core, collections, Python libraries, system dependencies, and supporting tools. Think of an EE as a "complete kitchen set you can take traveling": all the equipment, ingredients, and recipes are in one box. Anyone who opens that box gets an exactly identical kitchen.

The main advantages of EE are very clear:

In the Red Hat ecosystem, there are three tooling layers working together: ansible-builder (building images), ansible-navigator (running images), and AWX/AAP (scheduling and managing image execution across many nodes). We'll cover all three one by one.

#ansible-builder: Building an Execution Environment

ansible-builder is the tool that takes an environment definition in YAML and turns it into a ready-to-use container image. It builds on top of the base ansible-runner image and injects the dependencies you define.

#Installing ansible-builder

pipx install ansible-builder

ansible-builder --version

#Defining the Environment: execution-environment.yml

The heart of ansible-builder is the execution-environment.yml file. This file has three main sections: build_arg_defaults, dependencies, and images:

---
version: 3
 
build_arg_defaults:
  ANSIBLE_GALAXY_CLI_COLLECTION_OPTS: "--force"
 
dependencies:
  galaxy: requirements.yml
  python: requirements.txt
  system: bindep.txt
 
images:
  base_image:
    name: quay.io/ansible/ansible-runner:latest
 
additional_build_steps:
  prepend_final: |
    RUN whoami

Let's break down each section:

The referenced dependency files are stored in the same directory:

---
collections:
  - name: community.postgresql
    version: "3.5.0"
  - name: community.general
    version: "9.2.0"

requirements.txt (Python)

psycopg2-binary==2.9.9
jmespath==1.0.1

bindep.txt (system packages)

libpq-dev

#Building the Image

With the execution-environment.yml file and its dependencies in one directory, run:

ansible-builder build \
  --tag quay.io/acme/ee-devnull:latest \
  --container-runtime docker

This command generates a context/ directory structure containing the Containerfile and all dependency files, then executes the build:

Ansible Builder is building your execution environment image.
The build context can be found at: context/
 
Complete! The build context can be found at: context/
Tag: quay.io/acme/ee-devnull:latest

Once done, the image can be pushed to a registry:

docker push quay.io/acme/ee-devnull:latest

#ansible-navigator: Running Automation Inside a Container

If ansible-builder builds the "kitchen", then ansible-navigator is the way to "cook" in it. ansible-navigator is a CLI that runs Ansible inside an EE container, complete with a TUI (Text User Interface) for interactive navigation.

ansible-navigator's advantage: you don't need to install Ansible on the machine at all. All you need is ansible-navigator and a container runtime (Docker/Podman).

#Installation & Basic Usage

pipx install ansible-navigator

Running a playbook with ansible-navigator:

ansible-navigator run playbooks/deploy.yml -i inventory/production.yml

Not just playbooks, ansible-navigator also wraps various other Ansible commands:

ansible-navigator doc ansible.builtin.apt

ansible-navigator inventory --list

ansible-navigator collections

ansible-navigator run -m ansible.builtin.ping all

#Configuring ansible-navigator

Default configuration is stored in the ansible-navigator.yml file. Here you specify the EE image used, output mode, logging level, and environment variables passed through:

---
ansible-navigator:
  execution-environment:
    enabled: true
    image: quay.io/acme/ee-devnull:latest
    pull-policy: missing
    container-engine: docker
    environment-variables:
      pass:
        - AWS_PROFILE
        - AWS_ACCESS_KEY_ID
        - AWS_SECRET_ACCESS_KEY
  playbook-artifact:
    enable: false
  logging:
    level: warning
  mode: stdout

Important points from the configuration above:

• image points to the EE we built with ansible-builder. All engineers just use this file and the same image.
• environment-variables.pass specifies the env vars allowed to be passed from the host to the container — a safe way to inject credentials without storing them in the image.
• mode: stdout makes the output formatted like a regular Ansible CLI. The default (interactive) mode shows a TUI navigable with the keyboard — very helpful for browsing task results one by one.
• pull-policy: missing means the image is only pulled if it isn't already local, speeding up repeated execution.

#Understanding the ansible-navigator TUI

In interactive mode, ansible-navigator shows a full-screen view with panels:

• The execution result panel displays task by task, complete with ok/changed/failed statuses.
• You can press keys to expand task details, view standard/error output, and even jump to the playbook source to see the code line being executed.

This feature is very useful when debugging long playbooks: you don't have to guess from long text logs, just navigate directly to the failed task and see its context.

#AWX / Ansible Automation Platform (AAP): Ansible for Enterprise Scale

At this level, we're talking about many engineers, many projects, and a need for control. ansible-navigator still runs on each engineer's laptop; for large organizations, Red Hat provides the Ansible Automation Platform (AAP), with AWX as its open-source upstream version.

AWX/AAP turns Ansible from a CLI tool into a centralized platform with a Web UI and REST API. Imagine AWX as the "central control building" where all Ansible executions are supervised, scheduled, and audited — instead of being run in a scattered, uncontrolled way on personal laptops.

#AWX / AAP Main Features

Let's use a real-world analogy: if an engineer's laptop is like a private car (flexible but uncontrolled), then AWX is like a city transportation system — all vehicles registered, official routes, schedules, and trip records. Nobody can "bring a private car" onto the production highway without permission.

#The Job Template Flow in AWX

Job Template is AWX's core concept: a complete package containing a project (Git repo), inventory, credentials, and execution options. The flow:

1. Project refers to the Git repository containing playbook/roles.
2. Inventory determines the target servers (static or dynamic from the cloud).
3. Credentials store SSH keys and vault passwords, locked by RBAC.
4. Job Template combines everything: pick project + inventory + credentials + playbook.
5. Launch runs the template, which triggers a job. All output and statuses are recorded.

As an example, launching a job template via the REST API (callable from a CI/CD pipeline):

curl -X POST https://awx.example.com/api/v2/job_templates/5/launch/ \
  -H "Authorization: Bearer $AWX_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"extra_vars": {"environment": "production"}}'

#Installing AWX

AWX is designed to run on Kubernetes and is installed via an operator. A short example once the Kubernetes cluster is ready:

kubectl create ns awx
kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/release/deploy/awx-operator.yaml

Then create the AWX instance with a Custom Resource:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  service_type: nodeport

After a few minutes, the AWX UI can be accessed and you can set up the admin, organization, then start creating projects and job templates.

#Comparison: ansible CLI vs ansible-navigator vs AWX/AAP

Now we have the full picture. Let's compare the three approaches in one table:

#Common Pitfalls

Adopting execution environments isn't free of traps. Here are the most commonly encountered:

1. Unpinned image versions. Without version pinning, an EE built today and next month could have different contents. Always pin versions in requirements.yml, requirements.txt, bindep.txt, and the base image tag.

2. Forgetting to install system dependencies. Python libraries like psycopg2 or cryptography often need system packages (e.g., libpq-dev). If they're not in bindep.txt, library installation fails mid-build. Test image builds regularly.

3. Secrets embedded in the image. Never put credentials inside an EE image. Use runtime injection mechanisms, like environment-variables.pass in ansible-navigator or the credentials store in AWX.

4. Running ansible-navigator without the right EE. If the image is wrong or pull-policy isn't right, execution could silently use the wrong image. Always verify the image used before running production playbooks.

5. Thinking AWX = install once, done. AWX requires maintenance: database backups, operator upgrades, and credential management. It's not a tool you can just leave alone.

#Conclusion

In episode 20, we brought you from the classic dependency hell problem on the control node to the modern solution used in industry. We understood Containerized Execution Environments as a way to ensure all engineers and CI run the exact same environment, then learned ansible-builder to build EE images from execution-environment.yml, ansible-navigator to run automation inside containers with a comfortable TUI, and finally AWX / Ansible Automation Platform for enterprise scale with RBAC, job scheduling, a credentials store, and audit logging.

Key points to take home:

• The "works on my machine" problem is born from environment inconsistency between machines, and the solution is containerization.
• An EE wraps ansible-core, collections, Python libraries, and system dependencies into one reproducible image.
• ansible-builder builds the image, ansible-navigator runs it consistently.
• AWX/AAP turns Ansible into a centralized platform with control, security, and a complete audit trail.
• Choose tools by scale: CLI for labs, navigator for consistency, AWX for enterprise.

In episode 21, we'll return to a deeper technical level with Advanced Inventory Management & Dynamic Inventory. We'll dissect dynamic inventory implementations for AWS, Google Cloud, Azure, and VMware, learn to filter instances by tag and region, create automatic grouping, and write custom inventory scripts integrated with CMDBs like NetBox. This will complete your knowledge of how inventory is managed in a real world full of servers being born and dying all the time. Keep your enthusiasm up!