Learn Ansible - Scaling Ansible for Enterprise

#Introduction

After episode 28, where we covered Windows automation — managing WinRM, IIS, Active Directory, and Windows Update from the same control node — in this episode we go one level higher: how to take Ansible from "it runs on a developer's laptop" to "an enterprise-class automation platform managed by many teams."

Ansible's journey is like a restaurant. When it first opens, one cook can double as chef, cashier, and waiter — just a small kitchen and a few menus. But when customers flood in and the team grows, you need a head chef organizing the menu, divisions separating the kitchen from the dining room, a centralized ordering system, and audited cleanliness standards. Otherwise, chaos: duplicate menus in two branches, different recipes at each chef station, and no one knows who's allowed to change the recipe.

Most Ansible teams start like a small kitchen: one developer running ansible-playbook from a laptop, inventory in a local file, everyone has production access. This runs smoothly until the number of hosts and team members grows. Then classic questions arise: Who runs playbooks to production? How do we separate dev and prod configuration without duplication? How does one playbook reach thousands of hosts without being slow? How do we ensure only certain people can change secrets?

In this episode we'll cover the four pillars of scaling: enterprise architecture (AAP/AWX/Semaphore, GitOps, distributed execution), repository structure (monorepo vs multi-repo, environment segregation), access control and delegation (RBAC, credential management, audit logging), and performance at scale (callback plugins, Mitogen, serial/forks).

#Main Discussion

#Enterprise Architecture Patterns

When Ansible is managed by a small team, the control node is whoever's laptop needs it. That doesn't scale for three reasons: there's no audit trail (who ran what), no credential centralization (passwords scattered across laptops), and no safe concurrency (two people can run conflicting playbooks).

The solution is a centralized control node: a platform that runs playbooks on the team's behalf, stores credentials securely, and records every execution. Here are the main options:

Common architecture patterns in enterprise:

1. Centralized automation — one AWX/AAP instance is the single point of execution. All engineers trigger jobs via the web UI, CLI (awx / ansible-runner), or REST API. No playbooks are run from laptops into production.
2. GitOps workflow — the Git repository is the source of truth. Pull requests are reviewed → merged → project sync in AWX/AAP pulls the latest version → the job runs. Automation is triggered by Git changes, not by humans.
3. Distributed execution (automation mesh) — for network flexibility and thousands-of-nodes scale, AAP supports hop nodes (execution nodes) that run jobs closer to targets, reducing latency and avoiding a single control node bottleneck.
4. Multi-team collaboration — each team has its own organization and team in AWX/AAP, complete with separate roles; they share collections/roles via internal Galaxy/Ansible Hub.

#Monorepo vs Multi-Repo

The most debated repository architecture question. Each has trade-offs:

An example enterprise monorepo structure that divides attention neatly — notice the separation of concerns: inventory, playbooks, roles, collections, and environment config each in their own place:

ansible-platform/
├── ansible.cfg
├── requirements.yml
├── inventory/
│   ├── production/
│   │   ├── hosts.yml
│   │   └── group_vars/
│   │       ├── webservers.yml
│   │       └── dbservers.yml
│   ├── staging/
│   │   ├── hosts.yml
│   │   └── group_vars/
│   └── dev/
│       ├── hosts.yml
│       └── group_vars/
├── playbooks/
│   ├── site.yml
│   ├── security-hardening.yml
│   └── windows-patching.yml
├── roles/
│   ├── nginx-website/
│   └── postgres-cluster/
├── collections/
│   └── internal/
│       └── platform-utils/
├── vault/
│   └── prod.secrets.yml
├── .github/workflows/ci.yml
└── docs/
    └── runbooks.md

#Environment Segregation: Dev, Staging, Production

Separating environments isn't just about folders, but also about security mechanisms that prevent cross-environment mixing. Recommended practices:

• Separate inventory — different YAML files for each environment (like the example above).
• Separate credentials — different vault passwords per environment; the production team holds access.
• Different vault IDs — ansible-vault encrypt --vault-id prod@prod.pass and --vault-id dev@dev.pass.
• Guard conditions — production playbooks protected with confirmation prompts or mandatory limiters:

- name: Site deployment
  hosts: "{{ target | default('all') }}"
  gather_facts: true
  pre_tasks:
    - name: Wajib menentukan environment target
      ansible.builtin.assert:
        that: env in ['dev', 'staging', 'production']
        fail_msg: "Set --extra-vars 'env=production' secara eksplisit!"
      when: env is defined

And an example of per-environment group_vars separating configuration without duplicating playbooks:

env: production
app_replicas: 8
log_level: warning
nginx_conf:
  worker_processes: auto
  keepalive_timeout: 65

env: staging
app_replicas: 2
log_level: info
nginx_conf:
  worker_processes: 4
  keepalive_timeout: 60

env: dev
app_replicas: 1
log_level: debug
nginx_conf:
  worker_processes: 2
  keepalive_timeout: 60

With this pattern, the same playbook (site.yml) produces the correct deployment in every environment just by pointing to the right inventory:

ansible-playbook -i inventory/dev/hosts.yml playbooks/site.yml
ansible-playbook -i inventory/staging/hosts.yml playbooks/site.yml \
  --vault-id staging@staging.pass
ansible-playbook -i inventory/production/hosts.yml playbooks/site.yml \
  --vault-id prod@prod.pass --limit "webservers"

#Access Control & Delegation: RBAC and Credential Management

In large companies, not everyone is allowed to run playbooks to production. This is answered with RBAC (Role-Based Access Control). In AWX/AAP, the RBAC hierarchy is:

• Organization — business unit (e.g., Payments, E-commerce).
• Team — working group within an organization (Platform, SRE, Database).
• User — individual; gets roles via a team or directly.
• Role — access rights like Admin, Execute, Read, Update, Ad Hoc.
• Job Template — a combination of project (repo), inventory, credential, and playbook; this is where permissions are enforced.

Thanks to this model, delegation becomes tidy: the SRE team can execute patching templates to staging but only read in production; CI service accounts can trigger specific templates; and no one ever needs to touch raw SSH keys.

Credential management is the most critical point. The principles:

1. Credentials live on the platform, not on laptops — SSH keys, passwords, and cloud API keys are stored in the encrypted AWX/AAP credential store, not in plaintext host_vars files.
2. Separate per environment — production and staging credentials must not be the same or share storage.
3. Automatic rotation — many enterprises integrate AAP with HashiCorp Vault or AWS Secrets Manager so passwords rotate automatically and their usage can be audited.
4. Never put credentials in a repository — even Vault-encrypted, "living" credentials should live on the platform; Git should contain data, not secrets.
5. no_log: true — make sure tasks handling secrets don't write them to job output; the platform audit log records who used which credential, not its contents.

Audit logging & compliance: every job run in AWX/AAP is fully recorded — who triggered it, which template, which credential, complete output, and success/failure status. This is the foundation of compliance (recall episode 27): if an auditor asks "who changed the production firewall config last night?", the answer is in the audit log, not a guess.

#Performance at Scale: Thousands of Hosts

Dealing with thousands of hosts is a performance challenge. Some key settings:

• forks — the default number of parallel processes. The default of 5 is too small; raise it according to control node capacity. Raising forks is usually the first effect you'll notice.
• serial — limits how many hosts are processed per wave; for patching or staged rollouts.
• throttle — limits concurrency at the task level, not the play level; useful for APIs with rate limits (e.g., restarting one at a time).

Example global settings in ansible.cfg:

[defaults]
forks = 50
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/facts_cache
fact_caching_timeout = 3600
 
[ssh_connection]
pipelining = true

An example of correct batching for a staged rollout — combining serial with a percentage and a max_fail_percentage:

- name: Rolling update ribuan host
  hosts: webservers
  become: true
  serial: "20%"
  max_fail_percentage: 10
  tasks:
    - name: Deploy versi aplikasi terbaru
      ansible.builtin.service:
        name: webapp
        state: restarted

Or throttle for finer per-task control:

- name: Restart service satu per satu
  hosts: all
  throttle: 1
  tasks:
    - name: Restart nginx (hanya 1 host pada satu waktu)
      ansible.builtin.service:
        name: nginx
        state: restarted

#Callback Plugins for Custom Output

Ansible's default output (the default callback) is enough for experiments, but in enterprise you often want more: seeing per-task duration, tidier YAML formatting, or integration with external systems. That's where callback plugins come in — they change how Ansible reports execution results.

Commonly used callbacks:

Enabling callbacks to show duration (timer) and tidy YAML output:

[defaults]
stdout_callback = yaml
callback_whitelist = timer, profile_tasks
 
[callback_profile_tasks]
sort_order = descending

With profile_tasks, at the end of the playbook you'll see a top slowest tasks table — the main weapon for finding which task is the bottleneck when running playbooks against thousands of hosts. For report integration into CI, use junit:

ANSIBLE_STDOUT_CALLBACK=junit \
ansible-playbook -i inventory/production/hosts.yml playbooks/site.yml

The JUnit output can be directly consumed by a GitLab CI / GitHub Actions pipeline for test reports, making Ansible automation part of the company's quality gate.

#Mitogen: The Extreme Performance Strategy

When Ansible is already optimized but still feels slow — usually because of the Python process overhead that must be copied to the remote for every task — Mitogen is the secret weapon. Mitogen replaces Ansible's SSH execution mechanism with memory-based, streaming lazy execution, reducing the remote Python processes spawned per-task.

User-reported effects are very significant: playbook execution can be 4–10x faster on many workloads, because Mitogen retains state between tasks without having to restart the Python interpreter for each task.

[defaults]
strategy_plugins = /usr/share/ansible/plugins/strategy
strategy = mitogen_linear

#Common Pitfalls

1. No centralized control node

Playbooks run from many people's laptops → no audit trail, credential leaks, and potential conflicts. An enterprise without AWX/AAP is basically not enterprise-grade.

2. One inventory for all environments

all can hit production. Always separate inventory files per environment and make sure production credentials never leave the platform.

3. forks left at the default of 5

For hundreds/thousands of hosts this is very slow. Raise it gradually while monitoring control node CPU/memory.

4. Rebooting/patching all hosts at once

Without serial, a playbook restarting services on all hosts simultaneously = outage. Always batch with serial and max_fail_percentage.

5. RBAC only in the UI, not on the network

If engineers can bypass the platform with direct SSH, RBAC is meaningless. Secure the network too.

6. Ignoring audit logging

Without execution logs, you're blind to "who did what" — AWX/AAP audit logging is a compliance feature often undervalued until the annual audit arrives.

#Conclusion

In this episode we've covered scaling Ansible for enterprise: understanding centralized architecture patterns with AWX/AAP/Semaphore and GitOps workflows, comparing monorepo vs multi-repo strategies with a tidy repository structure, applying dev/staging/production environment segregation with separate inventory and vaults, implementing secure RBAC and credential management with audit logging, and optimizing performance for thousands of hosts via forks, pipelining, fact caching, callback plugins, serial/throttle batching, and even the extreme Mitogen strategy.

In essence, scaling Ansible isn't just technical — it's about governance: ensuring automation code is centralized, who may change what, and a complete audit trail. With this foundation, you're ready to build an automation platform that can be held accountable at any company.

In episode 30 — the closing episode of this series — we'll cover the Production Deployment Checklist & Best Practices: a pre-production checklist (linting, Molecule testing, dry-runs, inventory validation), operational best practices, common mistakes to avoid, monitoring Ansible operations, and disaster recovery and continuous improvement. Keep your enthusiasm up!