Learn Ansible - Advanced Inventory Management & Dynamic Inventory

#Introduction

After episode 20, where we covered modern execution environments like AAP/AWX, ansible-navigator, and ansible-builder to solve environment consistency between engineers, in this episode we'll cover the foundation that often determines the success of all automation in a cloud environment: advanced inventory management and dynamic inventory.

Imagine holding a handwritten shopping list for dozens of stores that can change in number at any time. Every time a store opens or closes, you have to update that record manually — prone to being late, wrong, and exhausting. That's the real condition when DevOps teams still manage Ansible inventory statically in the auto-scaling era: new instances are created by Auto Scaling Groups within minutes, then destroyed when traffic drops. If your host list doesn't change automatically too, playbooks will run tasks against servers that no longer exist, or skip new servers that actually need configuration the most.

That's why dynamic inventory is no longer a "nice feature," but a mandatory requirement in the cloud world. In this episode we'll cover three main pillars: inventory plugins for cloud providers (AWS, GCP, Azure, and VMware), advanced plugin configuration like filtering, keyed groups, and composed variables, plus custom inventory scripts and integration with CMDBs like NetBox and ServiceNow.

#Main Discussion

#Why Isn't Static Inventory Enough?

Since episode 3, we've known static inventory in both INI and YAML formats. Static inventory works very well for a set of relatively fixed servers — for example, 5 on-premises VM servers that rarely change. The problem arises when infrastructure starts moving to the cloud:

• Instances are ephemeral — born and die following traffic scale.
• Metadata changes — public IPs change on every recreate, tags grow, regions can differ.
• Counts can reach hundreds to thousands — writing them one by one in a YAML file is a human error generator.

The solution is dynamic inventory: Ansible asks the cloud provider API directly for the host list each time it's needed, then assembles it into an in-memory inventory structure. You never store the host list, but rather the rules about how that list should be formed.

#Inventory Plugin vs Inventory Script

There are two eras of dynamic inventory approaches in Ansible:

Ansible now pushes the use of inventory plugins because they're more declarative, well-tested, and bundled into official cloud provider collections. However, understanding the JSON format produced by scripts remains important — we'll cover this in the custom inventory script section.

#AWS EC2 Inventory Plugin

The AWS EC2 plugin is named amazon.aws.aws_ec2. This plugin reads the list of EC2 instances from the AWS API and turns them into Ansible hosts. Requirements: the amazon.aws collection is installed and AWS credentials are available (usually via environment variables, the ~/.aws/credentials file, or an IAM role).

Install the required collection first:

ansible-galaxy collection install amazon.aws

Next, create the plugin config file. The filename must end with a certain pattern to be recognized as an inventory plugin — for AWS, files ending in aws_ec2.yml or aws_ec2.yaml are automatically read as the aws_ec2 plugin. Here's a complete example configuration:

plugin: amazon.aws.aws_ec2
regions:
  - ap-southeast-1
  - us-east-1
filters:
  tag:Env: production
  tag:Project: "{{ project_name }}"
  instance-state-name: running
hostnames:
  - tag:Name
  - dns-name
  - private-ip-address
keyed_groups:
  - key: tags.Environment
    prefix: env
    separator: ""
  - key: placement.region
    prefix: region
  - key: "tags.Role"
    prefix: role
compose:
  ansible_host: public_ip_address
  ansible_user: ubuntu
  instance_type: instance_type
  vpc_id: vpc_id
strict: false

Let's break it down one by one:

• plugin: amazon.aws.aws_ec2 — points to the inventory plugin used. Must be the first key.
• regions — limits instance search to specific regions. Searching all regions slows down execution, so always narrow it down.
• filters — filters instances by supported AWS attributes. Here we only take instances with tags Env: production, Project: <project_name>, and status running.
• hostnames — determines the hostname Ansible uses, in order of highest priority. The order above: use the Name tag value, fall back to dns-name, finally private-ip-address.
• keyed_groups — automatically creates groups based on metadata values. This is the key to automatic grouping, which we'll dig into shortly.
• compose — sets or overrides host variables with Jinja2 expressions. Here ansible_host is filled from the instance's public IP.
• strict: false — if true, an error in undefined keyed_groups/compose will fail the inventory; if false, the line is skipped with a warning.

#Filtering Instances: Tags, Region, and Status

Filtering is the mechanism to "prune" the instance list before it enters the inventory. In the AWS EC2 plugin, filters follow AWS EC2 DescribeInstances syntax. Some of the most commonly used patterns:

Filters are server-side — the filtering happens at the AWS API, not on the Ansible side. This means the returned list is already slim from the start, saving transfer time and parsing.

#Keyed Groups and Composed Variables

Keyed groups is a feature that automatically creates inventory groups from a metadata value. The analogy is an automatic filing system: every time a new document arrives, it's immediately placed into the matching drawer based on its label, without you writing each one.

With the keyed_groups configuration above, instances with tags Environment: production and Role: web automatically go into groups env_production and role_web. The benefits are big:

• You can target hosts: env_production without ever mentioning a single hostname.
• Playbooks become environment-aware — the same playbook can run on env_production or env_staging, just by changing the host pattern.
• New groups form by themselves when new metadata values appear, without needing to change the config file.

Meanwhile, composed variables let you compute host variable values dynamically. This is very useful for adjusting per-host connection details — e.g., ansible_host from the public IP, or a different ansible_user for different AMIs:

compose:
  ansible_host: public_ip_address
  ansible_user: "'{{ 'ubuntu' if 'ubuntu' in image_id else 'ec2-user' }}'"
  instance_role: "tags.Role"

#Verifying Inventory with ansible-inventory

Once the config is ready, we must make sure the result is correct before running playbooks. The main tool is ansible-inventory:

ansible-inventory -i aws_ec2.yml --list

ansible-inventory -i aws_ec2.yml --graph

The --graph output will look like this:

@all:
  |--@aws_ec2:
  |  |--@env_production:
  |  |  |--@region_ap-southeast-1:
  |  |  |  |--@role_web:
  |  |  |  |  |--web-prod-01
  |  |  |  |  |--web-prod-02
  |  |  |  |--@role_db:
  |  |  |  |  |--db-prod-01
  |  |  |--@region_us-east-1:
  |  |  |  |--@role_worker:
  |  |  |  |  |--worker-prod-01
  |  |--@ungrouped:
  |  |  |--legacy-server

The graph above shows how groups stack hierarchically from keyed_groups. You can directly target hosts: role_web, hosts: env_production, or even combinations like hosts: env_production:&role_web — exactly like the host matching patterns we learned in episode 3.

#Dynamic Inventory for Google Cloud, Azure, and VMware

Besides AWS, Ansible has official plugins for the major cloud providers. Their configuration patterns are very similar because they all extend the same constructed mechanism (filters, keyed_groups, compose). Let's look at each.

#Google Cloud Platform

The google.gcp.gcp_compute plugin fetches the Compute Engine instance list. GCP credentials can be a service account file or the google.cloud.auth plugin. Example configuration:

plugin: google.gcp.gcp_compute
auth_kind: serviceaccount
service_account_file: /opt/ansible/credentials/gcp-service-account.json
project: my-company-project
zones:
  - asia-southeast1-a
filters:
  - status = RUNNING
  - labels.env = production
hostnames:
  - name
keyed_groups:
  - key: labels.role
    prefix: gcp_role
  - key: zone
    prefix: gcp_zone
compose:
  ansible_host: networkInterfaces[0].accessConfigs[0].natIP
strict: false

#Microsoft Azure

The azure.azcollection.azure_rm plugin uses service principal credentials, usually set via environment variables (AZURE_SUBSCRIPTION_ID, AZURE_TENANT, AZURE_CLIENT_ID, AZURE_SECRET):

plugin: azure.azcollection.azure_rm
auth_source: env
hostname:
  - name
  - default
conditional_groups:
  production: "'prod' in tags.env | default('dev')"
keyed_groups:
  - key: tags.role
    prefix: az_role
  - key: resource_group
    prefix: az_rg
groups:
  azure: true
compose:
  ansible_host: private_ipv4_addresses[0]

#VMware vSphere

For on-premises and hybrid environments, the community.vmware.vmware_vm_inventory plugin fetches the virtual machine list directly from vCenter:

plugin: community.vmware.vmware_vm_inventory
strict: false
hostname: vcenter01.company.internal
username: administrator@vsphere.local
password: "{{ vault_vsphere_password }}"
validate_certs: false
with_tags: true
properties:
  - name
  - config.name
  - guest.ipAddress
  - summary.runtime.powerState
filters:
  - summary.runtime.powerState == "poweredOn"
hostnames:
  - config.name
keyed_groups:
  - key: guest.guestId
    prefix: vm_os
  - key: tag_category.Environment
    prefix: vm_env
compose:
  ansible_host: guest.ipAddress

#Inventory Plugin Summary per Cloud Provider

Here's a reference table of plugins you can use as a quick reference:

#Enabling Plugins in ansible.cfg

Some inventory plugins need to be explicitly enabled in ansible.cfg, especially in Ansible versions that restrict the list of allowed plugins:

[inventory]
enable_plugins = aws_ec2, gcp_compute, azure_rm, vmware_vm_inventory

Additionally, enable inventory caching so Ansible doesn't call the cloud API on every execution — this saves significant time for large inventories:

[inventory]
enable_plugins = aws_ec2, gcp_compute, azure_rm, vmware_vm_inventory
cache = true
cache_plugin = ansible.builtin.jsonfile
cache_connection = /tmp/ansible_inventory
cache_timeout = 3600

#Custom Inventory Script

Even though inventory plugins cover the majority of needs, there are times when you need to pull data from internal systems that don't have an official plugin yet — for example, an internal team API, an asset management database, or an ownership spreadsheet. That's where the custom inventory script comes in.

The contract is simple: the script accepts a --list argument (and optionally --host <host>) and outputs JSON. Since Ansible 2.4, all data can be sent at once via the special _meta key so --host is rarely used. Here's a minimal Python script skeleton:

custom_inventory.py

#!/usr/bin/env python3
import json
import sys
import argparse
 
 
def build_inventory():
    inventory = {
        "web_servers": {
            "hosts": ["web-01.internal", "web-02.internal"],
            "vars": {
                "ansible_user": "ubuntu",
                "nginx_port": 8080,
            },
        },
        "db_servers": {
            "hosts": ["db-01.internal"],
        },
        "_meta": {
            "hostvars": {
                "web-01.internal": {"ansible_host": "10.0.1.11"},
                "web-02.internal": {"ansible_host": "10.0.1.12"},
                "db-01.internal": {"ansible_host": "10.0.2.11"},
            }
        },
    }
    return inventory
 
 
def main():
    parser = argparse.ArgumentParser(description="Custom dynamic inventory")
    parser.add_argument("--list", action="store_true")
    parser.add_argument("--host", nargs="?")
    args = parser.parse_args()
 
    inventory = build_inventory()
    if args.list:
        print(json.dumps(inventory, indent=2))
    elif args.host:
        host = args.host
        print(json.dumps(inventory["_meta"]["hostvars"].get(host, {})))
    else:
        sys.exit("Argumen --list atau --host wajib diberikan")
 
 
if __name__ == "__main__":
    main()

The JSON structure above consists of:

• Groups (web_servers, db_servers) — the main keys are group names, containing hosts and optional vars.
• _meta.hostvars — contains host variables for each host. Everything is sent at once so Ansible doesn't call the script per host.
• Ungrouped hosts — can be placed directly as keys in the form of hostnames with a value of {}.

To make it usable, the script must be executable:

chmod +x custom_inventory.py
./custom_inventory.py --list
ansible-inventory -i custom_inventory.py --graph

This script doesn't need to wait for complex output — the important thing is the structure follows the contract above. You can replace build_inventory() with a call to a CMDB API, a database query, or CSV file parsing.

#Integration with CMDBs: NetBox and ServiceNow

An increasingly common pattern in large companies is making the CMDB the source of truth for the server list, rather than querying each cloud provider separately. Benefits: one consistent data channel, covering hybrid infrastructure, complete with application relationships, owners, and lifecycle.

#NetBox

NetBox is a popular open-source CMDB / IPAM among network and infrastructure engineers. Ansible has the official inventory plugin netbox.netbox.nb_inventory:

plugin: netbox.netbox.nb_inventory
api_endpoint: https://netbox.company.internal
token: "{{ vault_netbox_token }}"
validate_certs: true
group_by:
  - device_roles
  - sites
  - status
query_filters:
  - role: server
  - status: active
keyed_groups:
  - key: device_roles[0].name
    prefix: nb_role
compose:
  ansible_host: primary_ip4.address

#ServiceNow

ServiceNow ITSM also provides the servicenow.itsm.now inventory plugin, reading data directly from the CMDB table (cmdb_ci_server by default). Credentials are taken from the SN_HOST, SN_USERNAME, SN_PASSWORD environment variables, or an instance block in the file:

plugin: servicenow.itsm.now
table: cmdb_ci_server
query:
  - os: = Linux Red Hat
  - os: = Ubuntu Linux
sysparm_limit: 5000
keyed_groups:
  - key: manufacturer
    separator: ""
  - key: environment
    prefix: sn_env
compose:
  ansible_host: fqdn
  ansible_user: "'cloud-user'"

#Common Pitfalls

1. Hardcoded credentials in the inventory file

Putting AWS access keys or vCenter passwords directly in aws_ec2.yml. This file usually goes into version control, so credentials can leak. Solution: use environment variables, a separate credentials file, or Ansible Vault.

2. Forgetting to install the collection

Running the amazon.aws.aws_ec2 plugin without ansible-galaxy collection install amazon.aws results in an Unable to load inventory plugin error. Make sure the collection is installed on the control node or listed in requirements.yml (recall episode 13).

3. Wrong file naming

Naming the file inventory-aws.yml when the expected pattern is aws_ec2.yml. As a result, the file is treated as a static inventory and the result is empty or a YAML error. Use the exact suffix from the plugin table.

4. Targeting all regions without filters

A config without regions will scan all regions and can be very slow, even risking API rate limits. Narrow down regions and use filters as early as possible.

5. Misunderstanding underscores in keyed_groups

By default, a keyed group with prefix and separator produces names like env_production (underscore from separator). If you want names without a separator or without a leading underscore, set separator: "" and leading_separator: false.

6. Not verifying with ansible-inventory

Running playbooks directly without checking the inventory result. Always run ansible-inventory -i <file> --graph first to ensure hosts and groups are formed as expected.

#Conclusion

In this episode we've covered advanced inventory management thoroughly: why static inventory is no longer adequate in the cloud era, how inventory plugins work for AWS EC2, GCP, Azure, and VMware complete with filtering, keyed groups, and composed variables, verification techniques with ansible-inventory, custom Python inventory scripts that follow Ansible's JSON contract, and integration with CMDBs via NetBox and ServiceNow. We also examined common mistakes and their solutions.

With dynamic inventory, your playbooks are now environment-aware and scale-ready: new servers appear in inventory automatically, and groups form by themselves following metadata. This is one of the biggest differentiators between automation that only "runs in the lab" and automation ready to survive at large production scale.

In episode 22, we'll cover a challenging and interesting topic at the same time: Network Automation with Ansible — managing network devices like Cisco IOS/NX-OS, Arista EOS, and Juniper Junos programmatically using network collections, the network_cli, httpapi, and netconf connections, complete with automatic running-config backup and templated config deployment to many switches at once. Keep your enthusiasm up!