Learn Ansible - Monitoring & Observability Stack Automation

#Introduction

After episode 24, where we covered database management — automatically managing PostgreSQL, MySQL/MariaDB, and MongoDB — in this episode we'll cover something many teams actually forget: how do we know that everything we've built is still working well?

There's a classic saying in the engineering world: "You can't manage what you can't measure." A server might look "just fine" from an SSH perspective, when in fact its CPU is already at 95%, the disk is full, and the application is starting to slow down. That problem is only felt when users complain. Yet, if we had a good monitoring system, that problem would already be detected in the first 10 minutes — even before users experience it.

Now imagine a team managing 50 servers across three environments (dev, staging, production), all provisioned with Ansible. Without automation, installing Node Exporter on 50 servers means SSHing into 50 servers. Changing prometheus.yml means editing and restarting 1 server, then hoping all the new targets get picked up. That doesn't scale, and sooner or later it leads to configuration drift — one server has an old Node Exporter version, another isn't exposed to Prometheus, and so on.

That's why monitoring must be treated like infrastructure: code, versioned in Git, tested, and applied with Ansible. This concept is often called observability as code. In this episode we'll cover deploying the Prometheus + Alertmanager + Grafana + Node Exporter stack, templating prometheus.yml with dynamic targets, managing alert rules, Grafana dashboards as code, and the ELK logging stack (Elasticsearch, Logstash, Kibana) along with its lightweight alternative: Loki + Promtail.

#Main Discussion

#Getting to Know the Monitoring & Observability Stack Components

Before diving into code, let's first understand each component and its role:

The role split is simple: Prometheus + Grafana answer the question "is the system healthy?" (metrics), while ELK / Loki answer "why is the system broken?" (logs). Both must exist — metrics tell you what's wrong, logs tell you why.

Install the required collections:

ansible-galaxy collection install prometheus.prometheus community.grafana

#Role Structure for the Monitoring Stack

Instead of writing one giant playbook, we break it into roles so they can be reused — exactly the pattern we learned in episode 12. The project structure:

ansible-monitoring/
├── inventory/
│   └── production.yml
├── group_vars/
│   ├── monitoring.yml
│   └── all.yml
├── playbook-monitoring.yml
└── roles/
    ├── prometheus/
    │   ├── tasks/main.yml
    │   ├── handlers/main.yml
    │   └── templates/
    │       ├── prometheus.yml.j2
    │       ├── alertmanager.yml.j2
    │       └── alert.rules.yml.j2
    ├── alertmanager/
    │   └── tasks/main.yml
    └── node_exporter/
        └── tasks/main.yml

The main playbook is very concise — proof that modularity works:

- name: Deploy stack monitoring ke server monitoring
  hosts: monitoring_servers
  become: true
  roles:
    - role: prometheus
    - role: alertmanager
 
- name: Deploy Node Exporter ke seluruh node terkelola
  hosts: all
  become: true
  roles:
    - role: node_exporter

Notice the important pattern: the prometheus and alertmanager roles only run on hosts in the monitoring_servers group, while the node_exporter role runs on all hosts in the inventory. This way, adding a new server to the inventory just means Node Exporter will automatically be installed on the next run — no manual SSH.

#Templating prometheus.yml with Dynamic Targets

This is the core part of observability as code: the Prometheus config file is generated from a Jinja2 template so scrape targets are not hardcoded. Targets come directly from the Ansible inventory — the same single source of truth for the entire infrastructure.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
rule_files:
  - "/etc/prometheus/rules/*.rules.yml"
 
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
        labels:
          env: "{{ env_name }}"
 
  - job_name: "node"
    static_configs:
      - targets:
{% for host in groups['nodes'] %}
          - "{{ hostvars[host].ansible_host | default(host) }}:9100"
{% endfor %}
        labels:
          env: "{{ env_name }}"

Let's break down the Jinja2 logic above:

• groups['nodes'] — the list of all hosts in the nodes inventory group. Prometheus doesn't need to know IPs one by one; just reference the inventory group.
• hostvars[host].ansible_host | default(host) — takes each host's IP address from the fact Ansible collected, with a fallback to the hostname if ansible_host doesn't exist. This is why gather_facts matters in this playbook.
• {% for %} loop — maps one inventory host to one - "ip:9100" line.

The task rendering this template automatically restarts Prometheus via a handler when there's a change:

- name: Render konfigurasi Prometheus
  ansible.builtin.template:
    src: prometheus.yml.j2
    dest: /etc/prometheus/prometheus.yml
    owner: prometheus
    group: prometheus
    mode: "0640"
  notify: restart prometheus
 
- name: Render alert rules
  ansible.builtin.template:
    src: alert.rules.yml.j2
    dest: /etc/prometheus/rules/alert.rules.yml
    owner: prometheus
    group: prometheus
    mode: "0640"
  notify: reload prometheus

#Managing Alert Rules as Code

Alert rules define when Prometheus "cries out". Because they're files, these rules can be versioned, reviewed in Pull Requests, and applied consistently:

groups:
  - name: node-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU usage di atas 85% pada {{ $labels.instance }}"
 
      - alert: NodeDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.instance }} tidak merespons scrape"

Why is for: 10m important? Because of flapping — a CPU that spikes to 90% for 2 seconds then drops doesn't need to wake an engineer at 3 AM. The for parameter ensures the condition must hold for that duration before the alert is actually sent. This is one way to prevent alert fatigue, the condition where teams ignore alerts because of too many false alarms.

#Alertmanager: Routing & Notifications

Alertmanager's job is to receive all alerts from Prometheus, then decide where each alert is sent — email, Slack, Telegram, or webhook. Its template:

global:
  smtp_smarthost: "{{ smtp_host }}:587"
  smtp_from: "{{ alert_email_from }}"
  smtp_auth_username: "{{ smtp_username }}"
  smtp_auth_password: "{{ vault_smtp_password }}"
 
route:
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "ops-team"
 
receivers:
  - name: "ops-team"
    email_configs:
      - to: "{{ alert_email_to }}"
        send_resolved: true
    slack_configs:
      - api_url: "{{ vault_slack_webhook }}"
        channel: "#ops-alerts"

Three routing parameters you must understand:

• group_wait — how long to wait before sending the first alert batch. Helps group alerts that arrive together (e.g., one server dies → many up == 0 metrics).
• group_interval — the group checking interval to see if new alerts join the same group.
• repeat_interval — how often unresolved alerts are resent. 4h means the team's pager doesn't keep ringing for the same problem.

#Grafana: Datasource & Dashboard as Code

Grafana is the face of the entire stack. Here the dashboard as code principle applies: dashboards are no longer objects created by click-and-drag in the UI, but JSON files versioned in Git. This way, dashboard changes can be reviewed, tested, and applied consistently across all environments.

There are two ways to manage Grafana with Ansible: via provisioning files (stored in /etc/grafana/provisioning/) or via the community.grafana API modules. Both are often combined — let's look at both at once:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true
    editable: false

apiVersion: 1
providers:
  - name: "node-dashboards"
    orgId: 1
    folder: "Node"
    type: file
    disableDeletion: false
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards

- name: Deploy konfigurasi Grafana
  hosts: grafana_servers
  become: true
  tasks:
    - name: Salin provisioning datasource
      ansible.builtin.copy:
        src: files/datasources/prometheus.yml
        dest: /etc/grafana/provisioning/datasources/prometheus.yml
        owner: grafana
        group: grafana
        mode: "0644"
      notify: restart grafana
 
    - name: Import dashboard Node Exporter Full via API
      community.grafana.grafana_dashboard:
        grafana_url: "{{ grafana_url }}"
        grafana_api_key: "{{ vault_grafana_api_key }}"
        slug: node-exporter-full
        state: present

The provisioning file approach (apiVersion: 1) is the way Grafana recommends for automated deployment: just drop the JSON dashboard folder into the registered path, and Grafana loads it. Meanwhile, the grafana_dashboard module uses the Grafana REST API — useful for importing dashboards from the Grafana.com library or updating existing dashboards programmatically. Choose by need: provisioning files for your own dashboards (in the repo), API modules for community dashboards.

#Logging: The ELK Stack (Elasticsearch, Logstash, Kibana)

A metrics-based monitoring stack isn't enough — when an application errors, you need logs for root cause analysis. The most classic stack is ELK. Deploying it with Ansible follows the same pattern: Elasticsearch as the store, Logstash as the pipeline, Kibana for visualization, and Filebeat as the agent on every node.

Here's an example Filebeat configuration, templated to send application logs to Logstash:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/*.log
      - /var/log/app/*.log
    fields:
      service: "{{ service_name }}"
    fields_under_root: true
 
output.logstash:
  hosts: ["{{ logstash_host }}:5044"]
 
logging.level: info

- name: Deploy Filebeat ke node aplikasi
  hosts: app_servers
  become: true
  vars:
    service_name: web-app
    logstash_host: logstash.internal
  tasks:
    - name: Render konfigurasi Filebeat
      ansible.builtin.template:
        src: filebeat.yml.j2
        dest: /etc/filebeat/filebeat.yml
        owner: root
        group: root
        mode: "0600"
      notify: restart filebeat
 
    - name: Mulai dan aktifkan Filebeat
      ansible.builtin.systemd_service:
        name: filebeat
        state: started
        enabled: true

#Loki & Promtail: The Lightweight Alternative

Loki is designed following the Prometheus philosophy: labels as the primary index, store logs as-is. Instead of building a full index like Elasticsearch, Loki indexes log labels and lets the log contents be compressed — far more resource-efficient. Its companion, Promtail, is the agent that ships logs from nodes to Loki. Both configurations can be managed as templated files:

auth_enabled: false
 
server:
  http_listen_port: 3100
 
common:
  path_prefix: /var/lib/loki
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
  replication_factor: 1
 
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13

server:
  http_listen_port: 9080
  grpc_listen_port: 9095
 
clients:
  - url: http://{{ loki_host }}:3100/loki/api/v1/push
 
scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log
 
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          __path__: /var/log/nginx/*.log

- name: Deploy stack Loki
  hosts: all
  become: true
  tasks:
    - name: Render konfigurasi Promtail
      ansible.builtin.template:
        src: promtail-config.yml.j2
        dest: /etc/promtail/config.yml
        owner: promtail
        group: adm
        mode: "0640"
      notify: restart promtail

Notice the job: nginx, job: varlogs labels — that's what makes log search fast in Loki: a query like {job="nginx"} |= "ERROR" only scans those labels, not the entire log contents. This convenience again teaches the same principle: configuration (labels, paths, targets) is code, not something edited manually per server.

#Common Pitfalls

1. Hardcoded Prometheus targets

Writing server IPs one by one in prometheus.yml means adding a new server = manually editing the file. This makes drift happen quickly. Use templating with groups['nodes'] as above, or go further: file_sd_configs/http_sd_configs generated by Ansible.

2. Alerts never evaluated because of evaluation_interval

evaluation_interval determines how often Prometheus evaluates rule files. If the value is too large (e.g., 1m when for: 10m and the condition is fast), alerts arrive late. The standard is 15s — enough for most cases.

3. Invalid alert rules YAML

One indentation mistake in alert.rules.yml and Prometheus refuses to load all rules — alerts silently die. Make it a habit to add a validation task, e.g., promtool check rules /etc/prometheus/rules/, before restart/reload.

4. Plaintext Grafana API key

The Grafana API key needed by the grafana_dashboard module should live in Ansible Vault. Don't let it get committed to Git — you know the consequences from episode 14.

5. Node Exporter exposed to the public

Port 9100 should only be reachable from the Prometheus server. Exposing it to the internet = detailed server info (kernel version, disk list, temperature) leaks to anyone. Restrict it with a firewall/UFW (we cover hardening in episode 27).

6. Promtail without log read permissions

Promtail often silently fails to read /var/log because its group doesn't have access. Make sure the Promtail user is in the adm group (Debian/Ubuntu) or configure the right permissions.

7. Forgetting to plan Prometheus disk capacity

Prometheus's default retention stores 15 days of data. With many targets and high metric cardinality, the disk can fill up within days. Configure retention.time and monitor its disk volume — ironically, monitoring stacks most often get hit by their own disk full alerts.

#Conclusion

In this episode we've built the foundation of observability as code with Ansible: deploying Prometheus and Alertmanager, Node Exporter automatically spreading to all nodes, templating prometheus.yml with dynamic targets from inventory, alert rules and Alertmanager routing as code, Grafana with provisioned datasources and dashboards, and the ELK logging stack with its lightweight alternative Loki + Promtail.

The principle uniting everything: monitoring is infrastructure. It must be managed the same way — versioned, reviewed, idempotent, and applied automatically. This way, adding a new server isn't more manual work, just one more line in the inventory.

In episode 26, we'll take all these skills to the next level: Cloud Infrastructure Provisioning — managing EC2, VPC, S3, and IAM in AWS; Compute Engine and GCS in Google Cloud; and Virtual Machines and Virtual Networks in Azure — complete with multi-cloud patterns. Keep your enthusiasm up!