Learn GitOps with ArgoCD - Monitoring & Observability

#Introduction

In episode 21 we built a solid disaster recovery plan. But there's an assumption underneath it: we can know when the system starts to drift. ArgoCD is a controller that keeps reconciling — but who reconciles ArgoCD itself? The answer: you, through monitoring & observability. In this episode we open up the metrics ArgoCD exposes, build Grafana dashboards, compose alerting that's actually useful, and organize component logs.

Why does this matter? GitOps moves the complexity from "how to deploy" to "how to run deployment as a system". An ArgoCD that silently stops syncing (repo-server running out of memory, Redis down, expired credentials) is the same danger as an erroring application — only more dangerous because its failure is silent. Observability makes that silent failure loud.

#ArgoCD Metrics

ArgoCD exposes Prometheus metrics from three main components:

The most operationally valuable metrics are those reflecting the delivery status:

argocd_app_info{namespace="api"} 1
argocd_app_sync_status{name="api",sync_status="Synced"} 1
argocd_app_health_status{name="api",health_status="Healthy"} 1
argocd_app_reconcile_count{name="api"} 4820
argocd_repo_pending_request_total{repo="repo"} 3

argocd_app_sync_status and argocd_app_health_status are the core of ArgoCD observability: they tell whether Git and the cluster agree and whether what's running is healthy. argocd_app_reconcile_count hints at reconciliation activity — a number that stops rising can mean the controller is stuck.

#Setting Up Scraping

If you use kube-prometheus-stack, use a ServiceMonitor (adjust the port name to match the ArgoCD Service):

ServiceMonitor for ArgoCD

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-controller
  namespace: argocd
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-application-controller
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

#Grafana Dashboards

The Argo ecosystem provides official dashboards (ID 14592 for the controller, ID 14593 for the repo server) that can be imported directly into Grafana. After import, the dashboard shows: application count per sync status, per health status, reconcile rate, and controller errors.

For specific needs, build a custom dashboard. Three panels that are almost always useful:

• Apps OutOfSync — count(argocd_app_sync_status{sync_status="OutOfSync"}) — one number telling "how many applications are drifting".
• Sync duration — histogram_quantile(0.95, sum(rate(argocd_app_sync_duration_seconds_bucket[5m])) by (le)) — how long synchronization takes; getting worse means the repo server is slowing down.
• Reconcile rate — sum(rate(argocd_app_reconcile_count[5m])) by (name) — per-application activity.

count(argocd_app_sync_status{sync_status="OutOfSync",name!=""})

#Application Observability

ArgoCD observability doesn't stop at sync status. To understand the user experience, expand to application metrics:

• Deployment metrics — argocd_app_info with health and sync labels as a release timeline.
• Sync duration — time from a Git change to the Synced status; a rising trend indicates a repo problem or too many resources.
• Error rates — application HTTP metrics (http_requests_total from episode 20), not just deployment status.

A powerful pattern: connect application metrics to Rollout analysis (episode 19) so the delivery dashboard and the application dashboard speak from the same data. When a canary is rolled out, the application error rate is the same feedback the AnalysisTemplate uses to decide promotion.

#Alerting

Metrics without alerts are an archive. Use Prometheus rules with AlertManager to catch the three most common classes of problems:

groups:
  - name: argocd
    rules:
      - alert: ArgoCDAppSyncFailed
        expr: argocd_app_sync_status{sync_status="Unknown"} == 1
          or argocd_app_health_status{health_status="Missing"} == 1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Sync gagal atau resource hilang"
      - alert: ArgoCDAppOutOfSync
        expr: argocd_app_sync_status{sync_status="OutOfSync"} == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Aplikasi tidak sinkron lebih dari 15 menit"
      - alert: ArgoCDRepoServerDown
        expr: up{job="argocd-repo-server"} == 0
        for: 5m
        labels:
          severity: critical

An important note: the OutOfSync alert must have a sufficiently long for. By design, an application is always OutOfSync for a moment after a manifest change and before the sync finishes — an alert without for will fire on every normal deployment. The rule of thumb:

• Sync failure / health degraded → immediately (critical severity).
• Persistent OutOfSync → give it time (e.g. 15 minutes).
• ArgoCD component down → immediately (all syncs stop).

#Logging

Metrics tell what happened; logs tell why. ArgoCD writes logs to stdout — kubectl logs -n argocd deployment/argocd-application-controller — but for cross-time investigation, aggregate them with Loki or Elasticsearch.

Two classes of logs you must collect:

• ArgoCD component logs — controller (reconcile, sync operation), repo server (clone, cache), API server (request, audit).
• Application logs — the logs of pods managed by ArgoCD; the primary source when debugging.

For Loki with Promtail/Alloy, add a simple scrape_configs for the argocd namespace:

scrape_configs:
  - job_name: kubernetes-pods-argocd
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        regex: argocd
        action: keep

With logs collected, investigating problems becomes much faster: take the alert's time range, query the controller logs in that range, and find the root cause without opening a terminal in the middle of the night.

#Closing

This episode equipped ArgoCD with eyes and ears: Prometheus metrics from the controller, API server, and repo server; a ServiceMonitor for scraping; official and custom Grafana dashboards; alerting for sync failures, out-of-sync, and down components; application observability through sync duration and error rates; and log aggregation with Loki.

The points you should take with you:

• argocd_app_sync_status and argocd_app_health_status are the core delivery metrics.
• An OutOfSync alert needs for so it doesn't fire on normal deployments.
• The official ArgoCD dashboards are the starting point; custom panels answer the team's questions.
• Application metrics and delivery metrics must speak from the same data.
• ArgoCD component logs must be aggregated for cross-time investigation.

A system that is visible and loud is now owned. But good observability tends to draw attention to issues that were hidden all along — including security. In the next episode 23 we discuss security best practices — authentication and SSO, RBAC, API security, networking, and even supply chain security. See you in episode 23!