Learn GitOps with ArgoCD - Troubleshooting & Debugging

#Introduction

In episode 26 we made ArgoCD hard to knock over — multiple replicas, leader election, Redis with Sentinel, and chaos testing. But resilience against infrastructure failures doesn't cure bugs and configuration errors. In the real world, an SRE's daily work is spent answering questions more than shutting down servers: why is this application OutOfSync? Why did the sync fail mid-way? Why does the health check say Degraded when the pod is Running?

This episode is the debugging toolkit. The goal isn't memorizing forty commands, but learning a diagnostic way of thinking — reading signals from status, logs, events, and diffs, then narrowing down the root cause from Git, to ArgoCD, to Kubernetes. Because everything originates from Git, remember the GitOps Golden Rule: always start from the question "what's the difference between Git and the cluster?".

#Status: The Starting Point of Diagnosis

Before debugging, you must read the status quickly. The two key columns from argocd app get:

• Sync status — whether what's in the cluster matches Git (Synced or OutOfSync).
• Health status — whether the synced resources are running healthy (Healthy, Progressing, Degraded, Missing, Suspended).

The combination already tells a lot:

argocd app get api
Name:               api
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          api
URL:                http://localhost:8080/applications/api
Sync Status:        OutOfSync
Health Status:      Healthy

#Common Problems and Their Causes

#Sync Failures

A sync fails when a rendered manifest can't be applied. The most common causes: invalid YAML, missing destination namespace, CRDs not installed, or a resource already owned by another application. The first step is always looking at the operation details:

argocd app get api --show-operation
argocd app sync api --dry-run

--dry-run shows the sync result without actually applying — the safest way to test whether a manifest change will be accepted by the cluster.

#Health Check Failures

ArgoCD's health assessment checks whether resources are healthy by its rules — a Deployment needs availableReplicas equal to desiredReplicas, a Job needs to finish, etc. When Degraded, the cause is almost always at runtime, not in Git:

kubectl get pods -n api
kubectl describe pod api-5d4b6c7d9-8xk2m -n api
kubectl get events -n api --sort-by='.lastTimestamp'

Notice the patterns in the output: CrashLoopBackOff means the application keeps crashing, ImagePullBackOff means the image can't be pulled, Pending means a scheduling or resource problem. Each pattern leads to a different place.

#Authentication Issues

• SSO broken — Dex/OIDC login fails; check the argocd-server logs and the Dex configuration.
• Expired token — the CLI returns a 401 error; re-run argocd login <server>.
• RBAC denying — even though credentials are valid; this is an authorization problem, not authentication (covered in the RBAC section).

#Repository Access

The repo server can't pull from Git. First check the repo list and its credential status:

argocd repo list
argocd repo get https://github.com/timmu/manifests
argocd repo update https://github.com/timmu/manifests --username deploy-bot

Classic errors: authentication required, repository not found, or an unknown SSH host key. If the repo is private, make sure the credentials are stored as an encrypted Secret (episode 12) and ACL'd to the right project (episode 10).

#Resource Conflicts

Two applications managing the same resource is a design flaw, and ArgoCD will display it clearly. Find out who owns the resource via the resource tree and the tracking label:

kubectl get deployment api -n api -o jsonpath='{.metadata.labels.argocd\.argoproj\.io/tracking-id}'
argocd app get api --tree

#Basic Debugging Techniques

#Application Logs

argocd app logs streams the pod logs from all Deployment resources in the application — equivalent to kubectl logs but automatically filtered per application:

argocd app logs api --tail 100
argocd app logs api --since-time 2026-08-03T10:00:00Z
argocd app logs api --container main

#Diff Analysis

argocd app diff shows the exact differences between the desired manifest (from Git) and what lives in the cluster:

deployment.apps/api desired
  - spec.replicas: 2
+ spec.replicas: 3
  - resource.argoproj.io/requested-at: ...

A replicas difference often appears when HPA also changes a Deployment — not a bug, but worth noting so it isn't misunderstood as drift.

#Live Manifest Comparison

To see what actually exists in the cluster (including fields filled in by other controllers), compare the raw manifests:

kubectl get deployment api -n api -o yaml
kubectl get deployment api -n api -o jsonpath='{.metadata.ownerReferences}'
argocd app get api --resource deployment:apps/api

#Resource Tree

argocd app get <app> --tree shows the parent-child hierarchy (Deployment → ReplicaSet → Pod) complete with each node's health. This is the fastest way to find which node is Degraded in a complex application.

#Debugging with the ArgoCD CLI

Besides --show-operation, there are four commands you must master:

app manifests is very useful for verifying that the values ArgoCD uses are the expected ones — often the problem is wrong Helm values, not the chart itself. Combine it with --hard-refresh to bypass the cache:

argocd app get api --hard-refresh
argocd app manifests api | less
argocd app history api

#Advanced Debugging

#ArgoCD Component Logs

When the problem isn't in the application, look at the components:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100 -f
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server --tail=100 -f
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server --tail=100 -f

• Controller — compares Git and cluster; these logs show reconciliation failures.
• Repo server — clones and renders; repo access, cache, and manifest errors appear here.
• API server — logins, tokens, and RBAC denials.

#Network Debugging

The repo server needs outbound connections to Git; the controller needs connections to the target cluster. In a cluster locked down with network policies (episode 28), these connections often break silently:

kubectl run nettest --rm -it --restart=Never --image=curlimages/curl -- \
  curl -s https://github.com/team/manifests.git/info/refs
kubectl get networkpolicy -n argocd

#RBAC Debugging

It's not only "login failed" — often the user is valid but doesn't have permission. Verify identity and capabilities explicitly:

argocd account get-user-info
argocd rbac can run sync --application api
argocd rbac can get application --application api

argocd rbac can directly answers the question "can this user do X?" without trying — very useful when drafting project policies (episode 10).

#Performance Problems

We dissected performance problems in episode 25; what you need to remember here are the signals:

• Slow sync — check pending_request_total on the repo server; large repos or many applications without a cache cause repeated cloning.
• High memory — a controller close to OOM often shows stale application status. Raise the limits or use sharding.
• API latency — the UI and CLI feel slow; check Redis and the Ingress rate limiting.
• Failed repo clones — repo size, clone depth, or the Git provider's rate limit (GitHub limits anonymous access to 60 requests/hour). Enable credentials and --depth 1.

argocd_repoclientset_processors_run_count
argocd_repo_pending_request_total
argocd_app_reconcile_count

#Closing

This episode equipped you with a diagnostic toolkit: reading sync and health status as the starting point, recognizing the causes of sync failures, health checks, authentication, repo access, and resource conflicts, using logs, events, diffs, live manifests, and resource trees, mastering the CLI commands argocd app get --show-operation, logs, manifests, and diff, advanced debugging in component logs, networking, and RBAC, and recognizing performance problem signals.

The points you should take with you:

• Always start from the question: what's the difference between Git and the cluster?
• The sync + health status combination determines the investigation direction.
• --show-operation and app manifests verify what ArgoCD is actually running.
• The controller, repo server, and API server logs separate application problems from platform problems.
• argocd rbac can verifies permissions without trying.

These debugging skills become the foundation for running ArgoCD for many teams at once. In the next episode 28 we discuss multi-tenancy at scale — namespace, cluster, and hybrid tenancy models, isolation strategies, self-service patterns with ApplicationSet, and resource and cost management across teams. See you in episode 28!