Learn Helm Chart - Troubleshooting & Debugging

#Introduction

After episode 25, where we covered Helm's role in a GitOps architecture — Git as the source of truth, ArgoCD and Flux continuously reconciling the cluster — in this episode we come back down to terrain closer to daily work: what do you do when a release fails? GitOps is indeed automatic, but automation doesn't make failures disappear — it just makes them appear faster and more often, complete with stack traces we have to read.

The honest fact: most of an Helm engineer's time is spent not writing charts, but dissecting why something doesn't work. A deployment that seems successful while its Pod is in CrashLoopBackOff, an upgrade hanging in pending-upgrade status, a template erroring with a confusing message, or a release that "already exists" yet isn't visible in helm list. Without a systematic debugging method, all of this becomes guesswork that eats up hours of work.

In this episode we dissect the most common Helm failure patterns, learn standard and advanced debugging techniques, understand how to recover stuck or failed releases, then close with a complete walkthrough: a failing chart dissected step by step from the first error to the root cause.

#The Most Common Helm Failure Patterns

Before diving into techniques, it's important to recognize the families of problems. Each problem has a distinctive signature, and recognizing that signature narrows the diagnosis space drastically.

#Installation Failures

helm install fails early. Common causes: chart not found in the repo (wrong name/version), malformed YAML in the rendered manifest, or a resource already existing in the cluster with the same name. The symptom is visible directly in the helm install output, usually stopping before a release status appears.

#Upgrade Failures

helm upgrade fails mid-flight. This is more dangerous because the release can be left in a transitional status (pending-upgrade). Common causes: invalid new values (for example, a changed service name making selectors not match), resources that can't be updated (usually due to immutable fields like on PVCs or a Deployment selector), or changed dependencies. The key to understanding this: always remember that Helm 3 performs a three-way strategic merge patch — it compares the old state, the new values, and the live cluster state at once.

#Template Rendering Errors

The chart can't be rendered at all. Common causes: a misnamed template function, an undefined variable (misspellings like .Value vs .Values), a type mismatch when comparing values, or unbalanced if/range blocks. Go template error messages are confusing at first, but the template: mychart/templates/deployment.yaml:12:34: executing ... at <.Values.replicas>: can't evaluate field pattern tells you the file, line, column, and expression in question — that's already half the solution.

#Release Conflicts ("Already Exists")

The classic error: Error: rendered manifests contain a resource that already exists. This happens when a resource to be created already exists in the cluster but is not owned by that release — for example, installing the same chart twice without changing the release name, or a resource created manually then Helm trying to take it over. Helm 3 won't overwrite another party's resources.

#Hook Failures

Hooks (which we covered in episode 12) run as separate Jobs/Pods. If a hook Job fails — for example, a database migration error — the install/upgrade is considered failed even though the main resources were created successfully. The symptom: the release is in failed status with a hook mention, even though its Deployment itself is healthy.

#Forbidden Errors (RBAC)

Error: pods is forbidden: User "system:serviceaccount:default:default" cannot .... The ServiceAccount Helm uses (usually default in the namespace) doesn't have permission to create resources. This is a pure RBAC problem — either a wrong kubeconfig, or a chart demanding broader permissions. This message often appears in CI/CD when the credentials used differ from what you use on your laptop.

#Basic Debugging Techniques

Now let's build your debugging toolbox, from the fastest to the deepest.

#--dry-run and --debug

Both are your first friends when suspecting a rendering problem. helm template release chart renders the manifest to stdout without touching the cluster — perfect for inspecting the rendered YAML with your own eyes. Add --debug to install/upgrade to see the rendered manifest along with an old-vs-new state comparison, plus cluster connection details. The rule of thumb: if the rendered result is already wrong, the cluster will never save you.

helm template myapp ./my-chart --namespace default
 
# Set values from several files & see the full output
helm template myapp ./my-chart \
    -f values-staging.yaml \
    --debug

#helm get manifest and helm get values

When a release is already installed and having problems, don't guess — look at what actually exists. helm get manifest myapp shows all the manifests that release applied (the deployed version, not what's in Git). helm get values myapp shows the effective configuration values used, complete with labels indicating each value's origin (from values.yaml, override files, or --set). This is the instant answer to "why is it 1 replica when I set 3?" — most likely because there's a --set or a default value overriding it.

#kubectl describe and kubectl logs

Helm only delivers manifests; after that, Kubernetes executes them. When the resources exist but aren't healthy, switch to kubectl. kubectl describe pod <pod> dissects the pod's lifecycle details, including the Events at the bottom telling the failure narrative (ImagePullBackOff, CrashLoopBackOff, Probe failed). kubectl logs <pod> (with --previous for already-restarted containers) reveals the application error itself.

Dissect an unhealthy Pod

kubectl describe pod myapp-7f9d8c5b6f-abcde
 
kubectl logs myapp-7f9d8c5b6f-abcde --previous

#helm lint

Before blaming the cluster, make sure the chart itself is clean. helm lint ./my-chart statically checks chart structure, Chart.yaml validity, and template correctness. It's not a substitute for real testing — many bugs only appear when rendering with real values — but it quickly catches the most common error class.

#Walkthrough: Diagnosing a Failing Chart

Theory alone isn't enough. Let's dissect one real case from start to finish.

Symptom: your team runs helm upgrade --install api ./api-chart -f values-prod.yaml and gets an error:

Error: UPGRADE FAILED: template: api-chart/templates/deployment.yaml:34:21:
executing "api-chart/templates/deployment.yaml" at <.Values.image.repository>:
can't evaluate field image in type interface {}

Step 1 — read the error message carefully. This message gives us: the file (deployment.yaml), line 34 column 21, and the expression .Values.image.repository. Helm can't evaluate .Values.image — meaning the image field doesn't exist in the values used. Don't immediately assume the chart is broken; more often the values aren't set.

Step 2 — check the effective values. Run helm template api ./api-chart -f values-prod.yaml --debug to see the render output, or helm get values api if the release already exists. Maybe the values-prod.yaml file places the image configuration under api::

api:
  image:
    repository: ghcr.io/company/api
    tag: "2.1.0"

While the template reads .Values.image.repository (without the api: level). Template and values don't match.

Step 3 — verify with a simple template. To confirm, render the chart with explicit --set: helm template api ./api-chart --set image.repository=ghcr.io/company/api --set image.tag=2.1.0. If the render succeeds, the root cause is confirmed: a values structure mismatch, not a broken template.

Step 4 — fix it in the right place. Since this is GitOps, the fix goes in Git: adjust the values file, or — if you control the chart — move the template's read to .Values.api.image. Commit, and let the ArgoCD/Flux controller apply it.

Step 5 — verify the result. helm status api shows deployed status, kubectl get pods -l app.kubernetes.io/name=api shows the Pod Running, and kubectl logs is clean. Done.

#Recovering Stuck or Failed Releases

Failure doesn't always end with fixing values. Sometimes the release itself is stuck. This is the area that most often panics engineers — and it's actually the easiest to handle if you understand release statuses.

#Stuck Releases

A release can hang in pending-install, pending-upgrade, or pending-rollback — usually because an operation failed mid-flight (hook timeout, a Job not finishing) and Helm never got to finalize the status. If a release has been "pending" for hours, Helm considers it corrupted. The classic solution:

# View the status & release history
helm status api
helm history api
 
# Try rolling back to the last known good version
helm rollback api <last-healthy-revision>
 
# If rollback is also stuck, just release it
helm uninstall api --keep-history

#Failed Releases

helm list --all will show releases in failed status. Don't uninstall it before examining why it failed — a failed status often gives richer information than deployed. Use helm get manifest, helm history, and helm status to dissect it. If the cause is clear and fixable, repeat the upgrade with correct values. Helm will try to continue from the last point, and most failed cases can be saved without uninstalling.

#The Rollback Procedure

Rollback is the fastest and safest recovery procedure. helm rollback api 12 restores the release to revision 12. Keep in mind: rollback is not a perfect restore — it's a new upgrade operation applying the old revision's manifests against the current live state. That's why rollback can also fail (for example, if a resource was manually deleted). Use helm history api to pick the right revision, and don't forget that --wait --timeout can be used on rollback so it doesn't hang.

#Manual Cleanup

Sometimes a release is too broken to save. The safe sequence:

1. helm uninstall api --keep-history — delete resources but keep the history for audit.
2. If the uninstall itself is stuck, manually delete leftover resources: kubectl delete for the remaining resources (Helm marks them with the app.kubernetes.io/managed-by: Helm and helm.sh/release-name labels).
3. Delete the release state storage secret (details in the advanced debugging section).
4. Reinstall from scratch with the correct configuration.

#Performance Problems

Debugging isn't always about errors — sometimes it's about slowness. The three most common causes:

#Large Values Files and Complex Templates

A chart with hundreds of template lines and thousands of values lines slows down rendering and upgrades. Every template is re-rendered on each operation, and the lookup function (covered in episode 9) makes a Kubernetes API call — inside a loop, one operation can trigger dozens of calls. The rule of thumb: don't call lookup inside a repeating range or include; cache the result in a variable.

#Many Dependencies

Each subchart dependency adds rendering load. A chart with 20+ dependencies will be far slower than a single chart. Consider splitting them into smaller releases, or use library charts (episode 19) to share logic without carrying large subcharts.

#Slow Upgrades

An upgrade taking 10 minutes is usually waiting on something — a long hook Job, or a resource that never becomes ready. Use a reasonable --timeout and --wait only if you genuinely want to wait for readiness. In very dynamic environments, consider --wait=false and let the controller (or monitoring) ensure the application is ready.

#Advanced Debugging

When the basic techniques aren't enough, these are the weapons that separate a senior engineer:

#Local Template Rendering

helm template api ./api-chart -f values-prod.yaml > /tmp/manifest.yaml then inspect the file carefully. Save it as a diff baseline: render with old vs new values, then diff them. This is the fastest way to see "what actually changed" behind a failed upgrade.

#Inspecting Release Secrets

Helm 3's release state is stored as Secrets (not ConfigMaps like Helm 2). Every release has a series of secrets named sh.helm.release.v1.<release>.v<revision> — one per revision. Digging into them is useful when a release is corrupt or the history can't be read:

kubectl get secret -n api sh.helm.release.v1.api.v12 -o yaml
 
# Decode the release data
kubectl get secret -n api sh.helm.release.v1.api.v12 \
    -o jsonpath='{.data.release}' | base64 -d | base64 -d

The decoded value is binary protobuf — not easy to read, but useful for verifying metadata (for example, the chart version and revision actually stored). Deleting this secret (along with --keep-history) is the last-resort way to "clean up" a release that can't be uninstalled.

#Checking Resource Status

The combination of helm status api and kubectl get all -n api shows two layers of truth: what the release declares vs what exists in the cluster. The difference between them is drift — the main source of the "why isn't the app updated even though the chart was upgraded" mystery.

#Analyzing Event Logs

kubectl get events --sort-by=.lastTimestamp is the chronicle of recent events in the namespace. When a resource fails to be created or a Pod can't be scheduled (insufficient resources, image pull failure, quota reached), the event log tells that story more honestly than the resource status itself.

View all events in the namespace

kubectl get events --sort-by=.lastTimestamp -n api
 
# Filter events related to a specific pod
kubectl describe pod api-7f9d8c5b6f-abcde | grep -A10 Events

#Conclusion

In this episode 26 we understood that debugging Helm starts with recognizing problem families — failed installations, failed upgrades, template errors, release conflicts, hook failures, and RBAC — then applying the basic techniques in sequence: --dry-run/helm template to inspect rendering, helm get manifest/helm get values to see the actual state, and kubectl describe/kubectl logs to dive into the Kubernetes side. We also dissected recovery: rollback as the fastest procedure, uninstall with --keep-history, and manual cleanup as the last resort, plus advanced techniques like inspecting release secrets and analyzing event logs.

The core takeaways:

• Read the error message carefully first — the file:line:column pattern in template errors already points to half the solution.
• Local rendering (helm template) is always faster and safer than trial-and-error directly in the cluster.
• A failed release doesn't always have to be uninstalled — check the history and try rollback first.
• When resources are healthy but not as expected, look for drift between the release manifest and the live state.
• Failure is a process, not a single event — document every case so the team doesn't repeat the same diagnosis.

These diagnosis skills will be increasingly tested when you face old chart versions or large migrations. In the next episode, episode 27, we cover chart migration and upgrade: migrating from Helm 2 to Helm 3 with the helm-2to3 plugin, upgrading charts between major versions with breaking changes, application upgrade strategies (rolling update, blue-green, canary), and facing Kubernetes API deprecations. See you in episode 27!