Learn GitLab CI/CD - Troubleshooting, Debugging & Monitoring Pipelines

#Introduction

In episode 18 we got to know Auto DevOps and components. But what happens when a seemingly perfect pipeline suddenly fails in the middle of the night? Staring at red logs without knowing where to start is a moment every engineer knows. This episode covers GitLab's debugging arsenal — from highly detailed shell traces to an interactive terminal inside the runner — then closes with monitoring runner health via Prometheus and Grafana.

#Reading Pipeline Logs Correctly

First, resist the reflex to hand the entire log to another team. Every failed job provides information in three places: the failing script line (usually marked in the log), the exit code status, and the artifacts produced. Start from the first failed stage, not the last — failures often cascade.

Common causes of failure:

#Debug Trace with CI_DEBUG_TRACE

When the error isn't clear — for example an empty variable that should be filled — enable the shell trace. The CI_DEBUG_TRACE variable makes GitLab print every shell command before it executes, complete with its variable values:

variables:
  CI_DEBUG_TRACE: "true"

This is equivalent to running bash -x on every script. You'll see variable expansion live — for example $CI_COMMIT_SHA turning into the actual commit hash — so you immediately see whether a variable is empty, mis-expanded, or contains spaces that break the command.

The more recommended way is enabling the debug trace per run only, via the pipeline variables in the UI, without changing the file:

Pipeline -> Run pipeline -> Variables -> CI_DEBUG_TRACE = "true"

#Web Terminal for Interactive Debugging

Sometimes a static trace isn't enough — we need to type commands directly inside the failing job's environment. GitLab provides the Web Terminal: while a job is running (or in manual mode), you can open an interactive terminal from the job page in GitLab Premium.

Working in that terminal is like sitting in front of the same machine: trying ls to inspect the directory structure, rerunning the failing command line by line, and observing the environment variables that actually exist:

pwd
ls -la
printenv CI_PROJECT_DIR
printenv CI_JOB_TOKEN | head -c 8

This terminal is only available while the job runs and uses the same permissions as the job — so you can test fixes in conditions exactly matching the original failure, instead of guessing from your laptop.

#Monitoring Runner Health

Debugging fixes one failure; monitoring prevents recurring failures. GitLab Runner exports its own metrics in Prometheus format, just enabled in config.toml:

[[runners]]
  name = "prod-runner"
  url = "https://gitlab.example.com"
  token = "xxxxxxxx"
 
  [runners.metrics]
    listen_address = ":9252"

Metrics you can collect include gitlab_runner_jobs_total, gitlab_runner_job_duration_seconds, and gitlab_runner_process_runner_version. To monitor them, point Prometheus at that endpoint:

scrape_configs:
  - job_name: gitlab-runner
    static_configs:
      - targets: ["runner01.example.com:9252"]

Grafana then turns those metrics into a dashboard: runner CPU and RAM, job queue, and the success-failure ratio. With this dashboard, runner problems are visible before users report them — and troubleshooting no longer waits for midnight.

One of the most important panels is queue duration: how long jobs queue before a runner is free. In Prometheus, this metric is a histogram:

histogram_quantile(
  0.95,
  sum by (le) (rate(gitlab_runner_job_queue_duration_seconds_bucket[5m]))
)

A value consistently above tens of seconds signals the runner is short on capacity — time to add a new runner, not blame the pipeline.

#Taming Recurring Job Failures

Not every failure needs manual handling. GitLab provides retry for transient failures (runner crash, network hiccup) and timeout so a hanging job doesn't drain the queue forever:

flaky_build:
  script:
    - npm ci
    - npm run build
  retry:
    max: 2
    when: runner_system_failure
  timeout: 30m

Never use retry to cover up systemic failures — if npm ci fails three times in a row, adding more retries just delays the diagnosis. Use retry for clear reasons: runner_system_failure and stuck_or_timeout_failure are the safest candidates.

#Closing

• The CI_DEBUG_TRACE debug trace prints the entire shell trace to find wrong variable expansion.
• The Web Terminal gives an interactive terminal inside the runner for debugging real conditions.
• Runner metrics are enabled via listen_address in config.toml.
• Prometheus scrapes runner metrics, Grafana turns them into dashboards.
• Queue duration and job failure are the two most important scaling signals.

In episode 20 — the final episode of this series — we bring it all together: a complete production-grade pipeline case study from the first commit to team alerts. See you there!