Learn GitLab CI/CD - Directed Acyclic Graph (DAG) Pipelines with needs

#Introduction

In episode 8 we sped up the pipeline with caching. Now imagine another problem: your pipeline has five jobs in the build stage and three jobs in the test stage. The first test job only needs one build job, but it has to wait for all build jobs to finish — including build jobs completely unrelated to it. The bigger the team, the more jobs that don't depend on each other, and the more time wasted behind rigid stage rules.

This is the most common complaint in large pipelines: "the pipeline is slow even though the jobs are fast". This episode covers the solution — Directed Acyclic Graph (DAG) pipelines with the needs: keyword. You'll learn why traditional stage pipelines are slow, how DAG breaks that bottleneck, and when to use it wisely.

#Main Discussion

#The Problem with Linear Stage Pipelines

The default GitLab pipeline follows a strict rule: all jobs in stage N must finish before jobs in stage N+1 start. Imagine a restaurant that only serves set menus — every dish must finish cooking before any single one is delivered to the table. But in reality, a guest who only ordered soup shouldn't have to wait for the grilled steak.

stages:
  - build
  - test
  - deploy
 
build_a:
  stage: build
  script:
    - echo "build module A"
    - sleep 20
 
build_b:
  stage: build
  script:
    - echo "build module B"
    - sleep 20
 
test_a:
  stage: test
  script:
    - echo "test module A"

In this pipeline, test_a only depends on build_a, but it's forced to wait for build_b to finish — 20 seconds wasted. Now multiply that by dozens of jobs, and you understand why large pipelines feel like waiting in unnecessary queues.

#The Directed Acyclic Graph (DAG) Concept

DAG flips that paradigm: a job may start as soon as all its dependencies finish, without waiting for the whole stage to complete. The word "acyclic" enforces the rule: no dependency may form a cycle (A waits for B and B waits for A). The result is a directed graph where each job only waits for what it truly needs.

Back to the restaurant analogy: instead of serving set menus, DAG sends each dish as soon as it's ready — the guest who ordered soup never waits for the steak. GitLab implements this with the needs: keyword since GitLab 12.2, and the result can cut pipeline duration by 50 percent or more.

#needs: Basic Syntax

stages:
  - build
  - test
  - deploy
 
build_a:
  stage: build
  script:
    - echo "build module A"
    - sleep 20
 
build_b:
  stage: build
  script:
    - echo "build module B"
    - sleep 20
 
test_a:
  stage: test
  needs:
    - build_a
  script:
    - echo "test module A"

With needs: [build_a], the test_a job starts running immediately after build_a finishes — it doesn't matter whether build_b is still running. Two syntax forms can be used:

• Short list: needs: [build_a, build_b] — an array of job names.
• Detailed list: needs: - job: build_a artifacts: true — allows extra per-dependency settings.

#needs + Artifacts: true/false

Remember in episode 7 we covered artifact transfer between stages. With needs, that transfer becomes explicit and controlled. In the detailed form, you can choose whether a dependency brings its artifacts along:

deploy_backend:
  stage: deploy
  needs:
    - job: build_backend
      artifacts: true
    - job: test_report
      artifacts: false
  script:
    - ls backend-binary/
    - deploy.sh

artifacts: true downloads artifacts from build_backend — exactly the usual cross-stage transfer behavior. artifacts: false means the test_report job is only needed as a completion marker, without bringing its artifacts along. This pattern solves two problems at once: jobs run faster (no waiting for the whole stage) and only download relevant artifacts (saving bandwidth and storage).

#Time Comparison: Linear vs DAG

# LINEAR: test_a waits for build_b to finish (40 seconds + overhead)
build_a (20s) + build_b (20s) -> test_a (5s) = ~45 seconds
 
# DAG: test_a starts as soon as build_a finishes
build_a (20s) -> test_a (5s)   = ~25 seconds (build_b runs in parallel)
 
# Savings: 45 -> 25 seconds, ~45% saved

With 4 interdependent jobs, the savings are already noticeable. On enterprise pipelines with dozens of jobs — unit tests per module, linters, SAST, image builds per service — needs changes the total time from the sum of all stages to the length of the longest dependency path. That's the main reason many teams report cutting durations in half.

#When to Use needs Wisely

DAG isn't a total replacement for stages. Sometimes waiting for the whole stage is actually desirable:

• Deployment gates: deploying to production should wait for all tests to finish, not just some. Keep using a regular stage (without needs) here.
• Aggregate reports: a job that summarizes all other jobs' results (e.g. producing a final report) genuinely must wait for everything — no needs needed.
• Independent parallel jobs: unit tests per module that don't depend on each other are perfect candidates for needs — especially combined with parallel, which we'll cover in episode 10.

#Closing

In this episode 9, you've understood DAG pipelines:

• Linear stage pipelines force jobs to wait for the entire previous stage, even when they don't depend on each other — the biggest source of wasted time in large pipelines.
• DAG with needs: executes jobs the moment their dependencies finish, ignoring stage boundaries.
• needs: [job_a, job_b] lists dependencies explicitly; the detailed form enables per-dependency artifacts: true/false control.
• Savings of up to 50 percent or more on pipelines with many non-interdependent jobs.

The key point: needs makes a pipeline run as fast as its longest dependency path, not as fast as the sum of all jobs. Once your pipeline flows like a DAG, the single most impactful next step is multiplying the number of test jobs automatically. In episode 10 we'll cover matrix jobs & parallel execution — testing your app across many combinations of language versions and databases at once, and splitting large test suites across many parallel runners. See you in episode 10!