Learn Docker - Modern Build Engine: Docker Buildx & BuildKit

#Introduction

After deploying services to Swarm, releasing updates without downtime, and storing secrets with secrets in episode 17 — you might be starting to feel the injustice: the cluster is modern, but the image building machine is still ancient. Almost every docker build command we've used so far runs on the legacy builder — an architecture that's old and now officially deprecated. In this episode we replace it: getting to know BuildKit, the next-generation build engine, and Buildx, the CLI that drives it.

Why is this important, and not just a "technical upgrade"? Because in the real world, build time is money: every slow build minute is a minute not spent shipping fixes, and in CI/CD (episode 19) slow builds slow down the entire pipeline. BuildKit delivers three things you'll feel immediately: (1) parallel execution — independent build steps are done together, not sequentially; (2) advanced caching — including cache that can be carried to other machines; (3) build secrets — secret data used during the build that is never embedded in image layers, plus the foundation for multi-platform images — one tag, many architectures, from one command.

Imagine the legacy builder as a single-line production line in a factory: every component must pass through the same station, one at a time, and every reprocess starts from zero. BuildKit is a factory with many parallel lines, a warehouse for intermediate results, and a separate vault for secret materials. Which factory do you want when orders start piling up? In this episode we'll build it.

#Main Discussion

#BuildKit and Buildx: Two Parts of One Machine

These two terms are often mixed up, so let's separate them first:

• BuildKit is the engine — the build daemon/engine that executes Dockerfile instructions with a DAG (directed acyclic graph) architecture. Because build steps are represented as a graph, BuildKit knows which can be done in parallel and which must wait.
• Buildx is the steering wheel — a CLI plugin (docker buildx) that controls BuildKit. It provides features that don't exist in classic docker build: multi-builder, multi-platform, and explicit cache management.

Since Docker 23.0, docker build uses BuildKit via buildx by default. You can verify:

docker buildx version
docker buildx ls

NAME/NODE        DRIVER      ENDPOINT                  STATUS   BUILDKIT   PLATFORMS
default*         docker      default                   running  v0.15.1    linux/amd64, linux/arm64

The default builder uses the docker driver — meaning BuildKit runs inside the same Docker daemon. That's enough for most needs. Later we'll add a dedicated builder for multi-platform.

#Why the Legacy Builder Died — and What You Get

The legacy builder executes Dockerfiles linearly: instruction by instruction, each RUN runs in a temporary container, and its result is committed as a layer. It has two structural weaknesses: it can't work on two independent steps at once, and its cache is local (can't be carried to other machines). BuildKit fixes this with four features you'll use every day:

1. Parallel execution. Independent steps (e.g. different RUN commands, or two multi-stage build stages that don't depend on each other) are done simultaneously. A multi-stage build that took 2 minutes can drop to 1 minute without changing a single line.
2. Advanced caching. BuildKit caches each step's result with a hash that considers context, env, and arguments — not just line order. Intermediate results can be exported to a registry (--cache-from/--cache-to) so the next build on another machine jumps to unchanged steps.
3. Build secrets & mounts. The RUN --mount=... instruction gives temporary access to files/secrets without ever writing them to an image layer — ending the era of secrets leaking via ARG/ENV (which we covered in episodes 6 and 26).
4. Frontend support. The Dockerfile frontend can be selected (docker/dockerfile:1 = dockerfile.v1, :0.4 = experimental). BuildKit separates the "engine" from the "language" — which is why new Dockerfile features (like COPY --link) can ship without a daemon upgrade.

#Build Secrets: Using Secrets Without Leaving a Trace

This is the feature most often misunderstood, so we start here. The classic need: building an image that must download dependencies from a private registry (npm, Maven, Go proxy) requiring a token. The naive way — putting the token in ARG or ENV — plants it in an image layer. Anyone with the image can read the token with docker history. BuildKit solves this with a secret mount: the token is only available during the RUN execution, and is gone the moment the command finishes.

FROM node:20-alpine
 
WORKDIR /app
COPY package*.json ./
 
RUN --mount=type=secret,id=npm_token \
    NODE_AUTH_TOKEN="$(cat /run/secrets/npm_token)" \
    npm ci --omit=dev
 
COPY . .
 
CMD ["node", "server.js"]

docker build \
  --secret id=npm_token,src=./.npm-token \
  -t my-app:1.0 .

The breakdown: RUN --mount=type=secret,id=npm_token makes the file /run/secrets/npm_token available only inside the container for that RUN instruction. When it's done, the file doesn't exist — and because BuildKit never commits secret contents to a layer, docker history is clean. This isn't just best practice; it's the difference between leaking and not leaking. You can prove it: docker history my-app:1.0 won't show the token.

#Cache Mounts: Skipping Repeated Downloads

The second classic problem: every time the code changes, the entire npm ci/apt-get install/go mod download runs again — even though the dependencies didn't change. Cache mounts separate frequently changing data (downloaded dependency results) from image layers, so downloads survive between builds without bloating layers:

FROM node:20-alpine AS node-stage
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev
 
FROM golang:1.22-alpine AS go-stage
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
 
FROM ubuntu:24.04 AS apt-stage
RUN rm -f /etc/apt/apt.conf.d/docker-clean; \
    --mount=type=cache,target=/var/cache/apt \
    apt-get update && apt-get install -y curl

Note the pattern: RUN --mount=type=cache,target=/path makes the directory /path persist between builds on the same machine (BuildKit's cache), but its contents never become part of an image layer. The result: the second and subsequent builds skip downloads without enlarging the final image. It's a minutes-saving win that feels real in CI.

#Bind Mounts Between Stages: Importing a Build Stage

To take results from another stage while the build is running — for example using a binary from a builder stage — COPY --from (episode 7) is already familiar. BuildKit gives a more economical alternative for some cases: RUN --mount=type=bind,from=<stage>, which mounts a stage as a source without copying everything:

FROM golang:1.22-alpine AS builder
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app .
 
FROM alpine:3.20
RUN apk add --no-cache ca-certificates
COPY --from=builder --link /out/app /usr/local/bin/app
ENTRYPOINT ["app"]

The combination of --link on COPY (a BuildKit feature) copies only changed files between builds — unchanged layers are reused, speeding up the "change code → build" cycle.

#Multi-Platform Builds: One Tag for All Architectures

This is the feature that means you don't need 10 machines for 10 architectures. With buildx, one command produces linux/amd64, linux/arm64, and more — combined in one manifest list tag. When docker pull runs on an arm64 machine, Docker automatically grabs the right variant.

First step: create a builder instance that supports multi-platform. The built-in docker driver can only build for the host platform. A dedicated builder uses the docker-container driver — BuildKit runs in a separate container, able to build for any platform (with QEMU emulation when the host has no native node):

docker buildx create \
  --name multiarch \
  --driver docker-container \
  --bootstrap \
  --use
docker buildx ls

--bootstrap immediately pulls the BuildKit container image, and --use makes it the active builder. Now build a Go image for two architectures at once — this is the end-to-end example of this entire episode:

FROM golang:1.22-alpine AS builder
WORKDIR /src
RUN apk add --no-cache ca-certificates
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/app .
 
FROM alpine:3.20
COPY --from=builder --link /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder --link /out/app /usr/local/bin/app
ENTRYPOINT ["app"]

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ghcr.io/arman/my-go-app:1.0.0 \
  --provenance true \
  --sbom true \
  --push .

This is the core of this episode in one command: two architectures, one tag, plus two metadata pieces that come free from BuildKit — --provenance (the build trace: who built it, from which code, with which machine — the supply chain foundation) and --sbom (Software Bill of Materials, which we learned in episode 14). Both are pushed to the registry alongside the image, with no extra tools.

To verify that one tag really contains many architectures, docker buildx imagetools reads the manifest list in the registry without pulling the image:

docker buildx imagetools inspect ghcr.io/arman/my-go-app:1.0.0

The output will list the platforms along with each variant's digest — proof that the 1.0.0 tag serves linux/amd64 and linux/arm64 at once. It's also a useful tool to make sure an image claimed to be multi-arch really is before use, and to read the provenance/SBOM accompanying the image from episode 14.

#Pitfalls Most Often Encountered

1. Buildx flags used in legacy docker build. Some buildx-exclusive features (--secret, --platform, --push, --cache-to) don't exist in classic docker build. When using a Dockerfile with --mount, make sure you invoke it via docker buildx build or docker build (which is now BuildKit) — not an old daemon.

2. QEMU emulation speed for arm64. Building linux/arm64 on an amd64 host uses emulation — it can be 2-5x slower than a native build. The solution: use a native arm64 CI runner (e.g. GitHub Actions ARM), or build each architecture on its own native node and then merge the manifests. For occasional builds, emulation is still very practical.

3. --load multi-platform fails. As above — use --push for multiple platforms. --load is only for a single platform into the local daemon.

4. Secrets leaking via ARG/ENV. --mount=type=secret solves this, but only if you don't also write the value into ENV. Don't combine secrets with ENV — once it enters env, it settles in a layer.

5. Cache not reused on other machines. BuildKit cache is local by default. For CI or builds on many machines, export the cache to a registry (--cache-to type=registry) — we use this in episode 19.

#Conclusion

In this episode 18 we replaced the build machine: distinguishing BuildKit (the engine) and Buildx (the CLI), understanding why the legacy builder died and which features we gained (parallel execution, advanced caching, the dockerfile.v1 frontend), using build secrets (RUN --mount=type=secret) that never leave a trace in image layers, saving time with cache mounts for npm/apt/Go, importing stages with --mount=type=bind,from=... and COPY --link, and producing multi-platform images (linux/amd64,linux/arm64) in a single command with built-in --provenance and --sbom.

Core takeaways:

• BuildKit = the modern build engine (default); Buildx = the CLI adding multi-builder, multi-platform, and cache export.
• Build-time secrets → --mount=type=secret; repeatedly downloaded data → --mount=type=cache.
• One tag, many architectures = docker buildx build --platform ... --push .; remember multi-platform needs --push.
• --provenance and --sbom come free for supply chain (episode 14).
• arm64 emulation on an amd64 host is slow — prioritize native nodes in CI for large builds.

You now have images built quickly, safe from leaking secrets, and ready for every architecture. The next question: who builds them, and when? In the next episode, episode 19, we'll automate all of it — Integrating Docker in CI/CD Pipelines — connecting build, test, scan, and push to GitHub Actions, optimizing pipeline caching, and understanding DinD vs DooD so you don't build a security labyrinth on your runners. See you in episode 19!