Learn Docker - Advanced Docker Compose Features

#Introduction

After turning the entire stack — web, API, PostgreSQL, Redis — into one declarative compose.yaml file in episode 10, in this episode we face the next limitation: that file is still static. Image tags are written by hand, the database password is baked into the file, and there's no way to use different configuration for a development machine vs a production server. In the real world, this difference always exists: in development we want bind mounts so code can hot-reload, publish debug ports, and turn on helper tools like phpMyAdmin; in production we want specific image versions, resource limits, and not a single dev tool running.

Mastering Compose's advanced features isn't about memorizing syntax — it's about building one source of truth for every environment. Good teams don't copy compose.yaml and hand-tweak it on every machine; they write one composable configuration base: a base for all, an override for development, extra files for production, and variables for what truly differs between environments. The result: no more configuration that lives "far away" from the code, and no more "but it works on my laptop!" that can't be explained.

In this episode we'll dissect the three places to store variables (.env, env_file, environment), master ${VAR} interpolation, separate development and production with multi-file compose, turn optional services on and off with profiles, kill duplication with extends and YAML anchors, and close with docker compose config — the mirror that shows the final configuration before a single container is created.

#Main Discussion

#.env, env_file, and environment: Three Different Boxes

The biggest confusion in the Compose ecosystem is rooted in three things with similar names but different roles. Let's separate them firmly:

The mental model that saves you: .env is read by Compose; env_file and environment are read by the container. .env, by itself, never produces variables inside the container — it only supplies material for interpolation. Conversely, env_file and environment produce variables you'll actually see when you run env inside the container.

A kitchen analogy: .env is the recipe notes read by the cook (Compose) before starting; env_file and environment are the ingredients placed on the counter (container). The recipe notes may list ingredients, but the ingredients don't automatically move to the counter just because they're written in the notes — unless the cook deliberately puts them there.

#${VAR} Interpolation: Dynamic Compose Files

Interpolation is Compose's ability to replace ${VAR} inside a Compose file with a value from the shell environment or a .env file. That's what lets one file adapt. Notice how the same value can be "played" in different places:

services:
  api:
    image: my-app:${TAG:-latest}
    ports:
      - "${HOST_PORT:-3000}:3000"
    environment:
      NODE_ENV: ${NODE_ENV:-development}
      DATABASE_URL: postgres://${DB_USER}:${DB_PASS}@postgres:5432/app

TAG=1.2.0
HOST_PORT=8080
NODE_ENV=production
DB_USER=app
DB_PASS=super-secret

When docker compose up runs, Compose replaces ${TAG} with 1.2.0, ${HOST_PORT} with 8080, and so on. You can inspect the final result without running anything:

docker compose config

services:
  api:
    environment:
      DATABASE_URL: postgres://app:super-secret@postgres:5432/app
      NODE_ENV: production
    image: my-app:1.2.0
    networks:
      default: null
    ports:
      - mode: ingress
        published: 8080
        target: 3000

Note: the .env above does contain DB_PASS, and its value doesn't just appear in the container on its own — it appears because compose.yaml explicitly mentions ${DB_PASS} in environment. This nuance is what separates people who understand Compose from those who guess.

#Interpolation Syntax: Defaults and Mandatory Values

Interpolation isn't just ${VAR}. There are four forms, each answering a different question:

• ${VAR} — direct substitution; if there's no value, it becomes an empty string.
• ${VAR:-default} — use default if VAR is empty or unset. Most often used for optional values.
• ${VAR-default} — use default only if VAR is unset; an empty string is preserved.
• ${VAR:?error message} — mandatory. If VAR is empty or unset, Compose stops and shows the error message. This is the best guard for critical values (passwords, versions).

services:
  api:
    image: my-app:${APP_VERSION:?APP_VERSION wajib disetel}
    environment:
      LOG_LEVEL: ${LOG_LEVEL:-info}
      SECRET_KEY: ${SECRET_KEY:?Secret key dibutuhkan untuk produksi}

A strongly recommended habit: for critical variables, don't give a silent default — use ${VAR:?message}. A loud failure is far better than succeeding with an empty value discovered at midnight. For convenience variables (log level, ports), a silent default with :- is the right choice.

#Priority Order: Who Wins When Values Collide?

When the same variable is defined in many places, there's a set hierarchy. For interpolation sources (the values Compose uses to replace ${VAR}), the order from highest:

1. Your shell environment (export TAG=...).
2. Values from the file pointed to by --env-file (if used).
3. Values from .env in the project directory (default, if --env-file isn't used).
4. The :- default inside the Compose file.

For variables that ultimately enter the container, the priority (highest to lowest):

1. docker compose run -e KEY=value (CLI).
2. environment or env_file whose values result from shell/.env interpolation.
3. environment in the Compose file.
4. env_file in the Compose file.
5. The ENV instruction inside the image (Dockerfile).

Two practical consequences: (1) you can override .env values from the shell without changing the file — that's what CI/CD uses to inject secrets; (2) environment always beats env_file if names collide — so don't define the same variable in both, since it only causes confusion.

#Multi-File Compose: compose.override.yaml

Now we move to environment separation. Compose supports merging multiple files. By default, when docker compose up runs in a directory, Compose reads two files: compose.yaml (the base) and — if present — compose.override.yaml (merged automatically). The override file usually contains adjustments for local development that shouldn't pollute the base file:

services:
  api:
    build:
      context: ./api
    volumes:
      - ./api:/app
      - /app/node_modules
    ports:
      - "9229:9229"
    environment:
      NODE_ENV: development
      DEBUG: "*"

The override file above adds a bind mount for hot-reload, publishes a Node debugger port (9229), and forces development mode — all without touching compose.yaml. Teams can commit compose.yaml as a shared contract, while each engineer uses their own personal override. (An override file usually goes into .gitignore if it contains local paths.)

#Production Mode: compose.prod.yaml with -f

For production, the common pattern is a third file merged explicitly with the -f flag:

docker compose -f compose.yaml -f compose.prod.yaml up -d

services:
  api:
    image: ghcr.io/my-org/my-api:${APP_VERSION}
    deploy:
      resources:
        limits:
          cpus: "0.50"
          memory: 512M
    restart: unless-stopped
  postgres:
    volumes:
      - pgdata-prod:/var/lib/postgresql/data
volumes:
  pgdata-prod:

An important point that often surprises people: as soon as you use -f, Compose stops looking for the default files and the automatic override. docker compose -f compose.yaml -f compose.prod.yaml up -d uses exactly those two files — compose.override.yaml is ignored. That's actually good: production shouldn't accidentally receive a local engineer's development tweaks. Alternatively, the file list can be set via the COMPOSE_FILE environment variable (e.g. in .bashrc or CI).

A healthy rule of thumb: compose.yaml holds what's genuinely common; compose.override.yaml holds development tweaks; the production file is selected explicitly with -f. Never make production depend on the existence of files you don't name.

#Merge Rules Between Files

So multi-file doesn't become a time bomb, you must know exactly how two files are merged. The rules are consistent and can be summarized:

• Scalar (image, restart, mem_limit) — the later-named file replaces it entirely.
• Map (environment, labels, build.args) — merged per key: same keys taken from the later file, different keys kept from the earlier file.
• Sequence/list — appended (combined, not replaced). Exception for ports, volumes, secrets, configs, which have uniqueness rules: entries are merged by their unique key (e.g. mount path for volumes), so the same path is replaced and new ones are added.
• Special exceptions: command, entrypoint, and healthcheck.test are always replaced, never appended — you don't want two commands stacking up.

The consequence that trips people most: because ports is appended, to replace a port in an override you must rewrite the old port — or use the !override tag for a total replacement:

services:
  api:
    ports: !override
      - "8443:443"

Without !override, 8443:443 will add to the ports already in the base file — not replace them. If you use !override, the base file needs to mention it to be clear: this is a relatively new Compose feature, so check your Compose version first (use docker compose version).

#Profiles: Turning Optional Services On and Off

Some services aren't always wanted. Development helper tools like phpMyAdmin, Mailhog, or debuggers aren't part of production, but they're very useful locally. Instead of deleting and rewriting, use profiles:

services:
  api:
    image: my-api:1.0
    ports:
      - "3000:3000"
 
  db-admin:
    image: phpmyadmin:5
    profiles: [dev-tools]
    ports:
      - "8081:80"
    depends_on:
      - db
 
  mailhog:
    image: mailhog/mailhog
    profiles: [dev-tools]
    ports:
      - "8025:8025"

The rule is simple: a service without profiles always starts; a service with profiles only starts when that profile is activated:

docker compose up -d
docker compose --profile dev-tools up -d
COMPOSE_PROFILES=dev-tools docker compose up -d

• docker compose up -d — only api (and db behind the scenes) starts.
• docker compose --profile dev-tools up -d — profiled services start too.
• COMPOSE_PROFILES=dev-tools — the same approach via an environment variable, suitable for CI.
• docker compose --profile "*" up — activates all profiles (useful for testing all services).

This is a very clean pattern: one Compose file holds the entire architecture, and profiles act as the "switch" for whether supporting tools start too. Don't be confused when a service "doesn't appear" in ps — first check whether it has a profile that isn't activated.

#extends and YAML Anchors: Killing Duplication

When several services share configuration, two tools avoid copy-paste: extends and YAML anchors. Both solve similar problems with different approaches.

extends — inherits an entire service definition from another service (from the same file or a separate one):

services:
  base:
    image: node:20-alpine
    environment:
      NODE_ENV: production
    restart: unless-stopped
 
  api:
    extends: base
    command: ["node", "server.js"]
    ports:
      - "3000:3000"
 
  worker:
    extends: base
    command: ["node", "worker.js"]

api and worker inherit the image, environment, and restart from base — then add their differing parts. The result: an environment change only needs to be made once. Note: base itself is not run unless it's explicitly defined as a service (which is exactly the behavior we want here).

YAML anchors — a pure YAML feature for sharing blocks within one file. Anchors are marked with &name (definition), used with *name (value), and can be merged into a map with <<:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
 
services:
  api:
    image: my-api:1.0
    logging: *default-logging
  worker:
    image: my-worker:1.0
    logging: *default-logging

The x-logging block is an extension field (the x- prefix is recommended and ignored by Compose) defined once, then *default-logging attaches it to several services. With <<, you can merge an entire map into a service — the fragment pattern:

x-base: &base
  restart: unless-stopped
  networks: [backend]
 
services:
  api:
    <<: *base
    image: my-api:1.0
  worker:
    <<: *base
    image: my-worker:1.0

When to choose which? extends is better for inheriting an entire service (especially across files in a monorepo), because it understands Compose's service structure. Anchors are better for small config blocks shared within one file (logging, restart, networks). Rule of thumb: start with anchors for small blocks, move up to extends once the need crosses files.

#docker compose config: The Mirror of the Final Configuration

All the features above — interpolation, multi-file, profiles — are magically combined by Compose before containers are created. So how do you verify the result is correct before executing anything? Use docker compose config:

docker compose -f compose.yaml -f compose.prod.yaml config
docker compose config --services
docker compose config --profiles
docker compose config --environment

• docker compose config — validates the files, performs interpolation and merging, then prints the final configuration. If there's a YAML error, a missing key, or an invalid value, this command fails with an error message — without touching a single container.
• --services — the list of services that will actually be created (after considering profiles).
• --profiles — the list of available profiles.
• --environment — the variables and values used for interpolation.

This is the best "dry run" Compose has. Get used to this flow in CI before deploying: run docker compose config as a validation step. If the Compose file is broken, you find out in seconds in the pipeline — not on the production server when up fails at midnight.

#Pitfalls That Most Often Mislead

1. Thinking .env automatically enters the container. .env is only interpolation material. To get variables into the container, use env_file or environment that explicitly mentions ${VAR}.

2. Defining the same variable in environment and env_file. environment always wins — this duplication only confuses the team about which configuration truly applies.

3. Using -f but thinking the automatic override still loads. As soon as -f is used, compose.override.yaml is no longer read. That's a feature, not a bug — but only useful if you know it.

4. Thinking lists merge by "replacement". ports, volumes, and other lists are appended unless you use !override. Accidentally, production ports can "stack up" with development ports. Use docker compose config to check the end result.

5. Storing secrets in .env and committing it. .env is easily scanned by security tools and often leaks. .env is for configuration; for real secrets use secret management (Swarm secrets later in episode 17) or inject from CI/CD — and make sure .env is in .gitignore.

6. A profiled service "disappearing". It's not gone — its profile just isn't active. Check docker compose config --profiles and --services before guessing.

7. An unset ${VAR} silently becoming an empty string. For critical values, use ${VAR:?message} so failures are loud.

#Conclusion

In this episode 11 we've freed Compose from a static file: understanding the firm difference between .env (read by Compose for interpolation), env_file, and environment (read by the container) plus their priority hierarchy, mastering ${VAR} interpolation with the :- default form and the mandatory :? form, separating development vs production with multi-file compose (compose.override.yaml automatic vs compose.prod.yaml with -f), understanding merge rules (scalars replaced, maps merged, lists appended, and !override for total replacement), turning optional services on with profiles, killing duplication with extends and YAML anchors (&, *, <<), and validating everything with docker compose config.

Core takeaways:

• .env for interpolated configuration; env_file/environment for variables entering the container.
• Shell env > --env-file > .env > default — that's what CI uses to override.
• One base compose.yaml, development tweaks in compose.override.yaml, the production file called explicitly with -f.
• profiles are the switch for optional services — services without a profile always start.
• docker compose config before up — the dry run that saves production.

Now you can manage a stack across many environments from one codebase. But there's one question we haven't answered: where do those images come from? So far we've pulled postgres:16-alpine, nginx, and their friends from a registry without ever asking how the registry works — and how you distribute images you've built yourself. In the next episode, episode 12, we'll dive into Docker Registry & Image Distribution: image naming conventions, docker login, docker tag, push and pull, immutable digests, and building your own private registry with TLS and basic auth. See you in episode 12!