Learn Docker - Container Observability & Monitoring

#Introduction

After ensuring in episode 14 that images entering the registry are truly clean — vulnerability-scanned with Trivy/Grype, with an SBOM, and signed with Cosign — in this episode we shift from what enters the system to what happens inside it: observability. A secure image is only the starting point; once it runs in production, the truly decisive questions are: is the application healthy? if not, why? and how do we know before the users complain?

This isn't a luxury. Imagine being a pilot asked to fly without speed, altitude, and fuel indicators. The flight could be smooth — until one day something goes wrong and you have no clue at all. Containers behave exactly like that: they run, they print logs, and they can die silently. Without a way to read them, you'll only know there's a problem when users report an error — and by then, all that's left is guesswork.

In this episode we'll build three layers of container observability: logging (recording what happens), health & metrics (measuring condition), and audit events (knowing who did what). We start with the most often neglected yet most server-sinking layer — logging — then move up to HEALTHCHECK, daemon metrics endpoints, and close with the industry-standard monitoring stack: cAdvisor, Prometheus, and Grafana.

#Main Discussion

#Observability: Three Layers That Must Exist

Before touching any commands, let's agree on the mental model. Observability isn't one feature, but three questions answered by data:

1. Logging — answers what happened? The sequence of events, errors, requests. It's the narrative story of the application.
2. Metrics — answers how healthy? Quantitative numbers: CPU, memory, request rate, latency. It's the aircraft dashboard.
3. Healthcheck & events — answers still alive, and who touched it? Explicit status that can be queried and an audit trail of activity.

The right analogy: logs are the aircraft's black box, metrics are the instrument panel, and healthchecks are routine pre-takeoff inspections. Running production without any of them is like flying with your eyes closed — it survives, but every minute is full of risk. Let's build all three one by one.

#Logging: The Container's Voice You Must Listen To

When an application inside a container writes to stdout/stderr, that data is intercepted by the Docker logging driver. The driver is a plugin that decides where those log lines go. The built-in default is json-file — every log line is written to a JSON file under /var/lib/docker/containers/<id>/. The docker logs command we've been using throughout only reads this driver's files.

Which driver you choose isn't a matter of taste — it determines three things: (1) whether docker logs still works, (2) where logs are aggregated when there are dozens of hosts, and (3) how fast your disk fills up. The last two questions are what will haunt you in production.

Architecturally, think of the driver as a water pipe: json-file holds the water in a bathtub (the host), loki/syslog/gelf forward it to a central tank (a log server), and journald connects it to the city sewer (the host's systemd). The pipe choice determines where the water can flow.

#Setting the Logging Driver in daemon.json

The default configuration can be changed globally via /etc/docker/daemon.json — the file the dockerd daemon reads at start (remember the brief mention in episode 24 that dissects the daemon). This applies to all containers created afterward, except those that explicitly choose their own driver:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After changing daemon.json, restart the daemon so the effect is applied:

sudo systemctl restart docker
docker info --format '{{.LoggingDriver}}'

Those two log-opts options aren't accessories — they're the most ignored disk-savers. max-size: "10m" trims the log file every time it reaches 10 MB, and max-file: "3" keeps only the last 3 files (a maximum of 30 MB total per container). Without them, json-file writes endlessly — and a single application printing verbose logs can eat hundreds of GB within weeks. We'll dive deeper in the log rotation section.

#Per-Container Override: --log-driver

Sometimes one container needs a different driver from the global policy — for example, a reverse proxy container whose access logs you want sent to loki while the rest stay on json-file. The driver can be chosen per container at docker run:

docker run -d --name web \
  --log-driver=json-file \
  --log-opt max-size=5m \
  --log-opt max-file=2 \
  -p 80:80 nginx

docker run -d --name app --log-driver=journald my-app:1.0
journalctl CONTAINER_NAME=app -f

The rule of thumb: set a sensible global policy in daemon.json, then override only in specific cases. Driver consistency matters because the log aggregator on the other side (Loki, Graylog, CloudWatch) must accept a uniform format from all hosts — wild mixed drivers are a recipe for lost logs.

#docker logs: Limitations You Must Know

It's crucial to understand the limits of docker logs: it only works for drivers that store logs locally, namely json-file and local. As soon as you use journald, syslog, loki, or awslogs, docker logs shows the error "configured logging driver does not support reading". Logs move to the destination system — in the journal, on a syslog server, in CloudWatch — and are no longer Docker's responsibility.

This is a sensible design decision: if logs are already sent elsewhere, Docker doesn't need to keep a copy. But you must know this from the start, because many teams "lose" logs when migrating drivers — the logs aren't lost, they're looking in the wrong place. Before switching drivers, make sure the new log reading path is ready: journalctl -u docker.service for journald, or a query on the Loki/CloudWatch server.

#Log Rotation: Preventing a Full Disk Before It Happens

The most common cause of production downtime isn't application crashes — it's a disk filling up from logs. The story is always the same: a service prints large logs, /var/lib/docker balloons, one partition fills up, and the entire node dies because processes can't write. It's a terrible incident because it looks like a "server problem" when the root cause is just one configuration line that was never set.

Let's look at the default behavior and how rotation saves you:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Imagine the mechanism like a notebook. Without rotation, notes are written until the paper runs out, then they keep piling on the floor (full disk). With max-size, each time the notes reach 10 MB, Docker closes that page, opens a new one, and keeps the old page — up to max-file: 3 as the page limit. Page 4 pushes out page 1. The total footprint per container is capped at 30 MB, no matter how long the container lives.

To check that rotation is actually working, look at the log files on the host — there should be several numbered files, not one giant file:

File log kontainer ter-rotate di host

sudo ls -lh /var/lib/docker/containers/$(docker ps -q --filter name=web)/*-json.log

#HEALTHCHECK: Early Failure Detection

docker logs narrates events; docker stats shows resource usage; but neither tells you whether the application actually works. A container can use 0% CPU and still be healthy, or use 100% CPU and still serve. What matters is: when asked, does it respond correctly? The answer comes from HEALTHCHECK — a command the daemon runs periodically to verify the application.

The HEALTHCHECK instruction in a Dockerfile has five parameters:

• --interval: the gap between checks (default 30s)
• --timeout: the time limit for one check (default 30s)
• --retries: how many failures before the status becomes unhealthy (default 3)
• --start-period: the grace period at startup — healthchecks run but failures aren't counted (default 0s, must be set for applications that need initialization time)
• --start-interval: (newer versions) a separate interval during the start period

A real example for a web service — note: we use wget, not curl:

FROM node:20-alpine
 
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
 
EXPOSE 3000
 
HEALTHCHECK --interval=10s --timeout=5s --retries=3 --start-period=20s \
  CMD wget -qO- http://localhost:3000/health || exit 1
 
CMD ["node", "server.js"]

Reading it: every 10 seconds, try to access /health; if it fails within 5 seconds, count it as a failure; 3 consecutive failures → unhealthy. During the first 20 seconds (start period), failures aren't counted so the application has time to load its dependencies. You could also use curl if the base image has curl — but mind the tempting tautology:

FROM nginx:alpine
 
HEALTHCHECK --interval=10s --timeout=3s --retries=2 \
  CMD curl -f http://localhost/ || exit 1

But don't rush to write curl into any image. This is a classic trap: slim/alpine/distroless images often don't include curl because they prioritize small size. If you write a curl-based HEALTHCHECK in an image without curl, the healthcheck fails not because the application is sick but because curl doesn't exist — the result: a healthy container is declared dead. The solutions: use wget (more commonly available), or use a tool that actually exists in the image (e.g. nginx -t), or call the endpoint over HTTP from Node itself. Always verify that the binary used by the healthcheck actually exists in the image.

To check health status:

docker inspect --format='{{json .State.Health}}' web | jq .
docker ps --filter health=unhealthy

docker inspect will show the recent check history — output, exit code, and timestamps — while docker ps --filter health=unhealthy instantly surfaces all problematic containers. This is the data orchestrators use for automatic restarts (we'll prove it in episode 17), and that load balancers use to stop sending traffic to sick replicas.

#Docker Daemon Metrics: Watching the Machine, Not Just the Apps

A healthy application doesn't mean a healthy host. The Docker daemon itself — which manages all containers — must be monitored too: how many containers are running, how many images are stored, how long pull operations take, and so on. Docker provides a Prometheus-format metrics endpoint that's enabled via daemon.json:

{
  "metrics-addr": "127.0.0.1:9323",
  "experimental": false
}

sudo systemctl restart docker
curl -s http://127.0.0.1:9323/metrics | grep docker_container_running

#The Monitoring Stack: cAdvisor + Prometheus + Grafana

Now let's assemble a real monitoring system. The architecture splits three responsibilities:

• cAdvisor (Container Advisor from Google) — sits on every host, collects container metrics (CPU, memory, network, disk) and exposes them in Prometheus format. It's like a sensor on every machine in a factory.
• node-exporter — exposes metrics of the host itself (not containers): host CPU, memory, disk, load average. This distinguishes "a busy node" from "a busy container".
• Prometheus — a time-series database that periodically pulls (scrapes) metrics from cAdvisor and node-exporter, stores them, and answers queries. It's like a stock clerk going around collecting sensor data.
• Grafana — a visual dashboard that pulls data from Prometheus. It's not a data source, but the screen that displays everything.

Here's the complete compose.yaml along with the Prometheus configuration:

services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    devices:
      - /dev/kmsg
    restart: unless-stopped
 
  node-exporter:
    image: prom/node-exporter:v1.8.2
    ports:
      - "9100:9100"
    restart: unless-stopped
 
  prometheus:
    image: prom/prometheus:v2.53.0
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    restart: unless-stopped
 
  grafana:
    image: grafana/grafana:11.1.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped
 
volumes:
  grafana-data:

global:
  scrape_interval: 15s
 
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
 
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]
 
  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

Run it with docker compose up -d, then access: cAdvisor at http://<host>:8080, Prometheus at http://<host>:9090, and Grafana at http://<host>:3000 (login admin/admin). In Grafana, add a Prometheus datasource (http://prometheus:9090) then import dashboards like "Docker monitoring" (ID 893) for a container overview, and "Node Exporter Full" (ID 1860) for host health.

Note the architecture: Prometheus scrapes cAdvisor and node-exporter by service name thanks to the custom bridge DNS (the episode 9 lesson). Add cAdvisor + node-exporter to every node when you build a Swarm cluster later (episodes 16-17) — once there are multiple hosts, Prometheus just needs one target per node added to monitor them all.

The most used Prometheus queries: container_cpu_usage_seconds_total for container CPU, container_memory_usage_bytes for memory, and node_filesystem_avail_bytes for remaining host disk. With cAdvisor + Prometheus + Grafana, you can answer "which node will run out of disk next week?" before the answer destroys production.

#docker events: Real-time Audit Trail

Finally, Docker's event stream is a real-time feed of everything that happens to the daemon: containers created, started, stopped, images pulled, volumes created, and so on. It's useful for debugging (why did the container restart?) and security auditing (who ran which image when?):

docker events --filter type=container --filter event=restart
docker events --filter type=container --filter container=web --since 5m
docker events --format '{{.Time}} {{.Type}} {{.Action}} {{.Actor.Attributes.name}}'

With --since 5m you can reconstruct what happened in the last five minutes — for example, why a container suddenly died. docker events is the "black box" that beginners rarely use but operators always reach for when an incident happens.

#Pitfalls That Often Sink Production

1. A full disk from logs without rotation. The max-size + max-file configuration should be set from day one — before an incident, not after. A routine docker system df check (episode 4) is the cheapest detection.

2. HEALTHCHECK using curl in an image without curl. slim/alpine/distroless images often have no curl. A healthcheck failing because the tool is missing produces false positive unhealthy status. Use wget or a binary that exists in the image, and test the command inside the container first.

3. docker logs goes silent after changing drivers. Switching to journald/syslog/loki/awslogs makes docker logs stop reading anything. This isn't a bug — the logs were relocated. Prepare the new reading path before migrating.

4. metrics-addr open to the public. An endpoint without authentication; bind to 127.0.0.1 or an internal network only.

5. Prometheus scrapes once, then "no data". Make sure the cadvisor and node-exporter jobs use resolvable service names (custom bridge DNS) and that targets don't keep changing. At http://prometheus:9090/targets, check the up status for each target.

#Conclusion

In this episode 15 we built the observability layer that was missing: understanding that the logging driver determines where logs flow — json-file for local, journald/syslog/loki/awslogs/gelf for aggregation — and that rotation (max-size, max-file) is the lifeline of a production disk. We also added HEALTHCHECK so the daemon knows the application truly responds (wary of images without curl), enabled the daemon metrics endpoint to monitor the machine itself, assembled cAdvisor + Prometheus + Grafana monitoring both containers and hosts, and used docker events as an audit trail.

Core takeaways:

• Set log rotation in daemon.json from day one; it prevents the most common full-disk incident.
• docker logs only works for the json-file and local drivers — plan a log reading path before switching drivers.
• HEALTHCHECK = an application that responds is considered healthy; use --start-period and a binary that actually exists in the image.
• Daemon metrics must be bound to 127.0.0.1.
• cAdvisor sees inside containers, node-exporter sees the host, Prometheus stores, Grafana displays.

Now you can see a single host clearly — logs, status, and metrics. But what if your application grows beyond one server and must spread across several machines that take over for each other when one fails? That's the point where we stop "running containers" and start "orchestrating". In the next episode, episode 16, we'll enable Docker's built-in mode for that: Clustering & Orchestration with Docker Swarm Mode — joining multiple nodes, introducing managers and workers, and understanding why a cluster architecture sits behind large systems. Make sure your hosts are ready, because we're going to build a real cluster. See you in episode 16!