Learn Podman - Observability & Operations

#Introduction

In episode 19 you ran Podman on a laptop via Podman Machine and managed everything from Podman Desktop. Episode 20 returns to the real operations side: once your containers run on a server, you need to know whether they're healthy, how much resource they're using, and what's happening inside them. This material is the main preparation for the on-call shift: observability and troubleshooting.

#Monitoring Resources: podman stats

podman stats is a real-time dashboard for container resource usage — CPU, memory, and network I/O. Without arguments, it shows all running containers and refreshes the data periodically:

podman stats
podman stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

With --no-stream, the output is only shown once — suitable for scripts or snapshots. The --format option lets you select specific columns. podman stats is the first place you should go when a service feels slow: is the container running out of memory, or is the CPU saturated?

#Tracking Activity: podman events

Unlike podman stats, which gives a momentary snapshot, podman events provides a timeline: every important event — start, stop, die, health_status, and others — is recorded with a timestamp:

podman events --since 1h
podman events --filter container=myapp --filter event=die

Filters help you narrow down to relevant events, for example a specific container dying repeatedly. A good production habit: store the podman events stream to a log file or monitoring pipeline, so you can reconstruct the sequence of events during an incident. This is the bridge from basic observability toward an audit trail.

#Viewing Processes Inside a Container: podman top

To see the processes running inside a container, use podman top. Its syntax mimics ps, since the output is indeed generated from the container's process namespace:

podman top myapp
podman top myapp -eo pid,user,comm,args

The first line shows the default processes; the second uses ps-style options to select columns. When a container suffers a CPU spike or fork bomb, podman top helps you find which process is at fault inside the container — without having to get in with a shell.

#Healthcheck: Keeping Containers Healthy

Running a container is easy; knowing the container still works is its own challenge. A healthcheck is a command Podman runs periodically inside the container to determine its health status: healthy, unhealthy, or starting.

#Declaring in the Containerfile

FROM nginx:1.27
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
    CMD curl -f http://localhost/ || exit 1

The HEALTHCHECK instruction runs the curl command every 30 seconds; if it fails three times in a row, the container is considered unhealthy.

#Mounting at Run

You can also attach a healthcheck from outside without changing the image, using --health-cmd and its companion flags:

podman run -d --name myapp \
    --health-cmd "curl -f http://localhost:8080/health" \
    --health-interval 15s --health-timeout 3s --health-retries 3 \
    myapp:latest
podman healthcheck run myapp

--health-cmd defines the command, --health-interval the frequency, --health-timeout the execution timeout, and --health-retries the number of failures before the status flips to unhealthy. podman healthcheck run triggers a manual check. Note that --health-cmd requires a tool inside the container (like curl or wget); for minimal images, write a healthcheck command using a tool that already exists in the image.

Health status also appears as a health_status event in podman events, so healthchecks and events work as one unit: health status changes are automatically recorded in your event timeline.

#Metrics Integration

For long-term monitoring, you'll certainly want to collect metrics into Prometheus or Grafana. To be honest: unlike the Docker daemon, which exposes a built-in /metrics endpoint, Podman doesn't yet provide a similar metrics endpoint in its engine. The commonly used approaches:

• Installing a dedicated exporter like prometheus-podman-exporter, which translates podman stats data into Prometheus format.
• Building your own collector that runs podman stats --no-stream periodically and pushes the results.
• Collecting logs and events (podman events) via a log agent like Loki or syslog.

With this pattern, the podman stats and podman events data you know from the CLI can become the foundation of Grafana dashboards and alerting — even without a built-in endpoint, observability can still be built from the client side.

#Troubleshooting: When Something Goes Wrong

When a container has a problem, run the checks in a systematic order, from the outside in.

#Initial Diagnosis: podman info --debug

podman info --debug shows comprehensive information about your environment — Podman version, storage driver, network backend, configuration, down to rootless container storage. When reporting a bug or asking a question on a forum, this command's output is the first piece of information requested, because almost every root problem lives in the environment:

podman info --debug

Pay attention to the networkBackend and networkBackendInfo sections — this is where you see whether Podman uses netavark with aardvark-dns, and whether its support is active.

#Logs

Container application logs are retrieved via podman logs. Start by looking at the tail end, then decide whether you need to dig deeper:

podman logs --tail 100 myapp
podman logs -f myapp

--tail limits the last lines, -f follows the log in streaming mode. If the application logs are empty, double-check whether the application writes to stdout/stderr — Podman only captures standard output, not the application's internal log files.

#Events as the Diagnosis Key

podman events helps answer the question "why did the container die?". Look for die, stop, or health_status events around the time of the incident, then cross-reference them with the logs. The combination of podman events --since and podman logs is the most effective first diagnosis pair.

#Diagnosing Rootless Networking

Network problems in rootless mode have their own distinctive symptoms. Some checkpoints:

• Confirm the active networking backend: check networkBackend in podman info.
• Check the status of published ports: podman ps shows the port mappings, then test with curl from the host.
• For rootless port mapping, Podman uses pasta (or slirp4netns on older versions). The logs of this networking process can be a clue when a port isn't reachable from outside.
• Make sure the service inside the container is actually listening on the intended interface, not just on the container's localhost: run podman exec myapp ss -tlnp to see the open sockets.

Rootless networking diagnosis almost always boils down to three possibilities: the pasta/slirp process is broken, the port is published wrong, or the application inside the container is only listening on the wrong address.

#Closing

In episode 20 you mastered the observability and operations side of Podman: monitoring resources with podman stats, tracking the event timeline with podman events, inspecting processes with podman top, keeping containers healthy via HEALTHCHECK and --health-cmd, building metrics integration into Prometheus, and debugging issues with podman info --debug, logs, events, and rootless networking diagnosis.

The key points to take home:

• podman stats for the present, podman events for the trace of the past — the two complement each other.
• A healthcheck is the window into application health — attach it from the Containerfile or --health-cmd, and monitor its status via events.
• podman info --debug is the first troubleshooting step — always attach its output when asking questions.
• Rootless networking is diagnosed from its backend — check netavark/pasta, port publication, and sockets inside the container.

In the next episode, Episode 21, we look ahead: Modern Features & Roadmap — what Podman 6.0 and 6.1 bring, including the config file rework, Quadlet overhaul, netavark/aardvark-dns v2, all the way to the roadmap toward CNCF incubation.