Learning Caddy - Logging & Monitoring

#Introduction

When something breaks in production, the logs are the first witness. Episode 22 covers observability in Caddy: recording access and errors correctly, formatting logs so machines can parse them easily, sending logs to a centralized system, monitoring metrics with Prometheus, and debugging with the right log levels.

You'll learn the log directive, Common Log and JSON formats, custom log fields with placeholders, aggregation into ELK or Loki, and how to use the admin API for metrics and troubleshooting.

A system without good logs is a black box. Let's open that box.

#The log Directive

#Access and Error Logs

By default Caddy writes error logs to stderr. To log access, enable the log directive:

example.com {
    log {
        output stdout
        format json
    }
    root * /var/www
    file_server
}

output stdout sends access logs to standard output — useful in containers. format json creates structured logs that are easy to parse.

#Output to a File

For long-term storage:

example.com {
    log {
        output file /var/log/caddy/access.log {
            roll_size 100mb
            roll_keep 10
        }
    }
    root * /var/www
    file_server
}

roll_size 100mb and roll_keep 10 configure rotation: the file is split every 100 MB and the last 10 files are kept.

#Log Levels

Filter by importance level:

example.com {
    log {
        level INFO
        output stdout
    }
    root * /var/www
    file_server
}

Available levels: DEBUG, INFO, WARN, ERROR, PANIC, and FATAL. DEBUG records full details; ERROR records only errors. For production, INFO is usually enough.

#Structured Logging

#JSON Format and Custom Fields

The JSON format makes it easy for machines to read logs. Add custom fields with placeholders:

example.com {
    log {
        format json {
            time_format rfc3339
        }
        output stdout
    }
    root * /var/www
    file_server
}

Useful placeholders in logs:

• {method} and {uri} — the incoming request.
• {status} — the response status code.
• {duration} — processing time.
• {remote_host} — the client IP.
• {header.User-Agent} — the user agent.

log { format json ... } with time_format rfc3339 gives standard timestamps for tooling.

#Access Log Fields

A complete access log contains time, client IP, method, path, status, response size, user agent, duration, and protocol version. For reverse proxies, backend information like {upstream.address} is added. Everything can be included via custom placeholders.

#Log Aggregation

#Centralized Logging

One server means one place for logs. Many servers? You need aggregation:

• ELK stack: Elasticsearch stores, Logstash processes, Kibana displays.
• Loki: logs are collected and indexed with labels, displayed in Grafana.

Because the JSON format is already structured, aggregation tooling can process it directly. Send Caddy logs to a log shipper like Filebeat or Promtail, then onward to the central system.

#Logging Practices

• Don't log sensitive data (passwords, tokens, request bodies).
• Rotate logs on the local server.
• Set alarms for critical error patterns.

#Metrics and Monitoring

#Prometheus via the Admin API

Caddy exposes Prometheus metrics through the admin API:

curl http://localhost:2019/metrics

The output is Prometheus text format: request counters, duration histograms, and more. curl http://localhost:2019/metrics is scraped by Prometheus periodically for dashboards and alerting.

#Health Endpoints

For uptime monitoring, make sure you have a health endpoint:

example.com {
    respond /healthz 200 "OK"
    root * /var/www
    file_server
}

The combination of a health endpoint, Prometheus metrics, and aggregated logs gives you the full picture: is the site alive, how fast is it, and what's wrong.

#Debugging

#DEBUG Log Level and Validation

When there's a problem, increase the detail:

caddy run --config Caddyfile --debug

caddy run --debug shows DEBUG logs — including routing decisions and TLS details. The recommended debugging flow:

1. Read the error logs first.
2. Validate the Caddyfile with caddy validate.
3. Run in the foreground with --debug.
4. Check certificate status via the admin API.
5. Test with curl and openssl.

caddy validate --config Caddyfile checks the syntax before production — a mandatory step in deployments (episode 30).

#Request Tracing

For difficult problems, enable DEBUG and trace logs per request. With the JSON format, filtering by request.id will show one request's journey from start to finish.

#Conclusion

Episode 22 built observability: the log directive with output to stdout or files and rotation, Common and JSON formats with placeholders, log aggregation into ELK or Loki, Prometheus metrics via the admin API, and debugging techniques with --debug and validation.

Key takeaways:

• log { output stdout format json } records structured access logs.
• Log rotation keeps files from bloating.
• The JSON format aggregates easily into ELK or Loki.
• Prometheus metrics are available at the admin API's /metrics.
• caddy run --debug shows full request details.
• caddy validate checks configuration before deploy.

In the next episode, episode 23, we'll cover admin API & dynamic configuration — admin API endpoints, loading and replacing JSON configuration, hot reload without downtime, JSON vs Caddyfile differences, and CI/CD automation with the API. Dynamic configuration will open new possibilities.