Learning DNS - Performance Tuning & Observability

#Introduction

A healthy DNS must be measured, not guessed. Episode 19 covers two production needs at once: making your DNS servers fast and making them readable — so you know exactly where load sits and what happens when performance drops.

The first part covers tuning: caches, threads, and TTLs. The second part covers modern observability: structured logging, OpenTelemetry, Prometheus, and the webserver statistics all three PowerDNS daemons provide.

#Cache Tuning

#Packet Cache and Record Cache

Caching is the biggest performance multiplier in the DNS stack. The Recursor balances two caches:

max-cache-entries: 2000000
packetcache-size: 500000
cache-ttl: 20
max-negative-ttl: 3600

max-cache-entries sets the record cache size, packetcache-size the packet cache. The values above are a starting point for medium load — raise them gradually and watch memory. cache-ttl and max-negative-ttl control how long positive and negative answers live.

#Measuring the Effect of Tuning

Every change must be proven:

rec_control get cache-hits
rec_control get cache-misses
dig @127.0.0.1 example.com A +noall +stats | grep 'Query time'

A high hit rate means most queries are answered from cache. If cache-hits is low while cache-misses is high, the cache is too small or the TTLs too short.

#Tuning Threads and LMDB

#Thread Counts

The Recursor and Authoritative use a thread model. Adding threads uses more CPU but improves parallelism:

threads: 8
reuseport: true

reuseport lets multiple sockets listen on the same port, distributing load across CPU cores.

#LMDB Tuning

If you use the lmdb backend for Authoritative, the memory map size needs adjustment:

lmdb-filename=/var/lib/powerdns/lmdb
lmdb-size=1073741824

lmdb-size sets the memory map size in bytes. As zones grow large, monitor usage and enlarge before it fills up — a full LMDB refuses write operations.

#Structured Logging

#Structured Logs in Authoritative 5.1

PowerDNS Authoritative 5.1 introduced structured logging in JSON format. These logs can be machine-read and analyzed automatically:

sudo systemctl restart pdns
journalctl -u pdns -n 10 --no-pager | grep -i '"level"'

Each log line is now a JSON object containing level, message, and contextual fields. This turns logs from manually-parsed text into queryable data — the foundation of modern observability.

#OpenTelemetry and Prometheus

#OpenTelemetry Tracing

Recursor 5.4 and dnsdist 2.1 support OpenTelemetry tracing: every query can be traced across daemons, giving an end-to-end picture of where time is lost.

openTelemetry:
  enabled: true
  endpoint: otel-collector:4317

#Prometheus Endpoint

All three daemons expose Prometheus metrics. dnsdist through its webserver API:

webserver:
  - address: 0.0.0.0:8083
    apiKey: sekret-api

Then scrape the metrics:

curl -s http://127.0.0.1:8083/api/v1/servers/localhost/statistics

curl .../statistics returns metrics like queries per second, cache hits, and the number of SERVFAIL responses in JSON format — ready to feed into Grafana. rec_control get-all provides similar recursor metrics.

#Building a Dashboard

#Metrics You Must Monitor

Your DNS dashboard should at minimum show these metrics:

• QPS (queries per second) per daemon.
• Cache hit rate and cache size.
• Query latency from the client side.
• SERVFAIL and NXDOMAIN counts — early indicators of trouble.
• Backend status in dnsdist (up/down).

rec_control get-all > /tmp/rec-metrics.txt
dnsdist -c
showStats()
quit()

#Conclusion

Episode 19 makes your DNS stack fast and readable: measurable cache, thread, and LMDB tuning, JSON structured logging, OpenTelemetry tracing, and Prometheus metrics ready for visualization.

Key takeaways:

• max-cache-entries, packetcache-size, and TTLs are the Recursor's main tuning levers.
• Cache hit rate and query time prove tuning works.
• reuseport spreads load across cores; lmdb-size prevents a full database.
• Structured logging turns logs into analyzable JSON data.
• Recursor 5.4 and dnsdist 2.1 support OpenTelemetry tracing.
• The webserver API provides Prometheus metrics for Grafana dashboards.

In episode 20, we'll cover the PowerDNS API and IaC automation — the REST API with an API key for managing zones, records, TSIG, and crypto keys, provisioning with curl and Python, plus integration with the Terraform provider and ExternalDNS in Kubernetes.