Learn NATS - Monitoring & Observability

#Introduction

An invisible messaging system is a time bomb. Episode 18 turns on the lights above NATS: you'll learn to monitor servers, clients, streams, and consumers through built-in monitoring endpoints, Prometheus metrics, Grafana dashboards, and $SYS account events for audit and alerting.

Without good observability, all the performance excellence of episode 16 can't be sustained in production.

#Built-in Monitoring Endpoints

#The Four Main Endpoints

The monitoring port (default 8222) provides JSON endpoints:

• /varz — general server status, version, message counts.
• /connz — list and details of all client connections.
• /jsz — JetStream status: streams, consumers, storage.
• /subsz — all subscriptions with their statistics.

curl -s http://localhost:8222/varz | python3 -m json.tool | head -n 30

curl -s http://localhost:8222/varz returns the full server status JSON. For production, make sure the monitoring port is only accessible from your internal network.

#Reading the JetStream Endpoint

The /jsz endpoint is the window into JetStream:

curl -s http://localhost:8222/jsz

The /jsz output shows the stream count, consumers, total messages, and storage usage. This is the data nats_exporter uses for Prometheus metrics.

#Prometheus Metrics with nats_exporter

#Installing nats_exporter

nats_exporter bridges the NATS endpoints to the Prometheus format:

prometheus:
  jobs:
    - name: nats
      static_configs:
        - targets: ["nats-exporter:7777"]

The targets: ["nats-exporter:7777"] configuration makes Prometheus scrape nats_exporter on port 7777. The exporter reads the /varz, /connz, /jsz, and /subsz endpoints, then exposes metrics prefixed with nats_.

#Metrics You Must Monitor

Some of the most important metrics:

• nats_server_in_msgs and nats_server_out_msgs — message volume.
• nats_jetstream_stream_messages — message count per stream.
• nats_consumer_ack_pending — messages awaiting ack.
• nats_core_subs and nats_core_connections — connection scale.

#Grafana Dashboards

#Visualizing Metrics

The official NATS Grafana dashboard provides ready-to-use panels:

messages per second    | pending per consumer
storage per stream     | active connections
request-reply latency  | redelivery count

The pending per consumer panel is your first problem radar: rising steadily means a consumer can't keep up. This dashboard speeds up diagnosis — one screen for the entire messaging health.

#Consumer Latency Metrics

#Keeping Response Time Healthy

Consumer latency shows how fast a message reaches the application after being published:

nats consumer report ORDERS

nats consumer report ORDERS shows per-consumer statistics including delivery latency and pending. Swelling latency often signals a slow consumer or a full storage disk — two things we'll cover in episode 19.

#$SYS Account Events

#A Source for Audit and Alerting

The special $SYS account receives system events: connections coming up, servers joining a cluster, authentication errors, and more.

nats sub '$SYS.>'

nats sub '$SYS.>' listens to all system events. Alerting can be built on top of it: send a notification on $SYS.ACCOUNT.NEW, a server going down, or repeated failed authentication attempts. $SYS events are the backbone of a proactive observability posture.

#Log Rotation

#Keeping Logs Under Control

The NATS server log can grow over time:

logging:
  debug: false
  trace: false
  file: "/var/log/nats.log"

The logging block directs the log to a file. Set up log rotation on the system side — for example logrotate — so the file doesn't fill the disk. For K8s environments, log to stdout and let the container runtime handle rotation.

#Conclusion

Episode 18 made NATS transparent: the /varz, /connz, /jsz, and /subsz endpoints for direct inspection, nats_exporter for Prometheus metrics, Grafana dashboards for visualization, consumer latency metrics for keeping responses healthy, $SYS events for audit and alerting, and log rotation so logs don't fill the disk.

Key takeaways:

• Port 8222 provides JSON endpoints for server, connection, and JetStream status.
• nats_exporter turns endpoints into Prometheus metrics.
• Grafana dashboards bring all metrics together on one screen.
• Pending consumers and latency are the main health indicators.
• $SYS events serve as a source for audit and proactive alerting.
• Log rotation prevents server logs from filling the disk.

In episode 19 next, we'll discuss troubleshooting — slow consumers and backpressure, endless redelivery, full storage due to max_bytes, lost quorum, subject collisions, plus the diagnostic tools nats server check, nats stream report, nats consumer report, and debug logs. Get ready to put out fires.