Learn NATS - Production-Ready Architecture

#Introduction

All the previous chapters lead to one question: how do you assemble everything into a production-grade architecture? Episode 21 answers it. You'll design reliable, secure, and observable NATS infrastructure — then lay out the application patterns that use it to its fullest.

This is the architect's episode: most of its content is design decisions that tie together all the previous lessons.

#The Cluster Foundation

#A 3 or 5-Node Topology

The first choice is the cluster size. 3 nodes for medium loads with tolerance for one dead node; 5 nodes for larger production workloads with tolerance for two.

server_name: nats-1
cluster {
  name: "nats-prod"
  listen: "0.0.0.0:6222"
}
jetstream {
  store_dir: "/data/jetstream"
  max_file_store: 100G
}

The jetstream block stores data in a persistent directory and caps the file store at 100 GB. Every production stream uses replicas: 3 so it stays available when one node goes down.

#Spreading Nodes Across Fault Domains

nats-1, nats-2, nats-3 (JetStream cluster)
  ├── client port 4222
  ├── route port 6222
  └── monitoring port 8222

The nats-1, nats-2, nats-3 (JetStream cluster) structure places the three nodes on different hosts or zones. Don't put all nodes on one machine — that's not a cluster, just a single point of failure.

#Accounts and JWT

#Isolation Between Teams

Production architecture uses accounts to separate domains:

• account ORDERS for order services and related streams.
• account PAYMENTS for the payments domain.
• account ADMIN for tooling and observability.

nsc add account ORDERS
nsc add user --account ORDERS order-service
nsc generate creds --account ORDERS --user order-service -o order-service.creds

nsc generate creds generates JWT credentials for a service. Each service uses its own creds, with permissions limited to the minimum. If cross-account communication is needed, use export-import as in episode 12.

#TLS on Every Connection

#End-to-End Encryption

All communication ports must use TLS: clients, cluster routes, gateways, and monitoring.

tls {
  cert_file: "/etc/nats/certs/server-cert.pem"
  key_file: "/etc/nats/certs/server-key.pem"
  ca_file: "/etc/nats/certs/ca.pem"
  verify: true
}

The tls block with verify: true enables mutual TLS on all connections. Clients use tls:// URLs with their own creds or certificates.

#Monitoring and Alerting

#One Center of Observation

A production architecture is incomplete without monitoring:

nats_exporter --> Prometheus --> Grafana --> alertmanager
$SYS events   --> notifications

The nats_exporter -> Prometheus -> Grafana flow brings the entire cluster's metrics into one dashboard. Alerting is built on top: full disk, rising pending, node down — all trigger notifications before they become incidents.

#Disaster Recovery with Mirroring

#Cross-Region Backups

Mirroring enables stream backups in another region:

name: ORDERS_DR
mirror:
  name: ORDERS
  external:
    api: nats://dr-cluster:4222

The mirror block in the DR cluster mirrors the ORDERS stream from the primary cluster. When the primary cluster fails completely, services are moved to DR and the stream is served from its copy.

#Application Patterns: Event-Driven Microservices

#One Event, Many Consumers

The first pattern: a service publishes the facts that happened, and consumers handle the rest.

orders --> publish orders.created --> stream ORDERS
                                        ├── notification service
                                        ├── inventory service
                                        └── analytics worker

The orders -> publish orders.created -> stream ORDERS flow shows one event consumed by many services without coupling. Each service subscribes to the stream with its own consumer — each with an independent read position.

#Application Patterns: Request-Reply APIs

#A Fast Internal API

The second pattern: request-reply for operations that need an immediate answer.

nats request auth.login '{"user":"arman"}' --timeout=2s

nats request auth.login represents the internal API pattern: a service queries status, does a lookup, or runs a verification, and waits for an answer. Disciplined timeouts keep callers from hanging.

#Application Patterns: Job Queues and Streaming Pipelines

#Async Work and Data Flows

The third and fourth patterns: job queues for async tasks, streaming pipelines for continuous flows.

event-driven microservices -> orders.created consumed by many services
request-reply API          -> auth.login, users.get
job queue                  -> stream JOB + pull consumer
streaming pipeline         -> ingestion -> enrich -> store

The streaming pipeline -> ingestion -> enrich -> store scheme forms a chain of stages: data comes in, gets enriched, then is stored. Each stage is a consumer on its own stream and can be scaled separately.

#Conclusion

Episode 21 assembled a production-grade architecture: a 3 or 5-node cluster with JetStream and replicas, accounts with JWT for isolation, TLS on all connections, centralized monitoring and alerting, disaster recovery via cross-region mirroring, and four application patterns — event-driven microservices, request-reply APIs, job queues, and streaming pipelines.

Key takeaways:

• A 3-node cluster tolerates one dead node; 5-node tolerates two.
• Every production stream uses replicas 3.
• Accounts and JWT separate domains with minimal permissions.
• TLS is mandatory on all ports: client, route, and monitoring.
• Monitoring and alerting prevent problems from becoming incidents.
• Cross-region mirroring is the foundation of disaster recovery.
• Four application patterns: event-driven, request-reply, job queue, and streaming.

In episode 22 next — the final episode — we'll discuss the alternative ecosystem & final reflection — an in-depth comparison of NATS with Kafka, RabbitMQ, MQTT, and Redis Streams, when to choose each, a recap of your journey from episode 0, and the future of NATS at the edge and in the cloud. The Learn NATS journey heads toward its final point.