Learn GraphQL - Building Resilient GraphQL Services

#Introduction

Downtime isn't "if" but "when". Episode 35 builds resilient GraphQL services — able to survive database, provider, even whole-region failures. The main goal isn't to avoid failure, but to keep its impact as small as possible.

We'll cover high availability, backup strategies, disaster recovery planning, circuit breakers, and graceful degradation.

#High Availability Design

#Multi-Region and Failover

High availability (HA) means the service stays available even through failures. The core designs:

• Multi-region deployment: run instances in several regions.
• Automatic failover: when one region goes down, traffic is routed automatically.
• Health checks (episode 24) as the basis for failover decisions.
• Redundancy: no single point of failure — primary and secondary databases, DNS with failover, caches that can be rebuilt.

region-a (primary) <--> region-b (secondary)
   |                         |
  traffic                  standby

For stateless GraphQL (episode 34), adding regions is relatively easy: deploy the same image, point DNS at it, and make sure Redis and the database are replicated across regions.

#Health Checks and Redundancy

Combine liveness and readiness (episode 24) with redundancy: a database with a promotable replica, a Redis cluster that can survive node loss, and a caching strategy that doesn't stop the service when Redis is down (falling back to the database directly).

#Backup Strategies

#Database Backups and PITR

A proper backup isn't just a copy — it must be restorable quickly:

• Scheduled backups: periodic full backups plus continuous WAL archiving.
• Point-in-time recovery (PITR): restore the database to a specific minute — saving you from a wrong operation or corrupted data.
• Backup testing: schedule periodic restore tests; a never-tested backup is an illusion of safety.
• Retention policy: keep backups per needs (for example 30 days) and regulatory policy.

A manual PostgreSQL backup is pg_dump -Fc database-name > backup.dump.

For managed databases (RDS, Cloud SQL), enable automated backups with built-in PITR.

#Disaster Recovery Plan

#RTO and RPO

A DR plan is measured by two metrics:

• RTO (Recovery Time Objective): the maximum acceptable time to get back to normal (for example 1 hour).
• RPO (Recovery Point Objective): the maximum amount of data that may be lost (for example 5 minutes).

RTO: how fast to get back to normal
RPO: how much data may be lost

The smaller RTO and RPO, the more expensive the infrastructure. Set realistic targets based on the business, then design the strategy (for example active-passive multi-region for low RTO, daily backups for non-critical data).

#Documentation and Testing

A good DR plan must be documented and tested periodically. Schedule DR drills: turn off the primary region, force a failover, measure recovery time, and compare against RTO/RPO targets. From the drill results, update the runbook and automation.

#Circuit Breaker Pattern

#Preventing Cascade Failure

A circuit breaker prevents one failure from spreading. Three states:

• Closed: normal, all requests are passed through.
• Open: after many failures, requests are rejected outright without calling the dependency.
• Half-open: after a pause, a small number of requests are tested; success returns to Closed.

Simple circuit breaker

class CircuitBreaker {
  constructor(fn, { threshold = 5, cooldownMs = 30000 }) {
    this.fn = fn;
    this.failures = 0;
    this.threshold = threshold;
    this.cooldownMs = cooldownMs;
    this.lastFailure = 0;
    this.state = "CLOSED";
  }
 
  async call(...args) {
    if (this.state === "OPEN" && Date.now() - this.lastFailure < this.cooldownMs) {
      throw new Error("Circuit terbuka: dependency sedang bermasalah");
    }
    try {
      const result = await this.fn(...args);
      this.failures = 0;
      this.state = "CLOSED";
      return result;
    } catch (err) {
      this.failures += 1;
      this.lastFailure = Date.now();
      if (this.failures >= this.threshold) this.state = "OPEN";
      throw err;
    }
  }
}

Use a library like opossum for a production-ready implementation. Apply circuit breakers to external dependencies: payment gateways, email services, and third-party APIs. Complement with a fallback — for example returning cached data when the dependency is down.

#Graceful Degradation

#Partial Failure and Feature Flags

Graceful degradation means the service keeps working (at reduced quality) when some components fail:

• Partial failure handling: if the feed fails, other pages still work — GraphQL already supports this via partial success (episode 11).
• Feature flags: disable expensive or experimental features without a deploy.
• Fallback responses: return cached data or a simple response when the primary source fails.
• Monitor degraded states: flag when the service is running in degraded mode.

Fallback with cache

async function getTrending(ctx) {
  try {
    const data = await ctx.trendingService.fetch();
    await ctx.redis.set("trending", JSON.stringify(data), "EX", 300);
    return data;
  } catch {
    const cached = await ctx.redis.get("trending");
    return cached ? JSON.parse(cached) : [];
  }
}

This fallback pattern keeps the API responding even when the primary source is troubled — much better than a total 500 error.

#Conclusion

Key takeaways:

• High availability means multi-region, automatic failover, and no single point of failure.
• Backups must be restorable, tested, and have a retention policy.
• RTO and RPO define recovery targets; DR drills are mandatory and periodic.
• Circuit breakers prevent cascade failure with closed-open-half-open states.
• Graceful degradation keeps services responding with fallbacks and feature flags.
• GraphQL partial success supports degradation without halting the entire request.

In the next episode, episode 36, you'll learn about GraphQL with microservices — gateway patterns for microservices, a federation deep dive, event-driven architecture with event sourcing and CQRS, message brokers like Kafka and RabbitMQ, and service discovery. Your architecture will scale to enterprise!