Belajar System Design - Design URL Shortener (bit.ly/lnkd)

#Pendahuluan

Setelah di episode 19 kita memahami cost-aware design, pada episode ini kita mulai case study pertama: design URL shortener seperti bit.ly atau t.ly. Case study adalah penerapan praktis dari semua konsep yang sudah kita pelajari — dari estimasi, arsitektur, caching, database, hingga bottleneck analysis.

URL shortener adalah case study klasik karena sederhana di permukaan tapi menyembunyikan complexity yang menarik: bagaimana generate short URL yang unik? Bagaimana handle 20K read/detik? Bagaimana analytics click? Di episode ini kita bedah semuanya.

#Requirements

#Functional Requirements

Shorten URL: user memberikan long URL → sistem return short URL.
Redirect: short URL → redirect ke long URL.
Analytics: track click (count, timestamp, geolocation).
Custom alias: user bisa pilih custom short code (opsional).
Expiry: URL bisa diatur expired (opsional).

#Non-Functional Requirements

Latency: redirect harus sangat cepat (<100ms).
Availability: 99.99% uptime.
Scalability: handle 500M URLs/bulan (write) dan 50B redirects/bulan (read).

#Estimasi

Estimasi throughput

Write:
  500M URLs/bulan = ~16.7M/hari = ~193 QPS
  Peak: 5x = ~965 QPS
 
Read (read:write = 100:1):
  500M × 100 = 50B redirects/bulan
  = ~1.67B/hari = ~19,300 QPS
  Peak: 5x = ~96,500 QPS
 
Storage (5 tahun):
  500M × 12 bulan × 5 tahun = 30B URLs
  Per URL: short_code(7B) + long_url(500B) + created_at(8B) = ~515B
  Total: 30B × 515B = ~15 TB
 
Cache:
  20% of read = 80% traffic (hot URLs)
  Cache size: 20% × 30B × 515B = ~3 TB (Redis cluster)

#Design

#Hash/Encode

Short URL generation

Long URL: https://example.com/very/long/path/to/resource
 
Option 1: Hash (MD5/SHA256) + Base62 encode
  MD5(url) = "d41d8cd98f00b204e9800998ecf8427e"
  Take first 7 chars → Base62: "k8dG3fA"
 
Option 2: Counter-based
  Auto-increment ID → Base62 encode
  ID 1 → "b", ID 2 → "c", ... ID 1000000 → "4c92"
 
Option 3: Custom alias
  User provides: "my-link"
  Store: my-link → long_url

Base62: karakter [a-zA-Z0-9] = 62 karakter. 7 karakter Base62 = 62^7 = ~3.5 triliun kombinasi — cukup untuk 30B URLs.

#Arsitektur

100%

#Flow Shorten URL

Shorten flow

1. Client POST /shorten { "url": "https://..." }
2. Service generate short_code (hash/counter)
3. Check duplicate → jika ada, generate ulang
4. Save ke DB: { short_code, long_url, created_at }
5. Return short URL: "https://short.ly/k8dG3fA"

#Flow Redirect

Redirect flow

1. Client GET /k8dG3fA
2. Service check Redis cache → HIT? → return long_url → 301 redirect
3. Cache MISS → query DB → save ke cache → return long_url → 301 redirect
4. Emit event "click" ke Kafka → analytics pipeline

#Database Choice

Skenario	Database	Alasan
Simple, small scale	PostgreSQL	ACID, simple query
High write throughput	DynamoDB	Auto-scale, low latency
High read throughput	Cassandra	Horizontal scale, write-optimized

#Caching Strategy

Cache-aside untuk redirect

Redis key: "url:{short_code}" → long_url
TTL: 24 jam (atau lama jika populer)
 
Cache hit rate target: 80%
  → 80% request dari Redis (~1ms)
  → 20% request ke DB (~10ms)
  → Average latency: 0.8 + 2.0 = 2.8ms

#API Design

API endpoints

POST /api/v1/urls
  Body: { "url": "https://...", "custom_alias": "my-link", "expires_at": "2026-12-31" }
  Response: { "short_url": "https://short.ly/k8dG3fA", "long_url": "https://..." }
 
GET /:short_code
  Response: 301 Moved Permanently → Location: long_url
 
GET /api/v1/urls/:short_code/stats
  Response: { "clicks": 12345, "created_at": "...", "top_countries": [...] }

#Bottleneck & Solutions

Bottleneck	Solusi
Read-heavy (100:1 ratio)	Cache aggressive (Redis), CDN untuk redirect
Short code collision	Hash + collision check, atau counter-based
Analytics write-heavy	Event log ke Kafka → async processing
Single DB hotspot	Sharding by short_code hash
Cache stampede	Mutex untuk reload, stale-while-revalidate

#Analytics Pipeline

Analytics flow

Redirect event → Kafka → Analytics Worker → Analytics DB (ClickHouse/PostgreSQL)
 
Metrics:
- Click count per URL
- Click timestamp (time series)
- Referrer, user agent, geolocation
- Device type, browser
 
Query patterns:
- Total clicks: SELECT COUNT(*) WHERE short_code = X
- Clicks by day: SELECT DATE(created_at), COUNT(*) GROUP BY 1
- Top URLs: SELECT short_code, COUNT(*) GROUP BY 1 ORDER BY 2 DESC

Note

URL shortener adalah contoh sempurna dari read-heavy system: 100:1 read/write ratio. Caching adalah optimasi paling berdampak — tanpa cache, database harus handle 20K QPS read; dengan cache (80% hit rate), hanya 4K QPS yang mencapai database.

#Penutup

Inti yang harus dibawa pulang:

Estimasi: 500M URLs/bulan → 193 QPS write, 19K QPS read → cache wajib.
Short code: Base62 encode dari hash atau counter — 7 karakter = 3.5T kombinasi.
Arsitektur: Service → Redis cache → DB; analytics via Kafka async.
Bottleneck: read-heavy → cache; analytics → event streaming; scale → sharding.

Di episode 21 selanjutnya kita akan membahas case study: design chat system (WhatsApp/Telegram) — WebSocket connections, message storage di Cassandra, fan-out ke group members, dan presence service. Chat system adalah tantangan berbeda: write-heavy, real-time, dan persistent!