The System Design Picture Book
Clients CDN LB app servers Cache DB primary Queue Workers System design is trade-offs on purposeevery box buys something and costs something
PLATE 0 — A big system is a few simple ideas combined: copy, split, cache, queue.
A picture book for grown-ups · with numbers

System Design: Scaling Past One Server

Nine spreads on how real systems handle more users, more data and more failures, with the numbers and trade-offs behind each choice.

Say it like a pro

"Stateless services behind a load balancer, a cache for hot reads, a replicated sharded database, a queue for slow work, and a CDN at the edge."

Use the tabs or the ← → keys.

1 · Requirementswhat must it do? what scale? 2 · Back-of-envelopeusers, QPS, storage, bandwidth 3 · API + data modelendpoints and entities 4 · High-level designboxes and arrows 5 · Deep divebottlenecks, failures ask questions first · state assumptions · name the trade-off
PLATE 1 — Don't start drawing boxes. Start by pinning down what you're building.
Chapter 1 · The method & the numbers

Estimate before you design

Rough math tells you whether you need one server or a thousand.

LatencyRoughly
CPU cache / RAM read1 ns / 100 ns
SSD random read100 µs
same-datacenter round trip0.5 ms
cross-continent round trip150 ms
Example: a photo app, 10 M daily users, 2 uploads each, 2 MB per photo
writes  = 20 M / 86,400 s        ≈ 230 uploads/s  (peak ×3 ≈ 700/s)
reads   = 100× writes            ≈ 23,000 reads/s
storage = 20 M × 2 MB × 365      ≈ 14.6 PB / year → object storage + CDN
rule: 1 day ≈ 10^5 s · 1 M/day ≈ 12/s · ×2–3 for peak
availability
99.9% = 8.8 h down/year · 99.99% = 53 min · 99.999% = 5 min
SLO / SLA
the target you aim for / the promise with penalties
read-heavy vs write-heavy
the ratio decides your whole architecture
Say it like a pro

"Before I choose anything: peak QPS, data size, read-to-write ratio, and the availability target."

Scale up bigger box · limit Scale out LB unhealthy, skipped Load-balancing algorithms round robin · least connections · weighted · IP-hash / consistent hash (sticky) Layers L4 (TCP, fast) vs L7 (HTTP: route by path/header, terminate TLS) DNS / global (GeoDNS, anycast) → regional LB → servers
PLATE 2 — Make servers identical and disposable, then add more of them.
Chapter 2 · Scaling & load balancing

Stateless servers behind a balancer

A server that keeps no per-user state can be cloned freely. Put sessions in Redis or a signed token, files in object storage, and the LB can send any request to any server and drop a dead one.

# nginx as an L7 load balancer
upstream app {
    least_conn;
    server 10.0.0.11:3000 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:3000 max_fails=3 fail_timeout=10s;
    server 10.0.0.13:3000 backup;
}
server {
    listen 443 ssl;
    location / { proxy_pass http://app; proxy_set_header X-Forwarded-For $remote_addr; }
}
health check
LB probes /health and removes failing nodes
autoscaling
add/remove servers on CPU or queue depth; set min, max and cool-downs
single point of failure
the LB itself needs a pair (active-passive or anycast)
Say it like a pro

"Scale out stateless tiers behind an L7 balancer with health checks and autoscaling."

App Cache Database 1 get 2 miss → read DB 3 store with TTL Cache layers, nearest first browser → CDN edge → reverse proxy → app (Redis/Memcached) → DB buffer pool The hard part: invalidation TTL (simple) · delete/update on write · versioned keys (user:42:v7) stampede: many misses at once → lock one refill, add jitter to TTLs eviction: LRU / LFU when memory fills
PLATE 3 — The fastest query is the one you never send to the database.
Chapter 3 · Caching & CDNs

Remember answers close to the user

// cache-aside with Redis (Node)
async function getUser(id) {
  const key = `user:${id}`;
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit);                       // fast path

  const user = await db.one('SELECT * FROM users WHERE id=$1', [id]);
  await redis.set(key, JSON.stringify(user), { EX: 300 + Math.floor(Math.random() * 60) });  // TTL + jitter
  return user;
}
async function updateUser(id, data) {
  await db.query('UPDATE users SET ... WHERE id=$1', [id]);
  await redis.del(`user:${id}`);                          // invalidate on write
}
Write policyBehaviour
cache-asideapp fills on miss (most common)
write-throughwrite cache and DB together; consistent, slower writes
write-backwrite cache, flush later; fast, risk of loss
CDN
caches static files and images near users; set Cache-Control: public, max-age=31536000, immutable on hashed assets
hit ratio
the number to watch; below ~80% rethink keys and TTLs
Say it like a pro

"Cache-aside with TTL jitter, invalidate on write, and a CDN for everything static."

primary replicate replica 1 replica 2 writesreads (may lag) A B C consistent-hash ring In this order 1 · index and tune queries · 2 · cache · 3 · read replicas 4 · partition big tables · 5 · shard by key (user_id) · 6 · move hot data to specialized stores Sharding costs cross-shard joins and transactions get hard · pick the key carefully hot keys (celebrity user) · resharding pain → use consistent hashing SQL for relations and transactions · NoSQL for scale, flexible shape
PLATE 4 — Copy for reads, split for writes. Splitting is the last resort.
Chapter 4 · Scaling data

Replication, sharding, and CAP

replication
primary → replicas; reads scale, but replicas lag (replication lag → stale reads)
sharding
split rows across databases by a shard key (hash or range)
CAP
during a network partition choose consistency (refuse some requests) or availability (serve possibly stale data)
eventual consistency
replicas converge given time; fine for likes, not for balances
ACID vs BASE
strict transactions vs basically-available, soft-state, eventual
Store typeGood forExamples
relationalmoney, orders, relationsPostgreSQL, MySQL
key-valuesessions, cacheRedis, DynamoDB
documentflexible recordsMongoDB
wide-columnhuge write volumeCassandra
searchfull-text, filtersElasticsearch
objectfiles, mediaS3
-- route by shard key
shard = hash(user_id) % N          -- simple, but changing N moves almost everything
-- consistent hashing moves only ~1/N of keys when you add a node
Say it like a pro

"Replicas for reads, shard by user_id only when a single primary can't keep up, and consistency chosen per feature."

API queue: ▮▮▮▮▮▯▯ workers dead-letter queue Why queue work respond fast, process later (emails, thumbnails, reports) absorb spikes: the queue is a shock absorber decouple services: the sender doesn't wait for the receiver The rules delivery is at-least-once → make handlers idempotent (dedupe by message ID) retry with backoff · cap retries · alert on queue depth and age
PLATE 5 — A queue turns a spike into a line, and a line into steady work.
Chapter 5 · Queues, streams & events

Do slow things later, and reliably

// producer: enqueue and return immediately (BullMQ + Redis)
await emailQueue.add('welcome', { userId }, { attempts: 5, backoff: { type: 'exponential', delay: 2000 } });
res.status(202).json({ queued: true });

// worker: idempotent, safe to run twice
new Worker('email', async (job) => {
  if (await alreadySent(job.id)) return;          // dedupe
  await sendEmail(job.data.userId);
  await markSent(job.id);
}, { concurrency: 10 });
queue
one consumer gets each message (SQS, RabbitMQ)
log / stream
ordered, replayable, many independent consumers (Kafka, Kinesis)
pub/sub
one event, many subscribers (SNS, Redis Pub/Sub)
outbox pattern
write the event in the same DB transaction, publish it after: no lost or phantom events
exactly-once
really "at-least-once + idempotency"
Say it like a pro

"Return 202, queue the work, make consumers idempotent, and watch queue depth and age."

Bucketcap 10 tokens+2 / second −1 200 ✔ 429 Too Many ✘ API gatewayauth · rate limit · routing · TLS · logging, in one front door Protocol choices REST/JSON: universal · GraphQL: client picks fields · gRPC: fast, typed, internal WebSocket / SSE: server pushes (chat, live data) · webhooks: server calls you paginate with cursors · version the API · timeouts everywhere
PLATE 6 — A limit protects the system from a single noisy client, or a bug.
Chapter 6 · APIs, rate limiting & gateways

Protect the system from its callers

// token bucket in Redis (Lua keeps it atomic), sketch
local tokens = tonumber(redis.call('GET', KEYS[1]) or ARGV[1])        -- capacity
local now = tonumber(ARGV[3])
local last = tonumber(redis.call('GET', KEYS[2]) or now)
tokens = math.min(ARGV[1], tokens + (now - last) * ARGV[2])           -- refill
if tokens < 1 then return 0 end
redis.call('SET', KEYS[1], tokens - 1); redis.call('SET', KEYS[2], now); return 1
429 + Retry-After
tell clients when to try again
idempotency key
client-sent ID so a retried POST doesn't double-charge
cursor pagination
?after=abc123&limit=50: stable and fast, unlike deep OFFSET
timeouts + retries
retry with exponential backoff + jitter, only idempotent calls
circuit breaker
stop calling a failing dependency for a while so it can recover
Say it like a pro

"Rate-limit at the gateway, cursors for paging, idempotency keys for writes, and timeouts with circuit breakers on every call out."

Global LB Zone A app DB primary Zone B (fails → traffic shifts) app DB replica Assume everything fails disks, servers, networks, zones, dependencies, deploys, and people
PLATE 7 — Redundancy turns "a part broke" into "nobody noticed".
Chapter 7 · Reliability

Design for failure

  • Remove single points of failure: at least two of everything, in separate zones.
  • Failover: automatic promotion of replicas; practise it.
  • Degrade gracefully: if recommendations are down, still show the page.
  • Bulkheads & backpressure: one slow dependency must not consume every thread.
  • Safe deploys: canary or blue-green releases with fast rollback; feature flags.
  • Backups: RPO (how much data you can lose) and RTO (how long you can be down) set the plan.
TermPlain meaning
RPOhow far back you restore to
RTOhow long to be back up
MTTRaverage time to recover (the one to drive down)
error budgetallowed unreliability; spend it on shipping faster
1 − 0.999 = 0.1% budget ≈ 43 min/month at a 99.9% SLO
two zones at 99.9% each, either one enough → ≈ 99.9999% in theory (ignoring shared faults)
Say it like a pro

"Multi-zone, graceful degradation, canary deploys, and tested restores against explicit RPO and RTO."

Metricsnumbers over time Logswhat happened Tracesone request's path gateway 240 ms orders 160 ms DB 70 ms payments 110 ms ← slow Four golden signals latency (p50/p95/p99) · traffic (req/s) · errors (rate) · saturation (CPU, pool, queue) alert on symptoms users feel, not every internal blip
PLATE 8 — You can't fix what you can't see, and averages hide the pain.
Chapter 8 · Observability

Metrics, logs, traces

percentiles
p99 = the slowest 1% of requests. Average latency hides them.
correlation ID
one ID threaded through every service and log line
SLI → SLO → alert
measure a user-visible indicator, set a target, page on burn rate
tools
Prometheus + Grafana · OpenTelemetry · Loki/ELK · Jaeger/Tempo
# Prometheus alert: error rate above 2% for 5 minutes
- alert: HighErrorRate
  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
  for: 5m
  labels: { severity: page }
  annotations: { summary: "5xx rate above 2%" }
  • Dashboards per service: the four signals plus dependencies.
  • Runbooks linked from every alert: what it means, how to check, how to fix.
  • Blameless post-mortems; track action items to done.
Say it like a pro

"Golden-signal dashboards, trace IDs end to end, alerts on SLO burn, and a runbook for each page."

LB + RL App ×N Redis KV store tiny.link/aZ3k9Q → https://long.example/… requirements: 100 M new links/month · 10:1 reads · 5 yrs retention · <50 ms redirect estimate: ~40 writes/s, ~400 reads/s, 6 B links ≈ 3 TB ID: 7 chars base62 = 62^7 ≈ 3.5 T. Use a counter service or pre-generated key ranges. read path: cache → DB → 301/302 (302 if you want analytics) scale: shard by short_id hash · replicas · CDN for hot links extras: expiry, custom aliases, abuse/spam detection, click analytics via queue trade-off named: counter = no collisions, a coordination point; hashing = stateless, collisions
PLATE 9 — Run the method on a real problem and say your trade-offs out loud.
Chapter 9 · Putting it together

Worked example: a URL shortener

POST /v1/links   { "url": "https://long.example/very/long/path" }  →  201 { "short": "aZ3k9Q" }
GET  /aZ3k9Q                                                         →  302 Location: https://long.example/…

CREATE TABLE links (
  short_id   CHAR(7) PRIMARY KEY,
  long_url   TEXT NOT NULL,
  owner_id   BIGINT,
  created_at TIMESTAMPTZ DEFAULT now(),
  expires_at TIMESTAMPTZ
);
  1. Requirements and numbers (above) → read-heavy, tiny records, simple key lookups.
  2. Storage: key-value or sharded SQL; a cache absorbs hot links.
  3. ID generation: base62 of a distributed counter, or random with collision check.
  4. Failure and abuse: rate limits, URL validation, safe-browsing checks, expiry cleanup.
  5. Operate: golden-signal dashboards, canary deploys, tested restores.
practise on
chat app · news feed · rate limiter · file storage · notification system
read next
"Designing Data-Intensive Applications" (Kleppmann) · the Google SRE book
Say it like a pro

"I'll confirm requirements, estimate the load, sketch read and write paths, then deep-dive the bottleneck and name the trade-offs."