System Design Guide 2026: Load Balancing, Caching, CDN, Databases, Message Queues, and Scaling

Distributes incoming network traffic across multiple servers to ensure no single server is overwhelmed. Algorithms: round-robin, least connections, IP hash, weighted. Layer 4 (TCP) or Layer 7 (HTTP). Software: NGINX, HAProxy. Managed: AWS ALB/NLB, GCP LB.

Content Delivery Network: geographically distributed servers that cache and serve content from edge locations near users. Reduces latency, offloads origin servers, and protects against DDoS. Static (images, CSS, JS) and dynamic (API acceleration). Providers: Cloudflare, CloudFront, Fastly.

Storing frequently accessed data in fast storage (memory) to reduce database load and latency. Levels: browser cache, CDN, application cache (Redis), database query cache. Strategies: cache-aside, write-through, write-behind. Invalidation is the hardest problem.

Partitioning data across multiple database instances. Each shard holds a subset of data determined by a shard key. Enables horizontal scaling of databases. Challenges: cross-shard queries, hot spots, rebalancing. Tools: Vitess (MySQL), Citus (PostgreSQL).

A copy of the primary database that serves read queries. Writes go to the primary, replicas sync asynchronously. Eventual consistency (slight lag). Simple scaling for read-heavy workloads. All major databases support replicas.

Async communication between services. Producer sends messages, consumer processes them. Decouples services, handles spikes, enables retry. Types: point-to-point (SQS), pub/sub (Kafka). Tools: Kafka, RabbitMQ, SQS, NATS.

Adding more machines to handle increased load. Requires: stateless services, load balancing, distributed data. More resilient than vertical scaling (no single point of failure). Standard approach for web applications.

Adding more resources (CPU, RAM) to an existing machine. Simpler but has hardware limits and creates a single point of failure. Good for: databases hard to distribute, initial scaling.

In a distributed system, you can guarantee at most two of: Consistency (all reads return latest write), Availability (every request gets a response), Partition tolerance (system works during network partitions). Since partitions are inevitable, choose CP (consistent but may be unavailable) or AP (available but may return stale data).

A hashing scheme where adding/removing nodes requires redistributing only K/N keys (K=keys, N=nodes). Nodes are placed on a hash ring. Keys map to the next node clockwise. Used by: Redis Cluster, DynamoDB, CDNs, distributed caches.

Controlling request frequency per client. Algorithms: token bucket (allows burst), sliding window (precise), fixed window (simple). Return HTTP 429. Implement at: API gateway, load balancer, or application. Prevents abuse and protects backend services.

Stops calling failing services to prevent cascading failures. Closed (normal) -> Open (fail fast) -> Half-Open (trial). Trips when failure rate exceeds threshold. Combine with retry, timeout, and fallback patterns.

Database transaction properties: Atomicity (all or nothing), Consistency (valid state transitions), Isolation (concurrent transactions do not interfere), Durability (committed data survives crashes). Standard for relational databases. NoSQL databases often relax ACID for scalability.

Alternative to ACID for distributed systems: Basically Available (system is available), Soft state (state may change without input), Eventually consistent (system converges to consistent state). Used by NoSQL databases, event-driven systems.

A server that sits in front of web servers and forwards client requests. Provides: load balancing, SSL termination, caching, compression, and security (hiding origin servers). NGINX, HAProxy, Caddy, Traefik. Every production deployment should use one.

Domain Name System: translates domain names to IP addresses. DNS-based load balancing distributes traffic geographically. TTL controls cache duration. DNS failover for disaster recovery. Services: Route 53, Cloudflare DNS, Google Cloud DNS.

Single entry point for API requests. Handles: routing, auth, rate limiting, transformation, caching. Decouples clients from backend topology. Tools: Kong, AWS API Gateway, Traefik, Envoy.

An operation producing the same result regardless of repetition count. GET, PUT, DELETE are naturally idempotent. POST is not. Use idempotency keys for non-idempotent operations to prevent duplicates.

A consistency model where updates propagate asynchronously. The system converges to a consistent state over time. Standard in distributed systems, NoSQL databases, and event-driven architectures. Trade-off: higher availability at the cost of temporary staleness.

Every read returns the most recent write. Requires synchronous replication or consensus (Raft, Paxos). Lower availability during partitions. Used by: relational databases, distributed consensus systems (etcd, ZooKeeper).

Dividing data across multiple nodes. Horizontal (sharding): different rows on different nodes. Vertical: different columns on different nodes. Range-based: partition by key range. Hash-based: partition by hash of key. Enables horizontal scaling.

Copying data across multiple nodes for redundancy and read scaling. Single-leader: one primary handles writes, replicas sync. Multi-leader: multiple nodes accept writes (conflict resolution needed). Leaderless: any node accepts reads/writes (quorum-based).

Agreement among distributed nodes on a single value. Algorithms: Raft (etcd, Consul), Paxos (Google Spanner), Zab (ZooKeeper). Used for: leader election, distributed locks, configuration management. Requires majority of nodes (quorum) to agree.

A probabilistic data structure that tests whether an element is in a set. False positives possible, false negatives impossible. Very space-efficient. Used by: databases (check if key exists before disk read), CDNs, spam filters, web crawlers.

A log where all changes are written before being applied to the database. Enables crash recovery (replay log after crash), replication (send log to replicas), and CDC (read log for change events). Used by PostgreSQL, MySQL, Kafka.

The process of choosing one node as the leader (primary) in a distributed system. The leader handles writes or coordination. If the leader fails, a new election occurs. Algorithms: Raft, Paxos, Bully. Tools: etcd, ZooKeeper, Consul.

A flow control mechanism where a system signals upstream that it cannot handle more load. Prevents overwhelming downstream services. Implementation: reject requests (429), queue and process slowly, reduce producer rate. Essential for streaming systems.

A centralized repository for structured and unstructured data at any scale. Store raw data and transform on read (schema-on-read). Technologies: S3, HDFS, Delta Lake, Apache Iceberg. Used for: analytics, ML training, data science exploration.

A system optimized for analytical queries on structured data. Schema-on-write, columnar storage, pre-aggregated. Technologies: Snowflake, BigQuery, Redshift, ClickHouse. Used for: business intelligence, reporting, dashboards.

Storing state changes as immutable events rather than current state. Current state derived by replaying events. Benefits: full audit trail, temporal queries, event replay. Challenges: schema evolution, snapshot management.

Command Query Responsibility Segregation: separate models for reads and writes. Write model handles commands, read model optimized for queries. Can use different databases. Benefits: independent scaling, optimized data models.

Mechanism for services to find each other in dynamic environments. Client-side: query registry (Consul, Eureka). Server-side: load balancer routes (K8s Service). DNS-based: service names resolve to IPs.

Understanding system state from external outputs. Three pillars: metrics (Prometheus), logs (ELK/Loki), traces (Jaeger/Tempo). OpenTelemetry unifies instrumentation. Essential for operating distributed systems.

SLI (Service Level Indicator): measurable metric (latency, availability). SLO (Service Level Objective): target for SLI (99.9% availability). SLA (Service Level Agreement): contractual commitment with consequences. Error budget = 1 - SLO.

A methodology for building SaaS applications. Principles: codebase in version control, explicit dependencies, config in environment, backing services as resources, build/release/run separation, stateless processes, port binding, concurrency via processes, disposability, dev/prod parity, logs as event streams, admin processes.

Two identical environments. Deploy to idle, test, switch traffic. Instant rollback by switching back. Requires double infrastructure during transition.

Deploy new version to small traffic subset (5%), monitor metrics, gradually increase. Catches issues under real traffic. Tools: Istio, Argo Rollouts, Flagger.

System continues to function with reduced capability when components fail. Example: show cached product catalog when catalog service is down. Provide fallback responses, disable non-essential features, prioritize core functionality.

Many clients simultaneously requesting the same resource after cache expiration or service recovery. Causes: cache stampede, service restart. Prevention: jitter on cache TTL, request coalescing, circuit breaker, staggered retry with backoff.

A node or partition receiving disproportionately more traffic than others. Causes: poor shard key selection (celebrity user, popular product), time-based partitioning during peak hours. Prevention: random suffix on keys, pre-splitting, separate hot data.

Server pushes notifications to a client URL when events occur. Client registers callback URL; server sends POST on event. Used for: payment confirmations, CI/CD triggers, integrations. Must handle: retries, signature verification, idempotency.

Full-duplex communication over a single TCP connection. Bidirectional real-time data. Used for: chat, live dashboards, gaming, collaborative editing. Alternative: SSE (Server-Sent Events) for server-to-client only.

High-performance RPC framework using Protocol Buffers and HTTP/2. Features: bidirectional streaming, code generation, strong typing, smaller payloads than JSON. Used for: internal microservice communication, mobile backends.

Storage for unstructured data (files, images, videos) with flat namespace and HTTP API. Services: AWS S3, GCS, Azure Blob, MinIO (self-hosted). Features: durability (11 nines), versioning, lifecycle policies, event notifications. Standard for storing user uploads and static assets.

Maintaining a pool of reusable database connections. Avoids the overhead of creating/destroying connections per request. Tools: PgBouncer (PostgreSQL), ProxySQL (MySQL), application-level pools (HikariCP). Essential for performance at scale.

Replicating data across multiple geographic regions. Benefits: low latency for global users, disaster recovery. Challenges: cross-region consistency, compliance (data residency). Services: CockroachDB, Spanner, DynamoDB Global Tables, Cosmos DB.

Flow control mechanism where a system signals upstream that it cannot handle more load. Prevents overwhelming downstream. Implementations: reject with 429, bounded queues, reactive streams (Project Reactor, RxJS).

HTTP mechanism for selecting response format. Client sends Accept header, server responds in requested format. Used for: JSON vs XML, language selection, API versioning via media types.

Endpoint (/health, /ready) reporting service status. Liveness: is the process running? Readiness: can it handle traffic? Used by load balancers and orchestrators for routing and restart decisions.

A lock mechanism that works across multiple machines. Ensures only one process can access a resource at a time. Implementations: Redis (Redlock), ZooKeeper, etcd. Use sparingly: prefer idempotent designs over distributed locks.