System Design

Imagine you’re running a service with 10 servers, each capable of handling 1,000 requests per second. You set up a round-robin load balancer—simple, elegant, fair. Every server gets its turn in sequence. Traffic flows smoothly until suddenly, at 2 AM, your monitoring alerts start screaming. Half your servers are overwhelmed, queues are growing, latencies are spiking, and the other half of your servers are nearly idle. What went wrong? The servers weren’t identical. Three of them were newer machines with faster CPUs and more memory. Three were legacy boxes running older hardware. The round-robin algorithm, in its mechanical fairness, sent exactly the same number of requests to a struggling legacy server as it did to a powerful new one. The legacy servers couldn’t keep up, requests piled up in their queues, and eventually they started timing out—cascading into a partial outage that woke up half your engineering team. ...

On September 20, 2015, Amazon DynamoDB in US-East-1 went dark for over four hours. The root cause wasn’t a hardware failure or a cyberattack—it was a feedback loop. Storage servers couldn’t retrieve their partition assignments from a metadata service, so they retried. The metadata service became overwhelmed. More timeouts. More retries. More overload. Engineers eventually had to firewall the metadata service from storage servers entirely, effectively taking DynamoDB offline to break the cycle. ...

At midnight UTC on January 1, 2017, deep inside Cloudflare’s custom RRDNS software, a number went negative when it should have always been at least zero. This single value caused DNS resolutions to fail across Cloudflare’s global network. The culprit? A leap second—one extra tick of the clock that most people never noticed. The bug revealed a fundamental truth that every programmer eventually learns the hard way: time is not what you think it is. It doesn’t flow uniformly forward. It jumps, skips, and occasionally rewinds. And if your code assumes otherwise, it will break in ways that are nearly impossible to predict. ...

A financial services company migrated their payment processing to a message queue architecture. The design seemed straightforward: producers publish payment requests, workers consume and process them. Six months later, they discovered their customers had been double-charged for approximately 3% of transactions. The queue was working exactly as configured—the problem was that “working” meant something different than they expected. Message queues occupy a strange position in distributed systems. They appear deceptively simple on the surface: put message in, get message out. But beneath that simplicity lies a maze of trade-offs involving durability, ordering, delivery guarantees, and failure modes. Understanding these trade-offs isn’t academic—it’s the difference between a reliable system and one that silently corrupts data. ...

In March 2013, Solomon Hykes demonstrated a new tool at PyCon that would fundamentally change how software gets deployed. He ran a simple command: docker run -i -t ubuntu /bin/bash. Within seconds, a complete Ubuntu environment appeared, ready to accept commands. The audience saw what looked like a lightweight virtual machine. What they were actually witnessing was something far more elegant: a single Linux process, wrapped in kernel features that had been maturing for over a decade. ...

A major e-commerce platform celebrated when their cache hit ratio hit 99.2%. Their dashboard showed beautiful green charts. Three days later, their database collapsed during a flash sale. The cache hit ratio never dropped below 98%. What went wrong? The team optimized for the wrong metric. While their cache served 99% of requests from memory, the 1% that missed were the most expensive queries—complex aggregations, joins across multiple tables, and expensive computations. A cache hit ratio tells you how often you avoid work, not how much work you’re avoiding. ...

The Oracle Real-World Performance group published a demonstration that should have changed how every developer thinks about connection pools. They took a system struggling with ~100ms average response times and reduced those times to ~2ms—a 50x improvement. They didn’t add hardware. They didn’t rewrite queries. They reduced the connection pool size from 2048 connections down to 96. Most developers configure connection pools based on intuition: more users means more connections, right? A typical production configuration sets the pool to 100, 200, or even 500 connections “just to be safe.” This intuition is precisely backwards. The correct question isn’t how to make your pool bigger—it’s how small you can make it while still handling your load. ...