KV Cache

The quadratic complexity of self-attention has haunted transformer architecture since its inception. As context windows expanded from 2K to 1M tokens, the O(N²) attention computation transformed from an annoyance into an existential bottleneck. Yet a counterintuitive discovery emerged in 2025-2026: computing only 5-20% of attention weights can match or exceed full attention performance. This isn’t compression with acceptable loss—it’s the revelation that transformers have been computing billions of unnecessary operations. The mathematics behind this phenomenon, and the engineering that exploits it, represents one of the most significant advances in LLM efficiency. ...

The LLM serving landscape has fundamentally shifted. What was once a simple choice between HuggingFace Transformers and early optimization frameworks has evolved into a sophisticated ecosystem where three engines dominate: SGLang, vLLM, and LMDeploy. The throughput gap between them—up to 29%—translates to tens of thousands of dollars in monthly GPU costs at production scale. This isn’t just about speed. Each engine embodies a fundamentally different philosophy about how to solve the same problems: memory fragmentation, computation redundancy, and the tension between latency and throughput. Understanding these architectures is essential for making the right deployment decision. ...

In March 2024, a team of researchers attempted to deploy a 70-billion parameter language model on a single NVIDIA H100 GPU with 80GB of VRAM. The model weights alone consumed approximately 140GB in FP16—already exceeding their hardware capacity. But even after applying 4-bit quantization to squeeze the weights down to ~40GB, the system still ran out of memory when processing contexts beyond 8,000 tokens. The culprit wasn’t the model size. It was something far more insidious: the KV cache. ...