LLM Inference

The quadratic complexity of self-attention has haunted transformer architecture since its inception. As context windows expanded from 2K to 1M tokens, the O(N²) attention computation transformed from an annoyance into an existential bottleneck. Yet a counterintuitive discovery emerged in 2025-2026: computing only 5-20% of attention weights can match or exceed full attention performance. This isn’t compression with acceptable loss—it’s the revelation that transformers have been computing billions of unnecessary operations. The mathematics behind this phenomenon, and the engineering that exploits it, represents one of the most significant advances in LLM efficiency. ...

The smartphone in your pocket has more computing power than the entire NASA control room that guided Apollo 11 to the Moon. Yet until 2024, running a useful language model entirely on that device seemed like science fiction. The revolution that made it possible wasn’t a single breakthrough—it was a cascade of engineering innovations that fundamentally rethought how neural networks run on constrained hardware. The Memory Bandwidth Abyss The first and most brutal constraint facing on-device LLMs isn’t compute—it’s data movement. When you run a 7-billion parameter model on an H100 GPU, you’re working with memory bandwidth of 3.35 TB/s. A flagship smartphone in 2026? You get 50-90 GB/s through its LPDDR5X memory. That’s a 30-50x gap, and it dominates every architectural decision. ...

The LLM serving landscape has fundamentally shifted. What was once a simple choice between HuggingFace Transformers and early optimization frameworks has evolved into a sophisticated ecosystem where three engines dominate: SGLang, vLLM, and LMDeploy. The throughput gap between them—up to 29%—translates to tens of thousands of dollars in monthly GPU costs at production scale. This isn’t just about speed. Each engine embodies a fundamentally different philosophy about how to solve the same problems: memory fragmentation, computation redundancy, and the tension between latency and throughput. Understanding these architectures is essential for making the right deployment decision. ...

The sequential nature of autoregressive language models creates a fundamental bottleneck: generating each token requires a full forward pass through billions of parameters. A 70B parameter model processing a single token must load roughly 140GB of weights from memory (FP16), and memory bandwidth—not compute—becomes the limiting factor. This is why a 70B model might generate only 20-30 tokens per second on an H100, despite the GPU being capable of orders of magnitude more computation. ...

In March 2024, a team of researchers attempted to deploy a 70-billion parameter language model on a single NVIDIA H100 GPU with 80GB of VRAM. The model weights alone consumed approximately 140GB in FP16—already exceeding their hardware capacity. But even after applying 4-bit quantization to squeeze the weights down to ~40GB, the system still ran out of memory when processing contexts beyond 8,000 tokens. The culprit wasn’t the model size. It was something far more insidious: the KV cache. ...