Inference Optimization

The mathematics of neural network pruning has been studied since the 1980s, when Yann LeCun demonstrated that optimal brain damage could remove redundant weights without harming performance. Yet for decades, pruning remained a niche technique—the computational savings rarely justified the engineering effort. Large Language Models changed everything. A 70-billion parameter model requires approximately 140 GB of memory just to store weights in FP16. At 50% sparsity, that drops to 70 GB—but only if your inference engine can efficiently skip the zero weights. The potential savings are measured in hundreds of thousands of dollars per deployment. The question is no longer whether to prune, but how to do it without destroying the model’s capabilities. ...

For years, the next-token prediction (NTP) paradigm has been the unquestioned foundation of large language model training. Given a sequence of tokens $x_{1:t}$, the model learns to maximize $P(x_{t+1} | x_{1:t})$. Simple, elegant, and remarkably effective—until you realize the fundamental inefficiency baked into this approach. The problem is that transformers spend the same computational budget predicting filler words (“the”, “and”, “is”) as they do on information-carrying tokens (“quantum”, “entanglement”, “superposition”). Research from Apple and EPFL reveals that over 50% of English text consists of function words—linguistic glue that carries minimal semantic weight. Yet models trained on NTP treat every token with equal reverence, creating a massive computational inefficiency. ...

A 70-billion parameter model requires 140 GB of GPU memory in FP16. A consumer RTX 4090 has 24 GB. This arithmetic gap defined the boundary between “enterprise AI” and “what you can run at home” until quantization mathematics cracked the code. The counterintuitive reality: reducing precision from 16 bits to 4 bits—a 75% compression—often preserves over 95% of model quality. Not through magic, but through a profound understanding of how neural networks encode information. ...

The sequential nature of autoregressive language models creates a fundamental bottleneck: generating each token requires a full forward pass through billions of parameters. A 70B parameter model processing a single token must load roughly 140GB of weights from memory (FP16), and memory bandwidth—not compute—becomes the limiting factor. This is why a 70B model might generate only 20-30 tokens per second on an H100, despite the GPU being capable of orders of magnitude more computation. ...