GPU Optimization

The LLM serving landscape has fundamentally shifted. What was once a simple choice between HuggingFace Transformers and early optimization frameworks has evolved into a sophisticated ecosystem where three engines dominate: SGLang, vLLM, and LMDeploy. The throughput gap between them—up to 29%—translates to tens of thousands of dollars in monthly GPU costs at production scale. This isn’t just about speed. Each engine embodies a fundamentally different philosophy about how to solve the same problems: memory fragmentation, computation redundancy, and the tension between latency and throughput. Understanding these architectures is essential for making the right deployment decision. ...

In 2022, training a transformer with 16K context length required either massive GPU memory or accepting severe approximations. Standard attention’s memory grew quadratically with sequence length—a 32K context demanded over 4GB just for intermediate attention matrices. Then Flash Attention arrived, reducing memory from $O(N^2)$ to $O(N)$ while computing exact attention, not an approximation. This breakthrough enabled GPT-4’s 128K context window, Llama’s extended sequences, and virtually every modern long-context LLM. The key insight wasn’t algorithmic cleverness alone—it was understanding that on modern GPUs, memory bandwidth, not compute, is the bottleneck. ...