Large Language Models

Fine-tuning a 7-billion parameter model used to demand 100+ GB of VRAM—roughly the memory of four A100 GPUs. Today, the same task runs on a consumer RTX 4090 with 24 GB. This 4× reduction didn’t come from better hardware; it came from a mathematical insight about the structure of neural network adaptations. Low-Rank Adaptation (LoRA), introduced by Microsoft in 2021, fundamentally changed how we think about model fine-tuning. The core idea is deceptively simple: instead of updating all parameters, inject small trainable matrices that modify the model’s behavior. But behind this simplicity lies deep connections to linear algebra, information theory, and the geometry of neural network weight spaces. ...

The race for larger context windows has defined LLM development for years. From GPT-4’s 128K tokens to Gemini’s 1M and beyond, the assumption has been simple: more context equals better performance. But a January 2026 paper from MIT CSAIL challenges this assumption entirely. Recursive Language Models (RLMs) don’t expand the context window—they render it irrelevant by treating prompts as external environments that models can programmatically explore, decompose, and recursively process. ...

Every transformer you’ve ever used treats every token with the same computational respect. Whether processing “the” or untangling complex mathematical reasoning, the model devotes identical FLOPs to each position in the sequence. This uniform allocation isn’t a design choice—it’s a constraint baked into the transformer architecture from its inception. In April 2024, researchers from Google DeepMind, McGill University, and Mila demonstrated that this constraint is not only unnecessary but actively wasteful. Their proposed Mixture-of-Depths (MoD) framework reveals a startling truth: transformers can learn to dynamically allocate compute across tokens, achieving 50% faster inference with equivalent performance. ...

When you ask ChatGPT about your company’s internal documents, it hallucinates. When you ask about events after its training cutoff, it fabricates. These aren’t bugs—they’re fundamental limitations of parametric knowledge encoded in model weights. Retrieval-Augmented Generation (RAG) emerged as the solution, but naive implementations fail spectacularly. This deep dive explores how to architect RAG systems that actually work. The Knowledge Encoding Problem Large Language Models encode knowledge in two ways: parametric (weights) and non-parametric (external data). Parametric knowledge is fast but frozen at training time, prone to hallucination, and impossible to update without retraining. Non-parametric knowledge—RAG’s domain—solves all three problems at the cost of latency and complexity. ...

For years, the AI community operated under a seemingly unshakeable assumption: the remarkable capabilities of large language models—from in-context learning to instruction following—inherently depend on autoregressive architectures. GPT, LLaMA, Claude, and virtually every dominant LLM shares the same fundamental design: predict the next token given all previous tokens. But what if this assumption was wrong? In February 2025, a paper from researchers at Renmin University of China challenged this paradigm with striking empirical evidence. LLaDA (Large Language Diffusion with mAsking), an 8B-parameter model trained entirely from scratch using diffusion processes, achieved performance competitive with LLaMA3 8B across diverse benchmarks. More remarkably, it solved problems that have plagued autoregressive models for years—the reversal curse being the most prominent. This isn’t merely an architectural curiosity; it’s a fundamental re-examination of how language models can learn and reason. ...

The Open LLM Leaderboard tells a surprising story: many top-performing models aren’t trained at all. They’re merged. A 7B parameter model, created by strategically blending weights from existing fine-tuned models, can outperform models 10x its size. This isn’t alchemy—it’s mathematics. Model merging represents a paradigm shift in how we think about model development. Instead of investing millions in GPU hours for training, practitioners are discovering that the collective intelligence embedded in existing open-source models can be combined to create something greater than the sum of its parts. The technique requires no gradients, no backward passes, and no training data. Just arithmetic operations on weight tensors. ...

For years, the path to better LLMs seemed straightforward: more parameters, more training data, more compute. The scaling laws articulated by Kaplan et al. and refined by Chinchilla painted a clear picture—performance improved predictably with model size. Then OpenAI released o1, and suddenly the rules changed. A model that “thinks longer” at inference time was solving problems that eluded models 10x its size. The breakthrough wasn’t just engineering—it was a fundamental shift in how we think about compute allocation. The question flipped from “how big should we train?” to “how long should we let it think?” ...