Reinforcement Learning

A math student solves a complex integration problem. Her final answer is correct, but halfway through, she made a sign error that accidentally canceled out in the next step. The teacher gives full marks—after all, the answer is right. But should it count? This scenario captures the fundamental flaw in how we’ve traditionally evaluated Large Language Model (LLM) reasoning: Outcome Reward Models (ORMs) only check the final destination, ignoring whether the path was sound. Process Reward Models (PRMs) represent a paradigm shift—verifying every step of reasoning, catching those hidden errors that coincidentally produce correct answers, and enabling the test-time scaling that powers reasoning models like OpenAI’s o1 and DeepSeek-R1. ...

On January 20, 2025, DeepSeek released R1—a 671B parameter Mixture-of-Experts model that achieved something remarkable: matching OpenAI’s o1 on reasoning benchmarks while being fully open-source. The breakthrough wasn’t just in scale or architecture, but in a fundamentally different approach to training reasoning capabilities: Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm that eliminates the need for reward models while enabling sophisticated reasoning behaviors to emerge naturally. The Problem with Traditional LLM Training Standard large language models excel at pattern matching and next-token prediction, but struggle with tasks requiring multi-step logical deduction, self-correction, and complex problem decomposition. Chain-of-thought prompting helped, but it required extensive human-annotated demonstrations and still couldn’t match the systematic reasoning humans employ. ...