Mixture-of-Experts

For years, the dominant approach to multimodal AI followed a simple recipe: take a pre-trained vision encoder (CLIP, SigLIP), bolt it onto a pre-trained LLM through an adapter layer, and fine-tune the connection. This “late-fusion” paradigm powered everything from GPT-4V to LLaVA, delivering impressive results with remarkable sample efficiency. But a fundamental question lingered: was this architectural shortcut an inherent advantage, or merely a convenient workaround? The answer arrived in 2025 with a paradigm shift that’s rewriting the rules of multimodal AI. Native multimodal models—trained from scratch on all modalities simultaneously—are proving that early-fusion architectures don’t just match late-fusion approaches; they exceed them in efficiency, scalability, and ultimately, capability. ...

When DeepSeek-V3 was released in December 2024, it achieved something remarkable: a 671-billion-parameter model that activates only 37 billion parameters per token. This isn’t a magic trick—it’s the power of Mixture of Experts (MoE), an architectural paradigm that has quietly become the backbone of nearly every frontier large language model. The math is compelling. A dense 671B model would require approximately 1,342 TFLOPs per token during inference. DeepSeek-V3 achieves comparable performance with roughly 74 TFLOPs—an 18x reduction in compute. This isn’t incremental optimization; it’s a fundamental rethinking of how neural networks scale. ...