When 1B Models Learn from Giants: The Complete Architecture of LLM Knowledge Distillation

The economics of Large Language Models present a brutal reality: GPT-4-level performance costs $0.03 per 1K tokens for input and $0.06 for output. Run that at scale—say, 10 million daily queries—and you’re burning $900,000 monthly. But here’s what’s fascinating: researchers have discovered that a 1.3B parameter model, properly distilled from a 175B teacher, can match 95% of its predecessor’s performance on specific tasks while costing 0.1% to run.

This isn’t magic. It’s knowledge distillation—a technique that has evolved from Geoffrey Hinton’s 2015 “dark knowledge” paper into a sophisticated ecosystem of methods that compress frontier AI capabilities into models small enough to run on your laptop.

The Distillation Paradigm: From Dark Knowledge to Bright Students

At its core, knowledge distillation flips the traditional training paradigm. Instead of learning from ground-truth labels, a smaller “student” model learns to mimic the behavior of a larger “teacher” model. The teacher’s outputs—whether logits, hidden states, or reasoning traces—become the curriculum.

The intuition is elegant: when a teacher model predicts “cat” for an image, it doesn’t just say “cat.” Its output distribution might be: cat (0.90), dog (0.07), fox (0.02), rabbit (0.01). This soft distribution contains “dark knowledge”—information about the similarity structure of the output space that hard labels discard. A student learning from this learns not just what the answer is, but why certain wrong answers are more wrong than others.

# The fundamental distillation loss
def distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):
    """
    Combine soft target loss (KL divergence) with hard target loss (cross-entropy)
    
    Args:
        student_logits: Raw outputs from student model
        teacher_logits: Raw outputs from teacher model  
        labels: Ground truth labels
        temperature: Softmax temperature (higher = softer distributions)
        alpha: Weight between distillation and hard target loss
    """
    # Soft targets: KL divergence between softened distributions
    soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)
    kld_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (temperature ** 2)
    
    # Hard targets: Standard cross-entropy
    hard_loss = F.cross_entropy(student_logits, labels)
    
    # Combined loss
    return alpha * kld_loss + (1 - alpha) * hard_loss

The temperature parameter $\tau$ controls the “softness” of the probability distribution. Higher temperatures produce softer distributions where more information about relative class similarities is preserved. The squaring of temperature ($\tau^2$) compensates for the gradient scaling that occurs when using softened targets.

White-Box vs Black-Box: The Two Worlds of Distillation

White-Box Distillation: Inside the Teacher’s Mind

White-box distillation assumes access to the teacher model’s internals—logits, hidden states, attention patterns. This is the regime where open-source models like LLaMA, Mistral, or Qwen serve as teachers, and where distillation achieves its most impressive compression ratios.

The key advantage is granularity. You’re not limited to mimicking outputs; you can transfer knowledge at multiple levels:

Black-Box Distillation: Learning from API Outputs

When the teacher is a proprietary model accessed via API (GPT-4, Claude, Gemini), you’re in black-box territory. You can only observe inputs and outputs—no logits, no hidden states, no attention weights.

This constraint has sparked an entire research direction. Methods like CycleAlign and GAD (GAN-style distillation) demonstrate that black-box distillation can achieve 85-90% of white-box performance through clever use of:

1. Output distribution approximation: Query the teacher multiple times with temperature > 0, collect responses, and approximate the probability distribution
2. Synthetic data generation: Use the teacher to generate training data that the student learns from directly
3. Iterative refinement: The student generates candidates, the teacher critiques, and the student improves

# Black-box distillation via response sampling
def black_box_distill(teacher_api, student_model, prompts, n_samples=10):
    """
    Approximate teacher distribution by sampling multiple responses
    """
    for prompt in prompts:
        # Sample multiple teacher responses
        teacher_responses = []
        for _ in range(n_samples):
            response = teacher_api.generate(prompt, temperature=0.7)
            teacher_responses.append(response)
        
        # Build approximate distribution
        response_counts = Counter(teacher_responses)
        approx_probs = {r: c/n_samples for r, c in response_counts.items()}
        
        # Train student to match this distribution
        student_loss = compute_distribution_matching_loss(
            student_model, prompt, approx_probs
        )

Logits-Level Distillation: The Mathematics of Soft Targets

The most common approach, logits-level distillation (also called response-based distillation), focuses on matching the teacher’s output distribution. The standard objective uses Kullback-Leibler divergence:

Where $z_t$ and $z_s$ are teacher and student logits, $\sigma$ is softmax, and $T$ is temperature.

However, recent research has challenged the supremacy of KL divergence. A 2025 study found that Mean Squared Error (MSE) on logits can outperform KL divergence, particularly when student and teacher have different architectures:

The explanation is nuanced: KL divergence assumes the student and teacher share the same vocabulary and output space. When vocabularies differ—or when the student’s capacity is significantly smaller—MSE provides a more stable optimization landscape.

Universal Logit Distillation (ULD): Bridging Different Vocabularies

A breakthrough came with Universal Logit Distillation, which solves the cross-tokenizer problem. When student and teacher have different vocabularies, standard KL divergence fails because the probability distributions have different support.

ULD reformulates distillation as an optimal transport problem:

Where $\Pi$ is the set of valid transport plans between teacher and student token spaces, and $C$ is a cost matrix measuring semantic similarity between tokens. This allows distillation between, say, a GPT-4 teacher and a LLaMA-based student with different tokenizers.

Feature-Based Distillation: Transferring Intermediate Representations

Output matching captures what the teacher predicts, but not how it reasons. Feature-based distillation addresses this by aligning intermediate representations between teacher and student.

The challenge is dimensionality mismatch: a 70B teacher and 7B student have different hidden dimensions. The standard solution uses projection layers:

class FeatureDistillation:
    def __init__(self, teacher_dim, student_dim, hidden_dim=None):
        self.projector = nn.Sequential(
            nn.Linear(student_dim, hidden_dim or teacher_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim or teacher_dim, teacher_dim)
        )
    
    def compute_loss(self, teacher_features, student_features):
        """
        Align student features to teacher feature space
        teacher_features: [batch, seq_len, teacher_dim]
        student_features: [batch, seq_len, student_dim]
        """
        projected_student = self.projector(student_features)
        return F.mse_loss(projected_student, teacher_features)

Flex-KD: Task-Driven Feature Selection

Not all features are equally important. Flex-KD (Flexible Knowledge Distillation) introduces task-aware feature selection:

1. Identify task-relevant neurons: Use gradient-based attribution to find which neurons in the teacher contribute most to the target task
2. Selective transfer: Only distill these “expert neurons” to the student
3. Dimensionality reduction: The student can have a smaller hidden dimension while still capturing task-critical information

Results on NLP benchmarks show Flex-KD achieves 98% of teacher performance with 10x fewer parameters, compared to 92% for vanilla feature distillation.

Chain-of-Thought Distillation: Teaching Reasoning

The most exciting frontier is distilling reasoning capabilities, not just outputs. When a teacher model uses chain-of-thought (CoT) reasoning, how do we transfer that thought process to a smaller student?

The Step-by-Step Approach

Distilling Step-by-Step (Microsoft Research, 2023) pioneered this direction. Instead of teaching the student just the final answer, it teaches the reasoning chain:

Teacher reasoning:
Question: If John has 3 apples and buys 2 more, then gives 1 to Mary, how many does he have?
Step 1: John starts with 3 apples
Step 2: He buys 2 more, so 3 + 2 = 5 apples
Step 3: He gives 1 to Mary, so 5 - 1 = 4 apples
Answer: 4

Student is trained to produce the full reasoning chain, not just "4"

The loss function becomes a weighted combination:

Where $\mathcal{L}_{reasoning}$ is the cross-entropy loss on the reasoning tokens and $\mathcal{L}_{answer}$ is the loss on the final answer token.

Distribution-Aligned Sequence Distillation (DASD)

A 2026 breakthrough, DASD-4B, demonstrates that a 4B parameter model can match DeepSeek-R1’s reasoning on mathematical problems through careful distillation of long chain-of-thought traces. The key insight: reasoning quality depends on the diversity of teacher reasoning paths.

DASD samples multiple reasoning paths from the teacher for each question, then aligns the student’s reasoning distribution to match the teacher’s distribution over valid paths:

# Simplified DASD training loop
def dasd_train_step(teacher, student, question, n_paths=5):
    # Sample diverse reasoning paths from teacher
    teacher_paths = [teacher.generate_reasoning(question, temp=0.7) 
                     for _ in range(n_paths)]
    
    # Score each path by final answer correctness
    valid_paths = [p for p in teacher_paths if is_correct(p, question)]
    
    if not valid_paths:
        return None  # Skip if teacher couldn't solve
    
    # Compute student loss against path distribution
    student_path = student.generate_reasoning(question)
    
    # KL divergence between student and teacher path distributions
    loss = path_distribution_kl(student_path, valid_paths)
    return loss

The CoT Distillation Paradox

A surprising finding from 2025 research: smaller models don’t always benefit from CoT distillation. When the student is too small (under 1B parameters), learning to reason can actually hurt performance on simple tasks—the model struggles to allocate capacity between reasoning and factual knowledge.

The solution is adaptive distillation: only use CoT distillation for questions where the teacher’s reasoning adds value, measured by a complexity classifier that predicts whether the question requires multi-step reasoning.

Self-Distillation: The Teacher Becomes the Student

What if the teacher and student are the same model? Self-distillation has emerged as a powerful technique for continual learning and model improvement without external supervision.

Self-Distillation Fine-Tuning (SDFT)

Introduced in January 2026, SDFT enables models to learn from their own high-quality outputs:

1. Generate demonstrations: The model produces multiple responses to a prompt
2. Filter by quality: Keep only the best responses (via self-scoring or external verifier)
3. Fine-tune on filtered data: The model learns from its own best outputs

def sdft_training_loop(model, prompts, verifier):
    for prompt in prompts:
        # Generate multiple candidates
        responses = model.generate(prompt, n=5, temperature=0.8)
        
        # Score and filter
        scores = [verifier.score(prompt, r) for r in responses]
        best_response = responses[np.argmax(scores)]
        
        # Fine-tune on best response (demonstration-conditioned)
        loss = model.compute_loss(prompt, best_response)
        loss.backward()
        optimizer.step()

The remarkable finding: SDFT can improve model performance without any new data. By learning from its own best outputs, the model internalizes its most successful reasoning patterns.

Preventing Catastrophic Forgetting

Self-distillation also addresses the catastrophic forgetting problem in continual learning. By using the pre-update model as a “teacher” for the post-update model, SDFT preserves previously learned capabilities:

This regularization prevents the model from deviating too far from its previous behavior while still adapting to new tasks.

The Practical Landscape: Frameworks and Tools

DistiLLM: Streamlined Distillation

The official PyTorch implementation of DistiLLM provides a unified framework for multiple distillation strategies:

from distillm import DistiLLM, DistillationConfig

# Configure distillation
config = DistillationConfig(
    teacher_model="meta-llama/Llama-3-70B",
    student_model="meta-llama/Llama-3-8B",
    distillation_type="multi_level",  # logits + features
    temperature=2.0,
    alpha=0.7,  # weight for distillation loss
    feature_layers=[16, 24, 32],  # which teacher layers to distill
)

# Train
distiller = DistiLLM(config)
distiller.train(train_data, eval_data, epochs=3)

Hugging Face Integration

The Transformers library now includes native distillation support through the DistillationTrainer:

from transformers import AutoModelForCausalLM, DistillationTrainer

teacher = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B")
student = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")

trainer = DistillationTrainer(
    teacher_model=teacher,
    student_model=student,
    args=training_args,
    train_dataset=train_dataset,
    temperature=2.0,
)

trainer.train()

Benchmarks: What Do You Actually Lose?

The critical question: how much capability is sacrificed in distillation? The answer depends heavily on the domain and distillation method.

The most striking result: on GSM8K (math reasoning), the 4B DASD-distilled model matches or exceeds its 671B teacher’s performance. This suggests that for well-defined reasoning tasks, the student can learn better than the teacher by focusing on high-quality reasoning paths.

The Trade-offs: What Distillation Cannot Capture

Despite impressive results, distillation has fundamental limitations:

1. Knowledge Degradation

Distillation transfers behaviors, not knowledge. A student model that learns to answer “What is the capital of France?” correctly may fail on “What country is Paris the capital of?” because it learned the specific pattern, not the underlying knowledge.

2. Distribution Shift Fragility

Distilled models excel on distributions similar to training data but can catastrophically fail on out-of-distribution inputs. A student distilled for medical Q&A may produce plausible-sounding but dangerous misinformation when faced with novel medical scenarios.

3. The “Clever Hans” Problem

Students can learn surface patterns that correlate with correct answers without understanding the underlying reasoning. In one experiment, a distilled model achieved 89% accuracy on a sentiment classification task by learning that reviews containing “not” are negative—failing when “not bad” appeared.

4. Compute-Optimal Trade-offs

Distillation requires significant upfront compute. Generating high-quality synthetic data from a frontier model costs $10,000-100,000 for a typical dataset. This investment only pays off if the resulting student model serves millions of queries.

Future Directions: Where Distillation Is Heading

Distillation for Multimodal Models

Vision-Language Models present unique challenges: visual features and text features must be distilled jointly. Early results from distilling GPT-4V into smaller VLMs show promising compression ratios (15-20x) with 85-90% capability retention.

Federated Distillation

Privacy-preserving distillation where multiple teachers contribute knowledge without sharing raw data. Each teacher computes gradients locally, and only aggregated gradient information is shared with the student.

Continual Distillation

Rather than one-time distillation, continually updating the student as the teacher improves. This requires efficient mechanisms to detect which capabilities have changed and selectively update the student.

The Bottom Line

Knowledge distillation has matured from an academic curiosity into a production-ready technology that makes frontier AI capabilities economically viable. A 1B model distilled from a 70B teacher costs $0.0001 per 1K tokens—300x cheaper than the teacher—while retaining 90%+ performance on specific tasks.

The key insight is that most of a large model’s parameters are insurance against edge cases. For well-defined domains, a properly distilled student can capture the teacher’s core capabilities with a fraction of the cost. As the field advances toward 1T+ parameter models, distillation will become not just an optimization, but a necessity for practical deployment.

The future belongs to hybrid systems: frontier models for novel, complex queries; distilled specialists for high-volume, well-defined tasks; and intelligent routing between them. In this architecture, distillation isn’t compression—it’s translation, making the insights of giants accessible to the masses.