How Vision Language Models Actually Work: The Architecture Behind AI's Ability to See

When GPT-4V describes a meme’s irony or Claude identifies a bug in a screenshot, something remarkable happens: an architecture designed purely for text somehow “sees” and “understands” images. The magic isn’t in teaching language models to process pixels directly—it’s in a clever architectural bridge that transforms visual data into something language models already understand: tokens.

Vision Language Models (VLMs) represent one of the most impactful innovations in modern AI, yet their architecture remains surprisingly underexplored compared to their text-only cousins. Let’s dissect how these systems actually work, from the moment an image enters the model to the final text output.

The Fundamental Challenge: Different Modalities, One Model

Language models operate on discrete tokens—sequences of integers representing words or subwords. Images, by contrast, are continuous grids of pixel values. A 224×224 RGB image contains 150,528 numbers that don’t naturally map to any vocabulary.

The key insight behind VLMs is that we don’t need to teach an LLM about pixels. Instead, we can translate images into a “language” the LLM already understands: sequences of embeddings with the same dimensionality as text tokens. This translation happens through a carefully designed pipeline.

The Three-Component Architecture

Every modern VLM follows a tripartite structure:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Vision Encoder │───▶│   Projector     │───▶│       LLM       │
│   (ViT/CLIP)    │    │  (MLP/Q-Former) │    │   (Transformer) │
└─────────────────┘    └─────────────────┘    └─────────────────┘
        │                      │                      │
    Image → Patches      Visual → Language       Token Sequence
      → Features        Embedding Space         → Text Output

Component 1: Vision Encoder

The vision encoder converts raw images into semantically meaningful feature vectors. Most VLMs use Vision Transformers (ViT) rather than CNNs because transformers naturally produce sequence-like outputs.

How ViT Processes Images:

1. Patchification: An image is divided into fixed-size patches (typically 14×14 or 16×16 pixels). A 224×224 image with 14×14 patches produces 256 patches.

2. Linear Projection: Each flattened patch (14×14×3 = 588 values) is projected through a learned matrix into the embedding dimension (e.g., 1024 for CLIP-ViT-L).

3. Positional Encoding: Since patches lose spatial information after flattening, learnable or sinusoidal positional embeddings are added.

4. Transformer Processing: The patch sequence passes through standard transformer layers with self-attention.

CLIP vs SigLIP:

Most VLMs use CLIP-trained vision encoders, but SigLIP is increasingly preferred. The difference lies in the training objective:

• CLIP uses InfoNCE loss with softmax normalization:

  $$\mathcal{L}_{\text{CLIP}} = -\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(\text{sim}(v_i, t_i)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(v_i, t_j)/\tau)}$$

• SigLIP replaces this with a simpler sigmoid loss:

  $$\mathcal{L}_{\text{SigLIP}} = -\frac{1}{N^2}\sum_{i,j}\log\sigma(\text{sim}(v_i, t_j) \cdot y_{ij})$$

Where $y_{ij} = 1$ for matching pairs and $-1$ otherwise. This eliminates the global softmax dependency, enabling better performance at smaller batch sizes and improved memory efficiency.

Component 2: The Projector (Connector)

The projector is the crucial bridge between vision and language spaces. Vision encoder outputs have dimensions like [batch, num_patches, vision_dim], but the LLM expects [batch, seq_len, llm_dim]. The projector performs this transformation.

Evolution of Projector Designs:

MLP Projector (LLaVA Style):

The simplest approach uses a two-layer MLP:

class MLPPProjector(nn.Module):
    def __init__(self, vision_dim=1024, llm_dim=4096, hidden_dim=4096):
        super().__init__()
        self.fc1 = nn.Linear(vision_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, llm_dim)
        self.act = nn.GELU()
    
    def forward(self, vision_features):
        # vision_features: [batch, num_patches, vision_dim]
        x = self.act(self.fc1(vision_features))
        return self.fc2(x)  # [batch, num_patches, llm_dim]

Q-Former (BLIP-2 Style):

The Q-Former is more sophisticated, using learnable query embeddings that attend to vision features:

class QFormer(nn.Module):
    def __init__(self, num_queries=32, vision_dim=1024, llm_dim=4096):
        super().__init__()
        self.queries = nn.Parameter(torch.randn(num_queries, llm_dim))
        self.cross_attention = nn.MultiheadAttention(llm_dim, num_heads=8)
        self.self_attention = nn.MultiheadAttention(llm_dim, num_heads=8)
        self.ffn = nn.Sequential(
            nn.Linear(llm_dim, llm_dim * 4),
            nn.GELU(),
            nn.Linear(llm_dim * 4, llm_dim)
        )
    
    def forward(self, vision_features):
        # Cross-attention: queries attend to vision features
        queries = self.queries.unsqueeze(0).expand(vision_features.size(0), -1, -1)
        attended, _ = self.cross_attention(queries, vision_features, vision_features)
        # Self-attention among queries
        refined, _ = self.self_attention(attended, attended, attended)
        return self.ffn(refined)

The Q-Former compresses variable-length vision features into a fixed number of query outputs (typically 32-64 tokens), providing consistent input size regardless of image resolution.

Component 3: The Language Model

The LLM receives visual tokens interleaved with text tokens and processes them identically. A conversation might look like:

[VISUAL_TOKEN_0]...[VISUAL_TOKEN_255][TEXT_TOKEN_USER_QUESTION]...[TEXT_TOKEN_RESPONSE]

The LLM doesn’t “know” which tokens are visual—they’re all just embeddings. The semantic understanding of visual content emerges from the alignment learned during training.

How Images Become Tokens: The Tokenization Pipeline

Let’s trace a concrete example: processing a 336×336 image in LLaVA-1.5.

Step 1: Patch Embedding

Input: 336×336×3 RGB image
Patch size: 14×14
Number of patches: (336/14)² = 576 patches
Output: [1, 576, 1024] vision features

Step 2: Projector Transformation

Input: [1, 576, 1024] vision features
MLP projection
Output: [1, 576, 4096] visual tokens (matching Vicuna's embedding dim)

Step 3: Token Concatenation

Visual tokens: [1, 576, 4096]
Text tokens for "What's in this image?": [1, 5, 4096]
Combined sequence: [1, 581, 4096]

The LLM processes this 581-token sequence normally, with attention operating across both visual and text tokens.

Fusion Strategies: Early vs Cross-Attention

VLMs employ two primary fusion strategies with distinct trade-offs:

Early Fusion (Concatenation)

Used by LLaVA, InternVL, Qwen-VL. Visual tokens are prepended or inserted into the text sequence:

Advantages:

• Simplicity—no architectural changes to the LLM
• Full bidirectional attention between all modalities
• Easy to implement and scale

Disadvantages:

• Quadratic attention complexity: $O((n_{vision} + n_{text})^2)$
• Limited context window: 576 visual tokens eat into the 4096 token budget

Cross-Attention Fusion

Used by Flamingo, IDEFICS. Visual tokens are accessed through dedicated cross-attention layers:

class GatedCrossAttentionLayer(nn.Module):
    def forward(self, text_features, vision_features):
        # Text queries attend to vision keys/values
        cross_attn_out = self.cross_attention(
            text_features, vision_features, vision_features
        )
        # Gating mechanism for training stability
        gate = torch.tanh(self.gate_param)
        return text_features + gate * cross_attn_out

Advantages:

• Attention complexity: $O(n_{text} \times n_{vision})$
• Can handle unlimited visual tokens
• Preserves text-only capabilities better

Disadvantages:

• Requires modifying LLM architecture
• More complex training
• Less bidirectional interaction

The Two-Stage Training Pipeline

VLM training follows a carefully designed curriculum:

Stage 1: Vision-Language Alignment

Goal: Connect the pre-trained vision encoder to the pre-trained LLM.

Data: Large-scale image-text pairs (millions to billions).

What’s trained: Only the projector (typically MLP). Vision encoder and LLM remain frozen.

Loss: Next-token prediction on image captions:

{
  "image": "example.jpg",
  "conversations": [
    {"role": "user", "content": "<image>\nDescribe this scene."},
    {"role": "assistant", "content": "The image shows..."}
  ]
}

def naive_dynamic_resolution(image):
    h, w = image.shape[:2]
    # Calculate patches dynamically
    patch_h = h // 14  # 14 is patch size
    patch_w = w // 14
    # Produce exactly (patch_h * patch_w) visual tokens
    # A 1080p image produces ~11,000 tokens
    return encode_patches(image, patch_h, patch_w)