Why SSH Doesn't Just Use Public Keys: The Hidden Architecture of Secure Shell

Every day, millions of developers type ssh user@server without a second thought. The connection establishes, the shell appears, and work begins. But beneath that familiar prompt lies one of the most elegant cryptographic protocols ever designed—a multi-layered system that somehow manages to be both simple enough for daily use and sophisticated enough to withstand decades of scrutiny.

The irony is striking: most people assume SSH “just uses public key cryptography.” After all, that’s what the ~/.ssh/id_rsa file is for, right? The reality is far more nuanced. SSH uses public keys for exactly one purpose—authentication—and a completely different mechanism for everything else. Understanding this distinction reveals why SSH has remained the gold standard for remote access since 1995.

A Protocol Born from a Password Sniffer

In early 1995, Tatu Ylönen was a researcher at Helsinki University of Technology when he discovered a password-sniffing attack on the university network. Someone was intercepting credentials as they traversed the network, capturing usernames and passwords in plaintext. The tools of the day—Telnet, rlogin, rsh, FTP—all transmitted authentication data without encryption.

Ylönen’s response was to design a replacement protocol from scratch. Within months, he had created SSH-1, which he released as freeware in July 1995. The software spread rapidly; by the end of 1995, it had 20,000 users across fifty countries.

When Ylönen needed a port number, he wrote to the Internet Assigned Numbers Authority (IANA), then run by Internet pioneer Jon Postel and Joyce K. Reynolds. His email on July 10, 1995, requested port 22—a number conveniently positioned between FTP’s port 21 and Telnet’s port 23. The next day, Joyce responded: “We have assigned port number 22 to ssh.” The protocol had found its home.

Three Layers, One Secure Channel

SSH’s architecture separates concerns into three distinct layers, each defined by its own RFC. This modularity isn’t just organizational—it’s fundamental to how SSH achieves its security guarantees.

Image source: Wikipedia - Secure Shell

The Transport Layer (RFC 4253) handles the cryptographic foundation. It establishes the encrypted tunnel, authenticates the server to prevent man-in-the-middle attacks, and negotiates all cryptographic parameters. This layer operates over TCP port 22 and can handle packets up to 32,768 bytes. Crucially, it automatically re-exchanges keys after 1 GB of data transfer or one hour—whichever comes first—limiting the amount of data protected by any single set of keys.

The Authentication Layer (RFC 4252) validates the user’s identity. Here’s where public keys actually come into play, but the protocol supports multiple methods: password authentication, public key authentication, keyboard-interactive (for one-time passwords and multi-factor authentication), and GSSAPI (for Kerberos and single sign-on). The authentication process is client-driven—when you’re prompted for a password, it’s often your SSH client asking, not the server.

The Connection Layer (RFC 4254) multiplexes multiple logical channels over the single encrypted tunnel. One SSH connection can simultaneously host a shell session, an SFTP transfer, and several port-forwarded connections—each as a separate channel with independent flow control. This design allows SSH to replace not just Telnet and rlogin, but also to provide tunneling capabilities that no other protocol offers.

The Handshake: A Five-Step Cryptographic Dance

When an SSH client connects to a server, the two parties engage in a precisely choreographed handshake. Understanding this sequence reveals why SSH needs more than just public keys.

Step 1: Version Exchange

Both sides announce their SSH protocol versions. Modern implementations support only SSH-2.0—SSH-1 was deprecated due to fundamental design flaws, including weak CRC-32 integrity checking and the inability to multiplex channels within a single session.

Step 2: Algorithm Negotiation

Client and server exchange lists of supported cryptographic algorithms: key exchange methods, ciphers, MAC algorithms, and compression methods. Each side independently selects the first algorithm on its list that appears on the other side’s list. This negotiation happens in plaintext—no encryption yet.

Step 3: Diffie-Hellman Key Exchange Initialization

Here’s where SSH diverges from what most people expect. Instead of using the server’s public key to encrypt data, SSH uses a Diffie-Hellman key exchange—typically Elliptic Curve Diffie-Hellman (ECDH) with Curve25519—to establish a shared secret.

The client generates an ephemeral key pair and sends its public key to the server. This key pair is temporary—it will be discarded after the handshake completes. This ephemerality is crucial: it provides forward secrecy, ensuring that even if someone later compromises the server’s long-term host key, they cannot decrypt previously recorded sessions.

Step 4: Key Exchange Reply

The server generates its own ephemeral key pair. Using both its private key and the client’s public key, it computes the shared secret $K$. Then it constructs an exchange hash $H$ containing:

• Both parties’ version strings
• Both parties’ algorithm negotiation messages
• The server’s host public key
• Both ephemeral public keys
• The shared secret $K$

The server signs this hash with its long-term host key and sends the signature to the client. The client can now verify that it’s talking to the legitimate server—not an impostor—by checking the signature against the server’s known host key.

Step 5: New Keys

From the shared secret and exchange hash, both sides derive six independent keys:

Why six keys? Using separate keys for each direction prevents replay attacks—if an attacker intercepts a client message and sends it back to the client, the MAC verification will fail because it’s using the wrong key. Separate encryption and MAC keys prevent subtle cryptographic vulnerabilities that arise when the same key is used for multiple purposes.

Why Not Just Use Public Keys?

This elaborate handshake exists for sound cryptographic reasons. If SSH simply used the server’s public key to encrypt session data, it would create several problems:

No Forward Secrecy: If an attacker records encrypted SSH sessions today and later obtains the server’s private key, they could decrypt all past sessions. The Diffie-Hellman exchange ensures that even with the private key, an attacker cannot derive the session keys—the ephemeral keys that generated them no longer exist.

Performance: Public key operations are computationally expensive. Encrypting every packet with RSA or ECDSA would be impractically slow. SSH uses asymmetric cryptography only for authentication, then switches to symmetric encryption (typically AES-256-GCM or ChaCha20-Poly1305) for bulk data transfer.

Perfect Forward Secrecy: The use of ephemeral keys during key exchange means that each session has unique keys that are never stored and cannot be reconstructed later. This property, called perfect forward secrecy, is a fundamental security feature that public key encryption alone cannot provide.

Public Key Authentication: Proving Identity Without Revealing Secrets

Once the encrypted tunnel is established, the client can authenticate. Public key authentication works differently than most people assume:

The client sends a message indicating it wants to authenticate using a particular public key. The server checks if this public key is authorized (typically by looking in ~/.ssh/authorized_keys). If authorized, the server sends a challenge—a random number encrypted with the client’s public key. The client decrypts the challenge with its private key and proves it did so.

The private key never traverses the network. Even the signature that proves ownership doesn’t expose the key—it’s a zero-knowledge proof that the client possesses the corresponding private key.

Modern SSH implementations prefer Ed25519 keys over RSA. A 256-bit Ed25519 key provides security equivalent to a 4096-bit RSA key, but with signatures that are 64 bytes instead of 512 bytes. Ed25519 also uses deterministic signatures (no random number generator required), eliminating entire classes of implementation vulnerabilities.

Trust On First Use: SSH’s Deliberate Vulnerability

SSH employs a security model called Trust On First Use (TOFU). When you connect to a new server, SSH displays a fingerprint of the server’s host key and asks if you want to continue:

The authenticity of host 'example.com (192.0.2.1)' can't be established.
ED25519 key fingerprint is SHA256:abc123...
Are you sure you want to continue connecting (yes/no)?

Most people type “yes” without verifying the fingerprint. This is technically a security vulnerability—if an attacker intercepts the first connection, they can present their own key and establish a man-in-the-middle position. SSH accepts this risk for practical reasons: there’s no universally trusted infrastructure for distributing host keys, unlike TLS’s certificate authority system.

Once a host key is stored in ~/.ssh/known_hosts, SSH will alert you if the key changes:

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

This alert is one of SSH’s most valuable security features—it detects when a server’s identity has legitimately changed (reinstallation, IP address reallocation) or when something malicious is happening.

SSH vs TLS: Different Problems, Different Solutions

SSH and TLS both provide encrypted channels, but they solve different problems with different assumptions.

TLS assumes a PKI (Public Key Infrastructure) where trusted certificate authorities vouch for server identities. SSH assumes administrators will manually verify host keys or use a separate system for key distribution. Both assumptions have trade-offs: PKI creates central points of failure (as seen with compromised CAs), while TOFU requires careful human attention that rarely occurs.

The Terrapin Attack: When Cryptography Meets Implementation

In late 2023, researchers discovered CVE-2023-48795, dubbed the “Terrapin attack.” This man-in-the-middle attack exploits a subtle flaw in how SSH handles sequence numbers during the handshake.

SSH packets include sequence numbers used for MAC calculation. The Terrapin attack works by dropping specific handshake messages before encryption begins, causing the client and server to have mismatched views of the handshake transcript. Because sequence numbers are incremented on both sides, the MACs still verify—but security extensions can be silently removed.

The attack requires a man-in-the-middle position and affects specific cipher combinations (ChaCha20-Poly1305 and CBC mode ciphers). The fix is straightforward: upgrade to OpenSSH 9.6 or later, which implements strict key exchange that cryptographically binds sequence numbers to the handshake transcript.

The Terrapin attack illustrates a broader truth: cryptographic protocols are only as strong as their implementations. SSH’s design is sound, but the interaction between sequence numbers, MACs, and the handshake created an exploitable gap.

Beyond Remote Shells: Tunnels and Port Forwarding

SSH’s connection layer enables capabilities that extend far beyond remote login. Port forwarding allows SSH to act as a general-purpose encryption wrapper for any TCP protocol.

Local Port Forwarding (-L) creates a tunnel from your local machine through the SSH server to a remote destination:

ssh -L 5432:internal-db:5432 user@jump-server

Now connections to localhost:5432 are forwarded through the SSH tunnel to the internal database server. The database traffic is encrypted without any configuration on the database server.

Remote Port Forwarding (-R) does the reverse—it exposes a local service through the SSH server to the outside world:

ssh -R 8080:localhost:3000 user@public-server

This is useful for development: your local web server on port 3000 becomes accessible at the public server’s port 8080, without deploying anything.

Dynamic Port Forwarding (-D) turns SSH into a SOCKS proxy:

ssh -D 1080 user@server

Configure your browser to use localhost:1080 as a SOCKS proxy, and all web traffic routes through the SSH tunnel. This provides a quick VPN-like capability built entirely on SSH’s existing infrastructure.

Key Re-exchange: Limiting the Damage

SSH doesn’t rely on a single set of keys for the entire session. By default, it automatically re-exchanges keys after 1 GB of transferred data or one hour of connection time—whichever comes first. This re-exchange uses the same Diffie-Hellman process as the initial handshake, deriving fresh keys from a new shared secret.

Key re-exchange serves several purposes:

Limits Ciphertext for Cryptanalysis: The more ciphertext an attacker has encrypted under the same key, the more information they have for cryptanalytic attacks. Regular re-exchange keeps the volume manageable.

Provides Fresh Forward Secrecy: Each re-exchange creates new ephemeral keys, extending forward secrecy to cover the entire session rather than just the initial handshake.

Recovers from Compromise: If session keys are somehow compromised during a long-running session, re-exchange limits the window of vulnerability.

The re-exchange is transparent—users and applications never notice when it happens. The SSH protocol handles all the complexity internally.

The Enduring Design

SSH has survived nearly three decades of cryptographic evolution with remarkably few fundamental changes. The SSH-2 protocol, standardized in 2006, addressed the design flaws of SSH-1 while preserving the core architecture. Modern implementations support newer algorithms (Curve25519, Ed25519, AES-GCM) but the protocol’s structure remains unchanged.

This longevity stems from the design’s fundamental soundness: separation of concerns into three layers, use of ephemeral keys for forward secrecy, derivation of multiple keys from the shared secret, and built-in mechanisms for key rotation. When Ylönen designed SSH in response to a password sniffer, he created something more enduring than a quick fix—he built a protocol architecture that could evolve while maintaining its security guarantees.

The next time you type ssh user@server, consider what happens in those few hundred milliseconds: a version exchange, an algorithm negotiation, a Diffie-Hellman key exchange, six derived keys, server authentication, and finally a secure channel. All of it happens automatically, reliably, and—when properly configured—invisibly. That’s the mark of excellent protocol design: complexity that serves users by being utterly unobtrusive.

References

1. Ylönen, T. & Lonvick, C. (2006). RFC 4251: The Secure Shell (SSH) Protocol Architecture. IETF.
2. Ylönen, T. & Lonvick, C. (2006). RFC 4253: The Secure Shell (SSH) Transport Layer Protocol. IETF.
3. Ylönen, T. & Lonvick, C. (2006). RFC 4252: The Secure Shell (SSH) Authentication Protocol. IETF.
4. Ylönen, T. & Lonvick, C. (2006). RFC 4254: The Secure Shell (SSH) Connection Protocol. IETF.
5. Harris, B. & Velvindron, L. (2020). RFC 8709: Ed25519 and Ed448 Public Key Algorithms for the Secure Shell (SSH) Protocol. IETF.
6. Baumer, D. et al. (2024). Breaking SSH Channel Integrity By Sequence Number Manipulation. USENIX Security Symposium.
7. Ylönen, T. (2017). The story of the SSH port is 22. SSH.com.
8. SSH Academy. (n.d.). What is the Secure Shell (SSH) Protocol? SSH.com.