On October 4, 2021, at 15:40 UTC, Facebook disappeared from the internet. Not just the social network—Instagram, WhatsApp, and even Facebook’s internal tools went dark. Engineers couldn’t access their own data centers. The outage lasted nearly six hours and affected billions of users worldwide.
The cause wasn’t a cyberattack or a data center failure. It was a BGP configuration error. Someone issued a command that withdrew the routes Facebook used to announce its presence to the internet, and within minutes, the company’s entire network became unreachable.
This incident exposed a uncomfortable truth: the internet’s routing infrastructure runs on a protocol designed in 1989, with security features added as an afterthought. Understanding BGP reveals how the internet actually works—and why it sometimes breaks in spectacular ways.
The Postal Service of the Internet
Border Gateway Protocol (BGP) is often called the “postal service of the internet.” The analogy is apt but incomplete. A postal service routes letters between addresses. BGP routes traffic between autonomous systems—independent networks operated by different organizations.
An autonomous system (AS) is a network under a single administrative control. Comcast, Google, MIT, and the French government each operate their own ASes. Each AS has an Autonomous System Number (ASN)—a unique identifier assigned by regional internet registries. Cloudflare is AS13335. Facebook is AS32934. As of 2025, there are over 100,000 registered ASNs worldwide.
When you visit a website, your traffic traverses multiple ASes. Your ISP’s AS hands off to a transit provider’s AS, which might pass through several other networks before reaching the destination. BGP is the protocol that determines which ASes your traffic passes through.
Image source: Cloudflare
How BGP Actually Works: A Conversation Between Routers
BGP operates on a simple principle: networks tell each other which IP addresses they can reach. Each AS runs BGP-speaking routers at its borders. These routers establish TCP connections on port 179 with their counterparts in neighboring ASes.
BGP uses four message types to maintain these relationships:
OPEN messages establish the initial connection. Two routers exchange their AS numbers, hold timers, and BGP version information. If the parameters are acceptable, the session proceeds.
KEEPALIVE messages maintain the session. By default, BGP sends a keepalive every 60 seconds. If a router doesn’t receive a keepalive within the hold timer (typically 180 seconds), it tears down the session.
UPDATE messages carry the actual routing information. An UPDATE can announce new routes (advertisements) or withdraw old ones. Each advertisement includes the destination prefix (like 192.0.2.0/24) and various attributes that influence path selection.
NOTIFICATION messages signal errors. If something goes wrong—a malformed message or an unsupported capability—the router sends a NOTIFICATION and closes the connection.
This architecture has a critical implication: BGP changes propagate hop by hop. When AS1 announces a new route, it tells AS2. AS2 decides whether to accept it, then tells AS3. AS3 tells AS4, and so on. A route change can take minutes to propagate across the internet, and a misconfiguration can cascade globally before anyone realizes what happened.
The Path Selection Algorithm: Why Your Traffic Takes the Route It Takes
When a BGP router receives multiple routes to the same destination, it must choose one. Unlike interior routing protocols (OSPF, IS-IS) that pick the shortest path by metric, BGP uses a complex decision process involving multiple attributes.
The selection algorithm evaluates attributes in a strict order:
1. Weight (Cisco proprietary) – Local to the router, not propagated. Higher weight wins.
2. Local Preference – Shared within an AS. Higher value wins. This is how network operators implement policy: “Send traffic through Provider A, not Provider B.”
3. Self-Originated – Prefer routes the local router originated.
4. AS Path Length – Prefer the route with the fewest AS hops. A path through AS1→AS2→AS3 beats AS1→AS4→AS5→AS6→AS3.
5. Origin Code – Prefer IGP (originated by BGP) over EGP (deprecated) over INCOMPLETE (redistributed from another protocol).
6. MED (Multi-Exit Discriminator) – Lower value wins. Used when two ASes connect at multiple points, allowing one AS to influence where the other sends traffic.
7. eBGP over iBGP – Prefer external paths over internal ones.
8. IGP Metric to Next Hop – Prefer the route with the lowest interior gateway protocol cost to reach the BGP next hop.
9. Oldest Path – If everything else is equal, prefer the route learned first.
10. Router ID – Prefer the path from the neighbor with the lowest router ID.
11. Neighbor IP Address – Final tiebreaker: prefer the lowest neighbor IP address.
This hierarchy explains why BGP is called a “policy-based” routing protocol. The first several criteria exist specifically so network operators can override the simple “shortest path” logic with business decisions. You might have a shorter path through Provider A, but if Provider B is cheaper, you set a higher Local Preference and traffic flows that way instead.
flowchart TD
A[Receive Multiple BGP Routes] --> B{Highest Weight?}
B -->|Tie| C{Highest Local Pref?}
C -->|Tie| D{Self-Originated?}
D -->|Tie| E{Shortest AS Path?}
E -->|Tie| F{Lowest Origin Code?}
F -->|Tie| G{Lowest MED?}
G -->|Tie| H{eBGP over iBGP?}
H -->|Tie| I{Lowest IGP Metric?}
I -->|Tie| J[Select Best Path]
eBGP vs. iBGP: External and Internal Worlds
BGP operates in two modes: external BGP (eBGP) between different ASes, and internal BGP (iBGP) within an AS.
eBGP sessions connect border routers from different organizations. These sessions typically span multiple network hops, though the routers must be directly connected by default (this can be overridden with “multihop” configurations). Routes learned via eBGP get propagated to all iBGP neighbors.
iBGP sessions distribute externally learned routes throughout an organization’s internal network. The key difference: iBGP routers don’t redistribute routes learned from one iBGP neighbor to another. This “split horizon” rule prevents routing loops, but it means every iBGP router must peer with every other iBGP router—a full mesh that doesn’t scale.
Large networks solve this with route reflectors: designated iBGP routers that are allowed to reflect routes between iBGP peers. A hierarchical route reflector architecture reduces the mesh requirement from O(n²) to O(n).
The Trust Problem: Why BGP Hijacking Works
BGP was designed for a smaller, more trusting internet. The protocol has no built-in authentication—when a router announces “I can reach prefix X,” other routers believe it.
This trust model enables BGP hijacking: a malicious or misconfigured network announces routes for IP addresses it doesn’t control. Traffic destined for those addresses gets diverted to the hijacker’s network.
There are two main techniques:
Prefix hijacking: The attacker announces a victim’s prefix (or a more specific sub-prefix) as if it belonged to their AS. Because routers prefer longer prefix matches, announcing a /24 when the legitimate owner announces a /23 will capture all traffic for that /24.
AS path manipulation: The attacker announces a route with a shorter or more attractive AS path, causing other networks to prefer the attacker’s route over the legitimate one.
In April 2018, attackers hijacked Amazon’s Route 53 DNS prefixes and redirected MyEtherWallet users to a malicious server. The attackers used a BGP hijack to poison DNS resolution, sending victims to a fake website that stole approximately $150,000 in Ethereum cryptocurrency.
Image source: Cloudflare
Real-World Incidents: When BGP Breaks
The Pakistan-YouTube incident of 2008 remains the textbook example. Pakistan Telecom (AS17557) wanted to block YouTube domestically. They announced a more specific route for YouTube’s IP prefix (208.65.153.0/24) than YouTube’s own announcement (208.65.152.0/22). Due to BGP’s preference for longer matches, traffic meant for YouTube worldwide started flowing to Pakistan.
The problem: Pakistan’s internal route leaked to their upstream provider, PCCW. PCCW propagated it globally. For about two hours, YouTube was unreachable for most of the internet. Pakistan’s censorship attempt became a global outage.
The 2021 Facebook outage demonstrated a different failure mode. A configuration change withdrew Facebook’s BGP announcements entirely. Without routes to Facebook’s DNS servers, DNS resolvers couldn’t resolve facebook.com, instagram.com, or whatsapp.com. The withdrawal cascaded: DNS resolvers worldwide cached negative responses, compounding the problem even after Facebook restored their routes.
Cloudflare’s monitoring showed the BGP updates clearly:
route-views>show ip bgp 185.89.218.0/23
% Network not in table
The prefixes weren’t just unreachable—they didn’t exist in the global routing table anymore. Facebook had effectively disconnected itself from the internet.
RPKI: Adding Cryptography to a Trust-Based Protocol
Resource Public Key Infrastructure (RPKI) represents the most significant security enhancement to BGP. It provides a cryptographic mechanism to verify that a network is authorized to announce a particular IP prefix.
RPKI uses a hierarchical certificate system modeled after the web’s PKI. Regional Internet Registries (ARIN, RIPE NCC, APNIC, LACNIC, AFRINIC) act as trust anchors, issuing certificates to their members for their allocated resources.
Network operators create Route Origin Authorizations (ROAs)—cryptographically signed objects that specify which AS is allowed to announce a prefix. A ROA for Cloudflare’s 1.1.1.0/24 would specify AS13335 as the authorized origin.
BGP validators check incoming route announcements against published ROAs:
- Valid: The announcement matches an existing ROA
- Invalid: The announcement conflicts with a ROA (wrong origin AS or prefix length)
- NotFound: No ROA exists for this prefix
As of 2025, over 54% of IPv4 routes and IPv6 routes in the global routing table are protected by RPKI ROAs. Networks that enforce RPKI validation reject Invalid routes, preventing many hijacking attacks.
Image source: Cloudflare
RPKI has limitations. It only validates route origin, not the AS path. A sophisticated attacker could forge a valid origin but manipulate the path. BGPsec, a proposed extension, would cryptographically sign the entire AS path, but deployment remains minimal due to complexity and performance concerns.
The Routing Table’s Relentless Growth
The global BGP routing table keeps growing. As of early 2025, the IPv4 table contains approximately 970,000 prefixes. The IPv6 table has grown to about 200,000 prefixes.
This growth creates operational challenges. Each router in the forwarding path must maintain the full routing table to make forwarding decisions. Memory and CPU requirements increase annually. More concerning is the rate of BGP updates—RouteViews collectors see thousands of updates per minute globally.
The growth isn’t uniform. Some of it reflects legitimate network expansion, but much comes from traffic engineering: networks deaggregating prefixes to influence routing. A network might announce a /16 as 256 individual /24s to prevent others from hijacking sub-prefixes, but this multiplies the table size.
BGP Communities: Tags That Influence Routing
BGP communities are optional transitive attributes that act as tags attached to route announcements. They enable sophisticated traffic engineering without changing the fundamental protocol.
Well-known communities have standardized meanings:
- NO_EXPORT (65535:65281): Don’t advertise this route to external peers
- NO_ADVERTISE (65535:65282): Don’t advertise this route to any peer
- LOCAL_AS (65535:65283): Only advertise within the local AS
Networks also define their own communities. An ISP might honor community 65000:100 to set Local Preference to 100, allowing customers to control their traffic without involving the ISP’s operations team.
This mechanism enables powerful traffic engineering: a content provider can tag routes differently for different peers, directing traffic along preferred paths without maintaining separate BGP sessions for each policy.
Convergence: The Time It Takes for the Internet to Agree
When a network change occurs—a new route announced, an old route withdrawn, a link failing—the change must propagate across the internet. This process, called convergence, can take anywhere from seconds to minutes.
BGP’s convergence time depends on several factors:
MRAI (MinRouteAdvertisementInterval): To prevent route flapping, BGP implementations wait a minimum interval (typically 30 seconds for eBGP, 5 seconds for iBGP) between sending updates for the same prefix. This dampens rapid changes but delays convergence.
Path exploration: When a route is withdrawn, routers may temporarily accept longer backup paths. Each path exploration step adds propagation delay.
Processing load: Large routing tables require more processing. A router receiving thousands of updates simultaneously may queue them, adding latency.
In practice, a route change typically takes 30-120 seconds to propagate globally. For a network like Facebook with thousands of prefixes, a configuration error can cause hours of instability as routes are withdrawn and re-announced across the internet.
The Invisible Infrastructure
BGP operates as infrastructure—not something users think about, but something that fails visibly when it breaks. Every website, every online service, every API call depends on BGP correctly routing traffic between autonomous systems.
The protocol’s design reflects its age: born in an era when the internet was smaller and more trusting, now stretched to serve billions of users and billions of devices. Security features like RPKI are being retrofitted, but adoption remains incomplete.
Understanding BGP reveals how fragile the internet’s foundation actually is. A single misconfigured router in Pakistan once took down YouTube globally. A configuration error at Facebook disconnected one of the world’s largest companies from the internet. Attackers have used BGP to steal cryptocurrency and intercept traffic.
The next time you see a “server not found” error that spreads globally within minutes, consider: the server might be fine. The routing infrastructure might just be having a conversation that went wrong.
References
-
Rekhter, Y., & Li, T. (1995). RFC 1771: A Border Gateway Protocol 4 (BGP-4). IETF. https://datatracker.ietf.org/doc/html/rfc1771
-
Lepinski, M., & Kent, S. (2012). RFC 6480: An Infrastructure to Support Secure Internet Routing. IETF. https://datatracker.ietf.org/doc/html/rfc6480
-
Huston, G. (2025). BGP in 2025. APNIC Blog. https://blog.apnic.net/2026/01/08/bgp-in-2025/
-
Cloudflare. (2021). Understanding how Facebook disappeared from the Internet. https://blog.cloudflare.com/october-2021-facebook-outage/
-
Cloudflare. RPKI: The Required Cryptographic Upgrade to BGP Routing. https://blog.cloudflare.com/rpki/
-
Kentik. A Brief History of the Internet’s Biggest BGP Incidents. https://nanog.org/stories/articles/a-brief-history-of-the-internets-biggest-bgp-incidents/
-
DeepStrike. What Is BGP Hijacking? How Internet Routing Attacks Work. https://deepstrike.io/blog/what-is-bgp-hijacking
-
NetworkLessons. BGP Attributes and Path Selection. https://networklessons.com/bgp/bgp-attributes-and-path-selection
-
MANRS. RPKI ROV Deployment Reaches Major Milestone. https://manrs.org/2024/05/rpki-rov-deployment-reaches-major-milestone/
-
Qrator Labs. Why BGP Hijacking Still Threatens Global Networks. https://qrator.net/blog/details/why-bgp-hijacking-still-threatens-global-networks/
-
APNIC. RPKI’s 2024 Year in Review. https://blog.apnic.net/2025/01/28/rpkis-2024-year-in-review/
-
Cloudflare. What is BGP? BGP Routing Explained. https://www.cloudflare.com/learning/security/glossary/what-is-bgp/
-
Huston, G. BGP Routing Table Analysis Reports. https://www.potaroo.net/
-
Wikipedia. BGP Hijacking. https://en.wikipedia.org/wiki/BGP_hijacking