an essay on the fabric of the Internet
by _dose, 05/2002
"You can read it?"
"No, I was just clearing my throat."
I'll keep the technical jargon to a minimum, or at least I'll try. A little.
Now for some administrative politics. There is a central organisation that deals with assigning 'numbers'. This is the Internet Assigned Numbers Authority (IANA). They control the delegation of ASNs, IP ranges and other numeric resources that require a central authority. Historically they have directly assigned IP blocks and ASNs to corporations or organisations, this is no longer the case. As the Internet grew, seperate organisations were set up to deal with this process. These organisations are called 'regional registries'. Broadly speaking (very broadly,) there are three of these. They are organised geographically. They are ARIN (American Registry for Internet Numbers), APNIC (Asian-Pacific Network Information Center) and RIPE (Reseaux IP Europeen, which translates as European IP Networks). These organistations can be found at,
When an organisation wants to build a network that actively participates in the global internet (as opposed to simply buying traffic from an ISP), it will request an ASN and IP allocation from the regional registry for their region. The most important part is receiving an ASN (I will elaborate on this later) and after that an IP allocation.
An IP allocation is one or more blocks of contiguous IP addresses that this organisation receives and is unique to this organisation. Actually, this is not entirely accurate. The IP allocation is tied to the ASN, but more on that later. An organisation can have more than one ASN, this is usually the result of mergers or acquisitions. The regional registries will not assign more than one ASN to one organisation, but once two organisations merge then the new entity holds both ASNs. Massive acquisitions have led to single organisations holding many ASNs, but usually they will consolidate all IP allocations associated with these ASNs to one single ASN and return the ASNs to the regional registries. They will however hold on to the IP allocations, as IP space is very valuable.
This brings me to a side note. IP addresses being sold by an ISP. Many ISPs charge money for assigning IP addresses to a customer company. This is acceptable as it is a one time administrative cost, but IP addresses are not the property of an ISP. IP addresses are granted to an ISP for connectivity purposes. Any ISP that requires a company to pay a recurring fee for handing out IP addresses is in violation of Registry policy. IP addresses are not property, and the only money you can be charged by an ISP for an IP allocation from the IP addresses they have been assigned is money that covers the administrative costs they have. An organisation may not, under any circumstances, make a profit on IP assignments. Many of them still do, because no-one feels like taking them to court over it. Just one of the many things that are wrong on the internet... But I thought I'd mention it. Smack the next sales person you encounter asking money for IP addresses with this little tidbit, wiil you?
So after reading all this, you might want a few interesting links.
Over here http://www.iana.org/assignments/ipv4-address-space you can find a listing of all IP allocations by IANA. The notation used is CIDR, so 212/8 is a massive block (class A in the old style) covering all IP addresses from 220.127.116.11 to 18.104.22.168.
Here http://www.arin.net/statistics/2001stats.html you can find the total number of IP requests processed and AS numbers assigned by ARIN in 2001.
This whole model works very well. The basics are pumped into each networking newbie and are referred to at all levels. For LAN technologies running our beloved TCP/IP protocols, that is. There are a lot of other networking technologies out there, and once you start mixing datacom and telecom technologies the entire model falls to pieces.
So IP is a network layer, not a transport layer. But that's just semantics. IP is what's used to transport data across a network. That's why I (and many others) call it a transport mechanism. At layer 3 we find the routers, pieces of network equipment that forward packets of data based on their IP addresses (which are logical, not physical).
You might be wondering now how IP packets go from one place to another. This is a very good question. On a local network you don't need IP, you can communicate with other machines just fine using layer 2. Basically you just send a signal onto the wire and, say, Ethernet will take care of the rest. But if you want to go outside you'll want your packets to be routed. Think of routing as sending a packet into a different domain. One that has no awareness of your physical networking media. You don't really think that the machines at http://www.google.com know if you're on a dial-up link, DSL or a LAN connection, do you?
If you want to 'go outside', your packets will reach a router. A router has multiple interfaces and a routing table. It receives your packet and looks at where that packet wants to go. It then looks into its routing table to decide which interface its going to forward that packet to. Most of the time your packet will have a destination address that it doesn't know about. So it'll send that packet to another router that is higher up in the hierarchy. If this packet has to go onto the Internet, it will be sent upwards and upwards in the network. Until it reaches the border of the network. Here it is passed to a router that is connected to the Internet.
So now we come to the Carriers. Carriers are networks that are vast and intercontinental. They exchange traffic with other Carriers. Examples of Carriers are KPN/QWest, MCI/Worldcom, Level3, etc. An ISP will usually sign a contract with one or more Carriers. The Carriers will then agree to accept traffic for any network and send traffic from any network to the ISP network if it's destination is that ISPs network.
So why were these ASN numbers necessary again? Well, these networks all build up their routing tables based upon the ASNs. At this level, the routers are not talking IP, they are talking another protocol, called BGP (Border Gateway Protocol). Using BGP these routers tell each other their ASN numbers and the ASNs they can reach.
The routers make decisions on where they send their packets based on AS policy. A Carrier will accept all traffic from an ISP router, but this costs money and if the router can send the packets directly to the network it wants to reach via the IX switch, it makes sense to use the switch because it is faster and cheaper. But if an ISP doesn't have a peering agreement with the network it wants to send data to then the router on the other side of the switch will not accept the traffic, so it has to be sent via the Carrier.
So at the heart of the Internet, routers don't look at IP addresses, they look at ASN numbers and what the best path is to send traffic to. Say a border router wants to send data to IP x.x.x.x. It will look up this address in its routing table - a table that is the result of BGP policy.
Consider this example of an AS topology,
AS-1 ----------- - AS-21 | \---- (Carrier)--------/ | | / AS-17 \ | [Exchange] / \ [Exchange] [ Switch ] / \ [ Switch ] | | / \ | | | / \ | AS-2 AS-3 -----/ ----- AS-42 ---- AS-33
AS-1 can reach the rest of the ASs via the following paths:
Over the exchange switch (via peering),
AS-1 - AS-2 AS-1 - AS-3and over Transit (via the carrier),
AS-1 - AS-17 AS-1 - AS-17 - AS-3 AS-1 - AS-17 - AS-21 AS-1 - AS-17 - AS-42 AS-1 - AS-17 - AS-42 - AS-33Apart from the fact that I will never be a graphic designer, the interesting details we can glean from the AS paths and the map are,
Now, if you'd prefer to view some actual graphics, you can find an AS map of the internet at http://www.caida.org/analysis/topology/as_core_network/AS_Network.xml. You might want to read the description to make sense of it, though.
Some say the Internet was designed to withstand a nuclear attack. This isn't true. The original ARPANet was designed to withstand the complete failure of one or two points, but the Internet as we know it today is a completely different animal. Many of the original protocols and concepts are still with us (much to the chagrin of security engineers), but the network has evolved into something different. Often I hear people saying "Well, if the US-EU links go down, we'll just route traffic via the Mid-East to Asia to the US West Coast, right?" Wrong. Take a look at this picture, http://www.telegeography.com/pubs/maps/internet/index.html - the wallpaper version at the bottom is a higher resolution. This picture shows us the intercontinental capacity of the IP datacom networks. Do you really think we could compensate for the US-EU loss by "routing that traffic somewhere else"? That said, it's not very likely that all the fibres across the Atlantic would fail at the same time.
If it were to happen, though, it would disrupt business communications between the EU and the rest of the world. It would disrupt private communications as well, but these are of lesser impact. Small to medium businesses would hardly notice, as most email and websites relevant to their operations are inside the EU region. There is no particular element of regional operation that requires the Americas to be reachable via the network. It might even improve productivity in many workplaces, as Hotmail, Yahoo and assorted web comics would also be unavailable :)
In this way it is trivial for one router to announce IP ranges that don't belong to it, and other routers will happily propagate this information over the internet. Let's say that ISP-A accidentally announces an IP range belonging to ISP-B. ISP-A will then receive an amount of traffic that is destined for ISP-B. These things happen on occasion, and are generally fixed very fast. Operator mailing lists will carry this information and if ISP-A doesn't fix it's announcement, other ISPs will start to ignore its announcements. Effectively isolating ISP-A.
These annoucements are however quickly noticed because an ISP will actively monitor the announcements of its address space. It is a bit different with IP address space that has not been assigned yet. This address space is called murky address space because it does on occasion appear in the internets routing tables. This is usually attributed to advanced spammers with a lot of networking skills wishing to hide their tracks. They break into an ISPs routers and adjust its BGP configuration to also announce unassigned address space (which they use to send SPAM) after which they remove the announcements and effectively disappear. These, and other advanced attacks of SPAM, scams, etc are attributed to the dot-com fallout, which left a lot of very skilled people with no jobs.
Currently, there are approximately 25,000 ASN assignments in use. The total number of unique networks will be significantly lower after the enormous amount of corporate mergers over the last few years. When two or more companies merge (or one is aquired by the other), the ASNs and IP allocation for both will still be held by the resulting company. It is considered proper behaviour to merge the ASNs and announce all IP allocation under 1 ASN and return superfluous IP ranges to the regional registry for re-allocation. However, this is a lot of work. And as obtaining IP addresses in the first place requires a lot of paperwork, most companies prefer to hold on to what they have.
A full Internet routing table holds approximately 120,000 unique routes today (mid 2002), up from 20,000 in 1994. A graph showing this growth can be found at http://www.mcvax.org/~jhma/routing/bgp-hist.html. This impacts not just the size of the Internet, but also the hardware necessary to process all these routes. Keep in mind that these routing tables are dynamic and change from minute to minute. Also, each packet traversing such a router has to be matched against the routing table (with its 120,000 possibilities) to determine its destination. Not just the capacity and speed of the hardware has improved to meet this challenge, the software performing the actual routing has become highly specialised to handle this. Additionally an Internet router usually has multiple high-speed interfaces (each usually between 100 Mbit/s and 1 Gigabit/s).
So how much traffic is there on the Internet? Also an impossible question to answer. A good starting point is however to view the traffic statistics of some major Internet Exchanges. One of the major European IXs is the AMS-IX (Amsterdam Internet Exchange), and its statistics can be viewed http://www.ams-ix.net/hugegraph.html here. Slightly over a year ago the aggregate traffic via peering alone here was 2 Gbit/s, now it is almost 6 Gbit/s. (As viewed in the monthly graph, in the daily graph we can see spikes over 8 Gbit/s). In this area, MFN - Metromedia Fibre Networks / Above.net have a large facility servicing IP traffic, with 9 public circuits carrying traffic to and from this location internationally. The traffic graphs for each of these circuits can be found here, http://www.mfn.com/network/ip_networkstatus.shtm#ams. The circuits are SDH STM-4 and STM-16s, 622.08 Mbps and 2.488 Gbps respectively. A high level overview of its network can be found here http://www.mfn.com/network/ip_networkmaps.shtm. And MFN/Above.net is far from being the only Carrier in the game.
Already there are plans on the table for interplanetary Internet communications. We have the Consultative Committee for Space Data Systems, http://www.ccsds.org/, to name one. And of course the Interplanetary Internet Project http://www.ipnsig.org/home.htm, in case you were afraid of ending up on Mars without pornography.
I bet you were expecting a description of IP version 6 here, right? Sorry, I'll write about that once it is more widely deployed.
I can usually be reached via various message boards, try ~S~ Seeker's messageboard . I usually drop in to read some posts there. Otherwise you could try dropping me a line at dose at remove-this at linux dot nl dot com. If that doesn't work - ask around.