How a small company caused Internet disruption

In a startling demonstration of exactly how little it can take to disrupt the Internet, a relatively innocuous network configuration update by a small Czech company last Monday briefly caused widespread router problems and traffic slowdowns around the globe.

The issue, which was addressed almost as quickly as it was caused, has evoked considerable discussion in some security and network-related blogs and newsgroups, particularly because all it took to cause it was one apparently inexperienced administrator.

[ Keep up on the latest networking news with our Networking Report newsletter. And discover the top-rated IT products as rated by the InfoWorld Test Center. ]

The trouble was seemingly caused by the manner in which the company, SuproNet, provided a crucial bit of information for telling network routers how to reach its site — or IP address — from other locations.

According to one description of the problem, on a blog maintained by Internet monitoring company Renesys, there is more than one way to reach a block of IP addresses. So the directions that are provided by sites to routers include an attribute known as the Autonomous System (AS) path, which tells routers the specific list of carrier networks that have to be traversed to reach IP addresses. Those paths are then propagated throughout the Internet by network routers to direct traffic to a site from locations around the world.

If network administrators don’t want routers to select a particular path they use a process known as prepending to artificially lengthen the path so that it is only chosen as a backup or secondary route to their sites. In this recent mishap, SuproNet lengthened its path for its secondary route by several orders of magnitude greater than was either needed or is customary on the Internet. As its routing announcements were propagated over the Internet, the sheer length of SuproNet’s path information caused them to essentially “tear down” or end their sessions with the immediate source of that data.

“What we think happened next is the Internet equivalent of a massive buffer overflow,” Earl Zmijewski, vice president and general manager at Renesys, wrote in the blog post. “While most of the core routers run by major ISPs fared well, processing the ridiculous path and sending it on, others choked,” causing widespread network disruptions and slowdowns around the globe, he said.

Zmijewski told Computerworld today that while SuproNet’s AS path length was unusually long, that alone should not have created the cascading set of problems around the Internet. Instead the problem has to do with a bug in Cisco Systems’ routers that makes its Internetwork Operating System (IOS) software susceptible to problems when they encounter such long AS paths.

“These Cisco routers were located all over the planet so it was a global event,” Zmijewski said. “What would happen is that these Cisco routers choked on the path [information] and assumed that the input was junk and by that they thought that whoever was giving it to then was wrong,” and essentially tore down connections with the source, he said. “You don’t want to propagate garbage so you turn it down.”

The matter was resolved fairly simply, when SuproNet changed the AS-path information after apparently being informed about the problems its routing update was causing around the Internet. Zmijewski said that as the change propagated, in a matter of a few minutes, routers started working as usual.

Danny McPherson, chief security officer at Arbor Networks, noted in a blog post that the problem was the result of some versions of Cisco IOS not allocating enough buffer space “for silly long” AS paths. “So they blow chunks when they receive the update,” such as the one announced by SuproNet, he said. Arbor provides a range of network security services for large ISPs and enterprises. According to McPherson, the problem seems to have “triggered a great deal of wide-spread routing system instability and underlying connectivity issues,” last Monday.

Ivan Pepelnnjak, chief technology advisor for NIL Data Communications in Slovenia, described in a blog post the bug in Cisco IOS as a new issue that is triggered on a router only when an inbound AS-path contains closes to 255 AS numbers. The blog provides technical descriptions of the bug and a fix for it.

Cisco did not immediately respond to a request for comment. It is not immediately clear if a patch is available for the flaw.

Computerworld is an InfoWorld affiliate.