Why you never let a vendor design your network

I happened across this breathtaking story in Ars Technica last week that details a discovery Brad Reese made many months ago. This didn’t get much attention at first, but it appears to be blowing up now. The story dovetails nicely (however sadly) into my discussion last Monday of the need for early IT involvement in IT projects.

To boil it down, West Virginia overspent about $5 million in federal money when designing and building a statewide network. Instead of specifying and purchasing adequate hardware for each location it needed to serve, the state bought a $20,000 Cisco 3945 router for every single location, regardless of size or capacity. That’s how it wound up with a library the size of a doublewide that’s open only three days a week, yet boasts a Cisco 3945 for its meager Internet connection. I imagine that if West Virginia had gone with more suitable hardware, it would have had enough money left over to keep the place open all week long.

[ Cash in on your IT stories! Send your IT tales to offtherecord@infoworld.com. If we publish it, we’ll keep you anonymous and send you a $50 American Express gift cheque. | Get the latest practical data center advice and info in Matt Prigge’s Information Overload blog and InfoWorld’s Data Center newsletter. ]

This is what happens when nontechnical decision makers ask vendors to design their networks and send them the quote. No matter how much Cisco denies it padded the specs or claims the routers were necessary for “future expansion”, the company can’t change the facts. Any network architect worth their socks knows this is a huge load of BS; moreover, so does Cisco.

No WAN this size — absolutely none — has identical requirements in every single location. Further, the Cisco 3945 is billed in Cisco’s own literature as a solution for medium-to-large installations. By any possible measure, it’s egregious overkill for nearly every location in the West Virginia network in which it was deployed. Many of those sites would’ve been perfectly served by a small branch router costing less than $1,000.

The cost of this baffling choice was gauged at $5 million in the article, based on the assumption that the locations could have been served by $5,000 Cisco 2800- or Cisco 2900-series routers. If we push the reality check even further and consider that $1,000 to $2,000 routers would have done the job, the waste is potentially much larger.

There’s no doubt this is a boondoggle and mistakes were made at every level of the deal. No amount of “futureproofing” or “forward thinking” can dislodge the fact that there is no rational reason for this purchase to have been made and, more important, this quote to have existed at all. This is a direct result of predatory salespeople and (at best) apathetic state officials. At worst, it’s yet another example of backroom dealings so over the top that they’ve actually found sunlight.

If you want to relate this story to those who don’t know a Cisco 3945 from a Cisco 870, try this analogy: Imagine that the state had received a grant to supply cars to officials in various towns and cities. These cars, perhaps 1,200 in all, are to be used for state activities such as transportation to meetings, site travel for state inspectors, and other mundane tasks.

The group responsible for procuring these cars sends an RFQ to a large car dealership, which responds with a quote for 1,200 Maseratis. The request is rubber stamped and paid for by the grant agency without anyone questioning why the state should be buying 1,200 luxury sports cars. Imagine the public furor when those cars hit the road.

I don’t know the intricacies of West Virginia’s network, nor do I know who was responsible for the design and implementation of the network itself. I think it’s a reasonable guess that those who signed off on the uncontested, amazingly bloated quotes from Cisco were not the same people who were responsible for the network design and construction. If they were, they have even more explaining to do. The engineers at Cisco who came up with the quote have some questions coming to them too.

There’s talk of debarment of Cisco in West Virginia, which would mean that Cisco would be forbidden from bidding on future projects. I doubt anything so reasonable will ever come to pass, but perhaps the state can try to sell several hundred of those routers on eBay and get the routers it should’ve bought in the first place — assuming they need routers for these locations at all. (I’m only partially kidding.)

If West Virginia had enlisted the assistance of a competent, independent network architect at any point during the design and procurement phase of this project, none of this would have happened. To an even reasonably skilled consultant, this quote and spec would have seemed absurd from the outset.

Unfortunately, this kind of budgetary disaster is still all too common when dealing with the “black magic” of large networks. Unless you know what you’re talking about or hire someone who does, you have to trust the vendor to do the right thing. That’s clearly not what Cisco had in mind for West Virginia.

This story, “Why you never let a vendor design your network,” was originally published at InfoWorld.com. Read more of Paul Venezia’s The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.