How to keep your servers moist

analysis
Mar 25, 20082 mins

I've done a lot of contracting but this one tops them all. I worked for a company as a contractor to design and oversee the building and migration of a new server room. This was a major move up for them and they wanted the works with waterless fire suppression UPS, generator cooling, and so on. They liked the design and all the features, and planned to go ahead with it. After the demolition was done on the space

I’ve done a lot of contracting but this one tops them all. I worked for a company as a contractor to design and oversee the building and migration of a new server room. This was a major move up for them and they wanted the works with waterless fire suppression UPS, generator cooling, and so on.

They liked the design and all the features, and planned to go ahead with it. After the demolition was done on the space and the new work started, the CIO left his position and the CFO stepped in for the interim. He had no understanding of IT. Based on recommendations from other sources, he decided the company could save money by replacing the fire suppression system with good old standard sprinklers. And that they could reduce the fire rating required for the room, thereby saving even more money.

Yes, my blood ran cold, too.

I protested the changes and explained the risks and issues but he would have none of it. He’d made his decision. Well, I get paid either way so his wishes were carried out.

The project was completed on time and well under budget. A bonus was paid to the CFO but none of the contractors. I’m of a mind, though, that karma has a way of restoring balance in the universe. Sure enough, not quite two weeks later, another contracting company was in there running some additional cabling when apparently someone clipped the sprinkler head with a ladder. We all know how well water and servers mix.

Not only did this bring the whole company infrastructure down but in an attempt to save more money, the disaster recovery site was offline as it was being moved to more economical hosting. Also the changes in design in the server room were not passed on to their insurance company, so — you guessed it — the company wasn’t covered for this type of accident. (The design and underwriting for the insurance was based on a waterless suppression system. Oops.)

I received some nasty calls after this incident but a detailed explanation, efficient use of archived e-mail, and the signed original designs and change orders kept me out of the ensuing fray. I never did hear about what new carer the CFO had moved on to but I don’t think it involved being any where near computers. At least, I hope it didn’t.

infoworld_anonymous

Since 2005, IT pros have shared anonymous tech stories of blunders, blowhard bosses, users, tech challenges, and other memorable experiences. Send your story to offtherecord@infoworld.com, and if we publish it in the Off the Record blog we'll send you a $50 American Express gift card -- and, of course, keep you anonymous. (Note that by submitting a story to InfoWorld, you give InfoWorld Media Group, its affiliates, and licensees the right to republish this material in any medium in any language. You retain the copyright to your work and may also publish it without restriction.)

More from this author