My kingdom for a RAID array!

analysis
Oct 16, 20074 mins

Hardware was only one of many costs to consider when it came time to purchase more disk space. But that was the only one management focused on. In the early 90s, I worked as the IT manager for a hardware distributor on the East coast. They were a fairly good-size company with a couple hundred employees and multiple branch offices spread throughout the country. At the time, we were operating on an IBM RS/6000 ser

Hardware was only one of many costs to consider when it came time to purchase more disk space. But that was the only one management focused on.

So I did my research and wrote a proposal to my boss recommending an external RAID array from EMC that would have hot-swappable hard drives, super-fast dual-channel SCSI interface (remember this was the early 90s — SCSI was fast back then!), redundant power supplies, and many other fine features. The price tag for this hardware was around $60,000. Considering the benefits of the technology, I thought this was a fair price and that our $50-million-a-year company could easily afford it. Since you’re reading this in “Off the Record,” you can guess what the VP said: “No. Too expensive.” I pleaded my case to him and even went above his head to the owner of the company. He came back with a resounding “no,” too. I was told to research non-RAID external arrays.

I looked at the usual players — IBM, EMC, and so on — and gave my employer my recommendation, all the while still pushing for the RAID 5 array. Ultimately, we settled on an external drive array from a company that I won’t name (they’re still in business). It was an array that I researched at the request of the vice president of the company. I tried to reject it as an untried, untested product, but it was forced upon me.

Within 60 days of installation, one of the drives failed and we were down for two days while we recovered our data off a backup tape. Less than a week after that another drive failed and we were down another two days. The unnamed company sent out a replacement array and that failed within 2 weeks. Ultimately, it wasn’t the drives failing, but the design of the enclosure that was causing the problems. They were using substandard parts and cables in a poorly designed chassis.

After the third failure in a month and more than a week without a computer system, the owner of the company called me and asked me what we could do to prevent this from happening. I told him, “You could spend $60,000 on the RAID array that I suggested in the first place.” I actually didn’t say it that nicely; I may have tossed in a profanity or two (stress and 100-hour workweeks will bring out the worst in me).

What did all of this ultimately cost the company? Hardware costs alone were only about $50,000. The bigger costs were in downtime and lost productivity. Without that system, the company couldn’t take orders or ship product. The owner estimated that each day of downtime cost the company $150,000, plus potential future revenue from customers that left us permanently because they couldn’t get their product. Using this number (as the owner did during his phone call with me), the company lost well over a half million dollars.

I didn’t get fired, and I didn’t get my RAID array for another couple of years. The company ended up purchasing an IBM drive subsystem and used that until the entire system was replaced. The IBM subsystem ran for almost five years, during which we had a couple of drive failures that we were able to catch before they caused any significant downtime.

infoworld_anonymous

Since 2005, IT pros have shared anonymous tech stories of blunders, blowhard bosses, users, tech challenges, and other memorable experiences. Send your story to offtherecord@infoworld.com, and if we publish it in the Off the Record blog we'll send you a $50 American Express gift card -- and, of course, keep you anonymous. (Note that by submitting a story to InfoWorld, you give InfoWorld Media Group, its affiliates, and licensees the right to republish this material in any medium in any language. You retain the copyright to your work and may also publish it without restriction.)

More from this author