IT sweatshop pays the price for budget cuts

In today’s economic climate, it’s not uncommon for employees to take on new roles, whether or not they’re qualified. To the budget people upstairs, this may sound like a wonderful idea. In real life, the benefits of such a strategy do not always translate.

I saw this for myself a couple of years ago while working for a major company at its North American headquarters — coincidentally, where the organization’s CEO and everyone else at the top of the corporate food chain were stationed as well.

IT management at the company didn’t treat any of their staff well, but the employees in this location were consistently handled like the proverbial stepchildren. New IT staff would come in on a temp-to-hire basis, assume the job responsibilities of three people, run full steam ahead until they burned out, and then quit/get fired. Very few actually made it to the point where they were hired on full-time. Even if they did, they usually didn’t last much longer since the pressure only got worse.

When I arrived, I found that the only other IT staff member had been there a mere two months with minimal training. The last full-time IT person had left a month before, so she’d been on her own since and was frazzled beyond words.

Not only were we overworked, but the users at this site were some of the neediest and most demanding backstabbers you could find anywhere. Add to that a ridiculously complex configuration for the machines, file synchronization that never worked, roaming profiles that corrupted at least once a week for each user, and servers that were less stable than a 20-something former child star.

We were hired as the local IT staff for the usual desktop support responsibilities. The offsite tier 3 support team was responsible for managing the servers and network equipment.

After a couple of the employees burned out and another was fired, the director of IT came up with a brilliant idea that would save the company all sorts of money: He would have the local IT support people be responsible for server and networking equipment at the headquarters.

In theory I’m sure this sounded like a great idea, but in practice he was begging for trouble. I had some server experience from a previous job, so I was able to keep an eye on that. But the networking equipment — fuhgeddaboudit. I didn’t even know what each piece of equipment did, much less how to troubleshoot it. They were simply magical boxes with twinkling lights that made stuff work.

When I expressed my concerns to the director, he started with an empty promise to have someone from the network team “come down and show us some stuff.” Then came a threat masked in all of the political horse crap he could muster: “We encourage all of the people that we hire to continue growing and bettering themselves, and this is what we would like to continue to see from you. So you can help the team out by taking on this role.”

OK, if you want to hand over one of your most important IT site’s infrastructures to a person who just told you he didn’t know a router from VCR, fine. Just don’t be surprised when something goes wrong.

And go wrong it did. One chilly April morning, I walked into the data center to rotate the backup tapes. In the background over the whine of the air conditioning, I heard a muffled high-pitched “beeeeeeeeeeep” noise. What was it? Maybe the AC?

Before I could investigate, my BlackBerry rang and I was off to take care of an exec’s Outlook problem. An hour later, everyone in the building lost all network connectivity. My BlackBerry went nuts with administrative assistants and execs alike calling me desperately to inform me of the outage that I was already aware of.

Literally running, I barged through the data center door and started looking at the network equipment. The real problem was that all of the stuff that was blinking before still appeared to be blinking. Everything looked like it was still working. I called the senior network admin, and begrudgingly he agreed to assist me with the issue, though he was irritated with the fact that I wasn’t able to handle this myself.

Over the next hour he guided me through checking connections and doing rudimentary troubleshooting while he juggled other tasks, but to no avail. Frustrated and stressed out, I banged my head on the back of the rack. Then something caught my eye below: One of the UPS units in the bottom of the rack was powered off. Tracing the single power cord connected to it, I discovered that it plugged into a green box labeled “Cisco.” Its lights weren’t blinking anymore, either.

I powered the UPS back on, the screeching noise started again, and a red warning light for the battery started flashing. Then the unit powered itself off again. The mystery of the screeching noise was solved.

On a hunch, I took the power cord for the green box and plugged it into another UPS. A minute later, like magic, everything started working again and the site was back up. As the networking person on the phone explained, “I guess that would do it. That was the core router for the site.”

The director of IT congratulated me on resolving the issue, still self-assured that dumping this responsibility on local IT was the right decision.

A short while later, I accepted a desktop support position at another company. The director of IT eventually made a stupid move that resulted in a major lawsuit for the company, and he was shown the door.

Execs, please take note: Having the right people for the right job is way too important to take for granted. Also, being too aggressive with cost cutting may make the bean counters happy but eventually will blow up in your face.

Do you have a tech story to share? Send it to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “IT sweatshop pays the price for budget cuts,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.