Cheapskate execs pay the price for outsourcing IT

Why is it so hard for some executives to understand that systems require maintenance in order to run properly? No, the technology does not magically work day after day all by itself. In fact, employees on the back end toil around the clock to make it so. But as in this story, IT’s arguments are often ignored — until, of course, something breaks.

One day, the people a bit higher up in the line of command at my company told us that we were going to outsource our website to save on upkeep and labor costs. Rather than have IT manage it in-house, they wanted to hire two third parties: a Web hosting service and a Web developer. The only part managed in-house would be the website’s content, which would be updated by a marketing employee.

[ More about the IT job on InfoWorld: “11 signs your IT project is doomed.” | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the Off the Record newsletter. ]

They said they wanted IT to be out of managing the website so that we could focus on more business-oriented initiatives. Less work for us and more involvement in the business side? That sounded good. Even so, we were reluctant to give up control and had many questions about security and maintenance. Our questions were waved away with, “Oh don’t worry, it’s all taken care of.” Nor were the higher-ups definitely interested in hearing about possible disadvantages. This change made sense to them from a fiscal standpoint — end of story.

They proceeded to outsource our site to a Web hosting service that charged a very low monthly rate. They also contracted a Web developer to create and set up the new website on the Web host. We were told all this in passing, again with the ominous words of “it’s all taken care of.”

What could possibly go wrong?

This arrangement worked well for a couple of years, and the suits were quite proud of themselves. But one day we found out that our site had been blacklisted by Google and was now classified as an attack site. The marketing person who had been maintaining the content asked me to find out what was going on.

Based on quick investigation, it appeared that our site had been hacked. I thought the hack might be from an unpatched hole in the CMS software, because I noticed that the CMS was an old version that was no longer supported. But I wondered why the CMS hadn’t been updated.

I asked the execs and found out they hadn’t been in touch with the Web developer since he first created and set up the website. I tried to contact the developer, but no luck. I don’t know if he was on vacation, moved, or whatever, but he was definitely not available.

There was some good news: The developer had given the passwords to the marketing person, who even knew where they were and passed them on to me. At this point, the higher-ups were begging IT to rectify the issue, given that nobody could really access our site without getting infected with Trojans.

Upon logging in, I was horrified when I verified that even though the marketing person had kept the content current, the back end hadn’t received even one update in two years!

Two years is an eternity on the Web

The first thing I thought of was to restore a recent copy of the website, but I was again horrified when I found that the most recent backup was the initial one the Web developer had done a couple of years ago.

Next I checked to see if there was actual malware hosted on our website. There wasn’t any, so the main problem seemed to be scripts that were directing our visitors to malware sites. Why ISPs don’t shut down such sites, I don’t know — but that’s beside the point.

It took me quite a few hours to look at the scripts, the .htaccess file, hidden iframes, and the rest to try to get rid of the malicious scripts. As a precaution, I changed all the passwords on the site and scanned the marketing person’s computer for malware — fortunately, I didn’t find anything. After everything looked OK, I made a backup of the site and put in a request to Google that we be removed from the blacklist.

So much for not involving IT.

The higher-ups had been too cheap to pay for maintaining the back end of the site — why spend money to update when everything is working? But the mess showed them they couldn’t just leave things alone. Though the Web developer had called it a day the second after he finished setting up the site, at least the Web hosting provider offered a yearly service to perform maintenance on the back end, which the higher-ups signed up for. Baby steps, I guess.

I wonder how much of a hit our company’s reputation took from the experience. My guess is that in itself cost way more than what they were trying to save.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “Cheapskate execs pay the price for outsourcing IT,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.