Never Outsource DBAs… or Other Critical IT Staff

Outsourcing critical IT functions has always been a mystery to me. I realize it saves the company money in certain aspects, and even saves some admin headaches. However, when you get down to it, it’s really just a trade-off. You’re trading one set of complications for another. I’ve been on both sides of the hosted data center and I have to say that I just don’t get it. What I really don’t get is companies that outsource their helpdesk and network admin functions.

Let’s walk through this and see if we can come up with a logical reason for going with a separate data center and helpdesk staff. What are the key benefits? Well, there’s typically enough savings in employee benefits that it’s worth your while to go the outsourced route. There’s also the cost of putting together and maintaining the data center. There’s the cost of hiring, training and maintaining the staff.

There may be a couple other smaller benefits, but those are the biggest ones and the ones that count.

Now let’s look at the downside to outsourcing your IT staff. For starters, you rarely have any control over the staff. So you have no control of the quality of the people handling your business. You also have no control over your operations. When you sign your contract with your hosting company there are usually backup clauses in there that state how often, and to what extent your data will be backed up. However, hosting centers typically have their own interests in mind and they’re going to go the cheap route.

Here’s a good example:

A couple years ago, the national hosting company CIHost had a virus sweep their environment and brought down several of their SQL boxes. There were 7-9 effected, I forget how many exactly. Many customers lost months of data because the servers hadn’t been backed up to any reasonable degree in months. It was a hosting nightmare, and even worse, their customers trusted them. It just so happens that I was one of the DBAs brought in to get them back online. We sat up to all ends of the night for days working through trouble tickets and trying to restore corrupt DBs just to get back whatever data we could for the customers. When all was said and done, CIHost pretended that they had never heard my name and stiffed me for several thousand dollars.

There are other problems with outsourcing your IT staff. Quite often the outsourcing firm will make decisions that will increase your dependence on them, and not what’s best for your business. I have seen that from the inside where I created a solution that would put a company in a good position and my boss told me to suggest a solution that would cause them to have more support issues with us. I recommended the solution he told me to, and then sent the customer a different email from my personal account explaining what he should do instead.

You can also have your business hi-jacked, which is the worse thing that can happen. There’s a very large outsourcing company here in Dallas that a lot of companies use, and this company (we’ll call them EvilEmpire) takes things over whenever it can. If you ask for a hole in the firewall, they say no, it’s against policy. If you ask for a user name change, they say no. If you ask for a new admin to be added to a server, typically say no, and when they do say yes, it’s 3 days before you see anything. You never really know who’s doing what on your servers because they control the domain admins group and you can’t keep them out, and you can’t keep them out of your database if you want. In short, your entire business is hi-jacked. They tell you what to do and how to do it.

Here’s a little wake-up for you EvilEmpire… you work for us, not the other way around. You’re supposed to be making our business easier to run, not putting up walls every time we try to do something.

Remember what I said though. It’s not in their best interests to give us what we want, or what’s best for us. Their job is simple… keep as many support contracts as they can. If your business happens to run at the same time, then that’s ok with them, but it’s really not their #1 concern. I’ve even seem them use helpdesk functions as a way to pressure you into acquiesce to other demands, like bringing on one of their DBAs. It’s amazing that the same day you refuse to bring one of their guys on-site, all your tickets take 2 days longer to complete than usual. The situation oddly clears itself up once your new DBA is in place.

It’s for these reasons that I really think it’s always best to house your own equipment, and to keep your own staff. Sure, you have other headaches, but they’re your headaches. If you drop the ball, it’s your ball to drop, and you are in full control of it. You can physically go in and make sure the backups were pushed to tape, and not just take somebody’s word for it. And in keeping with my previous blog on Keeping DBAs, you can make sure you keep the staff you want instead of relying on a consulting firm to have the policies it takes to maintain their staff.