Dealing with a bad vendor

Dear Bob…

I recently starting working at a company as a network engineer. I’m working with a good group of in-house people (I think). There is also an integrator who has had four people onsite, full time, for a decade! We are locked into a contract with them with a 120-day termination clause, which I am trying to get terminated. We have scaled back any hours beyond the contract minimum and have stopped buying product through them.

The problem is that someone is sabotaging my work. I would almost think I was going crazy but three of the existing staff, on different occasions, confided in me that this has happened to “smart” or “competent” people in the past, and these people have quit as a result. For the most part we have pulled admin rights off the integrators accounts but these guys (and one gal) are technically smart and know passwords to all kinds of admin level accounts besides their own. In particular one of the people is managing our redundant PIX setup and I have little Cisco knowledge to take that on. So essentially I have an untrusted person in a position of high trust.

What can I do?

Untrusting – but not crazy

Dear Untrusting …

A few steps you can take. Which order you take them in depends on the political situation in your company. Other than the first step: Lay low and document. Be specific and detailed in every event that looks like sabotage or other malfeasance. Wait until you have at least three events documented before you do anything else.

Another step: Talk (privately) to Internal Audit, not about your specific concerns, but more generally about whether the level of access available to the vendor is in accordance with sound audit principles. If they are sabotaging your work (I presume we’re talking about non-physical sabotage: Introduction of bugs, deletion of documents in your private folders, undoing administrative changes – that kind of thing and not dropping servers on the floor), I’d be pretty sure that one of two situations is the case. Either the system generates an audit trail that will show which login made which changes, or it doesn’t. If it does, you’ll have direct evidence. If it doesn’t, it doesn’t pass an audit.

Going beyond my guesses, the short version regarding Internal Audit is that it’s in a position to require proper independent oversight of the vendor’s activities.

If system logs and audit trails don’t do the job, as a network engineer you’re in a position to do some network sniffing. If they’re logging in with administrator rights using accounts for which they aren’t authorized, you should be able to detect it. Add that to your documentation. If you don’t have the tools to do so, talk with whoever heads up Information Security, with enough information to at least validate that there is a cause for concern. Between InfoSec and Internal Audit, someone with some clout ought to care a bunch about this issue.

Best case: They run with it and you don’t have to carry the ball yourself. Worst case – they aren’t interested or your company doesn’t have such groups. In that case, meet privately with the CIO. If the vendor is, in fact, mis-using administrative accounts and there are proper audit trails, you can demonstrate malfeasance. You need to get the CIO on the right side of this issue before you take any concrete steps or you’ll be in danger of embarrassing your boss. This is generally considered to be a Bad Thing.

Unless whoever negotiated the contract with your vendor was a complete idiot, it will include a section covering termination for cause which will avoid the 120-day penalty.

One other suggestion: Don’t confide in anyone that you’re taking these steps. Ben Franklin once said that three people can keep a secret if two of them are dead. Be discreet.

But … don’t fall into the “junior G-man” trap. Don’t start thinking of yourself as an undercover detective. You’re just a network engineer doing his job. Part of that job is doing your best to detect intrusions, whether they’re from internal or external sources.

– Bob