Follow the money: IT unravels payroll puzzle

It can be infuriating — and humbling — to review the makings of a tech snafu. Yes, there are many reasons why details can be overlooked in any IT project: lack of communication between those in the workflow, limited resources, or a new technology deployed, to name a few. Most frustrating is the fact that if any one of those details had been addressed immediately, the problem likely could have been avoided altogether.

The issue we encountered during a four-day workweek began innocently enough. Nine months before, our company had negotiated a contract with a software vendor for a new payroll and time-keeping package. Being further down the food chain, I was not privileged — initially — to know the full details of the contract. My supervisor handled those details, and the rest of us on the small IT staff worked with the accounting department to get the new software working smoothly.

[ More from InfoWorld about the IT profession: The 12 most dreaded help desk requests. | Follow InfoWorld’s Off the Record on Twitter for tech’s war stories, career takes, and off-the-wall news. | Subscribe to the Off the Record newsletter for your weekly dose of workplace shenanigans. ]

On that front, events had progressed well. Data was converted from the old software into the new one. We paralleled several payrolls to verify deductions and such before shutting down the old package. The conversion and installation appeared to be a success, and the accounting employees had grown comfortable using the new software. In my mind, the project had been completed — which is why I was caught off guard the week of Labor Day when problems were reported.

Per my usual routine, I was in the office an hour before other employees arrived to make sure servers and other peripherals were running. Since this was a short workweek due to the Monday holiday, some of the payroll staff also came in early that day and began processing checks and electronic funds transfers. But after a few minutes, the manager of the payroll department was in my cube reporting that the software refused to function, saying its license had expired.

Not surprising for our company, no one in payroll knew if or when the license fee had been paid. No one had yet arrived for work from the accounts payable division. The IT director wasn’t in yet, either.

The manager and I made our way to his desk. As we walked through the payroll department, I glanced at an employee’s monitor. There, in the lower-left corner of the screen, was a notice in red that the license would expire the prior day.

When I asked about it, I got a range of replies. The payroll manager claimed to have never seen it, another employee said it had been showing up for a week, while the employee whose screen I was viewing said it had been there for a month. None of them had thought it necessary to investigate or to notify someone that the license was about to expire.

By this time my supervisor had arrived. He explained that he’d negotiated the software contract to be paid on a monthly basis that also included part of the original purchase amount along with the yearly maintenance fee. He verified the bill had been paid each month and began to make phone calls to his contacts at the vendor. Of course, as luck would have it, the vendor was in another time zone, so another half hour passed before we got a return call.

The vendor said the software wasn’t working because we were using old keys and the company had emailed new keys to us three months before. The rep even said the team had carbon-copied me as well since they had my contact information from ongoing communications we’d had about setting up and using the software.

Not wanting to outright deny ever having received the email, I went to my PC and checked my saved messages. I specifically create a directory for each vendor’s email and save all communications from them in that directory. I had no emails from this vendor the entire week that the rep claimed to have sent the keys.

I reported this to my supervisor, who was frantically searching his emails, also to no avail. In the meantime, the vendor reissued the keys, we applied them, and payroll was back up and running. But the missing emails were still a problem.

My supervisor and I looked through our emails again — nothing. Studying the calendar more closely, we discovered that another change had been implemented the week prior to when the software vendor said we’d been sent the new keys. We’d switched to a new email scanning and filtering service in the cloud to relieve some of the burden on the mail server. The service had required close monitoring at first, training it on what to permit to let pass to the mail server.

We traced what had happened. Initially, this software vendor had been flagged and blocked, which was discovered about a week into using the new email service. The settings were changed to allow future emails from the vendor to go through, and we verified that at that point normal weekly updates and communications came in without a hitch. Since the mail was flowing, no one had dug any further back; otherwise, they would have discovered the email with the keys was still blocked.

It’s hard to cover all the bases even though we try. We can make lists of details to address and check them twice. We can plead with users to report errors and messages to IT immediately. But even though it’s maddening, in the end it doesn’t hurt to get a reminder now and then to be more diligent on checking and following through with tech changes.

Do you have a tech story to share? Send it to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “Follow the money: IT unravels payroll puzzle,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.