Nice VoIP upgrade! Too bad you killed the sales lines

IT lives to solve problems, but we techies can do without the frustration of ironing out issues that could have been avoided if people had taken the time to document their systems’ workings. I’ve run into many of these situations during my IT career, such as this three-ring circus that should have been a simple phone fix.

The company in question had only one IT employee — whose background, it turned out, was in a different field and qualified as “IT” in only the most basic sense. Thus, I was hired as a consultant to come in once a month or so to take care of computer, printer, or network glitches. Little did I know I’d get caught up in a bigger task than my title would suggest.

[ Take a walk on tech’s lighter side with America’s funniest IT stories. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the Off the Record newsletter. ]

One day, the office manager asked me to take a look at the phone system — employees had been having problems with dropped calls and busy signals. The company hadn’t transitioned to much of a Web presence yet, so it did most of its business by telephone and bulk mailings.

I found out that the office manager, rather than the IT-in-name-only person, was in charge of the phone system. The office manager hadn’t made any changes to the phone system because she was worried she’d break something. She also hadn’t called the vendor in years because the last time a tech had come out to fix a problem, it cost a lot of money. Furthermore, she didn’t trust that the vendor had given good recommendations in the past.

She thought that since I could maintain the computer systems maybe I could fix the phone issues, too. I didn’t share her confidence, but decided to take a look. The company was paying me by the hour, and my fee was cheaper than the vendor’s.

I took a look at the PBX, and in the course of checking its specifications, I was quite surprised to discover its handling capabilities. Intended for 10 users, with a ceiling for 25, the system had a maximum of eight concurrent lines for both inbound and outbound calls, as well as voicemail checks. Considering the company had grown to a little more than 100 employees — a sharp increase from the five users at the start of business two decades earlier — it was no wonder there were so many busy signals.

I reported back that the firm was using the PBX way beyond its intended capacity and would need to upgrade. I recommended a move to a VoIP system — still relatively new then — and I would help work with a vendor to port the system. I found a capable vendor, and the company agreed on the upgrade.

I was a bit nervous at first, because prior to this I had never used a VoIP system. But I was pleasantly surprised when the upgrade went smoothly. We ported all the numbers over, at the same time keeping some analog lines should the VoIP system go down. The new system had 16 concurrent lines, which seemed to resolve the issues with busy signals and dropped calls, and it allowed the company to add capacity in the future.

The new VoIP system was very easy to maintain. The customer could take care of all the simple phone changes via a Web interface, and I showed both the IT employee and the office manager how to do it. They were thrilled.

A few months later, the company hired a finance guy who wanted to see if he could save money on the telecom bill. I located a different vendor to provide the PRI for the VoIP system at a much cheaper price, and the finance guy gave me the go-ahead. While handling that project, I asked him if the employees needed all those phone lines. He didn’t know and requested I consult the other managers on that question.

Nobody had any idea, and to our frustration there was no documentation whatsoever. A handful of numbers seemed to serve no purpose at all. When I called these mysterious numbers, quite a few of them kept on ringing without going to voicemail or a greeting; I figured they must not be in use.

After I’d made the calls, I wrote down the numbers that I thought could be dropped when we moved to the new PRI provider. This list was forwarded to the department managers, who were asked to verify if they could be removed. Everybody agreed that those numbers were extraneous, and we proceeded with the change on a scheduled Friday.

For the first two weeks, everything seemed to work fine — until some users told me that they hadn’t had calls to certain hunt groups since we moved to the new vendor. Nobody had mentioned hunt groups to me before.

It took a couple of hours to find the cause, but it turned out that some of the numbers we’d canceled were, in fact, being used to route to hunt groups. I guess nobody in the hunt groups picked up the phone when I was testing the numbers, and none of the employees remembered them when we’d passed the list around.

As these hunt groups were for external clients, I made a frantic call to the previous PRI provider to get the missing numbers to the new provider. Since we were no longer a customer, the old provider made no haste in helping us. Instead, the vendor gave us the runaround before finally offering the numbers, albeit as separate analog business lines. It was another three weeks before the new PRI provider was able to get those numbers and port them over as DIDs.

Most of the delay was due to the previous vendor, though the fault rested with us. I guess that’s the problem when you have no phone system documentation, changes are made on the fly, and current employees are unaware of decisions made in the past. On the bright side, the company cut its telecom bill in half and were happy in that regard. And it gave me a convincing illustration when talking to future customers about tech best practices.

Send your own crazy-but-true tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “Nice VoIP upgrade! Too bad you killed the sales lines,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.