A missed detail derails a server room move

None of us has time for long, puzzling, drawn-out tech problems. But the source must be found so the nightmare can end. And even though every detail is triple-checked, something small can get you — such as one wrong item on a contract proposal.

Recently, the office where I work consolidated space, and the server room had to move to a different location in the same building. Company policy requires three quotes for each part of the construction phase, so I made quite a few phone calls and sent a lot of emails.

[ For more stories about exasperating IT jobs, check out “10 users IT hates to support” and “7 blowhard bosses bollix up IT.” | Pick up a $50 American Express Gift Cheque if we publish your tech story: Send it to offtherecord@infoworld.com. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the anonymousOff the Record newsletter. ]

I was able to get quotes from several electrical, HVAC, alarm, and cable installers. But finding three quotes for companies that install network cabling was much more difficult. I contacted six companies, but only one even came out to review the requirements and give us a proposal.

I’ve been working with computers for more than 25 years, and I’ve been at my current position as a network administrator for 18 years. I don’t know much about electrical circuits, HVAC, or alarm systems, so I had to trust that the contractors knew what they were recommending. I do work with CAT 5 and CAT 6 cables fairly frequently, but I have very little experience with fiber. When the salesman asked if we wanted multi-mode or single-mode, I said we’d like the same type we currently had.

It took a few weeks, but we made decisions on which contractors to use and did the paperwork. The company said we could move forward with the one network cable installer that had given us a quote. Finally, we were ready to move on to the next phase.

Let the work begin

During the next month, I lost a lot of sleep, as I had to get to work early many days to let the contractors into the building. Company policy also required that visitors be escorted, which meant I wasn’t doing my usual job much of the time they were there. So I was also staying late to catch up on work.

The cable installers were at the office for three days installing copper and fiber, including some runs to the building’s phone closet in the basement. The cables were installed and tested, and the results showed the cables were working properly. We then set our sights on moving and hoped the project would be completed within days.

Boy, were we wrong.

The problem that wouldn’t go away

Saturday: My coworker and I got to the office bright and early and spent about 12 hours moving the server equipment, most of which would be set up the same way it had been before. There were a number of networking changes, but I’d worked on those ahead of time.

The item I’d been most concerned about was our Internet connection. We have a 20Mbps MPLS connection to another location, and then we get on the Internet from there. We also have a VoIP system that runs over the same MPLS connection.

There were a few bumps on the road during the day, including a Cisco command I’d overlooked. Our company’s Cisco expert was on vacation in an area he had warned us had spotty cell reception, so we crossed our fingers and hoped to reach him. Fortunately, we were able to contact him around 7 p.m., and he diagnosed the problem and gave us the fix in five minutes.

By 8:30 p.m. we were confident we had the critical equipment moved from the old server room to the new one. There were a few glitches we’d have to work on the next week, but the resources our users needed were working — or so we thought.

Monday: I was at work early to make sure there weren’t any problems users would notice. The file and printer server were working. Email was working. The Internet, intranet, and phones worked. We were ready for users.

But around 9 a.m., we started getting complaints about Internet and phone problems. We checked cables, network switch ports, and switch and router configurations. Finally, we came to the conclusion that there was another Cisco configuration we’d missed. The switch our Internet connection had been using for more than a year was now sitting unused on a shelf, and we decided to put it back in service. We put it in our rack and made sure to wire all the devices to the same ports they’d been working on before the move.

We breathed a sigh of relief that the Internet and phones were working better than they had been. We decided to leave it that way overnight and call our Cisco expert the next day and get the proper configuration on the switch ports we wanted to use.

Tuesday: Our Cisco expert looked at the switch configurations from his remote office and didn’t see a problem. But the Internet, phone, and intranet connection were still spotty, although they were up more than down. By the end of the day it seemed stable, but we weren’t sure it would last. And, of course, people were complaining.

Wednesday: The problems continued, so we contacted our ISP at about 9 a.m. to see if they noticed errors on their end. We were down most of the day, with a few up times. A tech finally called back at about 8 p.m. and agreed to replace our fiber-to-copper converter. I left at about 9 p.m., but my coworker waited for the tech to arrive.

Thursday: I arrived to find a note on my desk from my coworker saying he was leaving after 3 a.m. The tech hadn’t arrived with the new fiber-to-copper converter until about 2 a.m. The new one was DOA, so a different tech would arrive during the first shift with a working unit.

The new tech arrived at about 9 a.m., and it took him 30 minutes to install and configure the converter. The Internet came up within seconds, and the phones and intranet also worked. Success! Or so we hoped.

Thursday afternoon to Monday morning: No reports of Internet problems. We’d solved it! We figured that the fiber-to-copper converter must have gotten damaged during the move.

Monday afternoon through Tuesday: Intermittent phone and Internet problems yet again. We went through the possibilities again but were out of ideas on how to fix it.

Wednesday: We contacted the Cisco partner we work with, and one of their senior techs came out to look. He wasn’t able to find a problem that would explain what we were experiencing.

Getting closer

Thursday: My coworker and I looked through everything for the umpteenth time. What had we not tried? Could it be the new fiber cables? We called the cable installers and asked them to retest the cables.

Friday: A few minutes before 7 a.m., the cable installers arrived, but it wasn’t the same team that had installed the cables. It took them about a minute to diagnose the problem: The fiber run to the basement was multi-mode fiber, and we needed single-mode.

I looked again at the original proposal, and it did specify multi-mode. Oops! Was the fault mine or the salesman’s who had created the proposal? At this point we didn’t care. We’d lost more money in downtime than it would cost to put a new cable in.

We made an agreement with the salesman that we’d pay for labor and the installation company would pay for the materials. And I wanted to replace everything, so we also placed an order for new single-mode patch cables.

Monday (about two weeks after the move): The installers arrived at 6 a.m. The cable was run to the basement and terminated in just more than two hours. By noon, UPS delivered our new patch cables, and we got them installed in the basement and in our space. We then did a switch to the new cable, and within seconds the Internet, intranet, and phones were working!

Several weeks later, we’re still up. And no one has complained about the Internet connection, phones, or intranet. Maybe it’s fixed this time….

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, we’ll send you a $50 American Express Gift Cheque.

This story, “A missed detail derails a server room move,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.