A microwave mix-up nukes the network

You can never get too comfortable in IT. Even the best-implemented plans and top-of-the-line tech can fizzle in the face of shockingly mundane details, and all eyes are on IT until the problem is resolved. Here’s a recent story of a problem we dealt with at our company that had an almost comical ending for all the grief it caused.

Our company is spread out across five locations. The first is a headquarters office. The other four are scattered throughout a remote city an hour south of headquarters. About a year and a half ago, we set up a microwave point-to-multipoint system linking the sites and significantly boosting bandwidth over the discrete T1 circuits that had been the sites’ only connection to the core.

[ More from InfoWorld about the IT profession: The 9 most endangered species in IT. | Follow InfoWorld’s Off the Record on Twitter for tech’s war stories, career takes, and off-the-wall news. | Subscribe to the Off the Record newsletter for your weekly dose of workplace shenanigans. ]

We resolved several issues right after installation by upgrading firmware and replacing a bad run of equipment. Otherwise, it worked fine.

Soon after we set up the microwave system, we moved on to convergence. We had to migrate all of our internal telephone signaling traffic to our data network in order to keep our phone system current with the manufacturer’s latest offerings.

A year later, both systems had been quite stable — until two days ago.

It was just after lunch when the call came. The entire networking system in the remote city had gone berserk. One site was completely cut off, and the three others were having major connectivity issues.

It fell to me to troubleshoot because I was the principal on the microwave installation; also, I was the only one around who was part of the convergence project. I tried this, tried that, dialed into the system remotely after hours to make changes that couldn’t be done during business hours. Nothing worked.

I forced everything to reconverge over the T1 circuits, which fortunately hadn’t been disconnected and made plans to drive down to the city the next day. During this entire time, I was also dealing with the many people clamoring for results.

I got together supplies to deal with the most likely scenarios. At the top of the list was the site where the base station was located because that was the common point between all the failing locations. Also, as of the second morning, we couldn’t even log into the base station remotely. The second most likely culprit was the site that was completely cut off.

One of the guys from the radio shop traveled with me. We arrived at the first site with the tower where the base station was located. After restarting the base unit, things were no better. We could get into it now but still had problems with all sites. We tried switching to the failover unit already mounted on the tower, but that didn’t work any better.

After an hour, we put the primary unit back in service and headed to the second site, assuming the subscriber unit at that location had failed so badly it was affecting all the others. We’d probably have to replace it and re-align the new one.

Our branch office at the second site was located in a corner of a small strip mall. We pulled into the parking lot and looked up at the roof, where we saw a couple of people’s heads bobbing up and down. “What’s up with that?” we asked each other.

We found the access door unlocked, which was unusual, and climbed the ladder to the roof. There we discovered a crew with buckets of hot tar, mops, and rolls of roofing material in the process of replacing the entire roof covering on the strip mall.

They had taken our microwave unit, which was mounted on a post attached to a square base frame and anchored with cinder blocks, and moved it eight feet out of their way in order to work on the roof under it. But instead of pointing it in the same direction as before or off to nowhere to the east, they had managed to point it directly at the farthest building in our group of four offices — thus, blasting it with interference, disrupting the other three sites, and completely breaking the connection for this building.

The roofers said they could have our unit back in place by the middle of the afternoon, so we grabbed lunch, took our time getting back, and watched them return the unit to its original location about 20 minutes later. From there it was a quick job to fine-tune the alignment and verify that all sites were up and running again.

There is another side to all the nagging we get from people asking when things will be fixed: When the technology is working again, people can be grateful. We were hailed as heroes, and in the end, people even got a laugh out of the situation — until the next time anyway.

Do you have a tech story to share? Send it to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “A microwave mix-up nukes the network,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.