Sloppiness in the server room brings down the network

I’ve worked in IT for more than 10 years and have seen numerous examples of basic sloppiness. It’s amazing how many places overlook cable management, wire management, and labeling — at least in key areas such as servers and network racks.

One day when I was working as an IT contractor, I received a work order from a networking group that was located out of state; I was to go to a customer site and deal with an emergency situation. Some switches needed to be fixed and remote access to the switches set up. The estimated time on-site was two hours, and the last line in the work order read, “Network is down at location, so time is critical.”

[ Also on InfoWorld: “IT’s worst addictions (and how to cure them).” | Follow Off the Record on Twitter for tech’s war stories, career takes, and off-the-wall news. | Subscribe to the Off the Record newsletter for your weekly dose of workplace shenanigans. ]

I arrived and was greeted by a manager who was relatively clueless about the IT infrastructure. Moreover, there weren’t any on-site techs because the company’s help desk was also located out of state. She took me to “the room with all the wires,” a multipurpose area open to anyone that hosted a few servers, two shared workspaces for employees, and a radio charging area.

There were two rail racks. The one on the left had patch panels, switches, some shelves, and enough patch cables for a family of raccoons to make an apartment complex. The 15-, 20-, and 25-foot patch cables connected the patch panels to the switches, which were only 3 or 4 feet apart, and the slack in the cables hung to the floor. The rack on the right had a few servers on it. Surprisingly, the covers for the wire management were there — but weren’t being used.

I found the area I needed and called the designated contact to set up the remote management. The call went straight to voicemail, so I left a message and reached out to my original contact, letting them know I was unable to reach the support person.

While I was waiting for a return call from either contact, I made small talk with the manager to find out what the business did. She informed me it was a customer service firm and a 24/7 operation, and downtime really hurt the bottom line. I was surprised by this because the company had already been out of commission for well over 24 hours.

I finally got a call from a support person — not the one assigned to work with me, but someone the company found. We got started and discovered that the switches were not managed. The support person informed me there were old hubs in the warehouse that had been removed during an upgrade a few years prior. He wanted me to put one of them back into service and get a few workstations up and running until new switches could be shipped to the site. He said he would ship out new switches to arrive the next day.

I found the hubs after looking around the warehouse a bit and was able to set them on the rack in a way that sufficed for temporary use. However, I ran into a problem: None of the jacks in the building were labeled to the patch panel, and they had more than 100 drops. I managed to tone out 4 or 5 of the important ones and powered on the malfunctioning switches to see which jacks had equipment plugged into them. In the process, the problem switches started working just fine.

I called the new support contact to tell him the switches were working after all. He said they still wanted to change from nonmanaged to managed switches, and he asked me to come back the next day to do the upgrade. I also asked for approval to clean up the wiring, and he said he’d send the request up the food chain.

I returned the next day, opened the package with the new switches, and got started. I hadn’t received approval to clean up the wiring, so I was focused on the switches. But the disarray was so great that I ended up having to clean enough of the wiring just to be able to pull out the old switches. In the process, I discovered a total of three patch cables plugged into a problem switch that touched the ground and went back into another port on the same switch, causing a broadcast storm that had brought down the whole network.

The three days of downtime at this site must have cost the company tens of thousands of dollars in lost productivity and extra help desk fees, all because of a messy rack and far-flung tech support.

I’ve worked at many sites and different companies, and it seems that the organizations with better IT budgets and better-organized IT departments normally run better overall. This company may have saved time and money in the short term by leaving its cables in disarray, but the messiness took a large monetary toll when repairs were needed. The details matter, even down to the orderliness of a network rack.

Do you have a tech story to share? Send it to offtherecord@infoworld.com, and if we publish it, you’ll receive a $50 American Express gift cheque.

This story, “Sloppiness in the server room brings down the network,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.