Don’t touch that cable if you know what’s good for you

If there’s a single common element to most successful data centers, it’s at least one admin with a touch of OCD. At some level, obsession and compulsion are necessary to keep things running smoothly and reducing the propensity for the situation to go haywire.

It’s not just about keeping a close and detail-oriented eye on the software and systems, but also paying attention to how the data center is constructed and maintained. Basically, it comes down to determining when to put a “more urgent” project on hold to rip and rework a completely different subsystem, for no other reason than it wasn’t done right.

[ Also on InfoWorld: 6 things every IT person should know | When to cut bait on old IT | Get expert networking how-to advice from InfoWorld’s Technology: Networking newsletter. ]

One area where this has a tendency to crop up in general-purpose data centers revolves around cabling. When the data center was built, the cabling was immaculate. It was color-coded, neatly patched, wrapped to ladder racks and top-of-rack guides, gracefully strung down the rack sides, and carefully secured to each server and switch with an aesthetic that might just qualify as art. But that was The Before Time. That might have been two or three years ago, several server generations ago, several storage arrays ago, and possibly even several admins ago.

Now, like a Roman ruin, the origins of an expertly constructed and beautifully managed data center are still visible, but mired in the slough of the intervening years. The color-coding has long been rendered meaningless, though some vestiges remain.

Where a red cable used to always signify an unsecured link, now it might be an internal management link or even storage. Cables draped along those ladder racks have been covered over with a mishmash of other wiring, some of it coiled up in various places because there were no 15-foot cables to be had. Instead, 25-foot cables were called into service, their excess length gathered into haphazard rings and ziptied together so that no human may trace them ever again.

But this isn’t the case in all data centers. Some have the benefit of being highly fixed-purpose, where the equipment turnover is low, and upgrades usually involve a forklift. If every rack has and will always have 40 servers, you can probably reuse the original cabling without much fuss. But in data centers with high turnover of heterogeneous equipment, it can be a real challenge to maintain coherency in the face of a revolving door of gear, all with different needs.

In those spaces, it’s not uncommon to see bundles of cables hanging unattached within racks, yanked out during a refit or emergency, and left to dangle for months or years until someone asks where they go. Sadly, nobody really knows anymore.

In other racks, there might be a collection of links heading into smaller switches run so tightly that the switches can’t be removed from the rack without removing the cabling as well. When one of those switches goes bad, it becomes a much more tedious task to replace it.

In yet another rack, someone decided they didn’t want to bother racking some piece of gear, so it’s sitting on the server below it. When this is discovered, it’s guaranteed that the server forming the foundation needs to be replaced and can’t be moved unless and until the equipment riding on top of it is removed, which wasn’t in the original plan. Oh, and the original rack rails are long, long gone.

Eventually there comes a point when the mess begins to make every task more difficult — when you need to take a step back and decide to rip everything apart and put it back together again. Sure, you can try to bend space and time to replace that switch by removing rack side panels and shoehorning it out of there, but it might be a better plan to suck it up and fix the source of the problem. Congratulations, your 10-minute project turned into several hours at best and an entire weekend at worst.

Then again, this isn’t always a good idea. However tempting it may be to rework objects off the cuff, you need to be cognizant of the potential threat of an unplanned housecleaning. While it may seem that it’ll only take a few minutes to unplug a half-dozen ports and reroute the cables, you might instead find that several needed to be in certain ports due to auto-negotiation problems or some other specificity (though you made sure they were all in the same VLAN) — you’ve just significantly lengthened your day.

Sometimes that urge must be resisted and instead channeled into planning to do it right, with forethought and research. In other words, it could be best to tackle the whole thing at once, rather than while trying to do something else.

After a certain amount of time and a certain amount of churn, it’s definitely in your best interests to bite the bullet, schedule downtime if at all possible, and start anew. In a data center, it’s not spring cleaning, necessarily. It’s more like leap year cleaning — it should happen once every four years or so.

Otherwise, there will come a time when the avalanche of mystery cabling and long deceased components will force your hand. By definition, that’s guaranteed to occur at the worst possible time.

This story, “Don’t touch that cable if you know what’s good for you,” was originally published at InfoWorld.com. Read more of Paul Venezia’s The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.