A $1 part brings down the data center

Every IT pro knows embarrassing moments, although they’re certainly more palatable when someone else ends up red-faced. And no IT career would be complete without seeing a little detail cause a lot of grief, such as a $1 part bringing down expensive equipment. Here’s a story that includes both.

In the early ’80s, I was working as a mainframe computer operator for an electric company. We had recently spent a lot of money and upgraded to a couple of IBM 4341s. We were all proud of the machines, and the execs were no exception.

[ Take a walk on tech’s lighter side with America’s funniest IT stories. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the Off the Record newsletter. ]

One morning, the senior systems programmer came into the computer room to show off the shiny, new equipment. He was accompanied by the CEO, the then-equivalent of a CIO, and a member of the board.

From an operator’s point of view, a systems programmer was a god, and the senior systems programmer was the king of the gods. We were on our best professional behavior as we kept working but watched with awe as the group moved around the room led by the Programmer God, who was also pulling out all the stops to impress the execs.

In those days, computer rooms looked nothing like in the movies. Everything was controlled by a couple of Visual Display Units or dumb terminals. There were no banks of flashing lights. While we had tape drives, there wasn’t much to see as they spun, so the Programmer God had to get creative in his explanations to the execs, opening doors and panels to make the tour even more impressive. However, these facilities were loud, with fans, printers, card punches and readers, tape drives, and the air conditioner all assaulting our ears.

The group made its way to one of the mainframes, which was still pretty boring as the outside of the box had one red LED and one red switch. Apart from that, it looked like a waist-high bench with a flat top, measuring about three feet wide and 10 feet long. The mainframes seemed to be nothing more than unimpressive beige metal cabinets.

The Programmer God opened the side door to show the visitors what was inside. There wasn’t much to see, but at least the visitors got the feeling that they were sharing an inside look at something very important.

He finished talking and closed the door. As the latch closed, all the power to the computer room, including the lights, stopped. The situation was pretty embarrassing, considering who the visitors were and that we were the electric company.

As a horrible sinking feeling settled in the pits of our stomachs, the equipment went silent and the whine from the spinning disk drives died down. Suddenly we could hear the building’s air conditioning, the cars out on the street seven floors below, and the office’s background music, constantly in play but never before heard over the whir of the machines.

We stood still for a moment in the dark and silence, shocked. The Programming God croaked, “Was that me?”

Finally one of the operators sprang into action, found his way to the master console, and pushed the Power On button. It didn’t work. Muttering and apologizing, the Programmer God ushered his visitors out the door.

Back in the pitch-black computer room, we managed to find a flashlight and call our electrician (you have lots of those when you work for the electric company). The electrician arrived, and after 10 minutes of repeatedly asking us “what the hell did you guys do?” found that both 600-amp, 415-volt circuit breakers had tripped. To put this in perspective, these two circuit breakers had enough power to run over 60 average homes and should’ve been sufficient to juice our one largish room.

He reset both breakers, but the eagerly anticipated bang didn’t arrive. We had to hit the Power On button on the console before the system lurched into action. We breathed a sigh of relief, which lasted until we opened the mainframe door again to see if anything was wrong. Everything looked fine — then we closed the door and the lights went out.

We quickly agreed to keep our hands off the troublesome door and to call in an IBM tech. After resetting the circuit breakers, reactivating the power, and taping a large note that read “Don’t Touch the Door,” we managed to run normally for the rest of the day.

That evening, we were ready for the IBM technician when he arrived at 7 o’ clock. He kept asking us what we’d messed up this time and couldn’t quite believe that it all, er, hinged on a door. We asked him to close the door himself, but he insisted it couldn’t be the problem. After nearly an hour, he took his head out of the mainframe and proudly told us there was nothing wrong and we must have done something.

He then closed the door. And the power went out.

He wasn’t so smug after that. We got the power back on, and after a while longer, he figured out the problem.

As we understood it, the mainframe’s air-cooling system was the culprit. It checked the air temperature by using a “temperature sensitive” resistor. As the temperature went up, the resistance went down. If the resistance got too low, the machine shut itself down. The device was worth a dollar or two.

In our case, the resistor had come loose and shook when the door closed. Part of the insulation, of course on the wrong side of the resistor, had stripped back a bit, and when the resistor shook one of its wires touched the metal frame which caused a short circuit. Resistance went to zero. The mainframe thought the air temperature was at “a million degrees” and turned itself off, but did it so dramatically that the building power supply had a heart attack and dropped the circuit breakers.

This tiny artifact had left everyone standing in the dark and asking if they were to blame. For us lowly operators, it was highly gratifying to see the Programmer God, the IBM tech, and even the electrician embarrassed and wondering what they had done — and us not at fault.

Murphy was right: Things will go wrong at the worst possible time, such as during a demonstration to important people. But my favorite lesson is that a computer system that works is one where you haven’t found the bug — yet.

Send your own crazy-but-true tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “A $1 part brings down the data center,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.