Button it! A tiny detail snags a system setup

“What could go wrong?” These infamous words have prefaced the downfall of many a best-laid tech plan. But there was a time when I didn’t know any better.

In the early ’80s, I worked for a small startup division of a legacy electronics company producing one of the first interactive 3D graphics systems. Though these systems were sold as commercial products, they were handmade prototypes — wire-wrapped instead of soldered — on the inside, and each was slightly different.

[ Pick up a $50 American Express gift cheque if we publish your story: Send it to offtherecord@infoworld.com. | Get a dose of workplace shenanigans — follow Off the Record on Twitter. | For a quick, smart take on the news you’ll be talking about, subscribe to the InfoWorld TechBrief newsletter. ]

I managed the customer support tasks, such as installing systems at customer sites and training them on how to use it. But a combination of a Wild West environment and an understaffed team meant I was able to assign myself to other jobs as well — whatever needed to be done to gain a good standing with our customers, which happened to be aerospace companies. After one too many customer service fiascos, I stepped in to fill the gaps dotting our workflow.

For example, when an existing customer site was sent the wrong software upgrade tape, I appointed myself software librarian. And when a system was delivered to a customer facility dead on arrival, I became the quality assurance manager in charge of final outgoing inspection.

One small part, one big decision

This worked for a while, but the day arrived when I got the better of myself.

The parts buyer, who also wore the hardware operations hat, came to me and explained we had an order ready to go, but the system lacked a Reset button. It was the end of the month and the end of the quarter, and we needed to put the system on the truck so that we could book the revenue.

This wasn’t great news, but not necessarily a game-changer. I went to see if there was anything else wrong with the system. I was able to test it without the button: When I wanted to reset it, I grabbed two wires sticking out of the square hole where the button would mount and touched their stripped copper ends together. Everything else on the system was OK. We were just lacking one simple analog 12-volt press-to-connect single pole button.

The parts buyer implored me, “The button will be here Monday, which is when you’re going to do the installation anyway. Let me ship the system now, so the company can book the revenue this quarter. I’ll come with you to the customer’s site on Monday, bring the button, and install it myself.”

I agreed, glad he was willing to fix the problem and thinking to myself, “What could go wrong?” We shipped the system that day and met our accounting deadline.

On-site panic

On Monday, the button arrived, and we made the two-hour drive to the aerospace customer’s site.

The system was waiting for us. We unpacked it, and the parts buyer installed the button first thing. Foolishly thinking it was all downhill from there, we plugged in the system, powered it up — and nothing. It was dead on arrival.

Panicked, we tried to unearth the problem, but we were stumped. Never mind that it had checked out OK before we shipped it — it definitely wasn’t working now. There wasn’t even a video carrier coming out of the system, although the power supply was producing the right voltages, the boards were seated, and the fragile wire-wraps were in place.

Finally admitting defeat, we called our company’s chief engineer, who did the final stages of manufacturing on all the systems (though in those days nothing was documented). The phone call stretched into hours. Finally, he decided he needed to come to the site himself and left in time to make the trip before the end of the day.

When he arrived, the first thing he did was hook up his logic analyzer. A couple of minutes later he said, “This is strange, it’s constantly resetting.”

Then he removed the newly installed Reset button. The system came up with no problem. We all were amazed. And relieved. And angry that we’d missed something so simple.

To make a long story short, the supplier of buttons had sent the wrong part number — the button fit in the same square hole but was “press to disconnect” instead of “press to connect.” Unless the button was held down, the system would constantly reset.

I learned a valuable lesson that day, which I express as this homily: Test the way you will deploy. I sometimes add, “Unless failure is an option.”

To tell you the truth, faced with the same problem today, I’d probably make the same choice, but with a much better awareness of the risk involved, balancing certain loss of revenue against possible reduction of reputation. But I certainly wouldn’t say, “What could go wrong?”

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, we’ll send you a $50 American Express gift cheque.

This story, “Button it! A tiny detail snags a system setup,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.