Looking for bugs in all the wrong places

Where there’s smoke, there’s fire, and where there’s software, there must be bugs. Though we like to think the work we do as programmers is important and we’re the cornerstones of the industry, we make mistakes like everyone else. It’s just a matter of time before they’re discovered.

Time was, in fact, of the essence, when I worked for about eight years in a facility with a large data center that was home to a large travel reservation system. Reservation agents worked around the clock and used this system to book travel for car rentals, hotel rooms, and airline seats.

[ Pick up a $50 American Express gift cheque if we publish your story: Send it to offtherecord@infoworld.com. | Get a dose of workplace shenanigans — follow Off the Record on Twitter. | For a quick, smart take on the news you’ll be talking about, subscribe to the InfoWorld TechBrief newsletter. ]

Like many other companies at the time, we put in a lot of effort upgrading systems and running tests to verify that no problems would occur related to Y2K. It was a boom time for contractors and vendors. Many systems were upgraded, not because of actual problems, but simply because software and hardware vendors typically would not certify old products as being Y2K-compliant.

Tonight we’re gonna program like it’s 1999

The last evening of 1999 arrived, and I was assigned to be at the office for that shift. We had a large contingent of IT people prepared to identify and resolve any problems that might occur. Another shift was prepared to take over in the morning if there was a need. Jan. 1 was a Saturday, so most of us would not normally be working during this time.

The first part of the shift was mostly uneventful. We snacked on food that had been brought in and shot the breeze. Just before midnight, we manned our stations and set about the task of verifying the systems and software for which we had responsibility.

We counted down the last 10 seconds to issue in the new year and held our breath, our eyes glued to the systems. Midnight came and went. Nothing broke, blew up, crashed. The world did not end. The systems changed over just fine for the most part. I am aware of only one problem that occurred that was actually related to the year 2000, and the team that worked with that software spent several hours working to correct it. All our preparation efforts had paid off.

But with so many programmers and IT personnel closely scrutinizing every last detail, it wasn’t too surprising that we found other problems that had not been previously reported. For instance, we noticed the reservation system had a problem that prevented the users from displaying airline seat availability for the first hour after midnight. After the hour was up, everything worked normally.

The hour of truth

One problem prevented the software from working properly for the first hour of every year. The other problem had a similar effect for the first hour of every month. These bugs had been in the system for years! We found out later that the reservation agents using this application had noticed the problem, but simply tried again after 1 a.m., when the system worked properly, and never reported it.

Sure, it wasn’t a showstopper, and not many reservations were generally booked during that hour — at least, not in that era. With the growth of the Internet since then and anytime-anywhere Web access, it could very well have blown up into a major issue. In any case, we were able to find it and fix it simply because we happened to be looking for problems at midnight.

Look for one problem, find another

Now working at a manufacturing company, I stumbled into a similar scenario more recently. We have a system controlling a production line in the mill that has been in place for about two years. It was provided by a vendor and includes a website for reporting on the products that have been run through the line.

One day, I was chatting with another employee. As we were about to go our separate ways, he mentioned something about the format of the website. He said it was annoying to have to use a scroll bar to go back and forth to see the columns in a table. He suggested rearranging the table to push the less important information to the right, so that the most important information was easily viewed without scrolling. This table is seen by several employees every day to keep track of production and run reports. It has one row of information for each product that comes down the line during an entire shift.

Another programmer and I took a look at the code to see what could be done. In the process of analyzing the columns of data included in this table, we discovered there were two columns for the weight of the product: entry weight and exit weight. The entry weight was sensible, since it was provided to the system as a characteristic of the incoming product. But there should not have been an exit weight at all — there wasn’t even a scale in the line to weigh the product. Plus, the weight did not change during the process that the data in this table was following.

But there it was: The data clearly showed that each piece in the line weighed about 3 percent less when it exited the line — a phantom number. Not only was this information unnecessary, it was also wrong. In two years, no one had ever mentioned this discrepancy and had instead ignored it.

We fixed the original scrolling issue and took out the exit weight from the table. We still don’t know how the system came up with the new weight. But who knows how much longer it would have remained in the table if we hadn’t been fixing an unrelated issue?

I’m still amazed at how often I stumble across programming logic that is not logical or found bugs that might not otherwise have been mentioned. I guess the constant discoveries lurking on the next line of code keeps the job from getting dull.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, we’ll send you a $50 American Express gift cheque.

This story, “Looking for bugs in all the wrong places,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.