Panic mode

analysis
Jul 31, 20076 mins

Lack of knowledge + heat of the moment = bad decisions I started in the industry in the early eighties and worked as a technical salesperson for a local micro-computer company. We had a client that purchased a 286-based server running SCO Xenix to run a FoxBASE application in a multi-user environment ... pretty hot stuff in those days. The application had been written by a local consultant -- let's call him Dway

Lack of knowledge + heat of the moment = bad decisions

[Ever worked with a technology that seemed to defy all logic or simply drove you up the wall? Submit your story to offtherecord@infoworld.com]

About four weeks after the system was up and running, Herman was on-site doing some coding and the two office admins came into his office. They told him that the system was locked up and that they couldn’t continue their data entry. Their boss was exceptionally impatient and demanding, so they wanted to know what Herman could do to help them. Dwayne hadn’t thought it was important for Herman to have anything but user training for Xenix, so he couldn’t really do much.

The office admins put pressure on him to do something, so he called Dwayne … who had been given more extensive training by necessity, even if he didn’t want to learn it. Somehow, Dwayne had forgotten that the console had 12 distinct sessions and all he had to do was have the office admins Alt-F key to a different screen. Somehow he had forgotten that there were 15 terminals scattered around the office. Somehow he had forgotten the cheat sheet I had left him that told him how to identify and kill a process. Instead, he told his programmer that he should place a call to me to get help.

Herman called me, but I was out of the office for about 45 minutes (this was in the days before cell phones and pagers). He left a message and went to update the office admins. They pressured him more — they just couldn’t wait 45 minutes. They didn’t want to risk the wrath of their boss.

Herman pondered the situation.

He realized that when he turned off his computer at home and turned it back on, that seemed to cure almost all problems. He figured if it worked for his computer, it would work for this one, too.

So he flicked off the power switch on the system.

Xenix — being a Unix system — of course started up, discovered the file system had been improperly shut down, and immediately started running fsck to straighten things out. The screen output from fsck, however, can be somewhat frightening if you don’t know what it’s doing.

After seeing a bunch of “Deleting File XXXX” messages go by, Herman panicked … and turned the system off right in the middle of running fsck.

He restarted the system and a message popped up on the screen: “Failure to boot.” At this point, he really began to panic. He tried to figure out what he could do next and thought, “Maybe I can find the disks for the system and fix this.” So he asked where the disks were and the office admins found them in the safe.

He looked through the disks and saw that one was labeled “N1 Boot Disk,” so he inserted it in the floppy drive and restarted the system. The system came up and asked for him to insert the file system floppy. He again looked at the disks and saw a disk labeled “N2 File system,” so he inserted that in the drive and pressed Enter to continue.

Up popped a prompt that said “WARNING! Continuing with this procedure will erase the present contents of your hard disk. Do you want to continue? Y/N?”

He looked at the screen for a long minute … and then turned the computer off.

At that moment, the office admins came in and asked him how things were going. He fumbled and said he was working on it. They reminded him how important it was to get things back up right away.

He took a moment to ruminate and reasoned that if he were very careful, maybe he could use the disks to fix things. So he went through the floppy boot process again, pressed Y and Enter. Up until that moment, all that was wrong was that the boot files had been corrupted — a five-minute fix. After he started the procedure, the disk was repartitioned and formatted. But partway through the process, he panicked again and turned off the system during a floppy read.

I arrived on site about 15 minutes later to find one very nervous programmer. After a few minutes of questioning and forensics, it was pretty clear what had happened, so I said we’d have to restore from backup.

That’s when I found out that the office admins, who were tasked with doing backups, hadn’t thought backups were important and so hadn’t done them for three weeks. They’d been too busy working 50 hours a week doing data entry into the new system. They asked how soon we could have the system up — they needed to get final edited copy in their boss’s hands in about two hours.

At this point, I figured I’d better at least boot to floppy and see if any portion of the file system remained. That’s when I found out that programmer — during his panic — had managed to damage the boot/file system floppies.

Trembling, Herman asked, “How bad is it?”

I told him he’d single-handedly destroyed the system and that he’d better be ready to explain it to the hard-ass guy who ran the company.

When told that the only technical way to possibly recover the data was to use a very expensive disk recovery service, the hard-ass boss came up with a better solution. After I’d used backup copies of the floppies I’d kept for emergencies to restore the system, Herman got some fine on-the-job experience as a pro-bono data entry clerk for three weeks. Oh, and the office admins starting backing up twice a day.

I’ve used this story countless times over the last 25 years as an example of what goes wrong when well-intended people do really silly things in a moment of stress.

infoworld_anonymous

Since 2005, IT pros have shared anonymous tech stories of blunders, blowhard bosses, users, tech challenges, and other memorable experiences. Send your story to offtherecord@infoworld.com, and if we publish it in the Off the Record blog we'll send you a $50 American Express gift card -- and, of course, keep you anonymous. (Note that by submitting a story to InfoWorld, you give InfoWorld Media Group, its affiliates, and licensees the right to republish this material in any medium in any language. You retain the copyright to your work and may also publish it without restriction.)

More from this author