Overpromised, underdelivered — and the joke’s on them

The rock-bottom bid price should’ve raised a red flag. The database issues should’ve made our company reconsider. Instead, we soldiered on, but no Band-Aid could save this IT project. The one upside: I only watched this IT disaster from the sidelines.

I once worked in IT for a very large company that did everything from engineering and design to manufacturing and sales. It used a lot of computing power, primarily from two vendors, with about 75 percent of the business going to one vendor and the remaining 25 percent to the other.

[ For more real-life IT tales, check out the slideshow “Step away from the button! 6 touchy tech disasters.” | Pick up a $50 American Express Gift Cheque if we publish your tech story: Send it to offtherecord@infoworld.com. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the anonymous Off the Record newsletter. ]

The sales organization launched a new initiative to improve communications with its zone offices. The plan called for a minicomputer in each zone office and some very large computers at HQ.

Our company put out a request for a pilot project, and the vendor that had 25 percent of our business won with a remarkably low price of $50,000. It saw lots of potential and was willing to underwrite the cost of the pilot as a marketing expense. It sounded good to us and we’d been happy with the business thus far, so our project team signed on.

Strikes one, two, and three

However, the problems started early on. The vendor had fancy report-writing software it believed could be adapted to our needs to get a system up and running in short order. But it became obvious very quickly that the report writer would not meet our requirements, and suddenly a small army of the vendor’s programmers showed up to write conventional code.

After months of slogging through code and performing reviews of completed screens, reports, email, and so on, the system was pronounced ready. To begin, though, the database had to be loaded. We watched as the vendor’s techs ran into problem after problem.

The hardware was terribly unreliable, and the techs determined it would take many reels of tape (this was back in the day) and a couple of weeks to load the database. In addition, the database checkpoint/restart routines were unstable.

The vendor decided to load two reels of tape, then do an image copy of the database. It would then load two more, and do another image backup. If there was a hardware failure, the vendor could go back to the last image copy, restore, and pick up from there without losing too much time.

Well, the system went down a lot. The backups took a long time. The restores that were required when the hardware failed took a long time. And the databases had so many indices, loading took forever. In fact, from start to finish, it took six weeks to do an initial database load. The problem was that at this point the database was six weeks out of date.

The vendor’s team then began to apply six weeks of transaction history to the database. Of course, this took two weeks to complete, plus more time to fine-tune the data and backups to coincide. In the end, it took about 10 weeks to load the database. We all breathed a sigh of relief when the vendor completed this step. We could finally get started!

Such was not to be. We began our pilot operations, but response time on the system was abysmal. Transaction response was measured in minutes, with some transactions taking half an hour or more. The nightly transaction updates from production to the pilot system took so long that the system was late coming up in the morning. Users were saying the system was unusable, and management was screaming.

Too little too late

The vendor had internal meetings and determined it needed to substantially upgrade the hardware to higher-end equipment and make some architectural changes. The vendor had a lot at stake — if it won this project, it would sell a significant amount of hardware and be well positioned to make serious inroads in the rest of the division.

A presentation was made by the vendor to senior management. The vendor had spent a lot of money to upgrade hardware and software, but wanted us to pay another $20,000. It wouldn’t cover all the costs, but essentially, the vendor was looking for a commitment that our company would see the project through.

Our management discussed it, then decided they had no confidence in the project and proceeded to cancel it. It was a Wednesday, the day before Thanksgiving, when the director called the vendor into his office to break the news. Needless to say, the vendor was sick.

That wasn’t the end of it. Over the next 90 days, the vendor took some of its top people, re-architected the system, and moved it to the company’s most powerful computer. The vendor came back in and demonstrated the “new and improved” version and explained that for the pilot project and the re-architected version, it had spent more than $1 million and indicated that we owed it to them to go forward, given the investment. The improved version did, indeed, work much better — but it was too late.

After some gnashing of teeth and wringing of hands, an agreement was reached: The project remained canceled, but our company would buy each of our zone offices some PCs from the vendor to compensate them for the efforts.

In the months that followed, all of the top people at the vendor who were associated with the project left — we assumed they were fired, but we don’t know for sure. Ultimately, the vendor also lost the 25 percent of our business that they did have. A little foresight and realistic expectations would have gone a long way to prevent such an ending.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, we’ll send you a $50 American Express Gift Cheque.

This story, “Overpromised, underdelivered — and the joke’s on them,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.