Our fault? Read the project audit and weep

How can tech projects go south? Let us count the ways: Cost overruns, change of staff, a full moon … And when it happens, you cross your fingers and hope all the precautions and documentation hold up.

There I was in the conference room at the HQ of a major multinational corporation. I was responsible for leading a software project going into its third year with a staff of 60 working on it. The cost had increased by 50 percent, and now I was being grilled by the executive sponsor (a VP whom I had never met before), a major audit firm that the client had hired to look into the situation, and key members of the project team. How could this have happened when we had done so many things right?

[ For more about exasperating IT jobs, check out “10 users IT hates to support” and “9 signs you should jump ship to a new job.” | Pick up a $50 American Express Gift Cheque if we publish your tech story: Send it to offtherecord@infoworld.com. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the anonymous Off the Record newsletter. ]

Rewind to the start of the project. My company had won the bid, but the client started out by saying the price needed to be cut by a third. We began the project by going through a series of workshops doing a cost-benefit analysis of major functions and trimmed the scope to get down to the target price. Since all key members of the client’s project team were a party to this exercise and the right people signed off, it seemed as though we had buy-in and everyone was happy.

Changes and more changes

However, over the next two years, the client’s project leader, “Dan,” received pressure from his team to add back the functionality that had been cut in order to meet the cost targets. Dan came to us with the requests, and we accommodated them, but also insisted on cost estimates, documented change control, and written approval of the time and cost impact of the change.

As time went on, we grew nervous as the cost of the project ballooned. We brought in our own audit team to assess the project. The results said we ran the project well, but there were concerns about the amount of change control and the fact that we didn’t regularly meet with, nor had even met, the client’s executive sponsor. In fact, we seldom met with anyone higher up than Dan.

After the audit, we put together a nice presentation for the client explaining the results of the audit and particularly raising alarm at the amount of change control that had been approved. However, we received a serious slap in the face — but not for what the report said. We were admonished for going around the project leader, and it was pretty uncomfortable for a few weeks. We were told to never do that again — not just by Dan, but also by the client’s senior management team.

A few months later the original funding was nearing an end. We weren’t concerned because the additional funding was there, we assumed, as a result of our documented and approved change controls.

Those approved changes weren’t actually approved

It was a dark day when Dan and I met about the matter and he explained that he didn’t have the money he had signed for in the change controls. In fact, he said he probably didn’t have the authority to sign those in the first place. It then became clear that he felt we had made an initial bid two years before, should keep to that number, and needed to help him out of a spot. My management was open to helping, but definitely not covering a 50 percent overrun.

I took Dan through my change control file and all the approvals he had signed, and told him we needed to either be paid the additional money or we would have to stop work when the current funding ran out. And that was how I ended up in the hot spot in the big conference room.

The client’s project team, the executive sponsor, the client’s auditors, Dan’s boss, and others were in the room meeting for an hour before we were scheduled to join them. As my senior project manager and I walked in, we could feel the hostility. All that was missing for this to feel like a true interrogation was a bare bulb and a whip.

Pleading the case

I introduced myself and launched into a presentation. I had function points of what we quoted on, what the function points were after we reduced scope, and what they had grown to with change control. The increase in function points directly correlated to the increased funding approved in the change controls.

I had schedules that listed the change controls, the amount of each, the date they were presented, and the dates they were approved. By the time I had finished, it was pretty clear we hadn’t just increased costs on our own and we believed we’d had client approval to incur the extra expense.

At this point the executive sponsor asked me who had approved all of these change controls. I paused at this point, expecting Dan to stand up and take responsibility. The room was silent. Finally I said, “People in this room.” The sponsor asked, “Who?” I gestured my hands broadly toward the gathered crowd. Finally, the sponsor figured it out, pointed his finger at Dan, and literally screamed, “Did you approve all of those change controls?”

At this point Dan admitted that he had, then started to explain that since we were still under the original funding, they could always stop work and the money didn’t have to be spent. This did not comfort the sponsor. It was ugly.

The last question was asked by the auditors, who wanted me to explain why we hadn’t raised a concern to senior management when we saw this spinning out of control. I explained the audit we conducted and provided the date of our letter to senior management expressing our concern. I then flashed a copy of the letter on the screen (at this point Dan’s boss was not very happy with me). Why none of the client’s senior managers would acknowledge receiving it is still an irritant to me.

Crisis weathered

The next few weeks were most interesting as the sponsor asked the audit firm he had hired to conduct an audit of our project while he did some work internally (presumably to figure out his next steps if the results pointed back at the client’s company). The audit firm in the end told the client that we were running a good project and the problem was in its shop.

The sponsor got the additional money for us but insisted on a fixed price to complete the project. We agreed to this with the proviso that there be a steering committee formed that met every two weeks and that the executive sponsor and a few other senior leaders attend all of these meetings. They agreed. We had a number of other requests to improve the effectiveness of the project, and the client agreed to most of them.

Dan’s role was reduced to assistant project manager, and a new project manager was installed by the client. “Bob” was an engineer who had built manufacturing plants. He was very disciplined and very reasonable, and he made many good decisions. He was a pleasure to work with. The executive sponsor, when he finally engaged, was an excellent leader and his attendance at steering committee meetings made a real difference.

With the new leadership structure and the other changes that we asked for, the project moved along quite well. Furthermore, shortly before a successful launch the sponsor told his board of directors that ours was the best-run project he had ever seen. Challenges can be overcome.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, we’ll send you a $50 American Express Gift Cheque.

This story, “Our fault? Read the project audit and weep,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.