Opinion: How to Fix a Failing IT Project
My last two postings were about the difficulties of estimating the cost in time and money of software development projects, and the perils of planning them. This time I want to look at what happens when one or both of these processes fails, and you end up with a train wreck, yet you still have to finish the project. How do you avoid turning a failed development project into a death march?
About 15 years ago, I spend almost 18 months running an organization whose only job was to fix broken projects, mostly large, complex corporate information systems projects my firm was developing for very big companies. It was not fun work, but at the time it was extremely important to my employer. My firm needed to stop a rash of expensive project overruns (and unhappy clients). and it also wanted to boost the confidence of project teams that if they got into trouble, someone would be there to help them out. We wanted our teams to look for opportunities to “push the envelope” in their development pace, because that’s where the highest margins were, but not to routinely crash and burn.
I got the job because I had a good track record delivering tough projects and because I’d led my employer’s efforts to develop a world-class approach to project management. The thinking of my bosses seemed to be, “Well, you seem to know how to do this, and anyway, we’re using your ideas. So go explain to the troops how to make the project work.” Not that all (or even most) of the ideas were mine of course, but that’s what you get for being in the lead.
The result was that I got to parlay some of my successes and consequent reputation into some operating principles and rules of engagement.
First, as a matter of principle, we never simply “shot” the project team. I’d seen this done all too often in past lives, and it was my observation that the problems we were there to fix were seldom the team’s fault, even if they were active contributors to the failure to recognize and solve the problems. I wasn’t being entirely altruistic here. Fixing broken projects requires willing collaboration and a lot of effort from the team and the client, and knowing that you’re going to get whacked doesn’t exactly foster willing collaboration or much effort. I did reserve the right to move some folks out of the engagement if they were truly the source of the problem, but in practice we only had to do this once, early on in the program.
Second, we would not solve the problem by requiring even greater effort and sacrifice from the project team. Our people are generally highly dedicated and already work hard. That wasn’t the problem. We needed to work smarter, not harder. And I didn’t want to create (or extend) a culture of “heroic fire fighters”: Too often, when you do that you get people starting fires so they can be heroes when they put them out. I wanted a culture that valued excellence in meeting demanding targets and creating satisfied clients, not in rescuing bad situations.
Third, we agreed to look at any project referred to us, but we were not obliged to agree to help. This was in self-defense. I only had a small team and we could easily have been swamped by engagement managers who wanted an easy way out of a transient difficulty or an awkward client relationship. In the end, we reviewed just over 300 projects (about 20 a month) and agreed to intervene on about 25 (about 8 percent). I personally worked on about half of these. We gave a lot of (I hope good) advice to the rest of the engagement managers and many of them came back to ask for additional input from time to time, so I don’t believe we turned anyone away who really needed us.
Of the projects where we intervened, we successfully fixed all but one. By success, I mean explicitly that we got the team back on track, re-engaged successfully with the client, delivered what they needed, and did all of the above very close to the original schedule and budget.
Before I go through some hard lessons learned, I want to talk a little about our only “failure.” This happened about halfway through the program, by which time we felt we could fix anything. Circumstances showed us just the opposite.
Fear of Failure
Next page: Fear of Failure
This was a project that was in trouble for a combination of reasons, but mostly because the original team that had proposed and started the work had selected an inappropriate technology as the core of their efforts, and had failed to create a design that could possibly have worked with the selected technology. By the time we arrived, the original team had gone (it had resigned almost as a group to go into competition with us, as it happened), and their replacements were truly struggling. The client was mad at us thanks to the change in personnel, the ensuing loss of momentum, and the lengthening delays as the new team came to see that the project couldn’t possibly be completed as proposed. They were about halfway through the budget, three quarters of the way through the original schedule and maybe 10 percent of the way through the work.
We took a week to do a complete project review and to try to repair the relationship with the client. The review gave us two options: Start again with a new and appropriate technology, or redo the design so that, with considerable engineering effort, the original technology could be made to work. The redesign would consume a large proportion of our best engineering talent for about six months, but would then be reasonably straightforward, provided that there were no major changes in requirements and the technology vendor delivered the next release of their technology on time and to specification. Starting again would be more expensive: The client had already purchased a lot of software licenses that would end up as wasted cost, and the design effort would have to be almost completely scrapped and redone. But it would be less risky, since we would not be so dependent on software capabilities that were not yet delivered.
As we tried to repair our relationship with the client, another option arose: Walk away. It soon became clear that the client intended to force us to keep to our original budget and to the original technology proposal. We would not be allowed to start again, no matter how much sense it made and how adversely it affected the project schedule. After a further two weeks of negotiations, I recommended that we walk (and pay a fairly hefty penalty). But I was overruled by my bosses, who decided to eat the overrun in order to maintain a business relationship with the client and preserve our reputation. So we put a new team leader in place (a volunteer from my group who was willing to take on the remediation commitment (in part because the client was in his home town and he would get a year working from home out of the deal).
For those of you keeping score, here’s what happened next: The requirements changed a lot, and the vendor was late with the next release, which also failed to meet specification in several critical areas. The project finished three years late and cost us more than $20m over the original budget to complete. A restart would have taken a year and cost us under $10m. I still don’t know why the client wouldn’t let us start again. They would have received a better answer sooner. Oh well.
Fear of Failure
Next page: Lessons Learned
So, here are the broader lessons learned.
First, in almost every case, we learned to “be lawyers” for a while. We recorded (and circulated our record of) everything we were doing, especially every interaction with the client. We acted as if we expected to be in court some day, even as we worked to avoid just such a situation. We never blamed anyone specific for anything specific, but we did build and document a case that made it clear what had happened, who had been involved (or should have been involved but wasn’t) and what the consequences were.
Second, we insisted that the client put some skin in the remediation game by assigning a senior member of the client’s team to the project and giving him or her the authority to resolve issues quickly. Most of the time, failure to do this was a large part of the reason the project was in trouble in the first place.
Third we built a “wall” around the project team so that they were insulated from client-generated random interruptions and “noise” (there is an incredible amount of this around a troubled project, and very little of it helps). We did not cut off all communications; that doesn’t help either. But we filtered and moderated all contact so that it flowed though the remediation team first. As the situation got back on track, we eased up on this isolation, but by then the most important behavioral lessons had been learned by both sides.
Then we started in on the work environment and project process.
First we put everyone on a “rational” work schedule. On many of the projects we worked on, teams were working 70 or 80 hours, six or seven days a week. We cut everyone back to a core of 40 hours and a maximum of fifty with no more than five days a week for anyone. (Except for the remediation team: We worked significantly more hours, although I insisted that everyone have a least one complete day off a week.)
Second, we reduced scope and accelerated delivery. In essence, we transferred the client’s attention away from what they hadn’t yet received to what they had received, by delivering something, and then offering to help them get the deliverables deployed and into effective use. I was often surprised to see how much the team had done that the client didn’t know about or had received but not deployed, because it was’t ready to do so. That’s where the senior executives assigned to the remediation effort came into their own, clearing obstacles in their own business that we couldn’t reach or influence.
Third we “rationalized” the working environment, reorganizing the people and the work spaces so that there was less wasted time and effort just being involved in the project. We often added an “administrative assistant” to the team to free up team members from routine chores that were necessary but didn’t add value. We updated everyone’s tools and technology to the latest version. We added some convenience and team-building factors to the work environment.
Fourth, we looked at the team itself. Did we have the right people? The right leadership? Were some folks burned out and in need of a rest? Did the team dynamic work, or were there people who just didn’t fit in and therefore couldn’t contribute effectively? Were key technical or business skills lacking? Over a couple of weeks of observation and interviews we talked at least once to everyone on the team (and to a lot of client people) usually individually over breakfast or dinner. We built relationship and influence maps. By the end of a month we had a plan together and a list of priorities. We took this first to our own management and then to the client. In almost every case we got quick agreement from both.
Finally, we sold it to the team. This was often the hardest part of the sales effort, particularly because we told them that they would have to carry out the plan without us. Sure, we would be available to advise and we would be monitoring their progress. But we would not be taking over the work. This was their project, not ours, and it was theirs to succeed with as well. This is a big step for a team that has been beaten down by circumstances. But given the chance to be successful, most teams will respond to the challenge. And as I mentioned earlier, in almost every case the team rose to the occasion and got the job done.
Without embarking on a Death March along the way.
Copyright © 2006 Ziff Davis Media Inc. All Rights Reserved. Originally appearing in CIO Insight.