Is your project on track? Marking the most of metrics

Is your project on track? Marking the most of metrics

Wayne Turk

The panic is on. You’ve just received an e-mail from your boss asking how the project is going. How do you find out how the project is going, and what can you tell him? Or maybe your first Interim Progress Review (IPR) is coming up. How do you show your boss that the project is on track and you have everything under control?

[ILLUSTRATION OMITTED]

The answer is metrics. It’s an easy answer, but metrics can be a tough process. Maybe this article can help make it a little easier.

Metrics–A Simple Definition

We all know what metrics are. Or do we? Let’s try a simple definition. Metrics are a concrete way of defining what a project will achieve and whether it has met or is meeting those goals. Or maybe I can give the simplest definition of all–metrics are measurements of progress.

In addition to telling your boss how the project is going, there are other reasons to use metrics. In some cases, policy or regulations require metrics be applied to a project. Mandatory or not, metrics allow you to set targets, assess your success at meeting those targets, measure benefits, help identify issues and problems, determine the usability of a product (especially an IT-related product), and provide feedback on efficiency and process effectiveness. All of those are good reasons to use metrics, and they boil down to one thing–metrics help you manage the project.

Types of Metrics

What kinds of metrics are there? I could define things like ordinal, nominal, ratio, or interval metrics, but I want to keep this article on a practical level. This article will identify and provide examples for some of the more commonly used metrics.

Yes or no (success or failure). Usually this type of metric has only one of two answers–yes or no, indicating whether a part of the project has been completed or not. Does something meet a requirement? Has a task been completed? It’s a pretty simple metric. Example: Is the weight within the parameters set?

Percentages. This metric asks how much of a task is complete. It also asks how much the product will fulfill the requirements, which is always a good thing to know. Example: What percent of the tasks scheduled during this period were completed on time?

Comparisons (sometimes related to percentages). This metric is a direct comparison of the current process or product (or even a service) with something else. Examples: How does our product compare with previous models? How much cheaper to build (or maintain) is this model compared to other, similar models?

Variance (another type of comparison). This type of metric, a mainstay of earned value management, is a comparison of what has occurred versus what was expected. Examples: How does the earned value compare to the baseline? How far behind schedule are we?

Numeric. This is a straightforward measure or count of something. Example: What is the average number of errors for first-time users during testing?

Rating scales. This metric asks how something measures up. Example: What is the measure of satisfaction for users with functions and features on a new software program?

Trends. This measures how things are progressing over time. Are they improving, staying the same, or getting worse? These metrics are very important to monitor. Example: Is the average time between failures improving?

Other Metrics

Here are some of the most common metrics and the questions that they answer.

* Time — How are we doing against the planned schedule?

* Cost — How close to budget are we?

* Resources — How much time, staff, and equipment are we using on the project?

* Scope — Is there scope creep, and is it within acceptable limits? (We all wish that there was none, but that is the impossible dream.)

* Performance — Are we meeting the requirements and specifications?

* Risks — Are the project risks tolerable?

* Quality — How is our quality? Are we finding and fixing quality problems?

Choosing Metrics

What metrics are best and provide the most useful information when managing a project? That’s hard to say because each project is unique, and the specific metrics in each area will vary by project.

Designing or choosing the appropriate metrics is one of the most difficult tasks faced by the program manager and other stakeholders. Defining and identifying good metrics is very hard, as well as potentially time consuming and expensive. To be useful, metrics must be quantifiable, measurable, and limited, in both scope and number. Additionally, they must measure things that are controllable.

There’s an old saying that still holds true today: “What gets planned gets measured. What gets measured gets done.” What managers must remember, though, is that what is measured becomes what is important–both to management and the project team. Remember, too, that when you measure something, you influence it, so you have to measure the right things or your metrics can lead you astray.

Start by determining what results are important to the project’s success. This is the basis for useful metrics. For instance, if time is the driver for a project, then tracking project milestones alongside the projected schedule is important and must be monitored closely. Generally, as I’ve mentioned, metrics are designed to reflect progress in the areas of budget, schedule, technical achievements, and performance, although there could be others. Over time, project metrics provide benchmarks and a history of progress that can provide lessons learned.

[ILLUSTRATION OMITTED]

Once you choose the right metrics, you have to use them. It is up to the program manager and the staff to monitor the metrics. You probably will have to report them up the chain–usually done during the dreaded IPR with the boss or client. However, that is what metrics are for. They exist so that everyone involved in the project can see the status and so that problems can be identified early and fixed.

The Overall Metric Picture

So how do you view your metrics? Dashboards are a quick, high-level look at metrics that show the overall status of areas of the project. Frequently, dashboards use stoplight graphics and colors like red, which means problems (usually significant problems that could impact success unless something is done); yellow, which means problems or potential problems that are correctable; green, which means things are okay (on time, on budget, etc.); and sometimes blue, which means things are outstanding (well under budget, performance much better than expected, etc.).

“Presenting in a simple dashboard or traffic-light display focuses attention on the areas that need attention. An hour of analysis to establish all is well in a particular area is 59 minutes [and] 55 seconds of wasted time if a traffic light can provide the answer,” said Neville Turbit in his article “Measuring Project Health,” published on ProjectPerfect.com.

You should set dashboard metrics and obtain a common set of understandings on the meanings of the graphics and colors before you start the project. For example, when should an aspect of the project be colored yellow instead of red? All stakeholders–including your boss and your customer–should agree to dashboard guidelines and report to those parameters throughout the entire project. That way, those involved in the project can see the status and feel comfortable that the dashboard reflects the project’s expected progress.

Stopping Metric Problems

There are always pitfalls, and you need to avoid as many of them as you can when establishing metrics. One way you can do this is make the metrics easy to capture. Ideally, the data-capture should be a part of the project management process and not an end unto itself. It should not be cumbersome, time-consuming, or costly. A poor metric is one that generates data in a costly manner without producing any suggestion of how the process could be improved or the problem resolved.

Additionally, metrics should not be thought of as a replacement for face-to-face communication. They should be the genesis of communication to assess the impact of an issue or problem, its cause, and some options for correcting it.

Project metrics should be useful, and they should be designed to reflect what is, not what should be. Project managers (or their staff) are often reluctant to provide data that might reflect negatively on the project. That is only human, but it must be overcome or it creates a false status of where the project really stands.

Another way to avoid pitfalls is to not shoot the messenger. All of us have been there. We bring bad news to the boss, and he explodes. The common result of that is that the metrics will likely be manipulated before they are reported, creating a false status.

The metrics for contractors should be developed jointly between the project staff and the contractor, although this can be very difficult and time consuming. It is essential, though, because the metrics must be acceptable to both, and the metrics have to show the status of the project using measurements that the contractor can control. If the metric is affected by something that the project staff does (such as the speed at which deliverables are approved/accepted), then the contractor is not going to accept the metric as a measure of his performance.

The metrics should be scaled to fit the project. A small project doesn’t need several metrics, while a complex ship or aircraft design project would need many. Pick the ones that you need–and need is the operative word. Don’t collect data just because you can. It’s a waste of time and energy if it is something that you are not going to use.

Finally, choose the right metrics, even if it’s hard to do. The wrong metrics are a waste of resources and may not be useful at all. They may even be misleading. If poor metrics are forced on you by someone higher in the chain, make the effort to show them a better alternative.

Metrics Software

There are plenty software products out there to assist you in tracking metrics for project management and portfolio management, including Artemis, Changepoint, CA Clarity[TM] PPM, DOORS, Primavera[R] Planview[R], and Microsoft[R] Office. Project managers must remember that these are only tools and need to be used wisely to get the data that’s needed and not just to get data. A good metrics program should provide reliable, useful information for good decision making.

You may find that you need only a few metrics to measure the project’s status. Don’t be concerned if there are only a few. A large number doesn’t necessarily make for better understanding or for good decision making. Too many metrics can make life confusing for the project team and cause people to manage the metrics rather than the product.

If you aren’t using metrics, start. If you are, take a look at the ones that you are using. Are they worthwhile? Do they tell you what you need to know? If not, you had better take the time to determine the metrics that you really need. Otherwise you could find yourself and your project in deep trouble.

The author welcomes comments and questions and can be contacted at rwturk@aol.com or wayne.turk@sussconsulting.com.

Turk is an independent management consultant with Suss Consulting and a retired Air Force lieutenant colonel and defense contractor. He has supported information technology projects, policy development, and strategic planning projects for DoD, federal agencies, and nonprofit organizations.

RELATED ARTICLE: 10 Steps to Collecting Project Metrics

1. Identify success factors. Review the business objectives and be sure that the project deliverables clearly address all success factors.

2. Define what information is needed to show that the project was successful:

* Cycle time (milestones) met

* Budget contained within approved changes

3. Assign metrics for each of the success criteria that provide an indication of whether the success criteria are being achieved.

4. Determine how you would collect the information, what the effort and cost of collection would be, and what value would be obtained.

5. Cover the big picture. Don’t focus on just one or two–we want metrics that cover all aspects of the project, such as:

* Value delivered

* Cost

* Effort

* Quality

* Acceptance of deliverables

* Cycle time

* Productivity

* Team performance.

6. Prioritize the metrics. Make sure we are getting the best bang for our buck–something that will deliver the most meaningful information with the least cost.

7. Compare actual results against initial targets. These can be fixed or within ranges. For example, budgets may be set within a range, whereas set milestones have a definitive date to be met.

8. Provide the process steps for collecting the information that answers the following questions:

* Who is responsible for collecting the metric?

* When will the metric be collected and reported?

* How will the metrics be reported (status reports, quarterly meetings, metrics reports, manually, thought dashboard programs, etc.)?

9. Collect the data. Once your work plan is in place, the project manager’s job is to monitor and control the project. Instead of reacting to singular events, the project manager knows where to focus and proactively concentrates on staying within the metric boundaries.

10. Analyze results. Collecting metrics on a weekly or monthly basis helps us with critical analysis so that we can follow up on critical trends and make process improvements, if necessary.

Modified from the PMI Metrics SIG Newsletter, March 2005.

COPYRIGHT 2007 Defense Acquisition University Press