The Pragmatic Manager
Increase Your Capacity and Finish
More Projects Manage It! Your Guide to Modern,
Pragmatic Project Management Behind Closed Doors: Secrets of
Great Management Hiring the Best Knowledge
Workers, Techies & Nerds Corrective Action for the
Software Industry Amplifying Your Effectivess Speaking Calendar Testimonials Resume Contact
What to Measure?
© 1999 Johanna Rothman. Originally published in Cutter's Business-IT Alignment E-Mail Advisor, March 24, 1999.
At a recent visit to an IT client, the new QA manager took me aside and said: "JR, here are the metrics I want to measure on a monthly basis. What do you think?" He had attempted to measure efficiency, cost of quality, and defect counts over all projects. (In reality, he hadn't actually measured those things, but those were the titles on the histograms.) I asked him what the reactions to these measurements were. The other managers had no reactions -- the measurements had dropped on the floor with a thud. None of the managers seemed to want to measure things; no one wanted to see his measurements. He had no idea why no one was interested. I asked, "What do you want to accomplish with these measurements?" He was stumped. He'd had no idea what he wanted to do with the measurements, he just knew he wanted to collect them.
Measurements are only good if you know what you want to do with them.
This IT shop was working on increasing their product capability (the number of products they could release per year), decreasing time to release, and increasing the perceived quality of the products. The measurements the QA manager took were inadequate for those goals.
Decide on your goal
Measurements make no sense without a goal. In many IT groups, increasing product capability (the number of products you can release per year) is a key goal. Additionally, many groups are concerned with how long it takes to release a product, along with how good their estimates were for the project schedule. Some IT groups measure the cost of quality, including defect counts, because their customers are willing to invest in product development but not service or maintenance.
Once you've decided on your goals, you can define appropriate measurements to determine whether or not you've met your goals, and you can understand how well your activities align with the rest of the organization.
Ask questions that will help you define your measurements
My client had some suspicions about why the department's perceived product capability was lower than they had expected. He wanted to answer these questions:
- Are people starting on the project on time?
- Do the people end their project work on time?
- Do we have key capabilities in only a few people? (Are the people a critical-path resource?)
- Do we estimate projects well?
Define measures to answer the questions
Once you've asked your questions, you can decide what to measure to get an answer to those questions. For example:
- When people start on the project vs. when they were needed
- How many of the same people are needed to do each project
- The difference in days between scheduled release and actual release
These aren't the only measurements the group will need, but they are a start to working on the particular issues this IT group wants to address. They can be collected quickly on a weekly basis. They are primarily trend-based data, so there aren't too many pieces of data on one chart, and it's easy to understand the data.
This is an example of the Goal-Question-Metric paradigm. To read more about Goal-Question-Metric, see Vic Basili and D.M. Weiss's "A Methodology for Collecting Valid Software Engineering Data" (*IEEE Transactions on Software Engineering*, Vol. SE-10, no. 6, November 1984).
Like this article? See the other articles. Or, look at my workshops, so you can see how to use advice like this where you work.