When Is Defect Data Not About Defects?

I taught my Pragmatic Project Management workshop in Israel last week. I was talking about defect charts and what they mean and how I use them. (No, I don't include priority or severity data on defect charts; just # opened and closed by week and # remaining open each week.

One of the participants explained that he also tracks # Fixed Private Defects. What are those? Fixed defects in a developer's sandbox, not checked into the mainline. Why are they not checked into the mainline? Because the organization does staged integration of several highly complex sub-projects, each of which has interdependencies with the other. And the build takes over a day.

This participant's defect data is not about defects. This data is about the cost of the build and possibly the product's architecture. When you can't build the whole product every day (this organization builds a couple of times a week), you create a bottleneck of fixes waiting to be integrated into the build. You slow down development–dramatically at the end of a project, badly enough during the project–to keep the build going.

Let's run a few numbers and see what it costs them to continue staged integration. Let's assume it costs just one person-day to create a fix. (That's optimistic, but not unrealistic.) Now, assume that for three months of a project, they have a weekly average of 50 “private-fixes” waiting to be integrated. (I'm assuming the 50 here, I have no data. But I think it's not far off reality.)

To keep staged integration going, they spend 50 person-days every week for 12 weeks. (Remember, this is a large project, really a program.) That's 600 person-days. I wonder if they could spend 600 person-days and fix the build so that they could build every day, without the need for staged integration. I don't know, but I suspect they could. </>

Watch for what your data is telling you. You might think you're measuring one thing, but you're really measuring something else. In this case, the defect data is not about defects; it's about the cost of an inadequate build system. What else are your defects telling you about your system?

Labels: measurement, project management