When Is Defect Data Not About Defects?

I taught my Pragmatic Project Management workshop in Israel last week. I was talking about defect charts and what they mean and how I use them. (No, I don't include priority or severity data on defect charts; just # opened and closed by week and # remaining open each week.

One of the participants explained that he also tracks # Fixed Private Defects. What are those? Fixed defects in a developer's sandbox, not checked into the mainline. Why are they not checked into the mainline? Because the organization does staged integration of several highly complex sub-projects, each of which has interdependencies with the other. And the build takes over a day.

This participant's defect data is not about defects. This data is about the cost of the build and possibly the product's architecture. When you can't build the whole product every day (this organization builds a couple of times a week), you create a bottleneck of fixes waiting to be integrated into the build. You slow down development--dramatically at the end of a project, badly enough during the project--to keep the build going.

Let's run a few numbers and see what it costs them to continue staged integration. Let's assume it costs just one person-day to create a fix. (That's optimistic, but not unrealistic.) Now, assume that for three months of a project, they have a weekly average of 50 "private-fixes" waiting to be integrated. (I'm assuming the 50 here, I have no data. But I think it's not far off reality.)

To keep staged integration going, they spend 50 person-days every week for 12 weeks. (Remember, this is a large project, really a program.) That's 600 person-days. I wonder if they could spend 600 person-days and fix the build so that they could build every day, without the need for staged integration. I don't know, but I suspect they could.

Watch for what your data is telling you. You might think you're measuring one thing, but you're really measuring something else. In this case, the defect data is not about defects; it's about the cost of an inadequate build system. What else are your defects telling you about your system?

There is No Such Thing as Percent Complete

Jason Yip's Hail Mary software development talks about what happens when you defer all the finishing to the end of the project. In the dashboard chapter of the book, I have a sidebar called "How Can We Have No Completed Work?" which talks about exactly the same thing.

The more serial your lifecycle, or the more you implement by architecture instead of by feature, the less completed work you have. Since I don't count partially completed work, it's possible to have 0 progress until the very end where things finally start coming together.

One of the assignments I gave my class this semester was to provide me a status report for the projects. Several of the teams decided to give me % complete measurements. I asked how they knew it was 80% complete. One team actually told me they'd done 80% of the work so far.

I told them to track the rest of the effort. I'm sure they fell or will fall into the 90% done schedule game.

Percent complete makes no sense. Features are done or not done. You can count done features and see how far along you are. You can't reliably count any percentage done.