With a little common sense and some dependable metrics, you can banish the “bug bucket” and keep the dreaded rework monster under control by creating a reasonable prediction of your project’s end.
A reader recently asked me, “When planning a project phase, how do you account for the bugs that you know will be created? Do you simply create a “bug bucket” and hope you put enough time in it? Or is there a better way?”
Yes. Allotting a random amount of time for rework is better than assuming it doesn’t exist, but predicting rework—how much your project will require and when it will end—is tricky. I use several techniques to predict rework, depending on the project and the organization’s culture: allowing each task to complete without incurring extra rework; predictive measures such as historical data and estimation quality factor; and planning and managing the project schedule using critical-chain project buffers or agile techniques.
To reduce or plan accurately for rework, first make sure you’re not creating more of it. Under pressure to meet their estimated dates, a project team may complete the development of as much as possible of their code by their scheduled date, and then open defects against their own code. (I call this incurring technical debt.) This team increases their need for rework by at least one order of magnitude. Why? Because they don’t complete the work when it’s fresh in their minds (the developers move on to the next task), so they leave half-baked software for other people to attempt to use in their development. When these downstream developers run into trouble with their software and can’t finish on time, they’ll open defects against their code, and march on toward the next task.
When the original team returns to complete their work, they find incomplete code and they’ve forgotten some of the important issues about it because of the time elapsed between their initial implementation and the rework. Trouble ensues.
If your project artificially meets deadlines when developers say, “The feature is complete, but there are a few things I need to take care of later,” stop. The project won’t meet the deadline—not if you know there are defects or incomplete work in the code. Avoid creating more rework by completing tasks, including all associated repair, testing, documentation and any other work necessary to complete the task before going on to the next task.
If you can avoid creating extra rework, you can use predictive measures to determine when the rework is complete. The three measures I use are historical information, estimation quality factor and fault feedback ratio.
To use historical information, gather data from a few previous projects. I gather such stats as the number of team members, the total project duration, the size of the project (lines of code and the number of requirements) and the percentage of the total project duration that was rework. I also gather defect trends throughout the project: how many defects were open, closed and remained open per week, so I can understand the previous projects’ profiles.
Then I ask myself a few questions about the project itself. How similar is it to any of my previous projects? Can I use previous data to predict this project’s rework effort, or should I scale it up or down due to the relative size of my current undertaking? By the same token, I review defect trends in each project to see if the defect profile still looks similar. If the defect trends look similar, I can continue to use the historical rework information.
If you’ve shortchanged the product in the past and incurred technical debt from not completing the product in past releases, the historical information is only a starting point. You’ll have to increase your rework estimate to account for technical debt work before you can apply it to this project. You’ll have more rework because you’re not starting from a stable base, and the developers may not remember why they took shortcuts.
Track Your Estimates
Whether or not you have historical records, you do have information about this project once you start it. Throughout the project, I use estimation quality factor (EQF) to track the overall schedule estimate and estimates with each developer.
In their book Controlling Software Projects: Management, Measurement and Estimation (Prentice-Hall, 1986), Tom DeMarco suggests you track the overall schedule estimate by periodically estimating when you think the project will end. I use this technique with the project team together and with individuals for their tasks.
At each project team meeting, I take a few minutes to ask people to predict when the project will be complete. If the team trusts each other and me, they tell me verbally. If there’s not a lot of trust, I ask them to write the end date on a sticky note. I then graph the dates on a flip chart and see where they fall.
More often than not, the developers and testers have a fairly realistic estimate of when they will complete their tasks. Based on the answers, I ask these questions:
- What have you seen or heard (or not seen or heard) that makes you think we’ll finish later?
- What have you seen or heard (or not seen or heard) that makes you think we’ll finish earlier?
You can ask the questions without the data, but then the discussion soon dissolves into a shouting match: The person who screams the loudest decides when the project will end.
In addition, when I’m managing a high-stakes project with a tight schedule, I ask each developer and tester to track each of their tasks’ EQF. If they can’t make progress at the project’s beginning (tasks take more time than anticipated), I know early on that we’re in trouble —and that gives me more time to implement a reparative process.
Measure Bad Fixes
While planning the project, or by using historical data and EQF during implementation, you can reliably predict the amount of rework necessary. However, starting with the work products, to predict the effect of bad fixes on rework progress, you can measure the fault feedback ratio (FFR): the number of bad fixes to total fixes. Even in the requirements, analysis and design phases of a traditional project, assuming that you track defects, you can see how many defects cause other problems or can’t be closed on the first attempt. If the developers have trouble in the initial phases, ask them if they think they’ve resolved the problems, or if they’ll have more trouble once they hit the coding phase. This gives them an opportunity to tell you if they’re resolving unknowns in the early phases, or if the requirements are so vague that they still can’t figure out what’s going on.
Especially in the coding phase, if the developers are spinning their wheels with a high FFR, they can’t make progress. Maybe it’s time to refactor, just to make the code clearer, or perhaps it’s time to redesign, reduce coupling and increase cohesion, or change modularity. But whatever the problem is, you must first understand that there is a problem, so you can work with the project team to determine what to do.
I find FFR especially helpful during final system test. One of my project teams thought they’d met their schedules all along during the project, but during final system test, they kept running into problems. At one point, they thought they were slipping a week every week—but I thought they were slipping two weeks every week because the defects uncovered significant design issues. Their FFR was 20 percent—indicating that one fifth of all fixes either weren’t really fixed or caused other problems. They instituted peer reviews on all fixes and brought the FFR down to less than 10 percent, which allowed them to predict an end to the project.
Manage Your Buffers
Sometimes, rework seems excessive because the project wasn’t scheduled accurately. If everyone estimates his or her tasks using the first possible date that a task is complete, assuming that the planets align and Murphy’s Law doesn’t squash your project, your project won’t meet its schedule. When that happens, it appears that rework has taken over the project—but that ain’t necessarily so, to paraphrase the immortal Ira Gershwin. Many people estimate project duration by predicting the duration of each task plus a buffer, or “fudge factor.” However, fudge factors allow project staff to wait before starting a new task, even if they finish a task early. And sometimes, fudge factors just aren’t hefty enough.
Separating the task estimates from the fudge factors can help to schedule the project to allow for rework during each task or in the planning stage. When I plan a project, I ask team members to estimate their tasks, keeping the fudge factor separate from their reasonable estimate. (A task estimate then looks like 80 hours plus 16 hours of buffer or fudge factor.) Then I gather all the individual task buffers from the critical path into one comprehensive buffer at the end of the project. If a task on the critical path takes longer, I remove that time from the comprehensive buffer. If a critical path task completes earlier and people start on their next task earlier, I add that extra time to the buffer. During the project, I monitor the buffer. If it empties faster than I expect, I know that the task estimates are incomplete. If the buffers empty at about the right time, we’re on target for both tasks and rework.
Agile techniques offer immediate feedback to developers, helping to create fewer defects and less rework. If you can, incorporate agile techniques such as test-first development, automated unit testing and pair programming—or at least another pair of eyes reviewing the code.
If you’ve got a large project with several agile teams and you’re not sure if you’re really going to catch all the defects in each iteration, plan on a defect search-and-fix mission on one iteration before you release it to the customer. Planning for one iteration of contingency is reasonable, especially if you think your project is risky due to size or newness to agile techniques. And if you’re using agile techniques, one iteration should be more than sufficient for detecting and fixing defects.
Developers create code—and defects. But shoving them under the rug isn’t the solution. If you don’t mask the real schedule by saying a task is complete when it’s really not and by making the developers context-switch, they’ll create fewer defects, and fix the ones they do generate with a reasonable rework plan.
Using historical information, EQF to help you analyze the schedule, and FFR to determine your developers’ stress and accuracy level, you can predict, plan and manage necessary rework activities. Critical-chain buffer management can help you reduce risk and glean early insight into the project schedule. Finally, agile techniques can help developers improve quality while they code rather than after the fact.
Guessing at rework with a “bug bucket” is better than nothing—but not much. A wiser approach is to get a clear view of the necessary fixes in your future—and then fit them into your schedule.
© Johanna Rothman. This article was published in October 2003, Software Development magazine.