by Johanna Rothman. This article was originally published in October 2000, on catapulse.com
During software projects, you can hear widely different attitudes toward fixing defects, depending upon priorities and motivations:
“We'll fix that when we have time. In the meantime, just keep developing! How can you possibly tell how much it will cost to fix later?” — An Engineering manager, eager to continue software development
“Let's find all the defects before system test. The customer will wait for the product.” — An Engineering manager, concerned that shipping a product with defects would prevent customers from buying the product
Many engineering managers have to make difficult decisions about when to fix which defects. The choice to fix or not depends upon many factors: the type of product; the risks associated with shipping known or unknown defects; your development processes; and the cost of fixing the defect.
The easiest time to calculate this cost is during the system test time, when people are 100% dedicated to finding and fixing defects. To begin, count the number of fixes made. You know how many people (developers, testers, writers, and others) were involved, the cost per person-day, and the duration of system test. The total cost can be surprisingly high, which is why the fix value is so important.
Doing the math
Let's look at some hypothetical examples that will help you understand both how to determine the cost and how the environment impacts this cost. Company A is in a rush to get its product to market. The company is developing a medium-sized, multithreaded application, and it has a “variable” culture. (See the sidebar on “Cultures and their implications”) Management doesn't recognize that engineering will continue to find defects through development and post-release, because Company A hasn't taken a systems approach to looking for problems.
Company B is creating a smaller, real-time, embedded system, and it has a “routine” culture. Company B's management has seen several disastrous releases, and they're now willing to spend a little more time finding and fixing defects because the company can't upgrade software once it is in the field.
Cultures and their implications
|I find Weinberg's 2 culture patterns particularly effective when describing the implications of the cost of fixing a defect.Variable culture: Management does not understand that quality is a management issue. Problems are fixed with great personal effort, not a systemic approach.Routine culture: Management agrees quality may be valuable, but there is no money or time allocated to make the quality happen. Teams fix problems, generally after a severe problem has occurred.
Steering culture: Management understands that quality is a systemic issue, and that can help the managers manage better. Problems are brought out into the open, discussed, and resolved in an orderly way.
Company C is developing a larger, multithreaded application, and is rushing to make a market window. It has a “steering” culture of software development, where management is on the lookout for problems that could arise and prevent meeting the desired ship date.
For each example, I'm using a standard rate of $500/person-day and a run-rate of $100,000 per person-year. In calculating the costs I have rounded slightly, because I care about the magnitude, not the specific dollars and cents, of the cost.
In Company A, during system test, seven people fixed only 143 out of 245 defects found, over a period of 65 days. It took 455 person-days (seven people * 65 days) to fix those 143 defects — that's about 3.2 person-days per fix. At $500 per person-day, the average cost to fix a defect was about $1,600 per fix, or $500 * 3.2 person-days. Because Company A has a variable culture, it was surprised that its employees found more defects than they could easily fix.
During Company B's system test, five people fixed all 165 defects detected over a period of 35 days. It took them 175 person-days (35 days * five people) to fix them — a rate of about one defect per person-day, for a cost of about $500 per fix. The company's routine culture helped it find the “easy” defects before system test, so it had the hard defects left.
In Company C, during system test, eight people fixed 50 out of 51 detected defects over a period of 30 days. This took 240 (30 * eight) person days, or about 4.8 days per fix. The cost per defect fix was $2,400. Company C's steering culture helped it detect the “easy” defects much earlier than system test, and also helped it create many fewer bad fixes while employees were fixing defects in system test.
At first look, it might appear that in a proactive, more steering culture, it costs more to fix a defect — but that's only part of the story. These costs were determined during the system test phase. It generally costs less find and fix defects before system test, and it almost always costs more to do so later.
Tracking fix costs outside of system test
While Company A didn't track the time to detect and fix a defect during the requirements, design, and development phases, it did track the engineering time required to fix a defect after the product shipped. After shipping, Company A ran into a common problem: Some customers were quite disappointed by product quality — most frequently, in the areas where the engineers were unable to fix the defects before release. When an unhappy Very Important Customer called senior management, senior management then demanded that Engineering fix the defect as an “emergency” fix.
Fixing the defect after shipping requires additional cost. The developers still had to find the defect, and the testers had to verify the fix. In addition, for each fix the writers had to develop release notes, the build engineers had to create a separate branch, and the testers had to do system-level testing to verify no other egregious defects were caused. Because the product was already released, Company A's additional system testing costs included:
- Adding regression tests to the test suite
- Testing more of the areas around the fix
- Performing extensive system testing on the different hardware/software platform combinations
During normal system testing, the testers skipped many of these steps because of insufficient time. They could not skip these steps after release, though — testers had to verify that this fix would not cause problems for other customers.
After the fix, the build engineers merged the fix back into the current development branch, and the testers verified that the fix still worked in the current development branch. Aside from the engineering work, middle management spent significant time tracking the progress of the emergency fix work and reporting that progress to senior management and the customer.
This extra work adds up. After release, Company A estimated that it spent 20 person-days per fix. At $500 per person-day, this comes to $10,000 per fix. Company A typically had 20 “emergency” fixes after each release, for a total post-release fix cost of about $200,000.
Company B tracked the time it took to fix a defect during implementation (before system test), and found the average time to be about two hours, or about $125 per fix. Company B generally didn't release fixes to its products once the product was in the field, so it had no extra post-release costs.
Like the others, Company C did not track defect fix time during the requirements, design, and implementation phases. However, Company C engineers used a build-every-night and fix-any-defects-immediately approach to product development. In effect, the engineers were finding and fixing defects throughout the entire lifecycle. They spent an average of one person-hour each day fixing defects from the previous night's build, or about $60 per defect. Using this approach — along with code walk-throughs and peer reviews — they generally shipped a very high-reliability, low-defect product, and were able to avoid most post-release defects costs. When they did have an “emergency fix,” their average time to fix was about 5 person-days per fix, or $2500 per fix.
Comparing the numbers
Table 1 shows a summary of the cost to fix a defect during the different parts of the lifecycle, and an estimated cost to the company of a defect.
Table 1: Defect resolution costs throughout the lifecycle
Notice that all three companies had different actual costs to fix a defect. This is because there is no standard cost to fix a defect. For instance, Company B had a less complex, smaller product than Company C, so we would expect different costs to fix. The cost to fix a defect changes for different projects and different organizations.
What we can compare, though, is how the ratio of fix costs changes during the product's lifecycle, which appears in Table 2. According to Pressman 1, the expected cost to fix defects increases during the product's lifecycle. Table 2 compares Pressman's expected figures to those found during development at Company A, B, and C.
|Expected Cost to fix (Pressman)||1 unit||6.5 units||15 units||60-100 units||1:6.5:15:60|
|Company A||Unknown||Unknown||$1600||$10,000||?:?:1600: 10,000== ?:?:107:166|
Table 2: Comparison of cost to fix a defect at different times in a project
There are a few things to notice here. First, even though the cost ratios don't match the generally accepted ratios according to Pressman, one trend is clear: The later in the project you fix the defects, the more it costs to fix the defects.
Second, none of these companies tracked their defect cost before coding started. Had they tracked defects found and fixed during requirements and design, they could have tracked the defect cost before coding started. Armed with that information, they might have chosen different activities designed find more defects earlier.
Understanding the costs
Company A had the highest cost, over $400,000 just to fix defects. When compared only with Company B, that might make sense, as Company A's product was larger. When compared with Company C's costs, though, Company A's costs seem extremely large. Though Company C had a larger, more complex product than Company A, and had much less cost to fix defects.
Company B had the lowest cost, just over $80,000 to fix defects. (It also had the smallest application.) Its costs would have been dramatically higher if it had had to fix a defect once the product was in the field. Company C had the largest, most complex project, with the most people, and was able to contain its defect costs to about $135,000. If you considered only the cost to fix a defect during system test, Company C's costs would look worse than the others. In reality, it was the best able to deal with defects, because it found the defects earlier and took less time to fix them than did the other companies.
What you can do
You can apply the defect cost-estimating technique to your own environment. First, follow the formula above to calculate the costs of the defects you find, and then determine how many defects you find and can fix at each stage. If your costs are higher than you would like, using the data can help you to change how you develop products. Remember, defects found earlier are less costly to fix, which means you can fix many more of them and keep overall costs down. Strategies such as peer walkthroughs during development, defining requirements and inspecting the requirements documents, and writing down and inspecting the design can help you do this.
If your customers won't wait for your product, then maybe you can afford to find and fix defects later in the project, if you absolutely have to ship now. Regardless, you're going to pay in some way. Either you take the extra time and pay less to fix the defects before you ship, or you incur technical debt for your product and fix the defects after you ship, when it costs more and you run the risk of alienating your customers. Remember — it will never cost you less to fix a defect than the moment you find it.
If your customers are pressuring you to ship faster, then using a technique such as build-and-test-every-night will help at the end of the project. In addition, judicious use of inspections during requirements and design will help reduce the number of defects and cost to fix them dramatically.
Copyright © 2000 Catapulse, Inc. All rights reserved.
Republication or redistribution of Catapulse, Inc. content is expressly prohibited without the prior written consent of Catapulse, Inc. Catapulse, Inc. shall not be liable for any errors in the content, or for any actions taken in reliance thereon.
Like this article? See the other articles. Or, look at my workshops, so you can see how to use advice like this where you work.