Using Quality to Drive Product Development Processes

© 1999 Johanna Rothman.

Abstract

Companies create a variety of products, and different releases of those products, for many reasons. These range from market-testing trial balloons disguised as ‘beta tests’ to releases forced by incompatible changes in operating systems. Some have many changes, some have few. Some can tolerate fairly glaring defects, others have to extremely reliable. Some are built on very speculative financial grounds, where others protect lucrative franchises.

The goals for quality, schedule, and features for a project are intimately related to the reasons for creating the release in the first place. Since the definition of quality is determined by the goals you’re trying to achieve, you can’t use a cookie cutter approach to the project management process in software development.

In my experience, each project has one, and only one, of these goals as its top priority, and then addresses the other two goals within that context. This prioritization determines the tradeoffs the project manager will make during the project, and should help the project manager determine the product development process to use in developing the product.

Different product development processes provide unequal leverage for the three possible top priorities of the software product development. This paper uses concrete examples to illustrate how to choose the quality priorities for a project. Once the priorities are set, it illustrates how to choose the appropriate product development process.

Define this project’s quality

Every project has its own goals. To determine the appropriate product development process for a project, you must first define and understand your project goals. The end goal in almost any project is to make money, either by selling the product or by serving an internal need, such as IT projects or internal tools. Choose the goal, which fits with the rest of the business objectives of the larger project. For example, if the goal is a time-to-market driven beta-balloon, then there are no “customers” at all ­ yet. But there is a business imperative to have “at least these features by this date.”

The three possible project goals for any given software project are [2]:

  • Minimize time to market, by delivering the release as soon as possible.
  • Maximize customer satisfaction, by delivering a fixed set of features, whether demanded by existing customers or believed to be required to grab the hearts and minds of potential customers
  • Minimize number of known and shipped defects

In my experience, a project can have one and only one of these goals as its top priority. The manager must address the other two goals within that context. If the manager accepts and understands this prioritization, she can actively decide to shape the development process to achieve the goal. As the project proceeds, she can make tradeoffs consistent with the priorities. If the manager refuses to accept that only one of these goals can be paramount, the project is likely to not make any of its goals.

When you start a project, you have to pick one of these three . as the predominant goal of the release. Choose which aspect of quality is most valuable to your customers, whether they exist or are represented by the company. The other two goals are traded off within the context of the primary goal.
This is an example of what happens when a project does not have one primary goal for a specific release:

SmartStore product developers worked on a major software release. The organization consisted of:

  • A product development group who believed that the most important thing to do in any release is to fix the bugs reported in the last one.
  • A management team that wanted (a) something to ship in about eight months for market reasons, and (b) a list of new features, and (c) some performance improvements.
  • A project manager who did not understand how to set project goals, nor how to track progress, or how to communicate project goals adequately.

SmartStore planned an eight calendar month time period from start to delivery. No one specified any explicit release criteria, but development was working towards a specific feature set with low defects (to maximize customer satisfaction). A month or so before the planned ship date, SmartStore’s managers realized the project would not make the ship date with both the features and low defects. At this point, management chose to focus the release goals on time to market-they would be content with any set of features that the development group could complete in the next 6 weeks (to minimize time to market). They decided to drop some features from the project and they made a new plan and schedule. The schedule was not terrible, it just did not take into account that the product developers needed to change how they worked.

Management then started measuring how well the project team met the new schedule. About two weeks before the new ship date, it was clear to everyone the project would not make the new ship date with any of the features completed (the feature freeze had been slushy and the product developers pushed into Beta anyway). Since they were now going to miss the second planned ship date, with fewer features and more defects, management said “We can’t make the features we want in the time we want, so let’s get the performance we want” (to minimize customer-visible defects). Performance had been an implicit development goal, but had never been a focus of the project. The project goals were briefly changed to focus on product performance.

After only two weeks of working towards product performance, the SQA Manager realized they might never ship a product. The SQA Manager chose specific product goals (a certain smaller set of features, moderate defect levels including performance no worse than the previous release, and an aggressive time to market) and formulated measurable release criteria. The project was complete when the release criteria were satisfied.

With all these goal changes, some members of the development team were unaffected, some felt jerked around by the apparent lack of management focus, and some were so utterly confused, they were unable to make progress. Even though the professed project goals changed, the product development activities did not change Throughout the entire process, the product developers spent most of their time fixing bugs to make a (lower priority) market ship target.

All of the management gyrations made the project even later because the project manager was busy trying to figure out how to accommodate what management wanted in the current development process.

The project manager was unable to adequately track what was really going on in the project, because he was confused by the lack of a single project focus. Since no one asked the technical staff if they were capable of meeting the new goals, the technical staff ignored the new goals. They continued working on reducing the inventory of defects in the existing product.

The product finally shipped about four months late, without some of the planned features, without the performance, and with more defects than originally hoped and planned. The lack of management focus during a critical phase of the project (when everyone realized the estimates were inadequate) allowed some of the technical staff to drift, some to focus on the new goals, and some to remain focused on the original goals. In the end, no one at SmartStore was happy with the release.

SmartStore’s product development process was not flexible enough to accommodate the shifting goals, and the project manager was not capable of explaining to the organization how the shifting goals would prevent the project’s success. If you’ve ever found yourself in the dilemma of shifting project goals, then consider how you focus your project goals on low defects, time to market, or features.

Some people are not easily convinced that there is only one top priority. Here is a check you can use to verify that choice: If it is three weeks before the scheduled ship date, and the defect count is higher than the developers prefer, and the last feature is not quite implemented, what does management choose to do? If management chooses to ship anyway, then time to market is the top priority. If management chooses to ship as soon as that feature is installed, then customer satisfaction through features is the top priority. If management chooses to hold shipment in order to fix the defects, then low defects is the primary definition of quality.

Note that the primary goal does not exclude a secondary goal from being extremely important. In the case of SmartStore, the top priority was time to market, with a specific feature set as a very close second. Low defects were only important if the features didn’t work at all.

If the management and the project manager had established these priorities at the start, the project manager would have had a clear goal to communicate to the developers, and a clear idea of how to measure the progress of the developers. With those, he could have brought the effort to completion on schedule.
When you are asked to manage a project with unclear priorities, your first job is to get clear priorities. You have to make it clear to your management that there’s no plan without them. If your management is hopelessly confused, you have to create them yourself (perhaps with other people on the project). It is better to manage the project with priorities you create than with none at all.

When the priorities are unclear, you have to ask more questions-when someone proposes to change the overall project priorities, the product itself shifts, and product shipment is at risk. No process can eliminate the impact of shifting priorities, but an appropriate process may help prevent priorities from shifting. If there is no way to prevent a shift, a conscious approach to the importance of priorities can help you minimize the impact.

One way to choose the right quality priorities for a project is to use Geoffrey Moore’s high-tech marketing model [4] to understand the market imperatives. Table 1 provides a view, based on his model, of the interaction between the definition of quality and the product lifetime. For more information, see [6]. The users (and potential users) have different perceptions of product quality over the lifetime of a product. The chart shows how the three possible priorities are sorted for different classes of users. As a product evolves, it moves from one target user group to another, and (in the model) the priorities shift accordingly.

Table 1: Quality Perceptions over Product Lifetime

Product Life/ Market Pressure Intro-
duction
Early Adopters Main-
stream
Late Majority Skeptics
  (Enthu-
siasts)
(Vision-
aries)
(Pragmatists) (Conservatives) (Laggards)
Time to Market High High Medium Low Low
Features Low Medium Low Medium* Medium
Low Defects Medium Low High High High

* Medium pressure for features does not mean pressure for new features. It means that the promised features must be in the product and must be working.

Choose a process based on your quality requirements

There are several well-known product development processes. Within an overall product development process, there are a number of techniques that can be used, such as cleanroom and concurrent engineering. See [3] for more explanation about product development processes. Different processes and techniques have varying effectiveness in terms of the goals of time to market, feature richness, or defect control. The following table is a comparison of process development processes and techniques, their strengths, and their product quality priorities.

Table 2: Processes, their Strengths, and Quality Priority

Product Development Process Process strengths and necessary conditions to make the process work First Priority Secondary Priorities Project Culture [5] Process’s prognosis for Success
Waterfall: A serial process that in its pure form does not consider feedback. I have only met waterfall projects that consider feedback. Known and agreed-upon requirements

Well understood system architecture

Requirements stable over project

Development team stable over project

Features Low Defects Time to Market Routine Many successful waterfall projects incorporate feedback.
Spiral, Staged-delivery, Evolutionary delivery Risk management Features Low Defects Time to Market Steering successful
Evolutionary prototyping, Design-to-schedule (frequently used in concurrent engineering) Rapidly changing requirements or incompletely known requirements Time to market Low defects Features Steering successful
Code and Fix (not recommended, but many organizations use it. [3]) Unplanned projects Time to market Low defects Features Variable Does not generally meet any of the desired product criteria
Product development techniques
Cleanroom Known customer requirements Development team capable of maintaining reviews during project Features Low Defects Time to Market Routine Successful when the project maintains the work style
Concurrent Engineering Most requirements are known, but not all Risk management by bringing in cross-functional team Time to Market Features Low Defects Steering successful

When you decide on your first priority of quality, you can then deliberately choose the most appropriate product development process to support that definition of quality. Sometimes you choose a combination of processes to meet the mix of quality attributes needed for this product at this time. In essence, you tailor the development process to produce the kind of quality that matches the overall business requirements.

If you are called upon to plan a project with unclear requirements, or other high-risk components, consider a spiral process to manage the requirements risk. To meet an aggressive time to market goal, consider concurrent engineering to make sure the whole company is ready for product shipment. To meet a specific time to market goal with low defects, consider a staged delivery process.

In the case of SmartStore, the project’s criteria for success were unclear. If the project manager had realized that a specific set of features was necessary, but that time to market was critical, the project manager could have communicated these priorities to the development team. SmartStore was in the Early Adopter phase of their market. As a result, time to market was much more important than low defects. If the project manager had realized that time to market was the first priority with a certain set of features, and low enough defects to use those features, the product development process would have reflected those priorities . And, then when management wanted to shift priorities, it would have been easier to explain the effects of changing the project goals.

This next case is an example of an organization choosing their definition of quality for their product in their marketplace, and then explicitly setting Beta and Release criteria to know when those goals were attained.

Example of process based on time to market

Messenger had a history of problems in the product development. All their previous projects were late, there were too many defects for the products to be sufficiently useful, and some necessary features were missing. Even at the start of this project, the Messenger organization was constantly in “catch-up” mode, trying to fix the deficiencies in previous releases. This project started as the others had, with vague project goals, and creeping requirements. Once the first project milestone was missed, the parent company changed management, including the project manager. New business and project management staff was brought in.

Messenger had an overriding goal: to make a specific time to market. They had a small number of customers, and were trying to break into the mainstream of the market. They had to have enough features in the product so that their potential customers could use the product and few enough bugs so that the product was mostly usable.

Messenger technical staff decided not to repeat the past. They worked with the project manager to define the value this project had for their current and future customers. They decided time to market was critical, with a small set of features. They built a project plan based on a design-to-schedule process, with elements of concurrent engineering. They divided the work into a set of relatively independent pieces of software. The technical staff worked in parallel on the different pieces. The plan called out specific times at which they would integrate the changes.

Messenger’s Beta ship and Release ship criteria were different because they used a design-to-schedule process. They developed the requirements, designed the architecture, and worked on features in priority order. That is, the highest priority feature was designed, implemented, debugged, and integrated first. The next highest priority feature was designed, implemented, debugged, and integrated second. This repeated for all the features before the release date. (Implement features until you run out of time.) Messenger was able to work on a number of features concurrently, because the staff worked in parallel. Since all the necessary features were implemented in parallel, they were sure they would make their release date.

Table 3: Messenger’s Beta and Release Criteria

Beta Criteria Release Criteria
1. All code compiles and builds for all platforms. 1. All code compiles and builds for all platforms.
2. All developer tests must pass. 2. Zero high priority bugs.
3. All beta customer must run and pass. 3. Document all open bugs in release notes with workarounds.
4. All current bugs are entered into the bug-tracking system. 4. All planned SQA tests run, > 90 percent pass.
5. First draft documentation is available and shippable. 5. Number of open bugs decreasing for last three weeks.
6. The code is frozen. (Debugging had stopped long enough to create a stable build and verify the Beta customers could use the product.) 6. All Beta site reports obtained and evaluation documented.
7. Technical support training plan is in place, and the people are in place. 7. Reliability criterion.
8. Less than 36 open high priority bugs 8. Final draft documentation available, complete and submitted to corporate organization.
  9. A working demo runs on previous release.
  10. Performance criterion.
  11. At least two Beta site references.

This project met the Beta and Release criteria. The product shipped on time, with almost all the features, and very few user-visible defects. The users found only 2 of the open defects, and were able to work around those problems. The users did not find other defects. Messenger met their quality goals with this design-to-schedule (and goal specification) process.

Example of process based on features

ExtendIt had problems in their previous projects: all their projects took much longer to complete than they expected and the projects were still missing major pieces of functionality. Since the projects took too long to complete, users started changing their requirements, frequently just before the release shipped. ExtendIt did not have change management processes that allowed them to deal with requirements changes-the only thing they knew how to do was hold the release to put the changes in. This led to even longer project schedules. It was common to have to wait for up to two years for a complete feature.
ExtendIt had an overriding goal: get a certain set of features to market. If each feature was not complete, the product would not be shipped. Time to market was a close secondary consideration-their market was growing, and they wanted to increase market share. This product had a significant number of pre-existing customers, and was firmly entrenched in the mainstream of their market.

ExtendIt went to a quarterly staged-delivery process called a Release Train. The product manager decomposed each release into several projects. Then, the product manager divided each project into a series of staged deliveries. The project manager worked with others to define the requirements for the release and with the developers to define the architecture. Then, the developers built and tested the components in parallel. This process reduced schedule risk by fixing the architecture at the outset and choosing an architecture built from components that could be developed in parallel.

As each component was integrated into the system, the system was tested to verify that it met the absolute ship criteria. If it failed the test, Project Management directed the developers to back out the feature or to wait for the next release cycle. Only projects that could meet their release criteria were allowed on the train.

ExtendIt ran short Beta test cycles. This is an example of Beta and Release criteria from a recent release:

Table 4: ExtendIt’s Beta and Release Criteria

Beta Criteria Release Criteria
1. Components 1, 2, 3 must be usable by Beta customers. 1. Components 1, 2, 3, must be complete.
2. All code compiles and builds for all platforms. 2. All code compiles and builds for all platforms.
3. All system tests must run, >90% pass. 3. All system tests must run, >98% pass.
4. System tests for Components 1, 2, 3 must run and pass. 4. System tests for Components 1, 2, 3 must run and pass.
5. First draft documentation is available and shippable to customers. 5. Final draft documentation is available and shippable to customers.
6. The code is frozen. 6. The code is frozen.
7. Technical support is ready to support Beta. 7. Technical support is ready to support Release Product.
8. For each major component, zero major bugs, and less than 30 minor bugs. 8. For each major component, zero major bugs, and less than 20 minor bugs.
  9. At least three Beta site references.

ExtendIt’s Beta and Ship criteria were very similar. The criteria are similar because the components had to be close to complete to make a meaningful Beta program. ExtendIt used a staged delivery process with elements of concurrent engineering, to get the required components and the company ready for the release.

ExtendIt’s release was successful: the project met all its milestones; components 1, 2, and 3 shipped; and the developers were not too tired to go onto the next project. The customers were amazed that they had complete components, amazed enough so that product sales improved.

Example of process based on low defects

ViewIt did not have customer-viewable problems in their projects. ViewIt’s products frequently met their customers’ perceptions of value, in all of time to market, feature set, and defects. ViewIt now needed to produce a next-generation machine vision application with very little downtime and high accuracy. They needed to emphasize low defects in this follow-on release. The customer had a previous version of the product, and was not willing to use software with more defects than the previous release. ViewIt’s next priority was time to market, since the software was promised on a very early delivery schedule to the initial customer.

ViewIt chose a concurrent design-to-schedule process. They could predict that some changes in the requirements were inevitable, so they modified this process to include an iteration cycle so that they could absorb these changes and control their impact. They involved representatives across the company from the start of the project, not just to verify product features, but to make sure everyone would be ready to for product shipment.

To ensure that the code would meet the defect metrics, the development team designed, implemented, and enforced a more stringent development process than that used on other projects in ViewIt:

  1. All code and unit tests had to be reviewed before it was merged into the project branch.
  2. All code went through a two-stage build process. The first stage accepted the code, created a product build, and ran a minimum acceptance test. If the acceptance test passed, the code was “promoted” to the accepted area.

These project rules were different from other ViewIt projects, because other projects did not have the same urgency of low defects and time to market.

In addition to the new process, the ViewIt project team created these product shipment criteria:

  1. All unit tests executed and passed.
  2. All code reviews complete, no outstanding issues.
  3. All system tests executed, >98% pass.
  4. Successful generation of executable images for all appropriate platforms.
  5. Code is completely frozen.
  6. Documentation review is complete.
  7. There are zero showstopper bugs.
  8. There are zero major bugs, and no more than 10 minor bugs.
  9. The product is ready to ship on July 1.

These defect-focused criteria refer to low defects as a primary definition of quality. The bug numbers were chosen based on other products, and the assumption that the severity of each the existing defects would be assessed. The ship date is still very important, which is why it was part of the ship criteria. (There was no formal Beta period for this product.)

ViewIt’s process was successful for this project. The project won a quality award from their largest customer. In addition to project success, some other project teams took on the review and two-stage build processes this team had found so successful.

Summary

Product quality is what the customers value [5]. The commercial success of a release depends on satisfying the requirements in the three quality areas discussed in this paper. Your job is to try to predict how marketplace (defined by existing customers, potential customers, magazine reviewers, etc.) will use these three criteria to evaluate your product, and then choose goals consistent with their expectations. All parts of your plan, including development processes, beta tests, documentation and training requirements, should be chosen to reflect your goals for these three criteria.

Those values should drive the project plan and the product development process. Additionally, creating and using Beta and Release criteria allow the project team to test their assumptions and progress as the project unfolds.

Here are the steps I use to determine the product development process for a given project:

  1. Decide what aspect of quality is most valuable to the customers: time to market, features, or low defects for the entire project. Decide what else the customers value.
  2. Create a balanced scorecard, indicating the trade-off levels of the three aspects of quality: time to market, feature set, and low defects.
  3. Expand the scorecard into beta and ship criteria to test your assumptions all the way through the project.
  4. Choose a product development process that meets what the marketplace values. Don’t be afraid to use different processes on different subprojects.

One of this paper’s reviewers offered an analogy about “typical” and “unique” aspects of the project. “Typical” means you’ve done a project like this before, you understand well how to do it, and you’ve succeeded at it. “Unique” means that you don’t really understand it, and you may not know how to do it at all, never mind doing it well.

When you plan the project: is this mostly a typical or unique project? Are there subprojects that are mostly unique or mostly typical? Many projects are mostly typical with some unique aspects. Can you organize to separate the two, and then use different planning and development approaches for the sub-projects you create that way?

Choose a product development process to provide the leverage you need to the product your customers want to buy at this time.

Acknowledgements

The following reviewers made valuable contributions to this paper: Marsha Browne, James Bullock, Benson Margulies, Steve Smith, Dan Starr, Jerry Weinberg.

References

  1. Crosby, Philip, Quality is Free. McGraw Hill, New York. 1980.
  2. Grady, Robert. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, NJ: Prentice Hall, 1992.
  3. McConnell, Steve. Rapid Development. Microsoft Press. Redmond, WA. 1996.
  4. Moore, Geoffrey. Inside the Tornado, New York: Harper Collins, 1995.
  5. Rothman, Johanna. “Defining and Managing Project Focus”, American Programmer (February 1998).
  6. Weinberg, Gerald M. Quality Software Management, vol. 1. , New York: Dorset House Publishing, 1992.

Like this article? See the other articles. Or, look at my workshops, so you can see how to use advice like this where you work.

Leave a Reply