Web Business Under Pressure: Using Requirements to Create Release Criteria

© 2000 Johanna Rothman.

The push to web-ify your business can feel overwhelming, especially when senior management is already demanding to know whether it is ready and when it will be released. If your organization is struggling and does not want to review all of the requirements in detail to get a project moving, you can create product release criteria to capture the critical requirements in a way that makes sense.

Let's first talk about what “web-ify” means. For some organizations, it means making the client-server systems run on browsers. For others, it means creating a parallel way to deliver goods and services, using the web as a means to push or pull information to and from customers. Making a client-server system run on a browser may be as easy as translating the current user interface to a browser (not a trivial task, but understandable). At the other extreme, using the web as an additional or new way to interact with customers may be a completely new and large project.

Other projects, even the ones already under pressure, may not have the same sense of urgency as your web projects. (No one wants to be left off the web bandwagon.) For your other projects, you may be able to discuss the requirements, negotiate a realistic schedule, and then plan and execute the project successfully. As an industry, we haven't figured out how to do web projects reasonably, especially the first few projects where we're moving the business to the web. We don't always think about all of our potential customers, what those customers really want, how to manage the updates to the web, and how to do configuration management on large numbers of items. If we don't think about those things, we don't retain our customers, we can't do follow-on releases, and we don't have a way to do frequent releases. Therefore, we might get the first release out, but we can't get ship the next release. Especially with a new, large web project, senior management frequently has a significant sense of urgency, and may want the project team to short cut its standard process. If that happens, you can still start gathering requirements to develop release criteriathe agreement with senior management about what is good enough to ship.

What are Release Criteria?

Release criteria are measurable, objective success criteria for your project. Some examples of release criteria are:

  • Publish the application by June 1.
  • Performance of <specific scenario> no slower than <comparable application>.
  • No critical open defects, all other defects discussed with project team and customer support.

These statements are all measurable and objective. Taken together, they represent what is critical for this project to succeed. Appropriate release criteria take both the business and system requirements into account in order to provide the customer with a valuable product.

One of my clients, NewStuff, develops underlying technology for other companies' web sites. NewStuff is still in the process of creating their market, and their customers have many requirements changes as they use the product. NewStuff decided to react with short release three-month cycles. They have these generic release criteria for each release:

  • Ship between 2-3 weeks before the end of the quarter.
  • No critical open defects, all other defects discussed with five major clients and customer support.
  • Fulfill at least two major and three minor attributes or functions per release.

In addition to their generic release criteria, they have specific criteria that help them know how well they have maintained performance, or added functionality to the product.

  • Engine's performance is at least as good as performance previous release.
  • Components A, B, C work in these five browsers (enumerated list).

Both the generic and release-specific criteria are objective and measurable.

Creating Release Criteria

Start by defining what's critical to your project. Then, draft release criteria and get agreement with the project team and senior management. Finally, judge the product against the release criteria.

1. Define what is critical to your project

How do you know what's successful for your web-based product? What combination of schedule, features, and defects create your product's place on the web? Your customers (potential and current) have specific expectations of your web-based application. How do they perceive defects? Is lack of performance a defect, or is performance a feature? Is application turn-on time important? Can you iterate on the web site, or does it have to be “complete” at its first sign of life?

Sometimes, the most critical thing for your project is the release date. If that's the case, then use the date and the staffing level to decide which requirements you can fulfill. Then create release criteria that hold the date as the primary criterion, with subsidiary criteria.

In my experience, the date by itself has not been the most critical attribute of success for web-ifying your business. I've found two possibilities for the most critical attribute:

1. The ability to complete a transaction on the web in the same way, with the same customer satisfaction, as you do currently. In that case, define release criteria that take performance scenarios, completed feature sets, and defect levels into account.

2. The ability to offer something completely different to your customers as quickly as possible. In that case, you may not need the entire product all at once. You can define release criteria for a specific release, and define multiple releases with criteria that become successively more constraining. NewStuff is at the beginning of this cycle. Their specific release criteria are more and more constraining for each cycle. For example, when they make enough money to have a product marketing department, they will evaluate their defects with product marketing, to represent more than their current five customers.

These two possible success attributes (the same thing on the web or something completely new) are opposites. You won't use the same techniques to create or to manage the project. You certainly won't have the same release criteria.

Once you've decided what's most critical to your project, negotiate how those critical attributes work with your project. This is where I take the requirements, and translate them to release attributes so the project team and senior management can see their relative importance for this project.

2. Draft and negotiate measurable criteria

Especially for web projects, I tend to quantify the release time and the risk of defects in the release criteria. Sometimes, senior managers are not aware of the interplay between time to release and defect rates. In general, the faster you try to complete a software project, the less time people have to test the product and verify their work, and the more likely the engineers are to hack. It can take much longer to release a product when the engineers hack.

Whether you use release dates or defect numbers in your release criteria, the first step is to create measurable success criteria. Sometimes the QA or test manager does this and then negotiates the criteria with the project manager. Sometimes the project manager does this and then negotiates with the project team. Either approach is fine, as long as you end up with measurable criteria that the project team can agree with and achieve.

On a recent web project, the project manager first came up with release criteria in the first column. After discussion with the project team, they agreed to the release criteria in the second column:

Original Criteria Release criteria after project team discussion
Private site available September 1.

Public site available December 1.

Private site available October 1.

Public site available December 1.

Zero high priority defects. Zero high priority defects.
Document all open defects in release notes with workarounds. Assess all open defects with marketing and support.
All planned tests run, understand why any failures exist. All planned tests run, understand why any failures exist.
Number of open defects decreasing for last three weeks. Number of open defects decreasing for last three weeks.
All Beta site reports obtained and evaluation documented and at least two Beta site references All Beta site reports obtained and evaluation documented and at least two Beta site references
Performance criterion. Performance criterion.
System requirements 20-30 complete. Other requirements planned for specific follow-on releases. System requirements 20-33 complete. Other requirements planned for specific follow-on releases.

Then the project manager presented the release criteria to senior management, so they could see what they would get from this release. The project manager was able to negotiate many of the tradeoffs in advance so that the he could make day-to-day decisions without having to worry about senior management interference.

3. Judge the product against the criteria

The release criteria are a subset of the product's requirements for the release. In a real sense, the release criteria are the project's requirements. You can use the criteria as requirements, and review the project against the criteria to decide when to release.

When a project is high urgency, as many web projects are, many senior managers or project managers say, “Well, we don't need to meet this requirement. We're just too stretched for time to make sure we have it all in. Anything we don't have in now, we can put into the next release.” Those of us who work on software projects for a living know the risks in that approach: missing the deadlines, missing critical requirements, and shipping a product the customers don't want to pay for or use.

Release criteria can be a vehicle to continually check the state of the product against what is good enough to release. Because you've already negotiated the release criteria with the project team and with management, you know early on if you have a problem with the project. When you do come across a problem, you have other alternatives than having to ship in the heat of the crisis.

Sometimes, you will have to change the product's requirements. When that happens, renegotiate the release criteria with the project team and senior management. Use the release criteria to generate the “what's the right thing to do for this project” discussion.


Many software projects are under significant pressure. Web projects tend to be under even more pressured, whether deserved or not. Use release criteria as a way to define your project's requirements and to make the product's critical requirements obvious to all concerned. Then, when management wants to know “Can we ship it yet?” you will be able to tell them what they want to hear.

Like this article? See the other articles. Or, look at my workshops, so you can see how to use advice like this where you work.

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: