“It Depends”

©1999 Johanna Rothman

“The Answer to the Great Question of Life, the Universe, and Everything is 42” –. Douglas Adams, The Hitchhiker's Guide to the Galaxy

Some people think there's a specific answer to questions such as:

“What's the correct staffing ratio for developers to testers in a software development organization?”

“What's the best project management software?”

“When should I create a branch (separate source code line) for this release?”

I wish there were One Right Answer. It would make product development much easier. Usually though, the only right answer is “It depends”. The answer you need depends on your situation: what you're doing, how quickly you're trying to do it, the culture you work in, and the results you desire and can tolerate.

In public forums, when I give the “It depends” answer, sometimes people get angry with me. They think I'm trying to hold some information back. I'm not; I just can't give them an answer without knowing more about their environment.

For example, when someone asks a question about developer-tester ratios, I now think they're asking: “How do I estimate the effort needed to do to test this product, and how many people do I need to do that work?” To do that, I first analyze the project, to see what it will take to do the testing:

What kind of user interface does it have? An API (application programming interface) allows the testers to separate the testing into pieces, which can make it easier to test. A GUI without a well-defined API generally requires more testers. In general, a product without well-defined boundaries, such as an API, requires more testing than a product with well-defined boundaries. Testing exposes the lack of definition late in the product development cycle, which causes more rework and more testing.

How much new product are we building? Are we re-architecting the product, and adding significant features? Or, are we filling in the feature set from the basics that are already there? A re-architecture effort often requires more testers.

How large or complex is the product? Sometimes, the larger the product, the more testing is required. The more complex the product, the more testing is required. If the complexity is hidden from the user, even more testing may be required.

How experienced are the product's users? A sophisticated user base may mean we choose to do different kinds of testing, which will affect the type and number of testers. A novice user base may require much testing in specific areas, with portions of the product not tested as thoroughly.

Once I've considered the product, I then think about the people available to do the development and testing:

How experienced is the development staff? Do the developers know how to create and run unit tests? Are the developers already doing design reviews, code inspections, and individual testing before the product reaches the system test phase?

How experienced is the testing staff? Do they understand the product's architecture and design? Do they know how to extend the tests that already exist? Do they understand the product?

What tools do the developers and testers have? Are they creating their infrastructure as well as creating the product? For the testers, is there a regression test suite? How much of the regression testing is automated? Do we understand how well it tests the current project?

Finally, I consider the project and its importance to the company and to its customers:

What is the risk of doing less testing? Will this software be used to run airplanes or heart implant devices, or is it a desktop application that can be easily rebooted? The more crucial the software is to people and our environment, the more testing I expect to plan and perform.

How much time do we have? Are we planning the testing at the beginning of the project, where we can consider automation to cut down on test repetition time, or is the testing supposed to start next week?

How good do we want it? What are our release criteria? What are we willing to spend on the total product development effort?

Many organizations tend to have similar projects, so they go through this exercise once, and keep their discovered answer as a rule of thumb. That's fine, until they create a different product, have a different team, or encounter a different situation. Then their rule of thumb doesn't work anymore.

When you're looking for a standard answer, consider whether there really is One Right Answer. You may need to analyze the problem before you can give an answer. It all depends-and what it really depends on is the analysis you perform of the tradeoffs.