Managing Product Development

A client was optimizing for what they thought was the bottleneck in their software development: the testers. In the assessment, I gathered some quantitative data about how long the testers took to test and how long it took for the other groups to perform their work. (They used a phased lifecycle.) The testers were busy throughout the entire project preparing for system test, and were completely consumed for final system test. The requirements and design part of the project took over half the project’s duration, whether that project was 18 months, 9 months, or 4 weeks. The testers part of the same projects were 5 weeks, 4 weeks, 3 days. Here’s a table showing the project durations:

The client was confused in a couple of ways. The first confusion was thinking if you remove time from the end of the project, you save time. Their data proved the opposite. If they could have reduced the requirements and design time, they could have easily saved tons of project time. The next confusion was in thinking testing was a bottleneck. It’s possible at one time testing was a bottleneck. They had effectively made requirements and design a bottleneck based on this data. The third confusion was in thinking optimizing for anyone’s 100% productivity will not produce any bottlenecks anywhere else.

As soon as you optimize for one group over another, you produce a bottleneck. In organizations where you need to produce more than one product, you’re better off optimizing for the strategically important projects, determining how to make each phase as short as possible (but no shorter), and making sure people have what they need to do the job. Agile project techniques help make the phases short, because you don’t waste time doing requirements and design on parts of the project you can’t finish (this organization’s problem). Artificially shortening a phase guarantees you spend more time and get less from it (as does artificially shortening a project).

A project is a system. Each group will have ebbs and flows to its work, but the best optimization you can do is to see that each person on the project has a constant, maintainable flow of work.

David Anderson has an intriguing post, Lawyers, Unit Tests and Performance Reviews. David says “Individual team members can be set specific goals and behavior objectives…” and gives examples. I prefer that team members set their own goals with input from their managers. But the key here is that a technical person should be looking to increase his or her value to the group all the time, with some way to measure that at performance reviews.

Managers who perform continuous career development with their staff (which means every week in a one-on-one, not just in a yearly performance review) gain benefits for their group and the company. Their group increases its capacity, which in turns helps the company.

If you’re a technical person, consider improving one of the four areas of technical skills (See this post for one idea about how to think about it. And any improvement you make in your non-technical skills such as verbal and written communications, influence, negotiation, or facilitation skills will certainly improve your value. If you’re a manager, review your management skills and think about how you can improve them.How have you provided more value than you did a year ago?

I have a column this week on Stickyminds, Multiprojecting: The Illusion of Progress. Thank you Frank Patrick, for naming this kind of multi-tasking (and for reviewing the work-in-progress). Feel free to comment on Stickyminds, unless you want to comment here.

About 15 months ago, Denise Robitaille and I met for lunch. Denise was explaining how she helped her clients implement a reasonable corrective action program. I explained how I’d helped some of my clients figure out what was really wrong when they had problems. We were struck by the similarities of our approaches. So, we wrote a handbook together.

Here’s an example of how I’ve used this approach. A client was convinced that the testers took too long to complete testing, with the result that the testing was slowing down the release cycle. We first defined the problem statement to avoid blaming the testers, and then collected some data about who was finding defects and when. We discovered that for one product, the defect arrival rate was a hockey stick. (You can click on the image to see it larger.)

When we investigated why the defects arrived late, we found that the project manager wasn’t managing the project, that the developers never knew which requirements were top priority, that the developers performed no peer review or other early defect-catching techniques. In fact, the testers were doing as good a job as they could.

For another product, we found a more even distribution of defects. (Click on the image to see it larger.) The test time was still long, and the testers were still finding more defects than the developers had, but they had many fewer defects escape the release. This project manager had no release criteria, so the entire project was held hostage by the senior manager who kept requesting more data about the product state before he would allow release.

They had other projects with different problems. For one product. The testers hadn’t learned about combinatorial test techniques, so they did take longer than necessary in the test part of the project. If they had stuck with their original problem statement, that the testers took too long for testing, they never would have seen each project’s root cause, and would have been unable to take appropriate corrective action.

Corrective action involves defining the problem, gathering data (quantitative and qualitative) to understand the root causes, developing a plan (and obtaining buy-in for that plan), executing the plan, verifying that the plan worked with appropriate results, and then closing out the plan. This handbook helps you do so.

Program management is the art of managing an effort of multiple projects with multiple deliverables.

If you google “Program Management,” you’ll see a bunch of interesting posts, including Chris Pratley’s Program Management. To me, Chris is describing project management, albeit of a large project. When Chris talks about his “program” management, he’s discussing the coordination of a single product’s release. When I talk about program management, I’m discussing how to coordinate multiple product releases, with people who perform project management managing their piece or release. Take a look at Program Management Definitions for their discussion of how to define program management. (Yes, I realize one of their definitions is the large project with sub-projects. I still think that real program management involves multiple deliverables.)

Multiple deliverables and their tradeoffs are the major problems of program management. Program managers coordinate the work of many people, projects, and deliverables (and they should all read Frank’s excellent Multi-Project Management with TOC). They use negotiation and influencing techniques to facilitate the work across the organization. Program managers need to understand enough about the product (solution-space domain expertise) to make sure the technical decisions don’t negatively affect the company’s strategy. (If one project makes a decision that prevents another project from meeting its release date, that’s a negative affect.) One of my clients once told me program management was like grease and glue: grease to ease the way for different parts of the company to succeed, and glue to bring everything together.

Even if you’re managing projects to one deliverable, you may already be using program management techniques. If you’re managing to multiple deliverables, make sure you use program management techniques, including: working across the organization, solving problems outside of meetings, making sure each project’s goals are defined, making sure each project’s staff are working towards those goals, and creating a real program team. The program team needs a reason to consider each other’s concerns (it’s that grease thing again).

Want to make sure you complete your project as early as possible? Define release criteria. Release criteria are the few critically important objective criteria that define what “done” means for your project. Sometimes, it’s a combination of date, defects, and feature completion. Sometimes it’s just the date. The formula for defining release criteria is:

1. Define project success.
2. Define what’s critically important that this project accomplish.
3. Draft release criteria that are: Specific, Measurable, Attainable, Relevant, Trackable.
4. Gain Consensus on the criteria from management and the project team.
5. Manage the project to the criteria.

Easy to describe, not always easy to do. Read more about release criteria here. For ideas about what to consider measuring, remember the six sides of the project.

In response to my most recent Pragmatic Manager (about shortening project startup times), a colleague wrote: “I am working on a lifecycle definition team in my department and finally convinced the team that Agile Development was a Methodology using an Iterative Model lifecycle.” My colleague has neatly described the methdology (the practices) and the lifecycle (agile). A lot of people confuse “lifecycle” with “methodology.”

A methodology is a set of practices, sometimes built around a lifecycle. But many methodologies allow you to select the lifecycle that makes the most sense. Agile development is a set of practices, including frequent builds, pair programming or other built-in peer review, test driven development, frequent retrospectives (and the others) with a lifecycle that produces iterations of chunks of development. (Agile lifecycles are not just iterative. The short iterations and the focus on delivering useful product at the end of each iteration is different from other iterative lifecycles.) When you take the agile lifecycles plus the practices, you have a methodology.

Ok, so some of you are saying, “Semantics, JR. Puhlease.” We use words to convey what we mean. It’s hard enough to understand what everyone is talking about without changing the meanings of the words. Lifecycles are the way you lay out a project. A methodology is the set of practices you use with your chosen lifecycle. I would rather discuss the merits of specific practices or lifecycle in a particular project context than get stuck on the words we’re using to describe them.

I’ve been thinking about Martha Stewart and her felony conviction this past weekend. I use this quote in the hiring book:

“Somebody once said that in looking for people to hire, you look for three qualities: integrity, intelligence, and energy. And if they don?t have the first, the other two will kill you.” — Warren Buffett (In Joan Magretta’s What Management Is, p. 194)

Stewart clearly has intelligence and energy — and her accomplishments including her company’s growth was proof of that. The jury decided she lied, which for me is evidence of insufficient integrity. Aside from Stewart’s personal problems, the company has to determine whether and how to continue.

I have no idea about the market for the kinds of things Stewart’s company sells. I’m not a consumer of her products. But, the company employs a bunch of people (I can’t get to the reports right now, so I can’t look it up). And, because head of the corporation lied and committed a felony, some of those people will probably lose their jobs.

We all have moments in our lives, when we’re tempted to lie — just a little. Or cheat — just a little. Or steal — just a little. Don’t do it. It’s bad enough to be unethical when you’re the only one depending on you. Once you’re a manager, your actions amplify those of the people who work with you. You cheat a little, or lie or steal or whatever — even just a little and they get cheated once the truth comes out.

Sometimes it’s hard to keep your integrity. You may even be faced with a choice of your integrity or your job. And you may feel as if you have no options. If you’re ever in that position, and you’re tempted, remember the rule of three, and determine what your three alternatives are. But don’t compromise your integrity. That’s the cornerstone of your management value — to you and to your company.

A colleague emailed me a few days ago, and asked “for a code base with a [given size], what
can we expect to see for numbers of defects per KLOC (given the actual industry average or given what the industry believes we should expect). We need some way of gauging whether or not our defect rates fall within the industry standard, or if we are better or worse than the industry standard.”

The question of “Are we producing code with fewer, about the same, or more defects than industry standard?” is a reasonable question. Unfortunately, I don’t think it’s a particularly helpful question.

I object to defects/kloc (defects per thousand lines of code) for these reasons:

• Defects/kloc treat all defects equally. So if you have developers who went to great lengths to make all the code solid but the writers didn’t have enough time to bullet-proof the help, the defects/kloc number is misleading. Or, if the developers prevented a whole bunch of serious errors but missed a bunch of not-so-serious errors the numbers look the same as if the developers missed errors over the whole project.
• Defects/kloc change over the course of the project. Depending on the practices the developers use, they will find defects at a different rate. If the developers are using inspection or peer review or agile practices, they will find many more defects at the beginning of the project. If they aren’t using any of these practices, they will find a great number of defects at the end of the project. If the project stops testing during the hockey stick of finding problems, the total defects part of the equation is wrong.
• Defects/kloc assume that there is an “average” consequence to each defect. Each defect is unique, and sometimes, it’s the sum of a bunch of non-related defects that matter to the customer’s experience of the product.

Ok, so what do I recommend instead of defects/kloc? If you must measure defects as part of your measure of how good the product is, measure the defect escape rate post release. At three months, six months, nine months, one year, and on at three-month intervals, count the number of defects that your customers found that you didn’t know about. That’s the numerator. The denominator is the total number of defects found (including these new ones). The better your perceived quality, the smaller the defect escape rate. The worse your perceived quality, the higher your defect escape rate. If the customers don’t find the defects, then they don’t matter to the customers. Those defects still may matter to you, but they don’t affect the customer’s experience.

To me, defects/kloc is something to measure when you want to see if your process is catching defects and dealing with them early. Snapshot code growth and defects/kloc weekly, compare the numbers each week, and you have some useful information you can use during the project to adjust course. But don’t use defects/kloc to reward or punish developers.

High or low defects/kloc is not indicative of how good the developers are; it indicates some sort of process problem or success (or coverup). The process is almost never a developer problem. Management decides where to spend the money. If the developers have a too-high defect rate, it’s almost always because management has overconstrained the problem so that the developers feel that they have to take shortcuts. To me, defects/kloc is a way to blame developers for inadequate management. That’s why I feel so strongly about it.

If you want to know about product quality, measure all six sides of the product quality equation. That will tell you about product quality more readily than defects/kloc will.

Joel’s discussion of project managers (MS calls them program managers) and developers got me to thinking about the differences between project managers and developers.

The difference between project managers and developers is where they deal with complexity and decision-making. PMs deal with complexity and decision-making between people. Developers deal with complexity and decision-making in design. From that, the need to deal with people across, up, and down the organization (PMs) vs. the need to deal with technical issues across, up, and down the product change each person’s focus. The more the PM deals with the people issues, the less attention the PM can pay to the technical issues. And vice versa.

Even people who aren’t empathetic by nature can become great project managers. (I give myself checklists.) But people who can’t grasp the technical complexity can’t become great developers (or great testers or great architects or any highly technical role). This doesn’t mean that people who can’t understand technical complexity can’t manage highly technical projects. They can — if they understand where the risks are and how to manage those risks. But the ability to understand the technical issues and translate those issues into the people problems — that’s the mark of a great project manager.