The Case for and Against Estimates, Part 2

In the first part of this series, I said I liked order-of-magnitude estimates. I also like targets in lieu of estimates. I'll say more about how estimates can be useful in part 3.

In this part, I'll discuss when I don't like estimates.

I find estimates not useful under these conditions:

  • When the people estimating are not the people doing the work.
  • When managers use old estimates for valuing the work in the project portfolio.
  • When management wants a single date instead of a date range or confidence level.

There are more possibilities for using estimates in not-so-hot ways. These are my “favorite” examples.

Let me take each of these in turn and explain how agile specifically helps these. That's not because I think agile is the One and Only Answer. No, it's because of the #noestimates discussion. I have used #noestimates in a staged-delivery project and on agile projects. I have not been able to do so on iterative or serial (waterfall/phase gate) projects. Of course, with my inch-pebble philosophy, I have almost always turned an iterative or serial project into some sort of incremental project.

People Estimate on Behalf of the Project Team

We each have some form of estimation bias. I have a pretty good idea of what it takes me to finish my work. When I pair with people, sometimes it takes longer as we learn how to work with each other. Sometimes, it takes much less time than we expected. I expect a superior product when I pair, and I don't always know how long it will take us to deliver that product. (I pair-write with any number of people during the course of the year.) Even with that lack of knowledge, we can pair for a short time and project to a reasonable estimate. (Do a little work and then re-estimate.)

When people who are not part of the project team estimate on behalf of other people, they don't know at least these things: what it will take the real project team to deliver, how the people will work together, and how/if/when the requirements will change. I have my estimation bias. You have yours. We might learn to agree if we work together. But, if we are “experts” of some sort, we don't know what the team will encounter and how they will handle it.

I too often see experts ignore requirements risks and the potential for requirements changes. I don't trust these kinds of software estimates.

Now, when you talk to me about construction, I might answer that we know more about construction. We have dollars per sq. foot for houses. We have dollars per road mile for roads. And, I live in Boston, the home of the Big Dig. Every time we remodeled/rebuilt our house, it came in at just 10% over the original number. We worked hard with the builder to manage that cost.

Those projects, including the Big Dig, were worth it.

How do we make software projects worth it? By delivering value as often as possible and asking these questions:

  • Is there still risk to manage?
  • Is there more value in the backlog?
  • How much more do we want to invest?

Software is not a hard product. It is infinitely malleable. What we deliver on Monday can change the estimate for what we want to deliver on Tuesday, Wednesday and Thursday. We can't do that with hard products.

When other people estimate, we can't use what we learn by working together and what we have learned already about this domain. Agile helps this specifically, because we deliver often and can re-estimate the backlog if we need to do so. We understand more about the remaining risks because we deliver.

Managers Use (Old) Estimates for the Project Portfolio

I have seen managers use estimates to value projects in the project portfolio. I wrote a post about that years ago: Why Cost is the Wrong Question for Evaluating Projects in Your Project Portfolio.

Here's the problem with old estimates. Estimates expire. Estimates are good for some time period. Not forever, but for some number of weeks. Depending on how you work, maybe the estimate is good for a couple of months. Estimates expire because things change: the team might change. The codebase and the requirements have certainly changed.

However, project cost is only one part of the equation. Value has to be another part when you think about the project portfolio. Otherwise, you fall prey to the Sunk Cost Fallacy.

You might say, “We use ROI (return on investment) as a way to value projects in the project portfolio.” Now you depend on two guesses: what it will take for you to complete the project and the sales/adoption rate for the release.

ROI is a surrogate measure of value. When I have measured the actuals (what it actually took us to finish the project and the actual revenue at three, six, nine and twelve months out, we almost always did not meet the projected ROI. And, because we chose that project with great-looking ROI, we incurred a cost of delay for other projects. “If we don't release this project because we are doing something else, what is the effect on our revenue/adoption/etc?” (See Diving for Hidden Treasures to read about the different costs of delay.)

People often say, “These two projects are equal in terms of project cost. If I don't use ROI, how can I decide between these projects?”

I have never seen this to be true, and it's quite difficult to predict which project will be shorter. Here are some options:

  • Use Cost of Delay as a way to value the projects in the project portfolio. See Diving for Hidden Treasures for ways to see Cost of Delay. See Manage Your Project Portfolio for many other ranking ideas for the project portfolio.
  • Determine the first releasable deliverable of value for each project. How long will that take? If you do one project, release something, does that provide you enough revenue so you can go to the other project and release something there?
  • Make all the deliverables small, so, if necessary, you could flow work from both projects through one team. The team can finish a feature/deliverable and move to the next one. I recommend using a kanban board and swarming over each feature so you get maximum throughput. Once the team has finished “enough” features, decide which project to spend more time on.

Agile helps the entire project portfolio problem because we can all see progress on an agile project: demos, product backlog burnup chart, and retrospective results. We know a lot more about what we finish and where we are headed. We can stop working on one project because we don't leave work in an unfinished state.

Management Wants the Comfort of a Single Estimation Date

I supply a range of dates for my projects: possible, likely, pessimistic. I sometimes supply a confidence range. I have met many managers who do not want the reality of estimation. They want a single date: September 1, 2pm.

The problem is that an estimate is a guess. I can only know the exact duration or cost when I'm done with the project. I can get closer as we finish work, but I can't know for sure months in advance. For a year-long project, I can guess as to which quarter/three month period. As we finish the project, I can spiral in on a date. By the last quarter, I can be within a couple of weeks of knowing.

Managers get paid the big bucks to manage the organization with assumptions, risks, and unknowns that we explain to them. When we work on projects, it's our job to manage our risks and deliver value. The more value we deliver, the fewer unknowns our managers have.

Agile (and incremental approaches) help us manage those unknowns. Nothing is perfect, but they are better than other approaches.

I've worked with several managers who wanted one date. I gave them the pessimistic date. Sometimes, I provided the 90% confidence date. Even then, there were times we had more problems than we anticipated. Meeting that date became impossible.

A single-point estimate is something we like. Unfortunately, a single-point date is often wrong. Management wants it for any number of reasons.

If one of those reasons is assurance that the team can deliver, agile provides us numerous ways to get this result without a single-point estimate: set a target, see demos, see the product backlog burnup chart.

I have nothing against estimation when used properly. These are just three examples of improper estimate use. Estimates are guesses. In Part 3, I'll talk when estimates might be useful.

(Sorry for the length of this post. I'll stop writing because otherwise I'll keep adding. Sigh.)

The series:

And, the book: Predicting the Unpredictable: Pragmatic Approaches to Estimating Cost or Schedule.

14 thoughts on “The Case for and Against Estimates, Part 2”

  1. If you have a small team of people in a single organization with little need for services or activities performed outside the group, estimates might sound like a frill because you can adjust internally to meet a target. On the other hand, if you have a dozen or more firms providing services at different points in time, some of whom will be paid for hours worked and others who will participate as a cost of continuing to do business with their current customer, and a hard “finish by” date, you have a scheduling problem. For these projects (Mars missions, changing outsourcing vendors), dependencies and sequence aren’t just Gantt chart issues, they are the hinge points of a moving contraption. So with those two extremes: at what point on the continuum is an estimate unnecessary?

    1. David, I have found the order of magnitude estimate helpful along with Cost of Delay to help evaluate the projects in the project portfolio. You are correct, a Gantt chart does not work for the project portfolio.

      In the program management book (I think you describe a program when you say “dozen or more firms providing services…”), I recommend deliverable-based planning, quarter-by-quarter and month-by-month. You might need a Gantt chart for the shorter-term deliverables (the ones in the next month or two) to keep everyone on track.

      One approach I have used is to plug the deliverables into a high-level view. You might do this with a Gantt. I have always started with stickies on the wall. I don’t much care how we start as long as we agree how we will monitor progress.

      I put the deliverables in time-order sequence at the top of the wall. We add the stickies to each deliverable that say how we will achieve the deliverable. If, at the end, you need more specific estimates and a Gantt, fine, do it. It all depends on how much detail you need from whom.

      I have some guidelines for my projects and programs:
      – I must see deliverables as in working product *at least as often* (more is good) as once a month. That means that when I have a program, I work with everyone to get a drop/build/whatever of working software and hardware at least as often as once a month. For hardware, this might be a demo. I’m a realist when it comes to money.
      – I work with everyone to make sure they understand what they have to do to achieve their deliverables.

      I have noticed that the more complex the work, the less I can trust the estimate. That’s one reason I like smaller estimates (which I’m working on describing in Part 3). It’s too easy to lose the risks and complexities when we say, “This will take 6 weeks.” It might. It might not.

      For visualizing the deliverables, I like a kanban board. You might like a Gantt.

      The key for me is the making deliverables small so we can see what we will deliver this month. When we have one-quarter roadmaps with monthly deliverables (more often is great), we can see what’s working and what’s not working.

      The longer the project/larger the program, the more I need to see deliverables, so I can see the risks and manage them.

      I hope this explains what I do for larger projects/programs.

      I think I still ask for some sort of estimate for a month’s work. Maybe that’s where you and I differ. (Okay, just one place 🙂 If we use agile and lean approaches, why not be able to keep the whole picture in mind (the six-quarter roadmap for example) and the next set of deliverables (one-quarter roadmap and one month deliverables) in the front of our minds? If we use agile and lean to deliver, that has been enough in my experience. My clients, too. But, maybe not for you.

      1. I should have been more specific: in the case I have in mind, many of the other firms are insurance carriers. They exchange information with the current outsourced administrator, but will have to change their integrations to point to the new vendor. Naturally, the incumbent is a participant, since everything has to change over on the same day and a lot of data has to be converted. For some of these firms, their participation will be as little as 60 hours comprising several tasks spread over several months. They need to know when since they have limited resources. We’re spread across several time zones, so I use a WBS to catalog the deliverables; we identify dependencies, estimate duration for each activity, and then prepare a schedule – standard PMBOK stuff. Just a project, but it will be managed as part of a global program.

        Like I said: this is one end of the continuum, where estimates are necessary to enable resource planning (people, test systems, etc) among multiple organizations. At what point on the continuum are they not necessary?

        1. Dave, I agree, you need your Gantt to help people see deliverables, especially what they need to deliver when.

          In the program management book, I suggest that if you have feature teams, you might not need to do substantial estimation. (I cannot imagine your program is agile. Is it??)

          Here are some criteria for thinking about when to estimate and when not to:
          – You will gain more value from delivering something small rather than estimating (Don’t estimate)
          – You will learn something from the estimation process. (Estimate)
          – You deliver value often enough that you don’t need to estimate the next chunk. (Don’t estimate)
          – You need to help people understand dates by which they need to deliver their chunk. (Estimate)

          There might be more cases. Where do you find more value?

          1. No, this one won’t be using Agile methods. All of the data sources and sinks are proprietary systems with published API’s, so the only development work is in the integrations and data conversion (which will use an iterative approach). And in several cases, there are existing connectors. The primary challenge is scheduling and communication among entities; all but the customer are providers who do this stuff every day and twice on Sundays, and the customer has me. So, no learning curve for anybody on this one – just measure twice, cut once, and on the announced date, switch over.

            Implementation projects like this are one example where Agile methods don’t apply. I use both Agile and traditional software engineering PM methods, depending on the problem at hand. If all you give me are OoM estimates, I will ask questions until we both have a better understanding of the WIP and work planned (learn something), and then we’ll revisit your estimates. Uncertainty is manageable because it’s usually reducible. If you are working in a business domain, you probably won’t encounter any unknowables – just things you don’t know yet. Manage accordingly.

          2. Dave, yes, I am not surprised you are using a more traditional approach with this program. Totally makes sense. I bet your people are accustomed to some sort of deliverable-based planning.

            It totally makes sense to ask for estimates here. Your people know enough about the domain and deliverables to be able to manage the risks. Yes, you’re using an iterative approach, which makes sense. I bet that manages the risk of big-bang integration. But, you don’t have the challenge of having to learn a lot during the project.

  2. Nice post Johanna.

    The dialog between between and Dave is interesting. IMO, the first question to ask is how much “software development” are we really going to be doing? Real software development is characterized by a significant amount of discovery and learning.

    If the engagement does require a significant amount of discovery/learning, then we have something that is more akin to manufacturing production line where we are executing a known set of instructions over and over again.

    1. Hi Eb, thanks, glad you like the post.

      I hesitate to differentiate between different kinds of software development. The kind of learning we do on more-development rather than more-integration projects is different. That I agree with.

      On systems-integration projects, if the players are familiar with the customers and the domain, it’s possible for us to have less learning and need fewer feedback cycles.

      I am thinking of two different clients as I write this. Client 1 is mostly a systems integrator, who has done these kinds of projects for years. They learn more when they bring a new integration partner on, and ask for weekly or monthly drops (depending on project duration).

      Client 2 is one who does a tremendous amount of internal software development for the bulk of the product. Because of the industry, they must allow other people to “play” in their product. They try to make all of their “components” so useful that their customers will buy all from them, and not integrate other people’s work. It’s quite difficult. In some ways, they have the worst of all worlds.

      I think of your example of a production line as a “checklist project.” The risk is in delivery, not in development. For example, when I finish a book, it’s all about the checklist 🙂 In some ways, I am running an integration project at that point.

      I suspect that many of us think our projects are riskier than other people’s. It’s all about the domain and the details.

      1. Hi Johanna –

        I wasn’t focused as much on risk but on the idea that knowledge work is characterized by a spectrum of known knowns to unknown unknowns and how much learning we will do is a often a function of that. This greatly influences our ability to estimate or influences how much effort we want to spend in being able to estimate.

        A team that is in the business of solely developing tele-medicine websites will experience something different from a team that develops websites for domains its not familiar with.

    2. As someone who started writing code in 1973, I would suggest that “real software development” is characterized by outputs. Good code is reliable, fault-tolerant, maintainable, auditable, and resource-efficient. The means to that end is subordinate to the end.

  3. Pingback: New PM Articles for the Week of June 6 – 12 - The Practicing IT Project Manager

  4. Pingback: Five Blogs – 13 June 2016 – 5blogs

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: