Every so often, managers want to compare teams. You've heard this in several ways:
- Managers want teams to “normalize” their story points, so they can better predict the cost or schedule for the rest of the backlog in large programs. This doesn't work, but managers want to do it anyway.
- Managers want to compare teams to see which teams are more productive. This is even crazier. Who can tell which teams are more productive? You can't tell which features are more difficult. You can't tell which projects are more difficult. Why not? Just because something looks easy does not mean it is easy. The teams will start to game the measurements if you try.
- If you ask the teams to increase their points, they will. This is an agile schedule game, Double Your Velocity. “That team is doing twice your points. Why can't you do twice as many points?” Have no fear. The team will at least double their velocity, just by doubling the points per story.
Comparing teams is not useful. There is no reason to do so. Each team is unique, composed of wonderfully unique individuals. What does matter is that the team is working well together, delivering product. That is the only thing that matters.
If the team delivers, without working well, the team is not sustainable over time. If the team does not deliver, even if they are harmonious, that is not sustainable over time. You need both. And, you cannot get both if teams are in competition with each other.
I was working as a manager in an organization several years ago, when one of my managers wanted to have a “friendly” competition, to see which team was “better” than another team.
“What outcome do you want from this competition?” I asked my boss.
“I want to know what team is better,” he said.
“Do you also want any shippable product?” I asked.
“Well, of course!” He looked at me as if I had three heads. “That's the whole point of this exercise. I want shippable product and friendly competition.”
“But these people have to work together. And you are asking them to compete with each other instead of collaborating with each other. I am the program manager. How the heck is this supposed to work?” By this time, I'm standing up, leaning over his desk, in his face.
I'm angry. I'm not happy about this situation. I've just helped the teams understand how to do deliverable-based planning so that we have sync'd their deliverables. (This is before agile.) Now, my boss is about to undo everything I have done. Argh. (Read Manage It! Your Guide to Modern, Pragmatic Project Management to learn more about deliverable based planning.)
I sat down. I calmed down. I explained that I needed collaboration, not competition.
I still have all my hair.
This is why I wrote Management Myth 20: I Can Compare Teams (and It’s Valuable to Do So). Managers have good intentions. And, they don't always realize the consequences of their intentions.
Hi Johanna
Your post on this subject is a timely one. Yesterday I asked Dean Leffingwell, the founder of the Scaled Agile Framework (SAFe), for his opinion on points comparison by management. I’d seen this happening on a recent SAFe implementation and was concerned about it. This was his response:
“Velocities are NOT normalized across teams. Estimating is. If you don’t normalize estimating, then there is no meaningful economics; you can’t figure ROI of you don’t know that the “I” is. If you want to scale agile, and there is no meaningful way to bid work across teams, and within a program, you will be blocked before you even try (executive view: “That agile thing; not ready for prime time. They use gummy bears per sprint as their main metric, and they can’t estimate work in any meaningful way. They just don’t get it. I know, let’s just have the PMs do a work break down structure; that we understand. And of course, we’ll need good specs to do that, so let’s stop and write the SRS now”.) Then Agile loses.”
Clearly then – in this popular approach to agility at scale – normalizing estimates between teams is seen as being viable. Moreover, it seems to be positioned as a remedy to the type of management dysfunction that Dean cites.
Ian, if you believe you can predict the estimation of a large program, and you want an estimate over working software, then SAFe is for you. SAFe is definitely more effective than waterfall. In my experience, staged delivery is more effective than SAFe. I have had more experience with staged delivery. I have had better results with staged delivery. I used staged delivery for more than a decade before transitioning to agile.
Let’s be clear. SAFe is RUP. It’s not agile. Staged delivery is not agile either. Both of these approaches are much more effective than waterfall. (If we could get people to consider alternatives to waterfall, the world would be a much better place. This is why I wrote Manage It! Your Guide to Modern, Pragmatic Project Management.
Dean is a very sharp cookie, and he has discovered a way to management’s heart. Good for him.
But don’t call SAFe “agile at scale.” It’s not. It’s RUP at scale. It’s an excellent RUP at scale. But it is not agile. The time from a feature into the backlog to the time a feature is out of the backlog is measured in months, not weeks. That is a very long time. SAFe has hardening sprints. What’s agile about a hardening sprint?
If you want agile at scale, you can consider the autonomy, collaboration, and exploration I am explaining in my blog and eventually in my program management book. Yes, I understand that management wants to predict a schedule. Yes, I understand that management wants to predict a cost. I also understand that those are the wrong questions, because I wrote the book on managing the project portfolio (Manage Your Project Portfolio.
Placating management doesn’t change the conversation. We have to help management understand why value is part of the conversation, along with incremental funding. And that means technical teams have to do their part, and deliver value on a regular basis. If the teams don’t deliver, the conversation is moot. Management can return to demanding the estimates, because what else can they ask for? (This is why I say Agile is Not for Everyone. It is not. The cultural transitions are quite difficult for many organizations. Many orgs would benefit tremendously from a great RUP or staged delivery implementation.)
But please don’t tell me SAFe is agile. It’s not. It’s RUP.
Hi Johanna,
You have some great points. However, here are a few misnomers:
1. SAFe is NOT RUP. It’s not a “process”. It’s a framework, and it’s available and open to the public for free. It leverages Lean|Agile practices and brings them to scale. It’s successfully been doing this for years, as noted on the Case Study page:
SAFe Case Studies
2. SAFe guides to “develop on cadence, and deliver on demand”. There is no “staged delivery” as inferred.
3. Nothing in SAFe compares teams, although, we do have teams estimate in a common way.
4. SAFe has a HIP sprint (Hardening, Innovation, and Planning). Anyone who has done enterprise software at scale knows that all of these activities take time and coordination.
5. SAFe is indeed not RUP, there are no lifecycle phases, and RUP certainly teams weren’t coached to use XP and Scrum for their development efforts, because RUP largely preceded them.
I’d like to invite you to an SPC class. You definitely have the background and knowledge to provide effective guidance on SAFe.
Cheers and best,
–Jennifer
Hi Joanna!
You wrote “In my experience, staged delivery is more effective than SAFe.”
I wasnt aware that SAFe precluded the use of staged delivery. I thought it was more of a framework than a prescription and not as rigid as that. Can you say more about what you see as the difference between staged delivery and SAFe’s “Agile Release Train” the precludes a staged delivery approach?
Regarding hardening iterations, I’m not a big fan of needing those – and for a single-team product I like to avoid those. However, Ive also heard of the use of so called “stabilizing iterations” in Scrum@Scale efforts that seem to be the equivalent of SAFe’s “stabilizing” iteration,
My understanding however is that once you are talking about multi-team programs (or even programs where there are both software and hardware to be delivered and the software development is physically capable of being more iterative/staged than the physical hardware development) that the issue of technical-debt comes into play at larger scale, and that even if you are reasonably good about refactoring and automated-test coverage, there needs to “spaces” that deal paying off technical debt that creeps in across team and integration boundaries, and that “hardening” or “stabilizing” iterations fit well there (and the original intent was to handle creeping technical debt rather than poor functional quality).
Hi Jennifer
Johanna says she has kept all of her hair; I confess to rapidly losing mine.
The problem is that understanding SAFe – so it can be placed in an agile context – is like nailing jelly to the wall.
Here’s just one example. You say that “nothing in SAFe compares teams”. Yet when I asked Dean himself, he indicated fairly clearly that there must be an ROI, reflected in normalized estimates, so that work can be bid across teams (I quoted his exact reply earlier).
Now, it’s entirely possible that my bamboozled brain has failed to perform some essential gymnastics, and which are important to understanding agility at scale. And perhaps there’s some manoeuvering that the likes of Johanna Rothman and Ken Schwaber have failed to grasp as well.
Yet even if that is the case, my observation has to be that you, Dean, and the SAFe consultants I have met are not reading from the same sheet. Never has “agility” seemed so mutable! Please accept that that SAFe has a genuine problem here, and that it is damaging to the framework’s credibility. I’m not surprised that certain thought leaders are reacting so negatively to SAFe…which is a shame because it has validated a market gap.
I would politely request that before SAFe consultants seek to teach, they do us first the courtesy of valuing our remaining hair, and get their house in order.
Hi Ian,
I read your response above, and it was quite clear:
“Velocities are NOT normalized across teams. Estimating is. If you don’t normalize estimating, then there is no meaningful economics; you can’t figure ROI of you don’t know that the “I” is. If you want to scale agile, and there is no meaningful way to bid work across teams, and within a program, you will be blocked before you even try”
We are not comparing teams. We are simply normalizing estimating so that the enterprise has a common way of estimating work.
I’m sorry about your hair. Mine is turning grey, and it’s making me unhappy as well.
Cheers,
–Jennifer
Pingback: Trust, Agile Program Management, & Being Effective
Hi Jennifer,
Three things strike me:
1)Doesn’t “bid[ding]” work across teams inherently imply a comparison? Seems like we’re parsing words. After estimation, and the ROI decision, you wouldn’t place work in a different team, that has a substantially slower velocity and similar run rate. That would blow up your ROI, so thus you need to compare.
2) Which leads me to my bigger conundrum. For cost estimates to work this way, not only must estimation be normalized, but the delta of actuals to estimates must be normalized too. (Or forced to a theorectical zero; Or each team’s avg. estimate “error” known and adjusted for, which points back to #1) Lets assume that you could successfully normalize estimates, which I think is dubious. A team who’s velocity is consistently under estimates by 15% is not giving you the same cost estimate as a team who is over by 15%. That’s within a reasonable margin of error for each team, but gives you a real cost swing of 30%! To me, the SAFe community is fooling itself if it thinks normalizing estimation is anywhere near accurate. Since it can’t be, why bother? Seems like a case of using dubious data to make a decision “because we have to use something.”
Missed the 3rd thing.
3) A team that reworks things more often is ultimately more costly than a team that doesn’t. Which leads us down the rathole of comparing teams again. Normalized estimates on their own are not accurate predictors of cost across teams.
Johanna,
Your comments above about SAFe — Any chance you’d turn that into a blog post of its own? If you do, I’d be happy to blog and re-tweet it, as well as put it in my “favorite resources page” on my web site.
HI Charles,
Uh, gee, that never occurred to me. If I can turn it into something that doesn’t just sound like me ranting out of nowhere, I will. When I get closer to releasing a draft of Agile Program Management, maybe I will. Thanks for the nudge.
I’d prefer the rant, but I’m “Crazy” like that, and understand why you might not prefer the rant. 🙂
I have every incentive to get behind SAFe except 2:
1. Overall, conceding that there are some tiny bits of wisdom hidden in there, I don’t believe it’s Agile, and I believe it will cause more harm than good.
2. I don’t think its leaders will move in a direction, at any reasonable pace, that is Agile.
Regardless of the extra sales it would probably bring me, I can’t do this to my clients.
Now I am not the smartest cookie in the jar, but normalising estimation just sounds like voodoo. Chad nails it on the head. No two teams are the same, no two pieces of work are the same, so my tiny brain is struggling to understand how you can normalise estimation. Now maybe I am missing something, but the only way I could see that you could normalise estimation is to convert hours into story points, but then you are no longer estimating, and 9/10 you will get this wrong. You may as well go back to the PM thumb sucking delivery times.
Dillon, right! I don’t get it either. I have said “velocity is personal” about a gazillion times. Okay, maybe not that many times, but close. If you want to “normalize” estimation, you are looking for perfect estimation which is impossible. You have to go meta and ask why you are looking for that. The real question is why are you looking for that in a supposedly agile environment? I don’t get it.
This discussion is focused on one particular aspect of SAFe, but surly not all of SAFe is bad, and not all of it is not Agile (or more to the point supplements Agile). I’d like to here Johanna (and others) take on what is good in SAFe – and what are the alternative’s especially from the design and requirements point of view. I’m referring to the SAFe notions of Roles of Enterprise and System Architects, collaboration of intentional architecture and emergent design, architectural runways, architectural features and Epics
Hi Yonatan, I certainly like the idea of intentional architecture and emergent design. I have long advocated “build 3 features and see where that drives the architecture.” I have said it in my talks and in Manage It! Your Guide to Modern, Pragmatic Project Management. That is an excellent way to see the architecture evolve. You do the simplest thing that works, refactor it, and continue.
I believe that SAFe’s notion of an Enterprise Architect is similar to my notion of a Program Architect. My program architect is optional. Some programs need architects, some don’t. It really does depend on the kind of product you have. Seeing your email address, I suspect you need architects to guide the business value of the product (which is what I want a program architect to do).
I don’t understand what a System Architect does in SAFe. I read the words and I don’t understand.
In my view of program management, architects, even the Program Architect, are embedded on teams. I can’t tell if the system architect is embedded or not.
It’s not that SAFe is bad. For many people, it is a step away from waterfall towards a more iterative and incremental approach. I object to it being called agile when it’s not. If you called a dog’s tail a fifth leg, would that make it a leg? No. It’s still a tail.
People, teams, and organizations should do what is most effective for them now, and continue on their agile journey. But if they think they are already doing agile, and they are not, why would they continue their agile journey?
Pingback: Can we introduce real Agile mindset in a large traditional organisation? | LeanArch.eu
I guess a good argument against SAFe next to Johanna’s ones is that SAFe does not focus on value but on system-level incrementation. It’s really something fishy going on with the term “system” in SAFe. You can’t find it in the glossary of SAFe so you must derive its definition yourself and that’s a hard task. However, in general it tends to become a component (according to Bas Vodde’s terminology) and by no means working, integrated part of a software product. Furthermore there is a need for a mysterious (I can’t find any meaningful definition of it) component integration team within that “framework”. SAFe argues no team can integrate all parts of a complex systems in enterprise. Which makes the term “system” even more creepy.
Hi Aleksander. Especially in large programs, I find it’s quite difficult to do system-level increments. Maybe that’s why many people who use SAFe tends to have hardening sprints.
I prefer to increment in much smaller pieces, as in the level of a story. I don’t even like component teams, because they make it more difficult to do continuous integration across the program and continuous delivery, as a business objective.
If everyone works on stories as feature teams, there is no reason they can’t integrate all the time, even in a complex system. No reason at all. They do have to be set up to do so.