Agile Has Not Crossed the Chasm, A Contrarian View
Note: I wrote this article in 2011. It was published in 2012. I hope at some point this article will be wrong.
I hear all the time from people that “Agile has crossed the chasm. Agile is everywhere. Agile has made inroads to every organization, to every industry, yes, to every type of software development. Agile is a flaming success!”
The reality I see is quite different. What I see is the intent for agile has crossed the chasm.
Agile Success is limited
Sure, there is plenty of success. Here’s what I’ve seen. Agile works for small co-located teams in small to mid-size organizations. If you have smaller projects, with one or two teams, composed of developers and testers working on software applications, where they do not have to ship their software out the door.
Agile also works in a very few larger organizations where there is significant understanding and buy-in on the part of senior management, where the product lines all support one another and the products or systems are related.
Bounce Back is too Common
Here’s the problem I see. We’ve seen several consultants and consulting organizations take the credit for the same large media organization succeed with their agile transition. Well, this same organization is on their fourth or fifth agile transition by now. They have the common problems of any significant process change: it’s difficult, culturally challenging, and the entropy of the organization is working against them. It’s easier to bounce back to chaos or waterfall.
I’ve seen agile project managers and managers in the organization draw fences around their teams and organizations, saying “I am successful. Do not make me help the rest of organization transition. I don’t care if the rest of the organization is waterfall. I will find a way to work with them. Don’t make me optimize for the greater good.”
Where is the Evidence?
So that’s just one organization. Where’s the evidence for saying that agile has not crossed the chasm?
How would we know anything has crossed the chasm?
First, let’s review how we got here. For years, we’ve heard the advice, “Start with a pilot project.” Well, if you read Moore’s Crossing the Chasm, (p.35), you’ll see that the visionaries—the very first people to adopt a new product — do just that:
[They] start out with a pilot project … followed by more project work, conducted in phases, with milestones, and the like. The visionaries’ idea is to be able to stay very close to the development train to make sure it is going in the right direction and to be able to get off if they discover it is not going where they thought.
That sounds like the advice many consultants provide to people starting their agile journey: start with a pilot project. Learn from that pilot. Add more projects, keep them going in the right direction and keep them going in the right direction.
Okay, so that’s the visionaries, the early market. That’s not anywhere near the chasm.
We are still in the early adopter phase of agile. Here are the problems we know how to solve, the problems we are still solving and the problems we barely have a clue about:
Problems we know how to solve:
- Small team agile (1-3 teams)
- Collocated teams
- Cross-functional teams
Problems we are still solving:
- Medium-team agile (4-8 teams)
- Project portfolio management in the small
- Geographically distributed teams
- Moving away from coaching
Problems we barely have a clue about:
- Agile “in the large”: Program teams, project portfolio management, moving to continuous integration for very large programs
- Program team agile (9 and more teams, up to 30-60 teams
- Integrating hardware
- Integrating waterfall teams
- Integrating silos of people into cross-functional teams
- A “recipe” to transition to agile for anyone (needed for crossing the chasm)
- Project portfolio management in the large (even though I’ve written the book, it’s not clear anyone has read it. And, in the large, you really do need a tool)
- What managers really do for an agile organization in the large
What do the Early Adopters Expect?
If you think about who the early adopters are, you can see that they tend to be the people who are the people whose problems we know how to solve or the people whose problems we are solving. The pragmatists are the people who have the problems we barely have a clue about. They want the recipe. We don’t have it.
Because although we know how to make agile work for one or two collocated cross-functional teams in small and medium size organizations, we do not know how to make agile work for up to hundreds of cross-functional teams who are geographically distributed who are making combinations of hardware/software products. We don’t have all the answers for first-line and middle managers.
That’s why none of us has written the book about how to transition to agile. We can’t. It doesn’t exist yet.
What does the market for the pragmatists look like?
“When pragmatists buy … they are in it for the long haul.” … “Pragmatists are reasonably price-sensitive. … in the absence of any special differentiation, they want the best deal. … Overall, to market to pragmatists, you must be patient.” — Moore, Crossing the Chasm
Well, to me, that sounds exactly like my clients—but agile is not ready for them. Many of my potential clients balk at the cost of certification training. They don’t understand why they need coaching—it seems like a huge expense. My potential clients are pragmatists. That’s why they are trying a breadth-first approach to agile.
We don’t understand how to package an agile transition for the pragmatists so they attempt a transition to agile they way they would try a software transition: breadth first. A breadth-first approach is not a horrible idea. However, combined with insufficient training and no coaching, it’s a setup for failure.
What Do Current Managers Need?
I see management desire to change their minds often, which is a great thing for features. It’s not great when it comes to team makeup, or to which project people are working on. When managers change their minds often and can’t commit to one thing, including an approach to projects, that means they don’t commit the resources, as in money, to an agile transition.
Potential clients often call me asking for the “quick and dirty” agile training because they don’t have the time to do the “regular” training. They also claim to have no money for coaching. Even when I explain the return they can expect for training or coaching, they expect management to be unwilling to spend money on training or coaching—whether it’s me or someone else.
Such managers are terrified of spending money on an “unproven approach” such as agile. So I often point those managers to Michael Mah’s data showing agile’s return on quality and release dates. And, because the idea of empirical projects is so foreign to these managers, they have real trouble understanding the empiricism of agile until they experience it. And that’s a problem because agile started as a development-driven approach.
Agile is not just for developers
And, even I, an agile advocate now, was slow to move to agile, because I didn’t see what the manager’s role in agile was. We have insufficient agile training for managers, so managers don’t know what their jobs are. I happen to offer an executive agile briefing, but few of my clients take it.
Managers don’t realize they are supposed to pay for training, so they undertrain the teams. The teams don’t realize they are supposed to report obstacles, so the agile “installation” never really gets installed, and the teams bounce back to either chaos or waterfall.
I recently taught a tutorial on making distributed agile teams work. A fellow consultant participated in the tutorial. We were discussing metrics the participants could gather to convince management to move from dispersed teams to cross-functional teams. The consultant said, “Use the Gantt.” I said, “What Gantt?” He explained, “Their management requires they have a Gantt.” “For a two-week iteration?” I queried, astonished. “Yes, every two weeks, they need another Gantt, explaining what each person will do.”
I asked him more questions. This action is not limited to one client, but occurs at many of his clients. This is not agile. This is management not understanding what they can see at the end of the iteration from an educated and trained team.
Agile requires tremendous team discipline. It also requires management discipline. Because it requires that management actually perform the job of management, not command-and-control overseeing, many managers have no idea what they are supposed to do— and flounder.
Agile is an Ecosystem of Systems
Agile is a system of development at the team level. If the team can’t get to working software every iteration, or with every feature, the agile system isn’t working. The team can’t cross their own chasm inside the organization. It also can’t attract other teams inside the organization. Working at the team level is both necessary and insufficient. Agile requires a management system.
To be effective in agile, the managers need to manage the project portfolio, be the champions for the team, provide meta feedback and coaching for the team members, initiate communities of practice, and build the organizational capacity of the team. This requires a light and deft touch—a true form of servant leadership.
Program managers, those who shepherd collections of projects across the organization where value is perceived in the total deliverable, not in each project— do not have a complete answer to how to do agile. For example, are release trains agile or not? I never thought so, because if you only integrate once a quarter, is that really agile? One well-respected member of the agile community thinks so. I’ve been working with some clients who are thrilled to be able to integrate everything once a quarter. I am working with people where the cost of continuous integration is quite high.
We don’t quite know how to do agile successfully with hardware/software products. We have pieces of that problem addressed, but managing projects with long-lead-time components or incremental funding is tricky.
We are only at the beginning of our agile journey
Because we are still experimenting with solutions, and because we—as an industry—are still implementing breadth first and not depth first, I claim we are still not across the chasm.
Companies and managers are still suspicious of agile. “Show me the proof!” is still a rallying cry for managers. Because of that— and because we are still missing many of the answers— I claim we are still in the early adopter phase.
© 2012 Johanna Rothman. This article was published in Cutter’s “Has Agile Crossed the Chasm” Executive Report, July 2012, along with commentary by other senior consultants. Contact Cutter to obtain the entire report.