Investing in Architectural Infrastructure: A Business Conversation

Meet Wendy, a new CTO. She was hired to make the company’s flagship product, BigProduct, releaseable more frequently. In fact, her predecessor was fired due to his “inability to release product quickly enough.” Wendy’s been able to deliver products in adverse circumstances, so she feels she’s ready for the challenge.

Wendy meets with her senior technical staff to discuss the product’s architecture. She asks, “What would it take to release products quarterly?” The senior technical staff discuss a number of areas, and they are all in vehement agreement about one thing: they can never get to a quarterly release until they rearchitect the build system. During Wendy’s interviews, she discovered the builds took more time then she would have expected. Still, she’s resolved problems like that before without rearchitecting the infrastructure.

Right now, the build takes weeks to settle down: at least a few days to resolve the circular compile references, several days of not-quite-automated smoke tests, and more days of fixes to the code base so the smoke tests pass.

The slow builds are a root cause for Wendy’s predecessor’s inability to release product quickly enough. Here’s why:

  • The testers are weeks behind the developers, so the developers receive feedback about their work many weeks (sometimes months) after they thought they were done. The delayed feedback leads to the developers multi-tasking and rework during the rest of the project.
  • The technical staff are averse to fixing anything during the last six weeks of the project, because they are afraid they won’t have time to run enough tests to know the fix was appropriate.
  • The project managers can’t estimate when the fixing will end. They end up releasing the product based on the date, not on any objective release criteria.
  • The defects remaining open at the time of a release require patch releases, which customers hate to install. Creating each patch release requires multiple builds and multiple sets of tests. The developers can’t take shortcuts with the patch releases, because the risk of not detecting egregious problems is too high.
  • Releases start late and end later, in a never-ending spiral of not enough people an time. Distractions, such as the delayed fixing work, means the subsequent release is starved for people.

All of these problems add up to an unwieldy system that’s difficult to develop, to test, and to release.

So it turns out that rearchitecting the build system will be necessary after all. Worse yet, because of BigProduct’s layers, the senior technical staff’s estimate is that it will take nine months and twenty of the most senior people to rearchitect the build system. The way the technical staff thinks the project should be done means the company won’t be able to release product during those nine months, and the senior people won’t be available for other work.

Wendy is dismayed. It’s one thing to rearchitect the product, but this is rearchitecting the product’s necessary infrastructure. The price is too high, not just in time and people but also in delay of future releases. Wendy is confident that she can break the problem into smaller pieces to be able to release more frequently, but she knows there will be at least a three- to four-month period of no releases.

BigCorp Thinks Small

Most organizations don’t want to upgrade their enterprise-wide software such as BigProduct, more than once a year. So why does BigCorp have such an emphasis on quarterly releases anyway? Wendy’s company has owned the top end of the market for years. Now, there’s significant pressure from the low end of the market. BigCorp believe that it can compete in this growing market if it develops its own low-end offering, NotSoBigProduct, as a one-off from the BigProduct Code base and releases both products more than once a year. More frequent releases will allow the salespeople to sell into new accounts more readily, and to add features to lure potential customers in new markets.

BigCorp is competing both in the Mainstream as well as in the Early Adopter markets [3] and needs a release strategy that will allow it to compete in both. If the build system prevents the company from releasing in time to capture the Early Adopters, then it’s likely they will stop owning the Mainstream market. (The Early Adopters are creating a new market, even if the market is only a one-off the current market.)

Although you might think the high-end product would cannibalize the lower-end product, Wendy’s company doesn’t think it will. They prevailing view is that the high-end product will become irrelevant without the low-end product. BigCorp’s competitors will capture the low-end market and the customers won’t upgrade their products to BigProduct; they’ll upgrade to the eventual higher-end products from the competitors. . It’s clear to Wendy this is about eventual corporate survival, not the whim of the salespeople.

Time To Have A Conversation

Rearchitecture problems can be based on the product, but in my experience, are even more likely to be about the product infrastructure (the builds, tests, upgrades, release, installation, and so on). Whether you’re talking about the product or the product infrastructure, wherever you’ve cut corners in the past and incurred technical debt, you’re bound to have a rearchitecture conversation in the future.It helps if you have data on what the current architecture costs you now. Estimate what the rearchitecture effort will buy you and be ready to use that in your pitch.

If you’ve ever been faced with the business decision of whether to rearchitect, you know how difficult a conversation this can be. Wendy needs to figure out how to sell the rearchitecture project to her peers. This is how she’ll approach the selling problem:

  1. Before she has the conversation about rearchitecture, Wendy and her staff will develop at least three alternatives for the rearchitecture project. Then she can present her options to the senior executive team and explain why she chose the one she did.
  2. For the rearchitecture conversation, Wendy will first determine the principles [1] behind each person’s negotiating position, the “what’s in it for me” (WIIFM) for each person who needs to be persuaded.
  3. Using those principles, Wendy will make the case for the rearchitecture, making sure she discusses both the costs and the benefits. She’ll prepare even for the illogical reasons she shouldn’t go ahead with the rearchitecture.
  4. Finally, Wendy will make sure to close the deal with her peers.

Let’s follow along as Wendy prepares for and holds the conversation.

Develop at least three worthy alternatives

The first step in planning the rearchitecture conversation is to develop at least three worthy alternatives to the problem that you think is demanding rearchitecture [4]. After discussing the problem further with her staff, Wendy developed the alternatives shown in table 1.

Table 1: Worthy Alternatives for the Rearchitecture Project

Alternative Time and People required Advantages Disadvantages Overall assessment of risk and return
1. Rearchitect the build system all at once. (Wendy’s staff presented this as her only alternative.) Nine months, 20 senior developers Finishes the build rearchitecture in less than one year. No public releases until after the build rearchitecture is complete
2. Plan for an iterative rearchitecture of the build system. Three  months, five senior developers for each iteration, four to six iterations expected Can still release at least as often as releasing now, the longest period of inability to release is about three months Extends overall build rearchitecture project. Reduces technical risk, doesn’t help the problem of knowing if the product works
3. Continue as things are Won’t rock the technical staff boat Wendy will probably have to look for a new job because she won’t be able to deliver Low technical risk and  high job risk
4. Rearchitect the rest of the build process, such as the automated test Six  months, 10 senior testers Can be completed in parallel with a release, requires different people Builds will still take a week to complete, too long for more frequent releases Moderate technical risk; some reduction of time for the build process; still high product release risk
5. Plan to iterate on the build rearchitecture and also break the automated test rearchitecture into chunks Three months for each iteration, five senior developers and five senior testers, four to six iterations required Can still release at least as often as releasing now, the longest period of inability to release is about three months Requires more people than alternative 2 and means that Wendy has two infrastructure projects ongoing Moderate technical risk. High schedule risk for the infrastructure projects. Reduced product release risk

Like most projects with a single, “big bang” deliverable, Alternative 1 is risky, both technically and for the organization. If Wendy chooses this alternative, she will be committing herself to at least nine months of no releases when the company wants to release more frequently. On the other hand, Alternative 1 is the most technically straightforward — by rearchitecting the build system in one fell swoop, the technical staff won’t have to juggle multiple projects. Wendy can see why this alternative appeals to them, but the overall risk is nevertheless too high for her. Furthermore, Alternative 1 won’t — by itself — move the organization to quarterly releases.

Alternative 2 helps with the builds, but not the testing of each build. Since about half the build process time comes from some sort of testing activity, Wendy is concerned this alternative isn’t suitable either. What use are faster builds if you still don’t know whether the builds work?

Wendy’s ruled out alternative 3, although it is a possibility. Wendy thinks this company has potential to grow even larger and wants to take on this problem.

Alternative 4, where just the testing activities are automated and modified is tempting. But Wendy realizes that even if she could reduce the testing activities to one day, she’d still be dealing with long build times. She’s worried she won’t have reduced the problem enough in six months, and she’ll be under more pressure to deliver faster.

Alternative 5 incurs a moderate amount of technical risk, and some product release risk, but is the fastest plan to move the organization to quarterly releases. Wendy realizes she’ll only get one chance to sell her plan to the other executives and the Board, so she decides on alternative 5. She’s concerned about her staff’s ability to manage iterative projects and the interdependencies between the projects, so she needs to address those issues as she prepares for the rearchitecture conversation. Alternative 5 is the safest in terms of ability to release the product, no matter what happens while alternative 5 progresses.

Wendy developed this chart with her immediate staff, so they’re all aware of the advantages and disadvantages. Wendy gathered her staff’s opinions, and then decided on alternative 5. Each person on her immediate staff has committed to supporting her choice.

Determine the WIIFMs

Before you can have the rearchitecture conversation, make sure you understand each person’s WIIFM. These are the principles each participant brings to the negotiation. If they become stuck during the discussion, you can help them recognize their principles. These WIIFMs help you know how to sell an alternative to your peers.

Top management understands cost, schedule, and customer perception of the company and the product. To understand what’s important to each person, Wendy needs to define the problem in ways that clarify the current situation for her peers—she needs to uncover the WIIFM for each person. Table 2 shows how Wendy thinks about the WIIFMs:

Who WIIFM How the rearchitecture effort affects that WIIFM How the selected alternative affects the WIIFM
Sales VP Increase revenue by selling new products and continuing to sell selected older products.Increase number of reference accounts A faster build process will help push products out the door faster.A reorganized build process will allow for product reorganization, which may allow for different selling. Alternative 5 supplies both build speed and information about how useful the builds are faster than the other alternatives.>
Service VP Sell more service contracts.Reduce the cost of providing service. With more frequent releases, the service contract is more valuable.Fewer patch releases reduce the cost of providing service Alternative 5 helps increase the number of overall product releases and decrease the number of patch releases.
HR VP Retain staff. Reduce cost of acquiring new staff. There will be more technical challenge in performing more technical work and less organizational and problem-fixing work.Right now, new people take over six months to fully integrate into the organization because the product is so unwieldy and difficult to understand. With a new build system, it should be faster to bring people up to speed. Alternative 5 will enable BigCorp to integrate people into the organization faster, by making it easier for new developers to see how their changes affect the system.
Marketing VP Drive how the product line is perceived in the market. Create a product lifecycle (release cycle) that entices customers by planning for product line improvements and changes More releases show the company is continually improving the product and is reaching out to new customers. More releases allows for more frequent and more adaptable product line planning. Wendy’s staff can’t release more often than once a year now. The more frequent the releases, the more flexibility the company has in creating product line improvements. Alternative 5 helps create good-enough releases, not just more releases that don’t work. When there’s only one release a year, the product managers try to force all the product line improvements into that release. Alternative 5 allows the product managers to slot features into the most appropriate release. They can choose when to respond to customer requests.
CFO Increase profit margin.Be accurate in assigning software effort to capital or expense outlays Reducing build time will allow more frequent releases, which will increase the profit per release.Right now, the software stays in R&D too long. Fixing the build system will allow more cost to be expensed rather than capitalized. Alternative 5 helps not just the speed of the builds but also decrease the frequency of patch releases, reducing overall expenses.
Senior management as a group Obtain value from product development investment.Use the products to move the company forward (to gain more market share and profit margin) More frequent releases allow the company to obtain value more quickly. More frequent releases provides more flexibility in product bundling, which can help market share and profit margin. Alternative 5 is a stepping-stone to the company’s product strategy. By itself, it doesn’t help change the product line or bring in customers. But without it, the company can’t proceed with its product plans using its current code base as an asset.

 

Make the Case

The costs are clear. Alternative 5 will cost 10 people for at least one year, probably closer to 18 months. That’s $1M – $1.5M for the rearchitecture project. And, during that time, there will be times when an early release or an interim release will not be possible. Wendy needs to make the benefits as clear as the costs to her peers.

Benefits are about value. When you want to make the case for a rearchitecture project, you need to discuss the costs of continuing to perform the work the current way versus. the cost and value of changing the way you perform work.

Wendy is going to make her case by explaining the costs of continuing with the build system the way it is. She’ll discuss the customer experience of having to wait for new features and having to install patches continually, disrupting normal work. She’ll also discuss the impact of the rearchitecture project on development cost and project time.

Before Wendy approaches her peers, first she decides where she thinks people stand, so she can consider what she’s going to have to discuss at the operations meeting. One way to think about that is to use some sales ideas. In this table, Wendy writes down how she thinks her peers will react when she proposes her rearchitecture alternative. Economic buyers release the money for the product. Technical users can veto a sale. User buyers judge the impact of a product on their job, but don’t veto a sale. Sometimes, you’re lucky enough to have a coach buyer, someone who helps you close the sale. In this case, the Marketing VP is Wendy’s coach buyer.

Table 3: Buying Influence Chart [2]

Economic buyers release money

  • CEO (Ed)
User buyers judge impact on job

  • HR VP (Frank)Sales VP (Don)
  • CFO (Ken)
Technical users screen out requests

  • Service VP (Vince)
Coach buyers guide for this sale

  • Marketing VP (Mark)

Wendy thinks if she talks to the Mark, the Marketing VP, she can persuade him to help her make her case. Wendy’s concerned that Vince, the Service VP will have serious problems with her proposal, because she won’t be able to spare people to create patches for already released products. Wendy’s sure the HR and Sales VPs are not going to be happy with the proposal because of the cost and delay.

Wendy has a private meeting with Mark and explains her case. She asks for his support, and he agrees, as long as she can guarantee they won’t have to go longer than four months between the last release under the old build system and the new release under the new build system. Wendy agrees.

Now Wendy’s ready to make her case. She’s asked for time in the operations meeting, where the senior executive team discusses their priorities, makes their decisions, and allocates funding for various projects. Here’s how she makes the case for the rearchitecture project:

“As you know, we’ve had trouble for the past couple of years when we tried to accelerate the rate of our releases. We know why—the build system is too slow. We’re using the same build system with the same organization as we did when we first built the product, and it’s not robust enough to get the job done. The builds take a couple of weeks to settle out which leads to lots of rework, insufficient testing, difficulty making fixes at the end of the project, and in general, an unwieldy product release mechanism. I know that you folks want more frequent releases, to reduce the cost of service, to better organize a product lifecycle, to attract more and different customers. ”

Wendy passes out two pages, and says, “This first page discusses where the time goes for each build. The second page (a copy of Table 1: Worthy Alternatives for the Rearchitecture Project) is an analysis of our alternatives to the current build system so you can see where I’m going.”

Wendy gives everyone a chance to settle down and then says, “Now let me walk you through these alternatives. The reason there’s more than one alternative is that I wanted you to see that I had done my homework—as well as making sure my staff did their homework—so you didn’t think I was forcing one alternative down our throats.”

Notice what Wendy did here. First she explained the general problem (the consequences of slow builds). She mentioned each person’s WIIFM. Then she followed it up with data about how bad the problem is (where the time goes for each build). Finally, she hands out the page of alternatives and talks about “our throats,” helping her peers realize that this is everyone’s problem.

Don, the Sales VP says, “Hey Wendy, this isn’t my problem, this is your problem. All I need to know is that you’re the one to blame when this doesn’t work.” He chuckles, thinking he can get this CTO thrown out too. Then sales can run the company.

Wendy replies, “Yes, Don, this is my problem to implement. But while the tactics are my problem, this is a strategic issue we need to resolve as a group. My predecessor brought this to your attention several years ago and you postponed work on infrastructure. At the time, the decision made sense. But we can’t postpone it any longer, and we have a hard decision here.” Notice how Wendy refuses to take blame, nor does she assign blame. Blaming anyone for previous decisions or this one is an approach that doesn’t help the organization. The entire organization needs to be behind the effort, not just one group.

Frank, the HR VP asks, “Wendy, is this going to affect our ability to recruit new engineers? You have some open reqs and if all they’re going to do is maintenance, I’m going to have a tough time recruiting people.”

Wendy answers, “No, only the most senior people who know the most about the product will be allowed on this project.”

Frank counters with, “Well, aren’t they going to be upset and want to leave? I’ve heard people say that working on the build system is like working on a stopped-up toilet.” The rest of the group chuckled.

Wendy replies, “No, I have the buy-in of my direct staff and both the top developer and test architects. They’re excited about finally being able to do this and not have to patch their way around the system.”

The conversation continued, with Wendy listening for positions, and emotions such as greed or fear. When she hears a firm position, such as “That can’t work here!” she knows she needs to look at the WIIFM for that person, to counter the position, and move the person back to the principle. When she hears greed or fear, she wants to understand what’s causing the person to think that way.

Ken, the CFO, wants some reassurance. “Wendy, are we going to spend more time in requirements and less time in development? We don’t spend much time in requirements now. How will this affect our ability to expense more and capitalize less?”

Wendy replies, “I don’t think we can know the requirements up front, so we may not spend more time in requirements. But all the time we now fix problems after the release—that time should be significantly reduced. It costs us about $18,000 to fix a defect post-release. That’s a huge expense. I anticipate that the alternative I want to proceed with should bring the cost to fix a defect post-release down to under $10,000. And, we’ll need to do fewer patches because we can fix more before we release. Right now, we spend about $432,000 on two patches/month, each month of the year. We can halve the number of patches, so even if we do one patch a month for $10,000, that’s still only $120,000. We’ll save money just on not having to do so many patches. I can’t anticipate the expense/capitalization issue yet, I just don’t know. It’s got to be better than what we do now.”

Logic isn’t all it’s cracked up to be

Even when you’ve marshaled all your data and know what you want to accomplish, you may still hear some illogical arguments. That’s when you want to use your coach buyer to help you through this sale. Vince, the Service VP, stands up and bangs his hand on the table. “Wendy, I can’t believe you think we’d fall for this stupid idea. Do you think we’re nuts??”

Wendy’s momentarily taken aback. She’s only been with the company two weeks and now one of her peers is yelling at her in a senior management meeting. Oh well, she’s been here before. But before she can say anything, Mark, the Marketing VP steps in. “Vince, hey, I don’t think this is stupid. Here’s what this proposal will do for me.”

Mark outlines why he likes the proposal. “But even more important for you, here’s what the proposal will do for you. You know those bug triage meetings you love so much when the developers tell you they can’t fix problems you want them to fix at the end of the release because it’s too risky? Well, they won’t have that excuse in a year. They’ll have to fix those problems. Your guys are going to love this.”

Wendy replies, “Right, Mark. Here’s how I envision the bug triage meetings working, starting in about six months and here’s how they’ll work in a year.” Wendy explains the details of how things will work to Vince.

Vince grudgingly sits down and says, “Well, I don’t like it. Right now, I know what I’m going to get. I’m not so sure what’s really going to happen when you start this project.”

Wendy continues the conversation with her peers, using logic and her coach to help her win over everyone at the meeting. Now all that’s left is to close the deal.

Close the deal

Wendy decided before the meeting she wanted an answer at this operations meeting, not the next one in two weeks. She asks Ed, the CEO if he’s heard enough and is comfortable making a decision. Ed is, and says, “I’m going to let Wendy run with this. She’ll do alternative 5, so she can assess progress as she goes. Wendy, we’ll want interim status reports on this. We don’t want to be left in the dark.” Wendy assures him and her peers that she’ll let them know what’s going on.

If you’re in a position where you’re not ready to ask for a decision to close the deal, say so at the beginning of the meeting. Explain that this is an information session, and the decision will be in a couple of weeks. You can use the intervening time to sound people out and see if you can turn any of them into coach buyers.

Summary

The conversation about whether to rearchitect is never easy. Begin by developing at least three alternatives for the rearchitecture and make sure you understand the advantages and disadvantages of each. Then make sure you’ve thought about the people to whom you need to sell and address their WIIFMs. Once you have the conversation, keep your cool, and keep bringing the conversation back to a jointly owned problem you will all solve within this organization’s context without blame.

Don’t forget to ask for a decision, to close the sale. Then, go to work!

Acknowledgements

I thank Laurent Bossavit, Esther Derby, and Rob Purser for their review of this article.

References

1. Fisher, Roger, William Ury, and Bruce Patton. Getting to Yes. Penguin Books, 1991.
2. Miller, Robert B. amd Stephen Heiman. Strategic Selling: The Unique Sales System Proven Successful by America’s Best Companies. William Morrow, 1985.
3. Moore, Geoffrey A. Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers. HarperBusiness. 1991.
4. Weinberg, Gerald M. Becoming a Technical Leader: An Organic Problem-Solving Approach. Dorset House, 1986.

Leave a Reply