Thoughts on Infrastructure, Technical Debt, and Automated Test Framework

I've had several conversations in email and with clients recently that have all been about this question: “What do we do about our infrastructure?” Either the project or the program has to create/update/upgraded their architecture or automated test infrastructure, pay down technical debt, or somehow do something that's not part of a story.

And, that's the part where I say, “Whoa, Nellie. How is technical debt not part of a story? Why does anyone care how you take care of the code base? Or the automated test infrastructure base? Why does anyone care how you curate the systems? Aren't you in charge of your environment?”

There's often a stunned silence. That's when I realize that while the outward part of the project or program looks agile, the project culture is not agile.

An agile project culture has an empowered team, a team who knows that they must leave the code base a little cleaner than they found it the day before. A team who knows that they must improve the automation every day. A team who knows the story is not done until the entire story is done, and that includes the automated tests and the automated test framework. A team who knows they have to work together to deliver business value every day, not just at the end of the iteration.

This problem is related to the feature-itis problem on the part of the product owner not wanting to take iteration time to schedule anything other than features in an iteration. If the product owner only sees what code developers can do, and doesn't look at what test developers can do, the project is not agile. If the code developers are the only ones estimating the backlog, that's a huge problem.

Here are some solutions:

  1. Call everyone on the project “developers.” Or, call everyone “testers.” I call the team the product development team, and everyone on the team product developers. You have to change the idea that one part of the team is responsible for code and one part is responsible for tests and that the two parts do not operate together. The two parts (plus all the other parts) are responsible for moving the features to a cohesive approach to done. The more you reinforce one group is testers and one group is developers, the less chance you have of getting to done.
  2. Make sure you have a team definition of done or that you somehow know what done means. It's not developer-done, or tester-done, it's demo-worth-done, or release-able-worth-done. I know of some teams that take a while and many discussions to agree on what done means. Discuss it. Don't worry if you don't agree right away. Keep discussing it. This discussion is critical to your success as a team.
  3. Stop estimating altogether. If you have an item on the backlog larger than something the entire project team can complete in a day or two, break it apart–not into tasks, but into smaller stories of business value. Now, you have as many stories as there are days in the iteration, more or less. Makes estimating much easier. You have more conversations about the stories, and much less estimating time.
  4. When you work on the code, wherever you are in the code, leave it a little cleaner than when you found them.
  5. When you work on the tests, leave them a little cleaner than when you found them.
  6. Have a product roadmap which includes the automation. If your product owner doesn't want to or can't own the automation, then the technical people must own the goals and the plan for the automation. But automation is a project. You should have a vision and release criteria. You should adapt to change in that project, just like any other project.
  7. As you work on a story, whomever you are, you help out wherever you are. If you are, by nature, a code developer, you start with the code. Here's a story to illustrate what I mean:

You happen to be Platform Paul and you do some development, some refactoring, maybe some rearchitecture, some unit test development, whatever it takes to make your code work and checked in. Fine, you are done.

You check with Tina the tester. She is having trouble with the system tests. You do not abandon her, saying, “My part is done.” Oh, no no no no you don't.

You say, “Hey, Tina, what's going on? What can I do?” She tells you. The two of you work on her automated test framework, refactoring it until it works for the new code you checked in and her tests. Once it works, and her tests work, now the two of you walk over to Willie Writer.

Willie glares at both of you and says, “I keep writing the online help, but you two kept refactoring all afternoon. I keep chasing you two and those guys!” as he pointed to the other two developers. The three of you laugh and then all three of you complete the online help in the next hour.

Willie and Tina and you do a little exploratory testing under Tina's guidance, because what do you know about exploratory testing? She calls it “session-based testing.”

Now, the three of you check with the other two developers who have finished the GUI and the middleware, and only now do you move the story to done. Because the feature required a little platform work, more middleware, and a little GUI to be complete.

Product owners, if you don't want to fund technical debt, you will create more of it. You will slow down the rate of creating features. I have example graphs of this in Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects.

You don't have to have the perfect automated test framework, not at the beginning of a project. You don't know what it is at the beginning of a project. You only know you need one. But you can write a little and refactor it. I wrote a column about that.

And, when it comes to creating technical debt, the one thing you must do is, stop. No matter what, you must not create more. And, if you wander into some code or tests that have technical debt, I do not see how you can be a professional and leave it there. At the very least, you can create a defect report that says, “We have technical debt here.” You know it's going to bite you on the tush at the most inopportune time.

I am not a fan of “go rewrite the system to avoid technical debt.” But you and I both know that technical debt slows down system development and often slows down system performance. I want to avoid rewrites. I want to clean up as I go. If you clean up every single one or two-day feature, you don't have to pay a huge price, ever.

12 Replies to “Thoughts on Infrastructure, Technical Debt, and Automated Test Framework”

  1. So a hard problem is where to start reducing your debt load if you’ve already got a mountain of it. It is much harder to “clean up” as you revise some code when there is a lot of pressure not to break what is already working (but still riddled with technical problems).

    I am looking forward to hearing your thoughts on this : How to start when there’s a really big mess? In that case, small, just being a professional clean-up acts may not even make a dent.

  2. Rebecca:

    First and foremost you have to understand that refactoring without tests is NOT refactoring. That’s called ‘flying by the seat of your pants’ and is extremely dangerous. Don’t do it. Ever. Another thing to understand (and most developers don’t IMO) is that refactoring is NOT rewriting. If you haven’t read or at least have a cursory understanding of Martin Fowler’s seminal book on the subject, then make the devs read the intro text so they at least understand the difference between refactoring and rewriting. I’d highly encourage them to keep going. Many developers rely on the refactoring tooling without understanding the underlying concepts which often makes them… a tad reckless.

    Once you have those two things down you get to the meat of your question which essentially is ‘where do I start?’ Use a higher level testing framework to wrap the nastiness in acceptance style tests so you can at least have some small assurance that what you do won’t break the world. Each language has it’s own. In Java, I prefer something like Spock (groovy) for higher level acceptance tests. Any web based application, regardless of language can leverage something like Selenium for high level UI testing (mind the data and UI brittleness here). The point is to wrap the big-nasty in as good a test as you can. Then begin breaking out smaller (tested) pieces while keeping the overarching test in place. At some point that big nasty will become a smaller set of much better tested methods at which time you can consider whether to retire the overarching test entirely if it’s too brittle, or keep it if it provides value.

    The previous is one idea of the how, but what about the ‘where’? Personally, I’d start with the most critical and probably nastiest area I could find. Case in point, I work in education and we had a several THOUSAND line method that handled the registration of a student in a course. We wrapped that in very course grained acceptance tests… the best we could, and began to break it apart. We chose it because it was vital to our application and because it was simply hideous. Some will advocate to start small. I personally prefer the opposite approach as when teams start small, they tend to get in a rut of small changes with small benefits. This fosters a fear in them of tackling the larger more complex problems at times.

    Lastly, if you’re going to do the above, my strong advice is to use a VCS such as git that has very good branch and merge support. Large scale refactorings often occur over a short period of time on a branch that should really be disposable. Why you ask? Because you’ll probably screw it up 4 or 5 times before you get it right. At least we did anyway.

    Anyway, you didn’t ask me the question, but it struck a cord with me so I thought I’d pile on with my 2 cents.

    BTW, excellent points on the agile side. I was chatting with some folk at a conference recently and it dawned on all of us that we’ve horribly screwed up the definition of ‘cross functional’. It doesn’t mean ‘I work next to the build box guy’. It means ‘I actually DO the other guys job sometimes.’ Great read!

  3. I think one problem is that people are used to living in debt. Debt is silent. It’d be nicer if technical debt made grinding noises in the transmission and flashing lights on the dashboard. Irritating and a little scary. Maybe velocity and fault feedback ratio charts should come with a soundtrack.

  4. While I like the *idea* behind #3, it scares me to share this with new teams. Saying “Stop estimating altogether.” then clarifying it further in the paragraph is dangerous. People might read that and not read the rest of it! You can’t break a story up without some semblance of estimating – even if it’s only enough to get to the point that you recognize you need to break it down. Also, without a form of estimation, how will the team know what they can fit in a day?

  5. So much truth and important lessons in this article. When you work with teams that do things “right”, it’s easy to forget how all you say in the article can be extremely difficult to achieve.

    Regarding #3, while a couple of days may not be enough to get some stories done, my philosophy is that spending a long time estimating, especially when it comes to task breakdown and ideal hours, is unproductive, boring and time consuming. At the time of estimation, one should have just enough information to say if something is achievable in a reasonable time scale. Other discussions and minor details should be addressed at the “last responsible moment”. In order for this to work, however, the team has to be mature and know very well what they are doing.

  6. Really powerful and inspiring! We are going through the hurdels at the moment to adopt ATDD practices in our teams. We customize and configure a vendor product for our business, which comes with some limitations like how much and how efficiently we can test our code (code for customization and configuration). Also lacking the mindset we need for test driven design and development. Will really appriciate if you have any suggestions.

  7. Nice post reminding than to deliver value through software you need often little bit more than just write code. As I have to leave the code cleaner than when I came, I will suggest to correct the “Infrastructructure” part of the title to “Infrastructure” ;o)

    1. Ahem, thank you! I did fix the title. Somehow, I didn’t see that! I miss the defects in my own code, and also in my own prose. Not a surprise to me.

  8. Pingback: Legacy Applications: getting out of technical debt | Cedar Hill Tech Notes

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.