I like continuous integration. A lot. I started being an aficionado of continuous integration back in my senior year of university . It was my very first (and last) team project in my college career. There were three of us. The project manager waited until the night before the project was due to get us all together (argh #1). I had completed much of my code several weeks before (argh #2: who can remember what they'd written several weeks ago?). We wrote code madly for hours, and then tried to make it work, starting about 9pm. It didn't work. We stayed up all night (argh #3). I finally went back to my dorm about 7am so I could shower and eat breakfast for my 8am class. The other two guys blamed me for leaving, saying I should stay with them. I explained nothing had worked yet, nothing was going to work if I went to my other classes (just about the only smart thing I'd said all night).
That's the night I realized that if you don't start putting software together a little bit at a time from the beginning it gets harder the more you have to put it together. I've been a fan of continuous integration ever since.
I am sure there are projects where continuous integration might not be the answer, and I have not met them yet. And, I admit, there are times when the cost of continuous integration seems pretty high.
The agilist in me says, “Do it more often. That way you see what the impediments are, so you can see what is so difficult and you can make it easier every time you do it.” The pragmatist in me says, “Not everyone knows how to do it more often. Have pity on these people. Do you want to make them suffer?”
I don't want to make people suffer. I do want people to realize that continuous integration is often well worth their time, even on a large program. So, here are some steps to help you move to more continuous integration, depending on where you are.
What Does Continuous Integration Mean to You?
The first question is this: What does CI mean to you? To me, it means that the software is Done. That is, it is compiled, tested, and not just demoable, but releaseable.
Now, it doesn't have to mean that to you, and it certainly doesn't have to mean that on a program, especially on a large program. But you should know what continuous integration means on your program. And, you should know who decides.
On a program, I recommend that a technical program manager suggest a strawman proposal of what done means, and see if the feature teams can live with it. Then, the feature teams should test that strawman for a limited time, such as an iteration, and see if that proposal makes sense to them. If you're working in kanban, set a time period, such as one week or two, to make sure you see some features flow through the system and see if the proposal makes sense.
Now, everyone knows what done means for the program, so you know what continuous integration can mean.
Where Are You in Your Journey to Continuous Integration and Agile?
The first step is to know where you are.
The ideal case: Are you finishing features every iteration, as often as every day or so? You are likely doing continuous integration across your program.
What I see in many teams practicing agile: Are you finishing features in a hockey stick, with most of the features finishing at the end of the iteration? This is tricky, and leads to staged integration, and makes for staged integration in a program if all the feature teams do this.
What I see in some teams quite new to agile: Do your features span iterations? If your features span iterations, you need to make your features smaller. The reason to make your features smaller has nothing to do with continuous integration yet. Making features smaller is all about seeing your progress and providing feedback to the product owner/customer and making sure you actually complete work inside the iteration. If you are using kanban, this is similar to seeing a board not change for weeks while the same features are still on the board, nothing moving. You are also likely doing staged integration.
If some of your feature teams do one thing, some of your feature teams do something else and you need a way to merge them all together. When I work with program teams, I find the teams doing some of each of these. And, the waterfall teams are doing something else entirely. On a program, we need a way to bring the entire product together on a periodic basis.
Stay tuned for part 2.
I’m curious how CI could be expensive to implement? Even back in my waterfall days, we had at least a daily build and test process (this was in the early 90s!). Granted, back then we had a person whose whole job was taking care of that and the version control, but it didn’t seem too costly.
The smaller teams I’ve been on had CI up and running in a day or two with open source tools and a cheap server. I’ve worked at a larger company with 25 teams where a company-wide build ran at least once per day, and most of the small teams also had their own CI.
I’ve talked to teams at global companies who don’t seem to have issues with CI. Now, they do have issues with things like maintainable automated tests!
I just think that a short feedback loop is something no team can do without. Some practices are a must IMO, and this is one.
I happen to agree with you, but not everyone agrees with you and me. Shocking, I know! Maybe in part 2, when I talk about # of teams and add waterfall, it will be clearer.
I’m a sysadmin who’s just learning more about CI, but I can easily see where the cost come could from (assuming I know enough to really see what’s it’s all about). If you’re doing incremental improvements (ie you have a daily build process already) maybe cost isn’t so relevant.
We’re not using CI – in fact we’re far from it. Our monthly releases are possibly about to go to quarterly as it’s impractical to handle the release process, data refreshes etc in a shorter timeframe across all our dev/test/preprod/prod environments. I’ve started looking at the various challenges, toolsets, resources, and culture that we might need to implement. There’s significant cost in servers, training, changed processes etc – and changing culture could be the most expensive in terms of time and effort.
I think with most Agile/XP practices there is usually initial resistance, “we don’t have time to write all these tests”, “you want us to check in how often”, etc. I know I carped about the practices a LOT when I was learning them the first time.
There are certainly costs associated with setting up a CI system, but once groups start using them they see the benefits greatly outweigh the costs. The tools have come quite a ways and setting up a build server is usually only a day or two, and many are open source. I am currently using TeamCity but have also used Hudson and CruiseControl.NET.
Change is never easy. Looking forward to part 2.
Lisa,
The cost of setting up a build server is like you say very small (a day or two). Making everyone integrate frequently is a whole different ball game. It means that developers need to learn how to slice work to small chunks, AND learn how to truely finish them. And this is hard.
The benefits of course amke this worthwhile.
@Ed Grigson
I think there is a distinction to be made between CI and release. As you point out, there can be a lot of work in “releasing software”. I am in the medical device industry, so although we use CI, we are not pushing releases out on a daily dasis. But one of the hidden benefits of CI is it helps to streamline your release process by moving you in the direction of automating more of your build, test, and documentation processes. All that will help with a formal release.
It sounds like there are some real challenges in your environment, so in your case there would definitely be more costs in implementing CI. Having to write scripts to automate data migration/refresh takes time and $$$. But the fact that your releases are obviously getting more difficult to perform and have a high overhead (going from monthly to quarterly) may indicate that trouble is brewing.
And you are correct, changing culture takes a lot of effort
Good luck
Agree with you Lisa, CI as a practice seems quite prevalent now and yes there are other bottlenecks that CI has now exposed, one of course is the need for extending those ideas prevalent in engineering into Operations, and the whole DevOps movement is focused around that.
The other ofcourse is the issue of building maintainable test suites, because CI is only as good as the feedback it provides, and if the test suite that is run as part of the CI process is not synchronized to test the current state of the application than the feedback is not as useful is it:) Even in the Continuous Delivery world having a malleable test suite that can evolve at the same pace as the application does is critical to validate that you are building the right thing.
How timely, I’m having a current debate lately with the Dev team that changing their process to embrace CI. I think they are so concerned about not breaking the build that they see CI as a no-win process for them.
CI practices are difficult and cumbersome to implement first and foremost because of the available tools. Yes you need to orient your team to a test culture mindset and you do need to change some day-to-day practices, but the thing is: it is an order of magnitude harder to bring your team to good CI practices when every single day you are fighting your tools. All those who had to configure Hudson (Jenkins) on their specific environment and needs, all those who had to configure CruiseControl, have to know what I mean.
Messing around with Ant XML scripting, setting up dependencies, making sure everything’s tuned up. I wouldn’t know a percentage for sure, but I’d say very often people just finally use minimum CI to automate tests. Because it’s just too much hassle to handle a CI server. There’s just not much in return.
I think CI doesn’t need to be this hard. I think all projects can take serious advantage of CI practices, not just those that are big and complex. Small to medium projects do benefit from being centraly monitored, from having proper quality metrics, trends – all valuable insight to just about any project.
I started working on an open source project a while ago, and that project has grown into Cintient. It is a PHP based, dead-simple to setup and use CI server. I can tell you that, for instance, you don’t need to mess around with Ant XML. You just configure your tasks visually, by clicking and dragging stuff and filling out small configuration forms. My hope is that it brings some change into Continuous Integration and finally make people stop talking about when is it worth the while to have CI or after when is CI more advantageous than not.
If you want to check it out, it’s up on GitHub: https://github.com/matamouros/cintient
Cheers.
Pedro.
PS: I’ve been wanting to extend this line of thought, I think I should write an article about this sometime… 🙂
Pingback: Is the Cost of Continuous Integration Worth the Value on Your Program?, Part 2 | The Agile Radar
The question in the title of your post is very good, Johanna. My answer to it is yes. Three years ago we started the journey with my team when I joined my company. What we had at the time was sub-optimal. Since then we have shipped 100 releases per year. Releasing has become a non-event. Three years ago it took weeks if not months to get a release out the door. Today we could release at least once a day and if needed probably even more often. Continuous Integration (CI) is a key element of this. We got our CI server license three months after I started and we have continuously built on it. For over a year I have now had a full time build master. The team is now doing things I didn’t even dream of. For example we have automated merges (the team proved me wrong!). We also have a process in place that automatically removes select pieces of code then builds and tests the system. It’s a brute-force approach but it works and is fully automated. It helps us to remove code we no longer need (of course we leave the tests in!). So, yes, CI is definitely helping us and we are now on our journey towards continuous deployment (CD).
Pingback: Is the Cost of Continuous Integration Worth the Value on Your Program?, Part 3 | The Agile Radar
Pingback: Agile Offshoring in Practice - Agile Project Manager
Pingback: Corporate Inertia - No Special Needs for IT | Greger Wikstrand