Back from Devoxx - Part 1 - Dev-QA-Ops Thoughts - About Branching

First published on April 19 2013 and tagged Tech . Java . Devoxx .

I spent 3 full days at devoxxfrimage late March. 3 full days really well spent.
The conference was fantastic, full of informative and entertaining presentations, live coding sessions and labs; and at every break, good old friends or friendly new faces for even more passion and sharing.

A big bravo and a big thanks to Antonio, Nicolas, Zouheir, Arnaud and all the devoxx team for such a great event. I felt the same passion and energy I felt back in 2005 when I first attended Javapolis in Antwerpen.

This year I particularly focused my attention on all things related to quality, devops and continous delivery.

You’ll find here a bunch of notes-to-self and various pointers on that will help me and my team at Adobe to do a better at improving our software engineering quality, continuous delivery and dev-qa-ops practices.

About branching strategy and feature toggling

About branching strategy and feature toggling

Feature branching

When building enterprise software we all spend quite a bit of time building/working/working-around code branching strategies and facing the consequences of choosing one strategy over another.

These strategies derives from the organization’s software development processes and are shaped by the answers to questions about how a company releases its software,

Git/dvcs eased the pain of feature branching and merging back into the team mainline (taken to production)
Obvious to say than doing local development is a feature branch in itself, impossible to avoid that. However, using branches for features or large-scale changes is seen by many as a bad ideaimage for several reasons, of which the most important are that it prevents both continuous delivery and refactoring

Feature Branching is a poor man’s modular architecture, instead of building systems with the ability to easy swap in and out features at runtime/deploytime they couple themselves to the source control providing this mechanism through manual merging.
– Dan Bodart

Feature Toggling

The alternative (mentioned quite a bit at devoxx this year) is Feature togglingimage

Feature flags and flippers mean we don’t have to do merges, and that all code (no matter how far it is from being released) is integrated as soon as it is committed. Deploys become smaller and more frequent; this leads to bugs that are easier to fix, (…) the amount of changed code is minimized.

This style of development isn’t all rainbows and sunshine. (…) after launching a feature, we have to go back in the code base and remove the old version (maintaining separate versions of all features on Flickr would be a nightmare).
– Flickrimage

Feature toggling also facilitates canary releasing, A/B testing, and makes it easier to roll-back should a new feature misbehave in production.

Cyrille (yet another devoxx speaker and cxf committer) summarized the many advantages of feature toggling in one of his former talk on monitoring and feature toggle pattern with jmximage :

  • dissociate deployment and feature activation

  • progressive activation of a feature i.e. canary testing

  • measure impact of a new version i.e. A/B testing

  • differ feature activation on production

  • trunk based development

Cyrille also hinted at the different coding and cleaning patterns:

  • smart reuse and mutualization

  • or brutal duplication allowing a cleaner and eased old version removal

I did a bit of googling to find feature toggling frameworks and techniques:

Feature toggling and trunk based development applied

  • Sucessfully applied, this techniques can allow several prod release per day, Facebook, stackoverflow and Flickr were mentioned for this by Axel in his architecting for continuous delivery talkimage

  • extract from a break chats:

    • ``here at Banque X we only maintain 2 branches, a trunk where all the bugs are fixed first and a production support branch''

    • ``here at insurance broker Y, we release a few times a week, we only have trunk, and we don’t bother with maven release, we release snapshots (but everything is traceable and repoductible), and we love it''

Not convinced ?

Feature Toggling is not a silver bullet and proper cleaning and technical debt management seems even more important in that context.

One might call this feature toggling technique: merge paranoia cf.

You can’t do decentralized versioning unless you also decentralize your testing and integration. Git has value when used as a SVN replacement. Git has more value when used as a DVCS. There is no good reason why you can’t do decentralized testing and integration with git. Rather the opposite: it has been designed with exactly this in mind. The whole point of git is divide and conquer. Break the changes up: decentralize the testing and integration work and solve the vast majority of problems before change is pushed upstream. If you push your problems along with your changes you are doing it wrong. Decentralized integration is a very clever strategy that is based on the notion that the effort involved with testing and integration scales exponentially rather than linearly with the amount of change.

Let’s make the build unbreakable

I’d agree with the above statement. With Git anyway, de facto, we do have a feature branch specific to each developer.

Using Git and prehooks techniques you can make the main line build unbreakable cf.

Note that if you are stuck on Perforce: you can use the shelving feature. Have the developers create shelved changelists and get Jenkins to unshelve the changes to build and run the tests. If the tests pass, the changelist can be submitted to the central depot.