quarta-feira, 27 de março de 2013

Feature toggles: good or bad?



Everyone who have read at least once about continuous integration or continuous delivery knows that it is not possible or,  at least it is very difficult, to do if you use more than one development branches. If you have never ever faced any problems using two or more development branches you are a fucking lucky guy or, probably, you are not used to refactor your code as you should be (in these rare cases, I strongly recommend you to take a look in this Martin Fowler & Mike Mason interview and Jez Humber Continuous Delivery book). On the other hand, if you have found yourself in an endless merging task or if you have been feeling insecure about some merging results. Do not panic! Keep reading. You are not alone. :-)


As you may already know, there is no problem with branches. The problems are all about merges. So, to overcome big and error prone merges the best strategy I would recommend is Branch by Abstraction. According to Martin Fowler, Branch by Abstraction is “a term coined by Paul Hammant to describe a technique for making structural changes to a code-base without a Feature Branch in a source-code control system”. To make that real, we are used to implement new features, or updating existing ones, using a very simple technique known as Feature Toggles. In short, a feature toggle is a way you develop your software in order to make it easy to enable or disable features through configuration changes.


This kind of strategy is widely used by companies such as Facebook, Flicker, Google, Amazon, and, of course, the ones I’ve been consulting. A very simple example of feature toggles can be seeing in the Chrome web browser by typing chrome://flags in the URL input. In this page you can enable or disable features of your browse and help Google to evolve it. ;-)



After some years of experience trying to improve clients’ delivery processes, we have found some advantages and disadvantages of using toggles instead of version control system branches. So, the main idea of this post is to share and discuss our experience with the community.


The advantages we have found are:


  • Ability to turn features On and Off at any moment
    • If you have found yourself adding a new bug in production, for example, you could just turn your feature Off.
  • Move forward, never backward
    • Once you can disable incomplete or buggy features, you can move on and fix it instead of rolling back your entire app (which in some cases is extremely painful).
  • Ability to put unfinished code in production
    • At the first glance it may sound a bit weird, but it helps you to reduce the amount of stuff you put in production at once. Small batches reduce the risk of going live (see Reducing IT risk using continuous delivery )
  • Reduce or eliminate the necessity to create branches in a source-code control repository
    • Your integration team will be very thankful, believe me ;-). This means less or no merges any more. Given them more time to making smarter tasks than merging.
  • Ability to make experiments that could not be done outside of production environment
    • For example, those intermittent issues which are impossible to reproduce outside of production.
  • Ability to enable a feature for only one group of users
    • For example, you can enable a feature for a specific region, role, user group, or any other rule you need.
  • Ability to enable and disable the features while the app is up and running
    • But you may set toggles at build time


The disadvantages are:


  • There are cases in which it is not possible or, at least,  it is very difficult to implement a feature toggle
    • Examples in the Java world
      • Static configuration files (i.e. web.xml, ejb-jar.xml e persistence.xml),
      • JPA mapping,
      • etc.
  • Every toggle requires an engineering process
    • First, avoid doing toggles for everything.
    • Second, you must be sure you need it.
    • Third, if you realize it is extremely necessary, it is important to provide a good design for its implementation.
  • Toggle On means: a pretty new technical debt
    • Once you enable your toggle and it is working 100% in production you have a technical debt that should be removed as soon as possible
    • If you do not remove the technical debt your software will be harder to improve, maintain, test and deploy. This happens because your code must care about all possible situations (i.e. toggle On and Off)
  • Features loosing priorities
    • Sometimes people forget to enable a feature toggle just because that feature loses its priority. It is very ease to happen, unfortunately. At least in my experience.
    • Sometimes people find an issue in production and then disable the toggle, but never fix it. In this case, the toggle remains there disabled forever.
  • Feature toggle could easily get out of control
    • Sometimes people lose the control of what is enable in which environment
    • The number of toggles will grow faster than you think. Take care.


Finally, the discussion about this subject

Any technique has its advantages and disadvantages. However, if you have noticed, most of disadvantages are a reflect of not given the right control feature toggles should have. With a good toggle management, frequent releases and focus (do one thing at a time) it is possible to solve most of these issues.


To help overcome these difficulties, we have been fulfilling the following information for every feature toggle:

  1. The Value
    1. Usually enable/disable, but can be a complex data structure informing the region it is enabled, the user grup, or etc
  2. A detailed description
    1. Detailed information about what this feature toggle does and why it is necessary
  3. The deadline (very important)
    1. When this toggle should die and how you should perform the clean up
  4. The responsible (very important)
    1. The engineer/team responsible to monitor and remove the toggle. Usually, this responsibility is given to the one which have chosen to add it.


Besides that we have developed a dashboard, which is revised periodically, which provides all toggles information in order to get the full benefits this technique could provide us. Moreover, as Martin Fowler already said “while feature toggles are a valuable tool in the box, they are a second-best option. The best thing to do with such features is to find a way to gradually release them into production as you are building them”.


If you like or not, please leave your comment. I will be very happy to listen to other teams experiences.

11 comentários:

  1. Very interesting approach to avoid branching and merging.
    Before reaching the end I was thinking on options that could help us could make responsible decisions on toggling on and off and the "Deadline" is a can't miss.
    As with every release delivery, I believe these toggles should have a limit of 2 or 3 releases to avoid an infinite number of toggles increasing the technical debt.
    Great post!!

    ResponderExcluir
    Respostas
    1. Hi Jorge,
      After reading your comment I think we should be adding the release number (or the creation date) in the toggle config. This would help us to track how old each toggle is.
      Thanks for the comment.

      Excluir
  2. Branches belong to your source code. You should use separate branches to develop features in isolation, while keeping at least one stable line of code. Feature branches are then integrated into these stable lines as the source code reaches relative stability. You can use continuous integration and multiple branches to makes different features available to different audiences, like end users and testers. Version control branches are a tool in the software engineer toolbox.

    Feature toggles can be used to progressively make a feature available while it’s being developed, but in a completely different sense. Develop a feature here doesn’t mean write source code, but to explore and improve the way users interact with the feature, to test its business value. By making the feature selectively available, you’re able to minimize the impact to end users in case its business value was misunderstood, and to better understand how different audiences can make use of it. Feature toggles are a tool in the product manager toolbox.

    Both options have their place in the overall product developer cycles, but they solve different problems and one is not supposed to take the other’s place.

    ResponderExcluir
    Respostas
    1. Hi Marcus,

      First of all, thank you very much for your comment. It is always good to listing to different opinions.

      Second, I agree with you when you say that branches are a good and necessary "tool". Actually, we use branches for POCs, releases, production support, and, in some cases, for developing new features. As I told in this article, "there is nothing wrong with branches, the problem is all about merges". Whenever you have many teams working in different development branches, merging would be a pain in the ass. Try make a refactoring in one line while you have 6 different teams working in different branches you will understand what I'm talking about. We have faced this kind of issue (during years) in some clients and can ensure you It does not worked very well. Going to production in those cases are very complex and risky.

      Third, I disagree with you when you say Feature Toggles and Feature Branches solve completely different problems. Actually, we have being using toggles in order to avoid branches during years and it have been working pretty well for us. Besides that, I agree with you when you say feature toggle is a tool in the manager toolbox. However, I consider it another plus over feature branches ;-).

      Excluir
    2. I've had a quite positive experience with Feature Toggles (I used to call it AllowedFeature, but Martin Fowler is better than me at naming stuff).

      They certainly decrease the number of branches you have to create. I also used slight variations to do access control and backend/frontend compatibility.

      I even used Feature Toggles with Feature Branches: for very complicated features that would take one week or more to stabilize I'd create a branch, use Feature Toggle in the Feature Branch, and merge it to the trunk when the feature was stable.

      Remark: In my case stable did not mean production-ready, hence the Feature Toggle.

      I don't necessarily think Feature Toggles are better than Feature Branches. It depends on how much complexity each one of them will add to your solution.

      Sometimes, disciplined Feature Branches with daily merges from the trunk would do it.

      Another issue is that over time Feature Toggles add a lot of complexity to the code. Specially when new features have to touch legacy code.

      Take care,
      Rafael

      Excluir
    3. "I don't necessarily think Feature Toggles are better than Feature Branches. [..] Sometimes, disciplined Feature Branches with daily merges from the trunk would do it "

      Couldn't agree more. :-). As I've said, we use feature branches for some complex development and these cases we are used to merge many times a day. This strategy makes merges much more simpler.

      Actually, we are working in some experiments using an automated merging strategy in a big project. If it works as expected I will be posting our solution. ;-)

      Excluir
  3. Very nice post, Fernando! I'll give my "2 cents" opinion as I don't have any experience on feature toggles.

    First, indeed feature toggles seem to be more manageable than enabling/disabling feature by branching and merging. However, by the time I was reading the post I was wandering whether the software product line (SPL) approach would be feasible instead of feature toggles. SPL puts the notion of features in the core of the development process. And once you have your process aware of "engineering features" you can generate different products by configuring (selecting/unselecting) features (according to your feature model). A feature model tells you which are the mandatory features, optional and alternative features, feature dependencies and so on. And this have to be carefully defined as an engineering discipline.

    Again, I don't have experience on feature toggles. But I think that in some cases there might be room to discuss about SPL adoption. SPL approach is another field which has been gaining the attention of great companies in recent years. Also there are recent studies about SPL and agile development.

    My impression is: do feature toggles as second best option (as you wrote) due to the disadvantages you listed. And as you wrote 'most of disadvantages are a reflect of not given the right control feature toggles should have' -- So, knowing that but also feeling necessary the use (and increasing use) of feature toggles, the approach may not scale in contrast of an increasing need for managing it. A more engineered discipline through feature-oriented software development may be a better way (maybe the case of SPL adoption).

    ResponderExcluir
    Respostas
    1. Hi Bruno,

      I'm not familiar with SPL and I would like to discuss more about it with you outside of this post. But the idea seems be very similar. The difference seems to be that in SPL approach you do not have to remove your toggles, right?

      Feature toggles instead of feature branches are mainly used to avoid merging. As a plus you have the ability to turn the feature off if something does not worked as you have planned.

      Excluir
    2. Merging development branches is much more complex than it sounds to be. Mainly when you have huge teams. The bigger the time you have between merges, the bigger is your problem.

      That is why companies such as Google, which refactor 50% of its code every month, do every thing in only ONE branch (see http://www.infoq.com/presentations/Development-at-Google).

      Excluir
  4. Hi,

    I really loved reading this post. I sounds that we share the same pain. :)
    I'm working in a company that has a main product but also has customer features.
    The current solution is by using branches, as you probably know, it's a merge hell !!

    My dream, is to bring us back to working on the trunk.
    I read the books, articles, etc... and I'm still not sure what is the right way to do it.
    Some guys advised me to keep working on branches:
    the core of the code will be on the trunk and the code that is feature like will be on branches.
    I prefer to go all the way to one trunk solution, but I'm not sure it's the best or even possilbe to a company that has a main product but is a customer oriented company.

    Please advise.



    ResponderExcluir
    Respostas
    1. Hi Avner,

      First of all, thank you for reading.

      Second, it sounds a very common issue whenever a company decided to NOT have a "closed box product". Imagine SAP working like you are saying. Imagine how many branches they would have? :-)

      My suggestion is to have only one product (for all client). This product must have a flexible Product Configuration, where you can flag features on/off according your needs. Moreover, you will need to put hooks around screens/rules/features/etc. So that you will be able to customize stuff for a given client. Every customization could be faced as an extension module. And each client has its own extensions. Whenever you would like to reuse a client extension in another client you could elect it to the main product.

      This way you will not need to use branches. Or, at least, you will reduce the number of branches. The difficulty of all of this is that it is EXTREMELY HARD to do this in an running software. The ideal scenario would be start from the begging thinking like this.

      Good look,
      Fernando Rubbo

      Excluir