Back from Devoxx - Part 4 - Dev-QA-Ops Thoughts - About continuous delivery and deployment

First published on April 26 2013 and tagged Java . Devoxx .

From Continuous integration to Continuous Delivery and Continuous Deployement

Most of us already do Continuous Integration by now. Through CI tools like jenkins we automated

  • builds

  • code analysis

  • dependency analysis

  • unit test

  • functional test

  • perf test

The next level brings us to continous Delivery: providing our product to our (internal and external) users that can then directly use it on their own systems. This means automating artifacts (the result of the builds) assembling and providing a API to fetch these artifacts (through artifact repositories such as nexus or artifactory)

The final step is to bring the products running on systems, it’s continuous deployement.
there you have 3 main requesters:

  • Dev Teams

  • QA Teams

  • Ops Teams (internal or customers)

Developers usually need to test their latest trunk code, the deployement decision should be left to them and the server should be their and operated by them.

QA need to test internal releases, just like for the dev team, the deployment decision should be theirs and operated on their QA servers. That’s were build promotion techniquesimage becomes very handy.

Ops need to install or update product from build promoted to release by the above QA.
They need to own the deployement decision together with the final customers
They also need to inform users about planned operations, to follow their advertized SLA.
They obviously operate the servers.

The challenge here is to make these 3 teams collaborate and speak the same language, the keys are to make:

  • the products precisely identified and securely fetchtable via APIs from a common referential

  • the operations on (fetching/installing/updating) the product automated. These ops automation should be used and common to all 3 teams.

One way to achieve that is for all 3 teams :

  • to leverage an artifact repository such as nexus or artifactory

  • to leverage and co-own/commit/review a common set of operations scripts on all (dev/QA/prod) servers

Towards continuous delivery and deployement

Ship native packaging

Shipping native packaging is something that was advocated and demoed this year and last year at devoxx by the ``devops mercenaries'' team. It looks very appealing to me. Cf.

Code the operations

If it’s not in the code repository, it does not exist

Common tooling and languages facilitates communications.

We all agree that each artifact should be versioned sufficiently that we can exactly reproduce the source code it was built from from our source control system. We should treat product deployement and configuration the same.

Tech stack such as Chef or Puppet and MCollective can help you achieve that.
Romainimage gave a very fun and informative session about it, I encourage you to watch the replay on parleysimage when it becomes available

Once you have your native packaging and/or deployement script ready:

  • Use jenkins as an operation console

  • Use build promotionimage techniques and dashboard to make everything visible and readable by all

Manage your secrets

Handling secrets/passwords/keys in a fully automated deployement is tricky.
It is a hard piece to streamline and automate safely.

Here is an open source solution that could of interest: https://github.com/hbs/ossimage to achieve the above.
It seems pretty solid, it’s in production in a major bank in France.
With this open secret server, you could `safely' commit your encrypted secrets on github image

Secure your artifact repositories

Maven support artifact repositories password encryption. Maven can retrieve the Master password from the ~/.m2/settings-security.xml file and use this master password to decrypt the password stored in your ~/.m2/settings.xml file. Maven will then send the decrypted password to the server.

For an extra level of security, you can store the encrypted master password on a removable storage device like a USB hard drive. To support this, your ~/.m2/settings-security.xml file would contain a reference to the location of the settings-security.xml file using the relocation element.

cf.

Artifactory goes one step further it provides a unique solution to this problem by generating encrypted passwords for users based on secret keys stored in Artifactory itself. By this, you can ensure users shared passwords are never stored or transmitted as clear text.
cf. http://wiki.jfrog.org/confluence/display/RTF/Centrally+Secure+Passwordsimage

One easy step to also ``secure'' the consistency of your artifacts repository is to prevent users from deleting or overwritting releases. cf. http://wiki.jfrog.org/confluence/display/RTF/Managing+Permissionsimage

Solve Jenkins false alarms

Jenkins build can fail for many other reasons than bad code checkins.
(JVM crash, scm/nexus/artifactory/etc.. connectivity issues, browser crash during selenium tests, jenkins/plugin crash, etc..) Vincent presented us a nice tip to prevent that using groovy script to prevent spamming developers with false alarm emails
cf. http://massol.myxwiki.org/xwiki/bin/view/Blog/JenkinsDontSendEmailForFalsePositivesimage
cf. https://wiki.jenkins-ci.org/display/JENKINS/Scriptler+Pluginimage

Implement proper monitoring

Ensure the server logs are made available to all
Logs are useful information to all 3 groups, dev QA and ops.
It should be accessible to all and a focus should be made to make sure the dev teams is producing useful and triggerable logs.

If it’s not monitored i’ts not in production

This other famous quote below can applied to code quality but also to prod monitoring:

If you cannot measure it, you can not improve it

Won’t you feel safer doing feature toggling (cf. the first part of my notesimage once you have a proper monitoring in place ? Won’t it be easier to manage:

  • beta test (feature flippers)

  • A/B testing (feature flags)

rethink your workflow and processes

Devops also requires organizational changes: here are a few ideas picked here and there during devoxx

  • cooperate from the start of the project

  • eliminate the silos

Hire a `Software factory/devops' specialist to :

  • work and support the dev team

  • work and support the QA team

  • work and support the ops team

His job ensure that the continous integration, delivery and deployement is :

  • always up and running

  • continously improved: every manual step in operating the aps should hunted and killed

  • ensure that areas of ownership and responsabilities are clearly defined between the dev qa qand ops teams

  • ensure all impediments and roadblockers towards efficient continuous delivery are taking care of

Implement a Bug fixing day: for instance in Vincent project, Xwiki, Every Thursday is bug fixing day: the Goal is to close the max number of bugs.

Slowly add new quality check over time

  • Everyone must be on board

  • Favor Active Quality (i.e. make the build fail) over Passive checks

  • Be ready to adapt/remove checks if found not useful enough

  • Quality brings some risks:

    • Potentially less committers for your project (especially open source)

    • project seen as ``less fun''

Further readings and other various pointers