Here is the final part of my notes on software engineering quality, continuous delivery and dev-qa-ops practices gathered during devoxxfr, after branching buid-promotion, and tests and code analysis, this last entry will be about continuous delivery, operations.
Most of us already do Continuous Integration by now. Through CI tools like jenkins we automated
The next level brings us to continous Delivery: providing our product to our (internal and external) users that can then directly use it on their own systems. This means automating artifacts (the result of the builds) assembling and providing a API to fetch these artifacts (through artifact repositories such as nexus or artifactory)
The final step is to bring the products running on systems, it’s continuous deployement.
there you have 3 main requesters:
Ops Teams (internal or customers)
Developers usually need to test their latest trunk code, the deployement decision should be left to them and the server should be their and operated by them.
QA need to test internal releases, just like for the dev team, the deployment decision should be theirs and operated on their QA servers. That’s were build promotion techniques becomes very handy.
Ops need to install or update product from build promoted to release by the above QA.
They need to own the deployement decision together with the final customers
They also need to inform users about planned operations, to follow their advertized SLA.
They obviously operate the servers.
The challenge here is to make these 3 teams collaborate and speak the same language, the keys are to make:
the products precisely identified and securely fetchtable via APIs from a common referential
the operations on (fetching/installing/updating) the product automated. These ops automation should be used and common to all 3 teams.
One way to achieve that is for all 3 teams :
to leverage an artifact repository such as nexus or artifactory
to leverage and co-own/commit/review a common set of operations scripts on all (dev/QA/prod) servers
Shipping native packaging is something that was advocated and demoed this year and last year at devoxx by the ``devops mercenaries'' team. It looks very appealing to me. Cf.
If it’s not in the code repository, it does not exist
Common tooling and languages facilitates communications.
We all agree that each artifact should be versioned sufficiently that we can exactly reproduce the source code it was built from from our source control system. We should treat product deployement and configuration the same.
Tech stack such as Chef or Puppet and MCollective can help you achieve that.
Romain gave a very fun and informative session about it, I encourage you to watch the replay on parleys when it becomes available
Once you have your native packaging and/or deployement script ready:
Use jenkins as an operation console
Use build promotion techniques and dashboard to make everything visible and readable by all
Handling secrets/passwords/keys in a fully automated deployement is tricky.
It is a hard piece to streamline and automate safely.
Here is an open source solution that could of interest: https://github.com/hbs/oss to achieve the above.
It seems pretty solid, it’s in production in a major bank in France.
With this open secret server, you could `safely' commit your encrypted secrets on github
Maven support artifact repositories password encryption. Maven can retrieve the Master password from the ~/.m2/settings-security.xml file and use this master password to decrypt the password stored in your ~/.m2/settings.xml file. Maven will then send the decrypted password to the server.
For an extra level of security, you can store the encrypted master password on a removable storage device like a USB hard drive. To support this, your ~/.m2/settings-security.xml file would contain a reference to the location of the settings-security.xml file using the relocation element.
Artifactory goes one step further it provides a unique solution to this problem by generating encrypted passwords for users based on secret keys stored in Artifactory itself. By this, you can ensure users shared passwords are never stored or transmitted as clear text.
One easy step to also ``secure'' the consistency of your artifacts repository is to prevent users from deleting or overwritting releases. cf. http://wiki.jfrog.org/confluence/display/RTF/Managing+Permissions
Jenkins build can fail for many other reasons than bad code checkins.
(JVM crash, scm/nexus/artifactory/etc.. connectivity issues, browser crash during selenium tests, jenkins/plugin crash, etc..) Vincent presented us a nice tip to prevent that using groovy script to prevent spamming developers with false alarm emails
Ensure the server logs are made available to all
Logs are useful information to all 3 groups, dev QA and ops.
It should be accessible to all and a focus should be made to make sure the dev teams is producing useful and triggerable logs.
Think production monitoring, think further: think jmx, jmxtrans, nagios, graphite:
Create smart dashboards and alerts
If it’s not monitored i’ts not in production
This other famous quote below can applied to code quality but also to prod monitoring:
If you cannot measure it, you can not improve it
Won’t you feel safer doing feature toggling (cf. the first part of my notes once you have a proper monitoring in place ? Won’t it be easier to manage:
beta test (feature flippers)
A/B testing (feature flags)
Devops also requires organizational changes: here are a few ideas picked here and there during devoxx
cooperate from the start of the project
eliminate the silos
Hire a `Software factory/devops' specialist to :
work and support the dev team
work and support the QA team
work and support the ops team
His job ensure that the continous integration, delivery and deployement is :
always up and running
continously improved: every manual step in operating the aps should hunted and killed
ensure that areas of ownership and responsabilities are clearly defined between the dev qa qand ops teams
ensure all impediments and roadblockers towards efficient continuous delivery are taking care of
Implement a Bug fixing day: for instance in Vincent project, Xwiki, Every Thursday is bug fixing day: the Goal is to close the max number of bugs.
Slowly add new quality check over time
Everyone must be on board
Favor Active Quality (i.e. make the build fail) over Passive checks
Be ready to adapt/remove checks if found not useful enough
Quality brings some risks:
Potentially less committers for your project (especially open source)
project seen as ``less fun''
facebook releases: https://vimeo.com/56362484