A Practical Guide to Extending Continuous Integration to Continuous Delivery

Introduction

I am assuming that because you are reading this blog you are familiar with the concept of Continuous Delivery and are already a supporter of the concept. If not, close your eyes for a moment and imagine a software system that has a bug (I know this is a bit hard to imagine). Now imagine a developer fixing that bug, completing his unit testing and checking the code back into source control. Now count to 10 and the code has been deployed to production and the bug has been fixed. This is the Nirvana and goal of Continuous Delivery. I have found that from a technology stand-point this is now a relatively easy thing to achieve for applications with small to medium levels of complexity written in ‘friendly’ technologies. However, expanding this concept to complex applications in enterprise environment becomes more difficult but in my opinion still achievable.

So why do so few people seem to be doing it?

For the past few years I have been trying to answer this question and apply Continuous Delivery concepts to complex systems in enterprise organisations. Some of the challenges faced include:

• Unfriendly technologies – this includes custom vendor programming languages and tools that do not immediately lend themselves to automated builds and deployments (Datastage and Webmethods are examples that spring to mind in this category).
• Multiple External Interfaces – Systems that interact and rely on multiple external interfaces for data and processing.
• Data – Systems that are heavily reliant on large volumes and synchronisation of data.
• Old technologies – Systems where there are legacy components (this is almost always the case).
• Team structures + Politics – One of the most challenging and slowest areas to combat and which can sink many good technology initiatives if not navigated correctly.
• Budgets – The ever present reality of money. It is important to understand that the goal of Continuous Delivery is to provide efficiencies and save money. This must always be kept in mind and any efforts assessed against this goal i.e. is there value add.
• Competing projects/priorities – The ever present reality of delivery. Continuous Delivery will assist with this and needs to be applied so that it is not seen as something delaying things.

I believe these and many more are just challenges and can be overcome. I am confident that one day this style of delivery will become the ‘norm’ in software development, potentially not fully automated all the way to production but at least to a test environment. What I aim to provide in this blog is a practical guide on how to make the next step from Continuous Integration to Continuous Delivery. It is a process I have used successfully and that I hope you will find useful. This guide assumes you have an established Continuous Integration process for your development practices. If not have a quick browse of http://jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html.

Continuous Delivery

Most development teams now appear to have established some level of Continuous Integration but are either unwilling or are have trouble moving to Continuous Delivery. One of the key underlying reasons why this transition is difficult is because Continuous Delivery covers three main practices;  Continuous Integration, Automated Deployment and Automated Testing. These practices generally span multiple teams across the software development life-cycle. Organising people can be difficult, organising teams in an enterprise environment is even harder.

If you look at Continuous Delivery from an engineering perspective there are 4 main components (1 covering each of the practices and 1 for choreography and reporting):

• Continuous Build/Integration –automated building of your application on check-in and execution of unit tests.
• Automated Deployment –automated deployment of your build artifacts and environment config.
• Automated Testing – automated acceptance/system testing of your application.
• Pipeline Choreography + Reporting – something to tie the above three areas together and to provide status reporting.

Continuous Delivery – Components

To implement Continuous Delivery you will need to have some sort of technical solutions and tools in each of these areas.

More on each of these components later, for now lets go through the set of steps that can hopefully be used to help with the transition from Continuous Integration to Continuous Delivery.

Step 1 – Setup another development environment

If another development environment is not practical you can get away with nightly builds in a shared development environment but I strongly encourage you look at establishing a separate environment. Suitable names I have seen in the past for this environment include the assembly, dev staging or dev test environment.

This additional environment provides you with a location to:

• test your continuous delivery process;
• test your automated deployment process; and
• run your automated integration and acceptance/system tests.

Ownership of this environment should initially be limited to a single team. It is important to contain the Continuous Delivery initiative to one team for the moment to avoid being dragged into any potential politics at this stage. It is important to note that this environment does not need to be, nor should be a full scale production like environment. In fact there are some key things about this environment that will need to be established. This may involve significant effort for some applications but the key is to start small providing as much value as possible early on and then to incrementally build.

More on this in step 4, for the moment I will simple list the key elements about this environment:

• External interfaces should be stubbed – You do not want to be reliant on external systems for your tests. This causes unreliable results and the environment and tests will become a pain to maintain.
• Cut-down static data – You need to control your data scenarios. Having random data will result in maintenance issues for your tests. Similarly you want a small data set to ensure efficient, quick turn-around times.

For now don’t worry about implementing a stub for every external interface and generating static data for every data scenario, just get an environment up and running which is fairly usable i.e. you can execute at least one use case on it. Remember, we want to add value quickly and incrementally.

Step 2 – Automate the deployment to this environment

The ease with which you be able to manage this step really depends on your maturity in the automated deployment space. However, if you don’t have an automated deployment framework don’t fear. For starters this can be a set of Bash, Ksh, Ant, DOS scripts <insert favourite> and some SSH keys to communicate between servers. Moving forward you can look at tools specifically designed for this purpose such as such as Control Tier, Chef, Run Deck, <insert favourite>… I feel it is important to separate your build and deployment processes. Some items you will need to consider during this step are:

• Environment Variable – How does your application handle application properties vs environment properties.
• Build Artifacts – where do you store your build artifacts

Jump to the step by step example of achieving this with Nexus, Tomcat, SSH and some simple shell scripts.

Step 3 – Integrate your Continuous Integration with your automated deployment

This step starts to tie together the underlying disciplines to form a Continuous Delivery framework. The aim here is extract the artifact built from your Continuous Integration and automatically deploy this into your new development environment after each code check in (with no manual steps required!). At the very basic level this can be achieved with a simple script with ties together the two components. I do however recommend that you look at a tool at this point to assist with the coordination and reporting. Using a tool will normally provide you with a nice interface and a history of build activity as well as a variety of notification mechanisms (email etc.) and security overlay. Make sure your tool is capable of filling your requirements in steps 4+5. I am most familiar and partial to the Atlassian suite so have provided the step-by-step demo using Bamboo but have a look around at what else there is available.

Jump to the step by step of how to achieve this step with Bamboo.

Step 4 – Automate your acceptance tests

First let me begin with a brief definition of the categories of testing:

• Unit: jUnit, testNG, easyB, that is what TDD and BDD is all about.
• Component: that’s where you want to use a spy/mocking framework: mockito, easyMock, rMock, jMock to test components in isolation, validating business rules.
• Integration: testing that components interact together correctly, using your favourite IoC (Spring, CDI, seam, etc.) and mocking framework, SoapUI can also be used, we are more interested in data and logic flows rather than business rules.
• System/Acceptance: the ultimate integration test, making sure all components are working together, using the GUI so you need something like selenium for browser based testing or QTP for native apps.

A useful diagram worth considering is the “test automation pyramid” taken from “The Clean Coder: A Code of Conduct for Professional Programmers” (R. Martin 2011) which gives an indication of what level of code coverage we should be targeting for each test category.

During this step we are not talking about the automation of unit tests. We are looking at some level of testing between component and system/acceptance testing depending on your application. My recommendation is to start with one simple system/acceptance test. Start with the obvious ones – i.e. 80% of usage. At this point you will need to start building out any mechanisms for controlling the data (if required) and stubbing out external interfaces. I won’t go into the details here of the variety methods that can be used to achieve this but I will mention this. Most developers when I talk to them about static data and stubbed interfaces immediately baulk at the effort that would be required to implement such a feat of programming wizardry. If you are thinking like this now I need to highlight at this point it is not as hard as you think it may be. I ask that you set aside a couple of hours, have a quick browse online about the various approaches and then sit down and figure out how you could achieve it for your application and your one test case that you are trying to write. Don’t try to boil the ocean here. Add value incrementally and ingrain the practices in your team and you will be amazed at the momentum that it creates once the framework is established.

The tools you use to test will depend on what type of testing you are planning on executing for your application, but for the purposes of this blog I will assume it is a web service under test and will use SoapUI as the tool of choice. One of the reason for choosing SoapUI as the tool is that it produces a nice JUnit style test output which integrates nicely with Bamboo in Step 5. Again I need to highlight that there are many tools out there that may be better suited to your needs so have a look around. I will also point out that everything is testable in an automated fashion, it really is just a question of difficulty (aka cost) versus value. Always look for the most cost effective manner to execute your tests. As a general rule this is normally at the unit test level. In some application it may not be possible to adequately perform units tests and so you may need to rely more heavily on integration/system tests (especially true when you are dealing with what I would refer to as difficult technologies). As an example, consider a system which delivers data via a webservice to a desktop application which then renders this data to PDF for users. To test at the PDF layer would be difficult and costly. Take a moment to consider where the majority of risk is, surely it is the data? In this example it then makes sense to push more of your testing down to the Web Service layer as it easy to implement and resolves the risk i.e. it adds value quickly and efficiently.

Always keep this in mind throughout the process. Some questions to ask yourself:

• Is this the best point to write my test?
• Can I test to the same outcome in a more efficient manner?
• Are there any alternative testing points that can be introduced to the system?

Jump to step by step of how to achieve this with SoapUI

Step 5 – Integrate your automated test into your build pipeline

This is the last step before you have a working POC for Continuous Delivery, the subsequent steps are about maturing and socialising your work. The integration of your test suite into your build pipeline is critical as it helps cement the practices into your development process. All the same reason and arguments that apply for Continuous Integration and unit tests apply here i.e. fast feedback, always having a working build etc. With your Continuous Integration process you should already have your unit test integrated into your build process. This is about extending the same concepts of Continuous Integration to your system/acceptance test suite and new development environment.

How you integrate will be dependent on the tools you are using but some key features you will want are:

• Notifications of failed tests (Email, IM, Build Lights <insert favourite>)
• History of test results and when / why they have broken

Jump to step by step of how to achieve this with Bamboo, Maven and SoapUI

Step 6 – Expand your test suite

Ok by this step you should be able to check some code in and the following should automatically happen:

• Your application is built;
• Your application is deployed to your new dev environment; and
• Your integration/acceptance test is run against your application.

You should be able to sit back smugly with your coffee and watch this all happen through your Continuous Delivery reporting tool.

Congratulations!

This step is now easy, expand out your system/acceptance test suite. Remember add value quickly and efficiently and ensure you are applying the most effective tests at the correct point in your application.

Step 7 – Go demo and talk to other teams

This can actually be one of the more difficult steps as it involves dealing with people, teams and office politics. This is why I have tried to defer this step until we have a working, solid version of Continuous Delivery which is adding real value to the business. Continuous Delivery and some of its concepts are rapidly gaining exposure but are still relatively new concepts. Many people outside of the development space will not see this concept as anything different that the 3 underlying practices; Continuous Integration, Automated Deployment and Automated Testing. Others will not see the value and see it as a ‘developers thing’. It is important to understand that Continuous Delivery spans more than just the development space and when executed correctly transitions all elements of software delivery. Now is the moment when you will need to start your road show and tackle any of these issues that arise. At this point your best tool would probably be a book on psychology, but some advice from experience is:

• Keep an Open Mind – people are resistance for a reason, try to understand the reasons behind their views. They may have been burnt by an automated testing project previously. They may be afraid they will no longer be required.
• Preach Value Add not Continuous Delivery – Always focus on providing value and return on investment in a timely manner. Avoid situations that result in a long turn-around prior to real value being added to the business. Don’t get lost or overly obsessed with terminology. It is great to talk in a common language but the important thing is to implement the process, what it is called is secondary.
• Find your Supporters – identify your supporters and leverage them. Don’t try to go it alone, identify people who are supportive of the benefits and understand the final end-state you are trying to achieve. In many cases you are changing the delivery culture. Changing a culture is difficult, acknowledge it and deal with it appropriately .
• Don’t get attached to Tools – During the journey to implement Continuous Delivery into your development practice you may need to tweak technologies and tools you have used. Do not become overly attached to the tools you have used in the above steps. Whenever you get involved in a discussion about tools just ensure you have your requirements ready i.e. integrates with Continuous Delivery reporting tool etc. and then evaluate each tool against these requirements. The process is paramount, the tools are secondary and there to assist the process.
• Work within existing structures – The question of the ownership about certain sections of the process will come up and will need to be resolved. Who writes the automated tests? Who manages the static data? No magic answers here and it depends on your organisation structure and skill sets in each of the teams. Just work through the advantages and disadvantages at each point and ensure the decisions make sense. Personally I am in favour of pushing as much as possible into the developers realm of responsibility as it makes sense that developers are responsible for the quality of work they produce. However, in larger more complex systems it may make sense to leverage other teams more.
• Work from a Basis of Proof– As an example to illustrate what I mean here consider the following. Imagine all deployments to the test environment are manually approved and only to occur over defined windows. Don’t insist that all deployments to the test environment must be automated. Use the additional development environment to perform your automated deployments and tests and stick with the manual process to test. Once confidence has been built up that the deployments to the test environment are seen to be of high quality then the manual deployment process to test can be discussed.

It is best to prove the system works and then start discussing some traditional entrenched processes. Hope you have found this blog useful, if so please drop me a note (brendan.haire@industrieit.com) and let me know.

Cheers Brendan

Step-by-Step Guides

Step-by-Step – Automate Deployments using Nexus, Tomcat and some simple Shell scripts

Note: This is a simple and brutal method to deploy a war artifact to a tomcat container. It is designed to show that automated deployment can be achieved easily. The process can be improved in many ways.

1. Extract the artifact from Nexus

OUTPUT_PATH=/myoutputpath/
NEXUS_REPO=http://mynexus
NEXUS_ID=snapshot
ARTIFACT_GROUP=industrieit.com
ARTIFACT_ID=continuous-delivery-app
ARTIFACT_VERSION=1.0-SNAPSHOT
ARTIFACT_PACKAGING=war

cd $OUTPUT_PATH;
wget $NEXUS_REPO/service/local/artifact/maven/redirect?g=$ARTIFACT_GROUP\&a=$ARTIFACT_ID\&v=$ARTIFACT_VERSION\&r=$NEXUS_ID\&p=$ARTIFACT_PACKAGING

2. Stop Tomcat

TOMCAT_HOME=/mytomcathome/

$TOMCAT_HOME/bin/shutdown.sh
while [ `ps -ef --sort=pid | grep "catalina.home=$TOMCAT_HOME" | grep -v "grep" | awk '{ print $2}'` ]; do
sleep 1;
done

3. Deploy Artifact

mv $OUTPUT_PATH/$ARTIFACT_ID*.war $TOMCAT_HOME/webapps/$ARTIFACT_ID.war

4. Start Tomcat

$TOMCAT_HOME/bin/startup.sh

This can be expanded to extract and set particular config variable if required. Again I suggest that you have a look at the available tools such as Control Tier, Chef, Run Deck, <insert favourite>… to support automated deployments before expanding too far.

Step-by-Step – Bamboo Integration of your Automated Build and your Automated Deployment

Note: Again this is designed to show a low effort way to integrate. The process can be improved in many ways.

1. Setup SSH key between bamboo agent and deploy server – If you are not sure how to do this see http://linuxproblem.org/art_9.html

2. Write a simple ssh wrapper script and check it into your source repository. This will be used to call your automated deployment script.

remoteServer=$1
cmd=$2

ssh ${remoteServer} ${cmd}

3. Make the wrapper script executable in source control – see here if you are using SVN http://www.mernin.com/blog/2008/02/making-scripts-executable-inside-subversion/

4. Configure a bamboo job to call the wrapper script after your build has been produced

• Set source repository to point to where your wrapper script is located
• Configure a Bash builder (see screenshot below) to call your ssh wrapper with appropriate server details and command to execute.

Step-by-Step – Automate your acceptance tests with Soap UI

1. Create a SoapUI project – http://www.soapui.org/Getting-Started/your-first-soapui-project.html

2. Create any necessary static data and stubs required for your test. Remember only focus on the single test for the moment.

3. Write an automated test – http://www.soapui.org/Functional-Testing/functional-testing.html

Step-by-Step – Bamboo integration of your SoapUI test suite with Maven

1. Check in your Soap UI project to a source repository

2. Create a Maven POM

• Create a Maven POM in the base directory of your SoapUI project
• Configure the POM as follows : http://www.soapui.org/Test-Automation/maven-2x.html (see full example below)
• Check-in the POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>
<groupId>com.industrieit</groupId>
<artifactId>continuous-delivery-demo-service-test</artifactId>
<packaging>jar</packaging>
<version>0.0.0-SNAPSHOT</version>
<name>Continuous Delivery Demo - Service Test</name>
<description />

<build>
  <plugins>
    <plugin>
      <groupId>eviware</groupId>
      <artifactId>maven-soapui-plugin</artifactId>
      <version>3.5.1</version>
      <configuration>
        <projectFile>src/test/resources/soapui-project.xml</projectFile>
        <endpoint>${endpoint}</endpoint>
        <junitReport>true</junitReport>
        <outputFolder>${basedir}/target/surefire-reports</outputFolder>
      </configuration>
    </plugin>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
  </plugin>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-install-plugin</artifactId>
    <version>2.3.1</version>
   </plugin>
  </plugins>
</build>
<properties>
  <endpoint>http://yourserver/yourwsdl.wsdl</endpoint>
</properties>
</project>

3. Configure a bamboo job to perform Maven build to execute the tests

• Set source repository to point to where your SoapUI project is located
• Configure a Maven builder (see screenshot below) to call maven goal “clean eviware:maven-soapui-plugin:test”
• Ensure that the build is listed as producing test results
• List the location of test results as “**/target/surefire-reports/*.xml”