Execute Jenkins Maven builds in Docker containers

When building software packages the build process itself might change some details in the underlying operating system or environment. This might also affect the next build as the environment is kind of „poluted“. So the best option would be to run the builds in an unchangeable environment or better: to run the different builds each in a new but identical machine. This could be reached via provisioning of heavy virtual os images. But this would mean to reserve a lot of system resources like CPU or RAM. So what we better need is a lightweight alternative for virtual machines. And one of those options is Docker. Docker containers provide process isolation on OS level while virtual machines offer isolation by creating hardware virtualization. For infrastucture use cases machine virtualization is an ideal option, but for our purposes (providing a recyclable build environment) software containers are the best choice.
With every build our build machine (jenkins) master will create a new runnable container from a docker image, run the build (including tests etc) and destroy it after everything was done. The build results can then be obtained from the jenkins master.

Here’s the overview:
overview_jenkins_docker

Versions

component version download
Ubuntu 14.04 http://www.ubuntu.com/download/desktop
Fedora 21 https://getfedora.org/de/workstation/download/
Java Hotspot 1.8.0_91 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Docker 1.10.2 https://docs.docker.com/engine/installation/
Jenkins 2.0 https://jenkins.io/
Docker Plugin (Jenkins) 0.16.0 http://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
Maven 3.3.9 https://maven.apache.org/download.cgi

Step 1: Create a Docker image

We need a dockerfile for the Docker image that will be used as Jenkins build agent. Therefor you need a working directory with following content:

  • /home/christoph/jenkins-docker-slave/docker.file
  • /home/christoph/jenkins-docker-slave/jdk-8u91-linux-x64.rpm
  • /home/christoph/jenkins-docker-slave/apache-maven-3.3.9-bin.zip

For production usage you might need a well configured settings.xml and a pre-filled local repository for Maven (to avoid long build times), but for now we rely on the defaults.

# Set the base image to Fedora v. 21
FROM fedora:21
 
# Set the maintainer
MAINTAINER Christoph Burmeister

# create users in the image
# Jenkins user is needed for the ssh access
RUN echo "root:password" | chpasswd
RUN useradd jenkins
RUN echo "jenkins:jenkins" | chpasswd

# update the yum
RUN yum -y update

# install unzip via yum
RUN yum -y install unzip

# install jdk, as on a clean fedora image, no java is installed
ADD jdk-8u91-linux-x64.rpm /home/jenkins/
# install java to /usr/java/jdk1.8.0_91/jre/bin/java
RUN yum -y localinstall /home/jenkins/jdk-8u91-linux-x64.rpm 
# Now JDK was installed at /usr/java/jdk1.8.0_91/ and linked from /usr/bin/java
RUN java -version 

# install maven
ADD apache-maven-3.3.9-bin.zip /home/jenkins
RUN unzip /home/jenkins/apache-maven-3.3.9-bin.zip -d /home/jenkins/
RUN chown -R jenkins /home/jenkins/apache-maven-3.3.9

# install svn client
RUN yum -y install subversion

# install and configure ssh server with ssh key
RUN yum -y install openssh-server
RUN sed -i 's|session    required     pam_loginuid.so|session    optional     pam_loginuid.so|g' /etc/pam.d/sshd
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''

# expose the ssh port
EXPOSE 22

# entrypoint by starting sshd
CMD ["/usr/sbin/sshd", "-D"]

That’s it for the dockerfile. Now you have to build and tag it in order to have an image in the local Docker registry that can be used later by Jenkins for creating the build slaves:

christoph@apollo:~/jenkins-docker-slave$ docker build -f docker.file -t fedora-jenkins-slave .

The above command builds the Docker image according to instructions in dockerfile and tags it with name „fedora-jenkins-slave“. The „.“ directory defines the location where the „ADD“ resources are placed.

With „docker images“ you can find all the images on the registry:

christoph@apollo:~/docker-jenkins-slave$ docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
fedora-jenkins-slave   latest              f810b8a77b58        12 hours ago        1.97 GB

You can try out the ssh access by starting a container:

christoph@apollo:~/docker-jenkins-slave$ docker run -t -i fedora-jenkins-slave

in a new terminal you can see the running Docker container by invoking „docker ps“ and „docker inspect“ to fetch the ip address and then you can ssh into the container with jenkins account:

christoph@apollo:~$ docker ps
CONTAINER ID        IMAGE                  COMMAND               CREATED             STATUS              PORTS               NAMES
b139441a46b5        fedora-jenkins-slave   "/usr/sbin/sshd -D"   12 hours ago        Up 12 hours         22/tcp              nauseous_poitras
christoph@apollo:~$ docker inspect b139441a46b5 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.2",
                    "IPAddress": "172.17.0.2",
christoph@apollo:~$ ssh jenkins@172.17.0.2
jenkins@172.17.0.2's password: 
Last login: Mon Apr 25 19:21:16 2016 from 172.17.0.1
[jenkins@b139441a46b5 ~]$ 
[jenkins@b139441a46b5 ~]$ ls -la /home/jenkins/
total 164860
drwx------  5 jenkins jenkins      4096 Apr 26 07:11 .
drwxr-xr-x  8 root    root         4096 Apr 25 19:21 ..
drwxr-xr-x 10 jenkins root         4096 Apr 25 18:40 apache-maven-3.3.9
-rw-rw-r--  1 root    root      8617253 Apr 25 17:05 apache-maven-3.3.9-bin.zip
-rw-------  1 jenkins jenkins        36 Apr 26 07:11 .bash_history
-rw-r--r--  1 jenkins jenkins        18 Okt  8  2014 .bash_logout
-rw-r--r--  1 jenkins jenkins       193 Okt  8  2014 .bash_profile
-rw-r--r--  1 jenkins jenkins       231 Okt  8  2014 .bashrc
-rw-rw-r--  1 root    root    160162581 Apr 25 09:55 jdk-8u91-linux-x64.rpm
drwxrwxr-x  3 jenkins jenkins      4096 Apr 25 19:21 .subversion
[jenkins@b139441a46b5 ~]$

The running container (id b139441a46b5) can be killed in the following way:

christoph@apollo:~$ docker kill b139441a46b5

Ok, this was the first step, the image is now ready to be used.

Step 2: Setup Jenkins with Docker plugin

For this step, you have to load the jenkins.war from their website. For this example we rely on the defaults to keep it simple. Prepare your working directory for jenkins:

  • /home/christoph/jtools/jenkins/home (empty)
  • /home/christoph/jtools/jenkins/jenkins.war

Then you can start the war directly:

christoph@apollo:~/jtools/jenkins$ java -jar -DJENKINS_HOME=/home/christoph/jtools/jenkins/home/ jenkins.war

After small initial configuration where you can leave the defaults, Jenkins is ready to use.

Now you have to setup the tool configuration with two different maven installations, where one is for the master and one is for the slaves (or better: agents):
maven_configuration

Then you have to install and configure the Docker plugin:
jenkins_docker_cloud
With „Test connection“ you will receive the Docker version if everything works.
Jenkins and the Docker server are running on the same machine, so you can use the unix sock connection. Otherwise you’ll have to expose the interface for the Docker server API to a tcp port.

and a corresponding template/image configuration:
jenkins_docker_image
Take care that you choose in „Labels“ field a specific value as this is the marker for the later job configuration.

Step 3: Prepare and run the build job

To keep this example simple, I don’t pay much attention to the parts of „unittestproject“ or its location in SVN-server. Let’s say, this is a default maven project within a default SVN location.

With this knowledge, you can create the build job:
jenkins_buildjob_config1
Note the value of „Label Expression“ field. It has to contain exactly the same as in image configuration.

jenkins_buildjob_config2

jenkins_buildjob_config3

By invoking the „Build now“ button, Jenkins will call the Docker server with the required image. Docker will then create the running container where the svn checkout and mvn build are getting executed. After the build has finished, the container is getting deleted automatically, so no poluted environment is left after the build.

01:51:07 Started by user Christoph Burmeister
01:51:07 Building remotely on docker_on_localhost-df25a5e49ae3 (fedora-jenkins-slave) in workspace /home/jenkins/workspace/unittestproject
01:51:16 Checking out a fresh workspace because there's no workspace at /home/jenkins/workspace/unittestproject
01:51:16 Cleaning local Directory .
01:51:17 Checking out https://svnserver/trunk/unittestproject at revision '2016-04-25T19:51:07.138 -0400'
01:51:22 A         src
01:51:22 A         src/main
01:51:22 A         src/main/java
01:51:22 A         src/main/java/eu
01:51:22 A         src/main/java/eu/christophburmeister
01:51:22 A         src/main/java/eu/christophburmeister/playground
01:51:22 A         src/main/java/eu/christophburmeister/playground/unittestproject
01:51:22 A         src/main/java/eu/christophburmeister/playground/unittestproject/MyClass.java
01:51:22 A         src/main/resources
01:51:22 A         src/test
01:51:22 A         src/test/java
01:51:22 A         src/test/java/eu
01:51:22 A         src/test/java/eu/christophburmeister
01:51:22 A         src/test/java/eu/christophburmeister/playground
01:51:22 A         src/test/java/eu/christophburmeister/playground/unittestproject
01:51:22 A         src/test/java/eu/christophburmeister/playground/unittestproject/MyClassTest.java
01:51:22 A         src/test/resources
01:51:22 A         pom.xml
01:51:22 At revision 2
01:51:22 
01:51:22 [unittestproject] $ /home/jenkins/apache-maven-3.3.9/bin/mvn -DDOCKER_CONTAINER_ID=df25a5e49ae3723f1223885531e7d61eb54d68b58159497216557a460751696e -DDOCKER_HOST=unix:///var/run/docker.sock -DJENKINS_CLOUD_ID=docker_on_localhost clean package
01:51:24 [INFO] Scanning for projects...
01:51:24 [INFO]                                                                         
01:51:24 [INFO] ------------------------------------------------------------------------
01:51:24 [INFO] Building unittestproject 0.0.1-SNAPSHOT
01:51:24 [INFO] ------------------------------------------------------------------------
01:51:28 [INFO] 
01:51:28 [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ unittestproject ---
01:51:29 [INFO] 
01:51:29 [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ unittestproject ---
01:51:34 [INFO] Using 'UTF-8' encoding to copy filtered resources.
01:51:34 [INFO] Copying 0 resource
01:51:34 [INFO] 
01:51:34 [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ unittestproject ---
01:51:45 [INFO] Changes detected - recompiling the module!
01:51:45 [INFO] Compiling 1 source file to /home/jenkins/workspace/unittestproject/target/classes
01:51:46 [INFO] 
01:51:46 [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ unittestproject ---
01:51:46 [INFO] Using 'UTF-8' encoding to copy filtered resources.
01:51:46 [INFO] Copying 0 resource
01:51:46 [INFO] 
01:51:46 [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ unittestproject ---
01:51:46 [INFO] Changes detected - recompiling the module!
01:51:46 [INFO] Compiling 1 source file to /home/jenkins/workspace/unittestproject/target/test-classes
01:51:46 [INFO] 
01:51:46 [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ unittestproject ---
01:51:48 [INFO] Surefire report directory: /home/jenkins/workspace/unittestproject/target/surefire-reports
01:51:48 
01:51:48 -------------------------------------------------------
01:51:48  T E S T S
01:51:48 -------------------------------------------------------
01:51:49 Running eu.christophburmeister.playground.unittestproject.MyClassTest
01:51:49 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.083 sec
01:51:49 
01:51:49 Results :
01:51:49 
01:51:49 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
01:51:49 
01:51:49 [INFO] 
01:51:49 [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ unittestproject ---
01:51:50 [INFO] Building jar: /home/jenkins/workspace/unittestproject/target/unittestproject-0.0.1-SNAPSHOT.jar
01:51:50 [INFO] ------------------------------------------------------------------------
01:51:50 [INFO] BUILD SUCCESS
01:51:50 [INFO] ------------------------------------------------------------------------
01:51:50 [INFO] Total time: 26.277 s
01:51:50 [INFO] Finished at: 2016-04-25T19:51:50-04:00
01:51:51 [INFO] Final Memory: 21M/280M
01:51:51 [INFO] ------------------------------------------------------------------------
01:51:51 Finished: SUCCESS

This example is only build up to „package“ phase. In a production environment when building up to „deploy“ phase we would have an artifact repository like Artifactory or Nexus outside the buildserver as distribution target for the packaged artifacts. But for now, this is the end of the process.

References