In Part 1 we gave the general outline of what we are trying to do, the tools we’re using, and the architecture of the application.
In this part (Part 2) we’re going to work on building the development environment with Ansible. This includes the Jenkins, Gitlab, a private Docker Registry, and a proxy server so we can point DNS to the site.
In Part 3 we configure the load balancers and the web site.
In Part 4 we configure Jenkins and put it all together.
The Ansible code can be found on Github in the Cisco Live 2015 repo.
Get the CoreOS Image
We’ll need an image to work with. While we can do this on the command line, its not something we’re going to repeat to often so I think we’re ok doing this the lame way and use the GUI.
A simple search takes us to the OpenStack page on the CoreOS site. I just used the current stable version. Its pretty simple. You follow their instructions:
I downloaded this to some Linux server that was on the Internet. From there, I went into the Cisco OpenStack private Cloud dashboard and under images created a new one.
You can also do this through the command line just to make sure you’re still connecting correctly:
Ok, now we have a base image to work with. Let’s start automating this.
Ansible Base Setup
I’ve set up a directory called ~/Code/lawngnomed/ansible where all my Ansible configurations will live. I’ve spoken about setting up Ansible before so in this post we’ll just go over the things that are unique. The first item we need to do is setup our Development environment. Here’s the Ansible script for creating the development node, which I gave the hostname of ‘ci':
This playbook does the following:
Creates a new server called ‘ci’
ci will use a security group I already created
ci will use a key-pair I already created.
ci will use the cloud-config.sh script I created as part of the boot up.
Once the node is created creates the following roles on it: registry, gitlab, jenkins, and ci-proxy
The metacloud_vars.yml file contains most of the environment variables. Here is the file so you can see it. Replace this with your own:
You can see I used a few images as I tried this out and eventually settled on using the same coreos image that my jenkins slaves run on. We’ll get to that soon.
You’ll need to create a security group so that all the services can be access. My security group looked as follows:
The other security group allows port 80 and 22 so I can ssh and go to the web browser.
The next important file is the files/cloud-config.sh script. With the script I needed to accomplish 2 things:
Get Java on the instance so that Jenkins could communicate with it.
Get Python on the instance so Ansible could run on it.
Make it so docker would be able to communicate with an insecure registry.
CoreOS by itself tries to be as bare as it gets so after trolling the Internet for a few days I finally cobbled this script together that would do the job.
Roles
A few directories with files were created:
Let’s go through each task:
ci-proxy
This role is creates a docker container that acts as the reverse proxy. So that when requests like http://jenkins.lawngnomed.com come in, the proxy redirects the request to the right container.
The task file below copies the nginx configuration file and then mounts it into the container. Then it runs the container.
The contents of the nginx default config file that will run in /etc/nginx/conf.d/default.conf is the following:
There could be some issues with this file, but it seems to work. There are occasions when jenkins and gitlab redirect to bad urls, but everything works with this configuration. I’m open to any ideas to changing it.
Once this role is up you can access the URL from the outside.
Gitlab
Gitlab requires a Redis container for key value store and a PostGrsql database. We use Docker for both of these and link them together. The Ansible playbook file looks as follows:
Notice that the gitlab_db_password is an environment variable created in the ../var/main.yml file. I set this up and then encrypted the file using Ansible Vault. See my post on how that is accomplished because its a pretty cool technique I learned from our Portland Ansible Users Group.
Jenkins
The Jenkins Ansible installation script is pretty straight forward. The only catch is to make sure the directory owner is jenkins and that you mount the directory.
Registry
No tricks, here, we’re just using the latest from the docker registry. This goes out and pulls the registry.
Loose Ends
There are a few parts that I didn’t automate that should be done.
The instance I created mounts a persistent storage device that I created in Metacloud. There are two pieces missing:
It doesn’t create the volume in OpenStack if its not there yet.
It doesn’t mount the volume onto the development server.
For speed, its better to pull the docker containers from the local registry. So technically we should tag all the images that we’re using and put them in the local registry. This is a chicken and an egg problem because you need the registry up before you can download images from it. So I left it that way.
There are still some things I needed to finish like putting some keys and other items I needed for Jenkins in the /vol directory. Its not perfect but its pretty good.
Creating the volume and mounting was pretty quick once the image was up. First I created the volume and assigned it using the Horizon Dashboard that Metacloud provides.
This was just a 20GB volume. Once the instance was up I ran a few commands like:
This way all of our information persists if the containers terminate and if the instances terminate.
Finishing up the Development Server
Once you get to this point, you should be able to bring it all up with:
That should do it! Once you are in you may want to tag all of your images so that they load in the local docker registry. For example, once you log in you could run:
At this point the idea is that you should be able to go to whatever public IP address was assigned to you and be able to access:
If you’re there then you can get rolling to the next step: Ansible scripts to deploy the rest of the environment.
In Part 3 we’ll cover Ansible for bringing up the load balancers and web servers. We’ll also snapshot an image to make it a jenkins slave.