You need to install Docker on the Virtual Machine in your preferred cloud. You provision an instance, SSH into it, install Docker, pull down the relevant images, etc. The steps are more or less similar on most public cloud providers and often you are just dealing with the mechanics of using their cloud console applications and getting this to work. Of course, if you have scripted your way through this, that is fantastic.
But what if there was a single client that you could run from anywhere to setup/control all these Docker hosts across cloud providers. Well that is what Docker Machine aims to do. It is part of the Docker Toolbox that has made available recently and key among them are the standard docker client and a utility named docker-machine that you can use to “create Docker hosts on your computer, on cloud providers, and inside your own data center.”
High Level View of Docker Machine
Take a look at the diagram below to understand what Docker machine can potentially do:
You see Windows over there, but this can be run on a Mac too. All you need to do is install the appropriate Docker Toolbox for your OS and you shall have all these tools. The diagram should make things obvious:
You can create one or more Docker hosts i.e. Docker Machines via the docker-machine utility.
When these hosts are created, the Docker daemon will be automatically installed for you.
You can set any one of your Docker Machine as the Active Machine via the Environment variables. In simple words, you are configuring the Docker Client to talk to your host i.e. a specific Docker Machine.
This active Docker Machine can then be controlled via the Docker client, and which means that you can pull images, start/stop containers and all the goodness that the docker client allows you to do.
The Magic behind Docker Machine – Drivers
Docker Machine is able to create VMs on not just your local machine but also on several public cloud providers, private cloud implementations and more. How is it able to manage this? It does this via drivers. When creating a Docker Machine, you have to specific which driver you want to use. For the local setup, it uses Virtual Box and it also provides drivers for AWS, Google Cloud, Azure, Digital Ocean and more.
In this blog post, we are going to look at create a Docker Machine on Google Compute Engine, all from your client. This should give you an idea about how you could potentially use other drivers too and control all your Docker Machines from one machine.
Pre-requisites
Before you move ahead with the commands, check out the pre-requisites:
You have the Docker Toolbox installed on your machine (Windows or Mac). Downloads for Docker Toolbox is available here. I have a blog post on setting up Docker Toolbox on Windows, if you are interested.
You have a Google Cloud Platform account with billing setup. There is a $300 credit if you are interested in a trial.
You have a Google Cloud Platform Project created and have a Project Id. If not, check out these simple steps:
Visit Google Developers Console and login with your account.
Click on Create Project. This will bring up the New Project dialog. Enter a name for the project and ensure that you have selected the correct Billing Account. An example screenshot is shown below:
Give any name to your project i.e. Project name. But do note the project ID, which is unique. This is important because when we use Docker Machine and use the Google Cloud driver, we will need to provide this project ID, so that the VM is commissioned inside this project.
Click Create to begin the process of creating the project.
Be patient while the project is being created and initialized.
Docker Machine – Basic commands
Assuming that Docker Machine is setup and available on your local desktop/laptop — you should be able to run it from the Command Prompt or Terminal.
The first thing we should do is to fire the basic Docker Machine command to list our hosts. I currently do not have any hosts, so when I run the “ls” command, I get the following output:
If you have already setup a Docker Machine or two, you would definitely get the hosts listed. For e.g. if I have two Hosts configured locally, I would get an output like this:
The important thing to note here is the DRIVER column. This column indicates the driver that is being used. In the output above, you would have noticed the virtualbox driver, which is the local driver.
To create a local Docker Machine Host, you would typically use the following command:
Docker Machine – Create Docker Host on Google Cloud
To create a Docker Machine on the Google Cloud, you would have guessed it by now that we need to provide the correct driver, in this case named google and ofcourse give some additional parameters like project ID, etc.
The command would typically be of the following format:
Let us take a look at what parameters we can provide to the docker-machine utility. Note that not all are mandatory and there are sensible defaults for most of them, if you do not give them. If you do a –help on the google specific parameters, this is what is currently available as parameters to pass:
If you have been using the Google Cloud and dealt with either the Google Developers Console or the gcloud utility, these options would be familiar to you.
The only mandatory parameter that you have to provide is the –google-project parameter, which is nothing but the project ID value for your Google Cloud Platform project.
I recommend that you closely study the other parameters too because they are important in the overall context of things. I would not suggest to go with all defaults, especially if you know what kind of loads you are going to run, where the Compute Engine VM instance should be located and more. Specifically, I suggest that you look at:
–google-machine-type : Currently it uses the f1-micro, which is the lowest in capability when it comes to memory and processing power. You might want to bump that up. Take a look at the list of machine types available. If you are just trying this to validate that Docker Machine works or for a demo, the f1-micro instance is just fine.
–google-disk-size and –google-disk-type : In case you want more space and different type (SSD)
–google-zone : You might want to host the VM instance in your preferred zone and not the default “us-central1-a”. Check out the various regions/zones.
Now, let us go ahead and create a default Docker Host on Google Compute Engine. The command is given below. Simply replace <your-project-id> and <your-docker-machine-name> with actual values for your Google Project ID and a name for your Docker Machine respectively.
An example run is shown below:
It will ask you to paste the URL in the browser and perform the authentication. This is to ensure that you login with your Google Account that is the owner of the Google Cloud Project. Please login and allow access to Docker Machine. On successfully authentication in the browser, you will be shown a code that you need to paste in the “Enter code:” above.
Once you enter the code, it will begin the process of create the Compute Engine instance, downloading Docker and more. You should start seeing a stream of output as given below:
Note: I have occasionally found this command to hang up indefinitely over here and you might face the same too. In case you do, you should break out of it, but give it some time.
Docker Machine – What did it create?
At this point, it should be clear that a Compute Engine instance has been created for you in your Google Cloud Project. But before that, let us run “ls” command on Docker Machine and see what we get:
On my local machine, if I am lucky, I will get the STATE column as Running and the URL column with the Docker Daemon URL pointing to the Google Compute Engine VM Instance External IP and port 2376, etc. But don’t worry, if you continue to see this in the Timeout state.
Additionally, we should also check out the VM instance that it created. You can do that in one of two ways:
Go to Google Developers Console. Go to your Cloud Project and navigate to Compute –> Compute Engine –> VM Instances. You should be able to see the f1-micro instance that was created.
If you have gcloud utility setup on your machine. You can run the gcloud compute instances list command for your project. A sample output from my local setup is shown below:
But this is not the end of our investigation. I suggest that you do the following to better understand what it created.
First up, let us go to Google Developers Console for our Cloud Project and navigate to Compute –> Compute Engine –> VM instances. On my machine, I see the following (ofcourse I just have one instance here and this is the Docker Machine that we created):
You will notice that the Name is the one that we provided while creating the Docker Machine. Note the External IP that has been provided to this instance. Now, click on the instance name i.e. google-docker-m1. This will bring up the page with the details as shown below:
I have highlighted a specific property of the instance titled “Tags”. This is important to see what has happened. In simple terms, this instance has been provided a tag value of “docker-machine”. But what does that really mean? The answer lies in the network configuration that is associated with your instance and which I have highlighted at the bottom i.e. default
What is interesting is when you click on the default Network that this instance is provided. Go ahead and click on default.
You will find in the section for Firewall rules, that a new rule that been auto-created for you. And this rule clearly specifies that TCP traffic will be allowed on port 2376. What runs on port 2376? The Docker Daemon ofcourse. So this will allow you to potentially connect to this Docker Machine from the Docker Client and execute commands (send those commands over to the Docker Daemon). This is exactly what happens behind the scenes in the next section.
The final piece of the puzzle is the column titled Target tags. The value here is docker-machine and this applies to all Compute Engine instances which have this value as one of the tags. This is a neat way of making sure that when you create further instances on Google Compute Engine, as long as you have the correct tags associated with the instances, the firewall rules that you have setup to target specific tags will get applied to the instance. It is simple and neat.
Note: Currently the firewall rule allows traffic from any source IP (0.0.0.0/0).
Docker Machine – How do we connect to it?
In the previous section, there was a message at the end that indicated that if you want to connect Docker to this machine, then it asks you to run the env command. Let us do that.
On Windows, I run the following command and get the details as shown below:
This indicates to you clearly what environment variables you need to set on your local machine to allow the Docker Client to connect to it automatically. In other words, if you go back to the diagram that we had in the beginning, these environment variables are going to point the Docker client (docker command) to the Docker Machine that we just just created i.e. docker-machine-m1
So, go right ahead and set those environment variables.
On my Windows machine, I just do the following as instructed:
Once you are done, you can now fire your favorite Docker client commands and it will execute straightaway on the Docker Machine hosted on Google Compute Engine.
Let us try a few things out:
C:\Users\irani_r>docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
This will pull down the latest Docker image for Nginx. Be patient as the file layers are being pulled. It should be pretty fast since this instance is running on the Google Network.
Now, if we do list the images again, we should see the Nginx image
Let us run a Docker container based on this nginx image:
Let us do a “ps” command:
Cool, isn’t it?
Now, all you need to do is set a firewall rule for allowing traffic on port 80 as shown below. Note that currently I have set it to allow all targets, which will include the docker-machine target.
If you have the correct firewall rule setup for allowing traffic on port 80, you should be able to visit the External IP for your GCE instance, which will indicate that the Nginx server is up and running well!
Next Steps
In this post, we saw how we can use Docker Machine, a part of the Docker Toolbox, to create Docker Hosts of a public cloud providers. We specifically took the example of Google Cloud in this post. Stay tuned for future posts where we will do the same for other providers like Digital Ocean, Amazon Web Services (AWS) and more.