In the previous blog post, we looked into Docker’s single-host networking for MySQL containers. This time, we are going to look into the basics of multi-host networking and Docker swarm mode, a built-in orchestration tool to manage containers across multiple hosts.
Docker Engine - Swarm Mode
Running MySQL containers on multiple hosts can get a bit more complex depending on the clustering technology you choose.
Before we try to run MySQL on containers + multi-host networking, we have to understand how the image works, how much resources to allocate (disk,memory,CPU), networking (the overlay network drivers - default, flannel, weave, etc) and fault tolerance (how is the container relocated, failed over and load balanced). Because all these will impact the overall operations, uptime and performance of the database. It is recommended to use an orchestration tool to get more manageability and scalability on top of your Docker engine cluster. The latest Docker Engine (version 1.12, released on July 14th, 2016) includes swarm mode for natively managing a cluster of Docker Engines called a Swarm. Take note that Docker Engine Swarm mode and Docker Swarm are two different projects, with different installation steps despite they both work in a similar way.
Some of the noteworthy parts that you should know before entering the swarm world:
The following ports must be opened:
2377 (TCP) - Cluster management
7946 (TCP and UDP) - Nodes communication
4789 (TCP and UDP) - Overlay network traffic
There are 2 types of nodes:
Manager - Manager nodes perform the orchestration and cluster management functions required to maintain the desired state of the swarm. Manager nodes elect a single leader to conduct orchestration tasks.
Worker - Worker nodes receive and execute tasks dispatched from manager nodes. By default, manager nodes are also worker nodes, but you can configure managers to be manager-only nodes.
More details in the Docker Engine Swarm documentation.
In this blog, we are going to deploy application containers on top of a load-balanced Galera Cluster on 3 Docker hosts (docker1, docker2 and docker3), connected through an overlay network. We will use Docker Engine Swarm mode as the orchestration tool.
“Swarming” Up
Let’s cluster our Docker nodes into a Swarm. Swarm mode requires an odd number of managers (obviously more than one) to maintain quorum for fault tolerance. So, we are going to use all the physical hosts as manager nodes. Note that by default, manager nodes are also worker nodes.
Firstly, initialize Swarm mode on docker1. This will make the node as manager and leader:
We are going to add two more nodes as manager. Generate the join command for other nodes to register as manager:
On docker2 and docker3, run the following command to register the node:
Verify if all nodes are added correctly:
At the moment, we have docker1.local as the leader. You can run “docker network” and “docker service
Overlay Network
The only way to let containers running on different hosts connect to each other is by using an overlay network. It can be thought of as a container network that is built on top of another network (in this case, the physical hosts network). Docker Swarm mode comes with a default overlay network which implements a VxLAN-based solution with the help of libnetwork and libkv. You can however choose another overlay network driver like Flannel, Calico or Weave, where extra installation steps are necessary. We are going to cover more on that later in an upcoming blog post.
In Docker Engine Swarm mode, you can create an overlay network only from a manager node and it doesn’t need an external key-value store like etcd, consul or Zookeeper.
The swarm makes the overlay network available only to nodes in the swarm that require it for a service. When you create a service that uses an overlay network, the manager node automatically extends the overlay network to nodes that run service tasks.
Let’s create an overlay network for our containers. We are going to deploy Percona XtraDB Cluster and application containers on separate Docker hosts to achieve fault tolerance. These containers must be running on the same overlay network so they can communicate with each other.
We are going to name our network “mynet”. You can only create this on the manager node:
Let’s see what networks we have now:
There are now 2 overlay networks with a Swarm scope. The “mynet” network is what we are going to use today when deploying our containers. The ingress overlay network comes by default. The swarm manager uses ingress load balancing to expose the services you want externally to the swarm.
Deployment using Services and Tasks
We are going to deploy the Galera Cluster containers through services and tasks. When you create a service, you specify which container image to use and which commands to execute inside running containers. There are two type of services:
Replicated services - Distributes a specific number of replica tasks among the nodes based upon the scale you set in the desired state, for examples “--replicas 3”.
Global services - One task for the service on every available node in the cluster, for example “--mode global”. If you have 7 Docker nodes in the Swarm, there will be one container on each of them.
Docker Swarm mode has a limitation in managing persistent data storage. When a node fails, the manager will get rid of the containers and create new containers in place of the old ones to meet the desired replica state. Since a container is discarded when it goes down, we would lose the corresponding data volume as well. Fortunately for Galera Cluster, the MySQL container can be automatically provisioned with state/data when joining.
Deploying Key-Value Store
The docker image that we are going to use is from Percona-Lab. This image requires the MySQL containers to access a key-value store (supports etcd only) for IP address discovery during cluster initialization and bootstrap. The containers will look for other IP addresses in etcd, if there are any, start the MySQL with a proper wsrep_cluster_address. Otherwise, the first container will start with the bootstrap address, gcomm://.
Let’s deploy our etcd service. We will use etcd image available here. It requires us to have a discovery URL on the number of etcd node that we are going to deploy. In this case, we are going to setup a standalone etcd container, so the command is:
Then, use the generated URL as “-discovery” value when creating the service for etcd:
At this point, Docker swarm mode will orchestrate the deployment of the container on one of the Docker hosts.
Retrieve the etcd service virtual IP address. We are going to use that in the next step when deploying the cluster:
At this point, our architecture looks like this:
Deploying Database Cluster
Specify the virtual IP address for etcd in the following command to deploy Galera (Percona XtraDB Cluster) containers:
It takes some time for the deployment where the image will be downloaded on the assigned worker/manager node. You can verify the status with the following command:
We can see that the mysql-galera service is now running. Let’s list out all services we have now:
Swarm mode has an internal DNS component that automatically assigns each service in the swarm a DNS entry. So you use the service name to resolve to the virtual IP address:
Or, retrieve the virtual IP address through the “docker service inspect” command:
Our architecture now can be illustrated as below:
Deploying Applications
Finally, you can create the application service and pass the MySQL service name (mysql-galera) as the database host value:
Once deployed, we can then retrieve the virtual IP address for wordpress service through the “docker service inspect” command:
At this point, this is what we have:
Our distributed application and database setup is now deployed by Docker containers.
Connecting to the Services and Load Balancing
At this point, the following ports are published (based on the -p flag on each “docker service create” command) on all Docker nodes in the cluster, whether or not the node is currently running the task for the service:
etcd - 2380, 2379, 7001, 4001
MySQL - 3306
HTTP - 80
If we connect directly to the PublishedPort, with a simple loop, we can see that the MySQL service is load balanced among containers:
At the moment, Swarm manager manages the load balancing internally and there is no way to configure the load balancing algorithm. We can then use external load balancers to route outside traffic to these Docker nodes. In case of any of the Docker nodes goes down, the service will be relocated to the other available nodes.
That’s all for now. In the next blog post, we’ll take a deeper look at Docker overlay network drivers for MySQL containers.
Tags:
MySQL
percona xtradb cluster
galera
docker
networking
swarm
container
multi-host
PlanetMySQL Voting: Vote UP / Vote DOWN