This article shows you how to set up automatic backup snapshots of instances on our OpenStack platform. These snapshots are full backups of a virtual machine and can be used to clone an instance or restore one. We'll also show you how to set up an hook for apt-get updates and installations. This way, you can quickly recover from a failed update. The snapshots have retention.
This article is a bit more technical than our usual blog articles.
We use a script on the instance together with cron to create the snapshots. In this example we'll set up the following schedule:
Daily snapshot, 'backup_type' set to 'daily', 'rotation' set to '7'.
Weekly snapshot, 'backup_type' set to 'weekly', 'rotation' set to '4'.
Monthly snapshot, 'backup_type' set to 'montly', 'rotation' set to '12'.
Yearly snapshot, 'backup_type' set to 'yearly', 'rotation' set to '2'.
You can change this if for example 7 daily snapshots are all you want.
The part of OpenStack that manages the instances, Nova / Compute has a function to create an image of an instance. This is a complete snapshot of the instance at the moment of snapshot creation. Under the hood a 'qemu snapshot' is created. The second function, based on this, is 'nova backup'. This creates a regular image, but with the type set to backup. It also has retention, so if there are more images than the given retention, the oldest one is overwritten.
Restoring a snapshot is as easy as creating a new instance based on a snapshot image and moving the floating IP. If you don't use a floating IP then you can reinstall the current instance via the command line.
Snapshots are a fast and easy way to clone or restore an instance to an earlier point. Let's say you're doing an upgrade of your application and it goes horribly wrong. Rollback fails and it's the middle of the night. Do you want to go and restore all the files and databases from the backup of 18 hours earlier, manually? Or do you want to revert the entire thing to a full snapshot from before the upgrade? If I could choose, I would know.
Do note that we still recommend to create regular backups. They allow for a more fine-grained way to restore files/folders/databases and have things like compression and incremental/differential support, thus backing up only the things that have changed since the last backup. This snapshot method is just a copy of the enitre disk, taking up more space.
Nova Backup
With the 'nova backup' command the snapshots are created. These are normal images, with their type set to backup. When doing a nova backup you need to specify a 'type' and a 'retention'. Based on the type the retention is performed. An example:
You create a snapshot with the type 'daily' and a retention of '7'. When there are 7 snapshots of type 'daily' and the new snapshot is created, the oldest snapshot with type 'daily' is removed.
Because of this you can set up different types with different retentions.
There is no scheduling in OpenStack, you need to create these backups yourself, from cron for example.
Installation
We'll use a simple script to create the snapshots. This script calls nova backup with the correct credentials. In this example we create a snapshot of the current instance, but you are free to adapt the script to create snapshots of more instances.
You need to connect to the instance via SSH. You can use a client like putty or the terminal.
We first install the required dependencies:
# Ubuntu
apt-get install dmidecode wget python-pip
# CentOS
yum install dmidecode wget python-pip
Recent Ubuntu releases have the OpenStack command line tools packaged:
apt-get install python-keystoneclient python-glanceclient python-novaclient
On older versions and CentOS you can use 'pip', the Python Pypi package tool:
pip install python-keystoneclient python-glanceclient python-novaclient
You also need a credentials file ('computerc'). Below you'll find an example file. Fill it in with the credentials you got in the provisioning email:
nano /root/.openstack_snapshotrc
export OS_AUTH_URL="https://identity.stack.cloudvps.com/v2.0"
export OS_TENANT_NAME="PROJECT_UUID"
export OS_TENANT_ID="PROJECT_UUID"
export OS_USERNAME="USERNAME"
export OS_PASSWORD="PASSWORD"
Test if the file is correct by sourcing it in the shell and issueing a command:
source /root/.openstack_snapshotrc
nova credentials
Example output:
+------------------+-------------------------+
| User Credentials | Value |
+------------------+-------------------------+
| id | f3[...]11 |
| name | image-test-1 |
| roles | [{"name": "compute"}] |
| roles_links | [] |
| username | image-test-1 |
+------------------+-------------------------+
The script itself is installed on the system with this command:
wget -O "/usr/local/bin/create-snapshot.sh" "https://github.com/RaymiiOrg/openstack-nova-snapshot/blob/master/create_snapshot.sh"
chmod +x /usr/local/bin/create-snapshot.sh
The script itself is very simple. It has two optional arguments; 'type' and 'rotation'. Both are optional, by default it is type 'manual' and rotation '7'.
You can now create a snapshot:
/usr/local/bin/create-snapshot.sh
Example output:
INFO: Start OpenStack snapshot creation.
SUCCESS: Backup image created and pending upload.
The server status in Skyline will also change:
While the snapshot is created you cannot create a new snapshot, or do other management actions via Skyline. To reboot your server during snapshot creation, use the command line (reboot).
The image will start to upload in Skyline:
Backup Snapshots
Now that the script, credentials and dependencies are installed we can set up the automatic backup snapshots. Open the following file in an editor:
nano '/etc/cron.d/snapshots'
Place the following:
# Daily snapshot
1 1 * * * root /bin/bash /usr/local/bin/create-snapshot.sh daily 7
# Weekly snapshot
1 1 * * 0 root /bin/bash /usr/local/bin/create-snapshot.sh weekly 4
# Montlhy snapshot
1 1 1 * * root /bin/bash /usr/local/bin/create-snapshot.sh monthly 12
# Yearly snapshots
1 1 1 1 * root /bin/bash /usr/local/bin/create-snapshot.sh yearly 2
If you do not need yearly snapshots, just remove the lines.
With cron, you can also setup other schedules. If you have an important server where people work on during the day, you can set up a snapshot every two hours during work hours. That way, you never loose more than two hours of work, instead of a whole day.
# Snapshot past every 2nd hour from 6 through 18
1 6-18/2 * * * root /bin/bash /usr/local/bin/create-snapshot.sh officehours 12
The above lines can be placed in the same cron file.
Snapshots before apt actions / updates
'apt', the package manager, has a hook named 'pre-invoke' (and 'post-invoke'). This hook is called before (or after) 'apt' calls 'dpkg'.
We can use this hook to create the snapshot before an 'apt' action, say an 'apt-get upgrade', 'apt-get install' or 'apt-get remove'. Create the following file:
nano /etc/apt/apt.conf.d/00openstacksnapshot
Insert the following:
DPkg::Pre-Invoke {"/bin/bash /usr/local/bin/create-snapshot.sh apt 5";};
This will create a snapshot before the actual action is done, allowing you to revert to a snapshot after a (failed) upgrade. For example, when the kernel is not working or with other issues.
Try it out by installing a package. You should now see the output as in the above screenshot added to the 'apt' command output.
Restore
The most important thing of backups is off course that they can be restored if needed.
Via Skyline you can create a new server based on an earlier snapshot image. Unassign the floating IP from the old server and assign it to the new server and bob's your uncle.
If you don't use a floating IP, but a public IP then you need to use the command line to reinstall the server.
You need a Linux machine with the command line tools and credentials installed. Follow the above instructions for that. When installed, you can use the below command to reinstall the instance:
nova rebuild --poll "INSTANCE_UUID" "SNAPSHOT_IMAGE_UUID"
Replace the parameters with the image and instance UUID's. Do note that all data on the server is deleted.
Conclusion
We advise you to regularly test if your backups are still working. Do a test restore at least once a month. You not only know that the backups are still working but you also have the knowledge to do a restore when needed.
Do you need help setting up snapshots? Or do you want more information about backups? Please let us know, contact us via support@cloudvps.nl or by phone.
More information on the nova command can be found here: http://docs.openstack.org/cli-reference/nova.html. A more technical article can be found here: https://raymii.org/s/tutorials/OpenStack_Quick_and_automatic_instance_snapshot_backups.html