Forward
Imagine a world of interconnected Clouds capable discovering, coordinating and collaborating in harmony to seamlessly carry out complex workloads in a transparent manner -- the intercloud. While this may be the dream of tomorrow, today's reality is a form of the intercloud called hybrid Cloud. In a hybrid Cloud model organizations manage a number of on-premise resources, but also use off-premise provider services or resources for specific capabilities, in time of excess demand which cannot be fulfilled via on-premise resources, or for cost effectiveness reasons. Both of these Cloud computing models have a common conduit to their realization -- open standardized APIs, formats and protocols which enable interoperability between disparate Cloud deployments.
Enter OpenStack -- a rapidly growing open source Cloud framework which removes vendor lock-in by providing open, vendor agnostic APIs. OpenStack consists of a number of distributed, scale-out ready services which allow organizations to build Cloud solutions in a more cost effective manner. Just as the name implies OpenStack is open source, open design and open teaming / leadership. As shown in a 2013 Forrester study, collectively OpenStack becoming the most popular private Cloud of choice (see diagram below).
SoftLayer is the premier Data Center and hosting provider in the industry today offering numerous pre-built managed services and solutions to fit many needs -- from web hosting to private hosted Clouds -- SoftLayer is the right choice. Not only does SoftLayer provide soft services, but they also offer fully dedicated bare metal hardware suitable for the most demanding workloads. SoftLayer also supports a comprehensive set of native APIs allowing developers and operators to take programmatic control over their resources. Wouldn't it be great if you could use OpenStack APIs with SoftLayer? As luck would you have it, you now can.
The folks over at the SoftLayer innovation team recently released a new open source project called Jumpgate. With Jumpgate you can use a number of your favorite OpenStack tools and APIs to bridge directly into SoftLayer. For example you can use the OpenStack nova CLI and APIs to boot SoftLayer virtual compute servers, or the OpenStack glance CLI to manage SoftLayer based images. You can even use the OpenStack Horizon dashboard to graphically manage a number of your SoftLayer resources including virtual servers and images.
In this article we'll cover the following topics:
An overview of Jumpgate including its high level architecture.
Step-by-step instructions on how to install and configure Jumpgate including creating an upstart job to manage Jumpgate as a service.
Detailed instructions on how to install and use the OpenStack nova CLI with Jumpgate.
A guide to setting up the OpenStack Horizon web-based dashboard with multiple regions so that you can manage SoftLayer resources from a single pane of glass.
Introducing SoftLayer Jumpgate
Jumpgate is an open source project started by the folks over at SoftLayer innovation labs which aims to bridge the gap between OpenStack native APIs and other proprietary vendor APIs -- an API adapter proxy if you will. Jumpgate is implemented in the Python programming language as a WSGI application and thus runs as a web based service. As a fully compliant python WSGI application, Jumpgate can be hosted via a Python server such as gunicorn or fronted by your favorite web server with WSGI support such as Apache with mod WSGI or nginx with NgxWSGIModule.
From an architectural perspective Jumpgate is stateless (no persistence) and lightweight making it easy to scale out suiting the needs of many deployment scenarios. Under the covers Jumpgate is pluggable, allowing you to easily snap-in or snap-out both request and response middleware "hooks" and service drivers.
Jumpgate request and response hooks permit you to write and plug-in Python code which can work with requests and responses as they enter / leave the Jumpgate application. For example you could easily implement a hook to support API rate limiting or fully featured request auditing. Out of the box Jumpgate includes a number of pre-built hooks including logging and authentication.
Jumpgate service drivers provide the functionality needed to translate OpenStack native API requests into native APIs requests and back again as OpenStack compliant responses. Think of these drivers as adapters. These drivers are pluggable per OpenStack service endpoint supported by Jumpgate -- image, compute, network, identity, etc.. Out of the box Jumpgate provides a set of SoftLayer drivers which allow you use a number of OpenStack client side tools immediately with SoftLayer Jumpgate. It's also possible to implement and snap-in your own drivers to support bridging between OpenStack APIs and your native API.
From an identity perspective, Jumpgate also includes a number of other plug points providing maximum flexibility for your solution needs. For example the Keystone compliant service catalog served by Jumpgate is based on a template file which look much like the Keystone default catalog. These endpoints can be modified to suit your needs and you can also plug in other remote service endpoints you wish Jumpgate to expose in the service catalog. Jumpgate also includes pluggable token and authentication drivers allowing you to control how authentication requests are fulfilled, as well as how tokens are structured, encoded and decoded.
The diagram below depicts the high level architecture of Jumpgate. On the left side of the diagram the generic form of Jumpgate is illustrated whereas on the right the SoftLayer specific drivers which realize the bridge are shown. You can also check out Nathan Beittenmiller's SoftLayer blog on Jumpgate.
From a development perspective, Jumpgate is a young active project which is growing / enhanced daily. As such, not each and every OpenStack API is supported with the out of the box SoftLayer drivers today. Instead, the most common OpenStack APIs were selected and initially implemented with additional APIs being added incrementally until the bridge is sufficiently supported. For a more complete list of the APIs supported, see the "compatibility matrix" on the Jumpgate developers page. That being said, please do file issues if you run into any bugs or if you have feature requests. Moreover, we are always looking for contributions so if you are feeling Pythonic please help out by following the contributors guide.
Let's now move to the hands-on instructions and get Jumpgate running.
Installing & Setting up Jumpgate
For this guide we'll be setting up Jumpgate and related components on a SoftLayer Cloud Computing Instance (CCI) which is running the Ubunutu 12.04 Operating System. However, you could just as easily provision Jumpgate off-premise SoftLayer as the SoftLayer API (SLAPI) endpoint is public facing and reachable over the public network. Moreover a number of the exact commands shown here are Ubuntu specific (i.e. apt-get, initctl, etc.), but they can easily be translated into your distro of choice.
In the steps that follow I will provide snippets from my terminal which include my shell prompt, followed by the exact command(s) run. I will also show an abbreviated section of the command's output using "..." to indicate additional output is removed for brevity. I tend run multiple commands on a single line using the bash && operator. This operator can be used to chain multiple commands on a single line and indicates bash should execute the commands left-to-right, one-by-one only running subsequent commands in the chain if the previous executes successfully. If a command should fail, you can address the failure and run the remaining commands one-by-one. Finally I'm performing all steps in this guide as a non-root user and thus use sudo to execute commands as root.
Before you begin
In order to prepare for the Jumpgate installation and setup, take a moment to consider the topology you wish to realize with this solution. In particular consider the following questions:
Will the components consuming Jumpgate be located on the same host as Jumpgate, or will they be remote?
If the components consuming Jumpgate are remote, which IP / interface will they be using to communicate with Jumpgate?
Which port will Jumpgate bind to on your host system? If you are planning to run OpenStack Keystone on the same system using Keystone's default ports, you will need to select a different open port for Jumpgate to bind on.
As a result of the questions above, you will need to determine the IP address Jumpgate will bind and listen on. If your solution will always have the Jumpgate consumers on the same host, you may choose to bind Jumpgate to the loopback IP of your host (localhost, 127.0.0.1) and thus not expose it externally. However if your Jumpgate consumers will be remote, you may want to restrict their access to the private IP of your host in which case Jumpgate would bind to the private IP / network.
The IP address you wish to bind Jumpgate on is referred to as the JUMPGATE_IP in the instructions below. Any place you see JUMPGATE_IP, substitute the IP Jumpgate will bind to.
Likewise the port you will bind Jumpgate to is referred to as JUMPGATE_PORT below. Substitute your selected Jumpgate bind port accordingly.
Install Prerequisites
The first thing we need to do is to install some prerequisites (dependencies) needed by Jumpgate and it's python requirements. Note that NTP is not required by Jumpgate, but my preference is to keep server times in sync and thus I illustrate its setup here.
Configure NTP for SoftLayer
Now we'll configure the NTP service on our system by pointing it to the SoftLayer time server which is available on the private SoftLayer network. If you're configuring Jumpgate off-premise SoftLayer (without a private link to SoftLayer's network) you'll want to substitute the SoftLayer fully qualified hostname for a NTP sever reachable from your host. More details on SoftLayer's time service can be found on the SoftLayer blog.
Update Python pip
Earlier we installed python pip using apt-get. However the version of pip installed in that step is not the latest. As some python dependencies require the later versions of setuptools which is included with pip, we'll manually upgrade pip in this step to get the latest version.
Clone the Jumpgate source code
We are now ready to get rolling with Jumpgate installation. Since we are installing Jumpgate from source we first need to use git to clone the github repo. For this guide I have chosen /usr/local/ as the root directory for the source.
Install Jumpgate Python Requirements
Python based projects often have dependencies on other python libraries / modules. Such dependencies are often times explicitly expressed using a 'requirements' file which lists all the py dependencies and optionally and version requirements per dependency (e.g. module foo requires module bar > 1.1).
We will use the pip command to install Jumpgate's python requirements from file. These requirements are defined in the tools/requirements.txt file within the Jumpgate source code project.
Install Jumpgate
We're now ready to install the Jumpgate python code itself. We'll use the standard python means of installing a project using the setup.py included with the project.
Create a service user and group
For this guide I've chosen to setup Jumpgate as a service which can be managed via Ubuntu's upstart. Using upstart you can manage jumgpate as an actual service including automatic start-up of the service on Operating System boot. My preference is to not run services as root unless absolutely necessary and thus I'll be creating a new group called srv and a service user to run Jumpgate called jumpgate.
Setup an etc directory for Jumpgate
Jumpgate requires a few configuration files which allow you define properties for Jumpgate's runtime behavior. Rather than using / referencing the configuration files under the Jumpgate source directly, we'll create an /etc directory for them in accordance with the standard file system layout for Linux applications.
First, let's create an etc directory for Jumpgate and setup the directory permissions:
Now let's copy the Jumpgate configuration files from the source project into the etc directory and change the permissions:
Configure Jumpgate identity templates
As Jumpgate provides a set of OpenStack Keystone compliant identity APIs, it must also host the service catalog which defines the service endpoints available to consumers of the API. Jumpgate allows configurable service endpoints via it's identity.templates file similar to how OpenStack Keystone's default_catalog.templates works. In this file you can define the services and their endpoints as returned in a Jumpgate identity catalog based response (such as a successful response to obtain a token).
By default these endpoints refer to localhost and port 5000 which is the default bind address / port for Jumpgate. However as discussed in the 'before you begin' section you may need to change these based on your desired topology.
Keep in mind these endpoint services and URLs will be followed by OpenStack compliant tooling such as the CLIs. Therefore the host / port should refer to Jumpgate bind address / port itself. Moreover note if you were developing a solution using other OpenStack services, you could define their service and endpoint in the catalog file to serve them from Jumpgate's identity catalog. In this case you would use the base URI to the service you are integrating.
First replace localhost with the IP you are binding Jumpgate on (remember to substitute JUMPGATE_IP in the command below with the IP you want Jumpgate on).
Next replace the usage of port 5000 with the port you wish to bind Jumpgate on (remember to substitute JUMPGATE_PORT in the command below with the port you want Jumgpate to listen on).
Setup Jumpgate configuration file
We are now ready to setup the jumpgate configuration file which defines various properties to control Jumpgate's runtime behavior. Jumpgate uses a standard ini formatted file just like all OpenStack proper services. There are a handful of properties you can change in this file, but for this sample we'll only edit the few necessary to get our demo working.
First we'll replace all usage of the loopback address (127.0.0.1) with the IP address we want Jumpgate bound on. You can do this manually using your favorite editor, or you can use the 1-liner below to do it for you. Remember to replace the usage of JUMPGATE_IP in the command below with the IP address you want Jumpgate to listen on.
We now need to setup an 'admin token' and 'secret key' for Jumpgate. The admin token is similar to OpenStack Keystone's admin token in that it can be used as the X-Auth-Token header value to gain authenticated access to Jumpgate -- think of this token as a special password for Jumpgate. Note that using this admin token does not signify you have access to the 'backend' API's Jumpgate is integrating with. Rather it signifies you can authenticate with Jumpgates authentication middleware permitting you to access the API controllers in Jumpgate.
Jumpgate contains a set of pluggable identity drivers which permit you to easily write and plug-in your own authentication, token and token encryption schemes. By default Jumpgate uses AES encryption to encrypt / decrypt the authentication token IDs it generates for it's identity driver. This AES support requires an AES key to seed the algorithm -- this is the 'secret key'. For this sample we'll generate a 16 char key, but feel free to generate your own AES compliant key and use it.
Here's one way to generate a random 16 character key:
The output from the command above can be used as the SECRET_KEY shown below.
We now need to edit the /etc/jumpgate/jumpgate.conf using your favorite editor (note if you are not running as root you'll need to use sudo to edit the file) and set the secret_key and admin_token values. Again the secret key is a valid AES key, and admin token is a random string of your choice. Below is a sample snippet of the conf file after it has been updated. Save the file once you have edited it.
Create a Jumpgate upstart job
To ease management of Jumpgate as a service and enable features such as auto-start at Operating System boot time, we will create an upstart job. Upstart is Ubuntu's init daemon and requires you to create a conf file to define a job. Using your favorite editor, create the file /etc/init/jumpgate.conf with the contents shown below. Note - you may need to use sudo to edit the file you do not have proper authority. Also remember to replace JUMPGATE_IP and JUMPGATE_PORT with the IP address and port respectively Jumpgate will listen on.
This conf definition sets up the logic necessary to manage Jumpgate via upstart. As you can see in the definition, we kick-off the gunicorn processes the runs the Jumpgate WSGI application as the jumpgate user. This job will auto-start on system boot after the filesystem and network interface becomes active.
Start Jumpgate
We are now ready to fire-up the Jumpgate job using initctl.
If you want to double-check that jumpgate is running you can use the initctl status command as shown below:
Testing Jumpgate with OpenStack nova client
At this point Jumpgate should be running and ready for action. To test it out and also demonstrate how you can use the OpenStack tools with Jumpgate, let's install OpenStack nova client and perform some operations.
Clone nova client source
We will be installing nova client from source and thus we need to clone it's github repo. For many Linux distros native packages exist which permit you to install nova client and other OpenStack components with you package manager. However my preference is to use the latest source, so let's get on with it and clone the repo.
Install nova client dependencies
Just as we did when installing Jumpgate from source, we now need to install nova client's python based dependencies based on its requirements.txt file. The command to do so is shown below.
Install nova client
We are now ready to install the nova client python source. To do so we'll follow the same process we did for Jumpgate by running the setup.py script.
Setting up your variables for use with nova client
We're ready to start working with nova client, but we first need to collect the variables we need to authenticate with Jumpgate. Just as with any other OpenStack client / service we need the following information:
Authentication URL --- This is the base endpoint URI to the Keystone compatible service which provides the identity APIs for the deployment. In our case this is the identity URI of Jumpgate as Jumpgate is serving as Keystone here.
Username -- Your username which serves as your identity in the authentication protocol implemented by the Keystone compatible service. Here we are using Jumpgate to bridge into SoftLayer and thus we'll our username we provide when logging into the SoftLayer portal.
Password -- The password associated with the username for SoftLayer. In this case we can use our SoftLayer password or API key. A SoftLayer API key can be obtained from the user management portion of the SoftLayer web portal. For more details on the SoftLayer API key see the SoftLayer documentation.
Tenant -- The project (aka tenant) your user belongs to. Here again since we are bridging into SoftLayer our tenant ID is our account ID in SoftLayer. Your account ID is shown in the upper right-hand corner of the new SoftLayer web dashboard and is typically a 6 digit number.
Now that we have the required identity information, we can use these values with nova client (or any other OpenStack client we use with Jumpgate). My preferred way of setting these is to define them as env vars in a file which can then be sourced. Using this approach you do not need to specify these values as parameters on each invocation of the CLI. For more details on the env vars for OpenStack env vars, see the OpenStack documentation.
Create new file in your home directory such as ~/jumprc and add the contents shown below. Substitute the following values:
JUMPGATE_IP -- The IP address of Jumpgate.
JUMPGATE_PORT -- The port Jumpgate is listening on.
YOUR_SOFTLAYER_API_KEY -- Your SoftLayer password or API key.
YOUR_SOFTLAYER_ACCOUNT_ID -- Your SoftLayer account ID.
YOUR_SOFTLAYER_USER_ID -- The ID you use to log into the SoftLayer web portal.
<code style="background-color: transparent; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; display: inline; font-family: Consolas, 'Liberation Mono', Courie