Stackato not only supports developers in pushing apps using any stack, but it is also built using many different technologies and programming languages. Ruby is still the predominant language for these technologies, with Python, Node.js and Go chasing closely behind.
In this blog post I am discussing how Ruby is used in Stackato. This will give some insight into the Ruby cogs that are turning in the background when you deploy your application on a Stackato cluster.
History
When ActiveState started working with the Cloud Foundry code base, everything was written in Ruby and Rails played a major part. We stripped it down and gave it a fresh set of clothes, but found the Ruby core to be a great language due to is flexibility and the speed at which you can develop. We definitely wanted to continue to build and grow Stackato in Ruby.
The A-Team
The Stackato development team is not just comprised of Ruby developers. We are polyglot developers building a polyglot platform for polyglot developers. We have a strong core of developers at ActiveState that can engineer themselves out of any situation. Picture the A-Team, where Murdock is a Ruby programmer, Face codes in Node.js, and Mr T compiles Go. Hannibal would probably be the voice of experience resulting from ActiveState supporting Perl, Tcl and Python developers for the past 15 years.
What is Polyglot?
The term "polyglot" in this context encompasses the utilization of many programming languages and technologies. Modern developers should be thinking in these terms, understanding the strengths and weaknesses of each and choosing the right tool for each job. These days, your average stack could easily consist of a few Node.js processes, a Rails app, and some backend workers written in Go or Python. A NoSQL database might sit along side a relational database, and a graph database might be needed as well. Data might passed over a message queue like RabbitMQ, and persisted in a data-store such as Hadoop for offline processing using Java map-reduce jobs. All these technologies can work together, and each does something better than the others.
Ruby, Ruby Everywhere
So, where can we find Ruby in Stackato? If I were to answer that fully, it would have to be a multi-part post. I have to get back to writing Ruby, so I will focus on the key components.
The Cloud Controller
This is at the centre of Stackato. It is a Rails application running behind Thin, which serves up a pure API. I say "pure API" because it is not concerned with assets such as JavaScript, CSS or images. It does not output any HTML for the web console. It simpl receives JSON HTTP requests and responds with JSON. Sounds pretty simple, eh!
EventMachine
The Cloud Controller runs behind Thin. If you are not familiar with Thin, it is built on top of EventMachine and enables Rails to receive and process HTTP requests asynchronously. I would guess that it is often the case that Rails developers use Thin and benefit from asynchronous HTTP request processing on the front-end of their Rails application, but then do little asynchronous processing within their application. I say this because adding Thin to a Rails application is a one-liner, but asynchronous programming is hard.
Within the Cloud Controller, EventMachine is used extensively beyond just receiving the HTTP request. Anytime we talk to a network connection, talk to the disk or spawn any external process, we do it in an asynchronous manner using EventMachine. When you push an application for the first time, it can be a heavy-weight operation. The Cloud Controller has to orchestrate a lot of things in the background. Some of these things are quick and some are slow, but if we were to block on any of them, then the Cloud Controller would quickly become unusable.
NATS
NATS is a "lightweight cloud messaging system" written in Ruby. The Cloud Controller is able to communicate with all the other components running within the Stackato cluster in a decoupled way by using NATS.
Messages are fired off and are received by the other components. When the other components have completed their tasks, they send a message back on the NATS bus. Other components have the option to listen to what is happening on the NATS and perform peripheral tasks, such as ensuring the DNS is correctly configured for deployed applications, logging activity, or managing scalability.
The NATS client that the Cloud Controller uses is also built with EventMachine, which means communication is asynchronous and does not block the Cloud Controller which can immediately handle any NATS messages that are pushed to it.
Even though Derek Collison describes his Ruby implementation of the NATS server as a "lightweight cloud messaging system" (or the longer form "lightweight publish-subscribe and distributed queueing messaging system"), he said it could be even lighter if he were to rewrite it in Go. Hopefully this will be a drop-in replacement once it is ready.
NATS usage is not limited to Ruby. We use NATS in almost all of Stackato's components. There are clients for Node.js, Go, Java and more.
Doozerd
Doozerd is not written in Ruby, it's written in Go, but we use it heavily across Stackato to manage configuration. Doozerd is a distributed versioned key-value data store that uses the Paxos algorithm to ensure that the configuration state is consistent across all doozerd nodes. With doozerd, we are able to push configuration updates to components in real-time. Using ephemeral values in the configuration state that are linked directly to our running process, we are able to receive instant notifications if a process dies.
Most of the manipulation of Stackato's configuration state, which is stored in doozerd, is managed by Ruby processes. We use both the fraggle and fraggle-block gems, which are clients to Doozerd. Fraggle is an asynchronous client built on top of EventMachine. Since we use EventMachine everywhere, this particular doozerd client gem gets the most usage within Stackato. The asynchronous nature of the fraggle Ruby client enables us to push configuration updates into our Ruby processes and have them take instant affect.
Kato
Kato is Stackato's trusty sidekick. It is a command-line tool present on all the virtual machines ("nodes") in a Stackato cluster. When you ssh into a node, most tasks you need to perform are covered by kato. For instance, you can join the node to an existing cluster, define the roles of the node, check status of the cluster, restart processes, assign administrators, update configuration settings, dump diagnostic reports, or run data exports. The list of tasks that kato performs is quick long, so check out the kato docs for more information.
Yes, kato is written in Ruby. It also uses EventMachine and talks to Doozerd using the fraggle client.
DocOpt
One recent addition to kato is the Ruby gem docopt, which we are very pleased with. Previously, we used OptionParser, and when we needed to handle subcommands we turned to Subcommand, which works with OptionParser. It did not scale well across kato, and the overall management of our options became unwieldily, even in simple cases. While searching for alternatives, we found docopt and have not looked back.
The great thing about docopt is that you write a usage file, which is human-readable and is shown to the user. The big win is that this usage file is the specification for the option parser. It's that simple. This means our documentation team can be more involved with constructing these usage files, which can also be repurposed for generating parts of the online documentation. Thanks to docopt, our kato option parsing is now more manageable, DRY, and concise.
Docopt is also available in Python, CoffeeScript, JavaScript, PHP, Bash, Lua and C. This makes it ideal for the polyglot environment at ActiveState's Stackato team.
Other Ruby Components of Stackato
There are many other components within Stackato written in Ruby. The Droplet Execution Agent (DEA), the Stager and service gateways that let your application talk to MySQL, MongoDB, and numerous other backends, are all written in Ruby. There is a Health Manager, written in Ruby, which monitors application health and will inform the Cloud Controller of anything that it needs to be aware of.
Conclusion
I have given you a taste of how Ruby is used in running a Stackato cluster. It is all so that you can easily deploy your own Ruby applications to your own cloud, whether it is with Rails, Sinatra, or Goliath.
-->
Trackback URL for this post:
http://www.activestate.com/trackback/3638