Back in late 2012 when I was working on building kato, the system administration tool for Stackato, we were looking for the perfect option parser. Up to that point kato was written in Python, but for various reasons we decided to jump ship and write it from scratch in Ruby. The number of options that kato had was already getting overwhelming, so it was important to find a Ruby solution for managing command-line options that would scale.
As you can probably guess, the discussion on ActiveState's development list for "What's a good option parser?" was lively. Everyone had their favorites. Maybe they had used in the past. Maybe they had one that was "only supported language X, but language X is better than Ruby anyway :)" Each candidate parser seemed to have its caveats and short-comings.
Since we already had a set of options that kato supported, we wanted to continue to support this if possible. We did not want to rewrite the interface to kato based the constraints of a new option parser.
Then somebody recommended "docopt". Nobody had heard of it.
Inside Out
The first most interesting thing about docopt was that it worked in the opposite way to most of other option parsers. Instead of defining everything in code and having the code generate the usage documentation, the usage documentation is written first and this is the specification of the acceptable command-line arguments. I thought this was genius and it would definitely be maintainable at scale.
Here is an example usage file from kato.
Language Agnostic
When choosing an option parser for kato, I was in the process of porting kato from Python to Ruby. Therefore, the possibility that one day we might port kato to another language was not out of the realm of possibility. At that time docopt supported Ruby, Python, CoffeeScript, JavaScript, PHP, Bash, Lua and C. It now also supports C#/.NET and Go.
Support across multiple languages means that you can take your usage files and support the exact same interface in multiple languages. The parsing and error handling is all contained within the docopt implementation, so there is much less code to port over than there would be with other option parsers. Unfortunately, our Python version of kato had not been using docopt.
Command Hierarchy
The first thing you might notice is that the above "kato config" usage file is rather small and succinct. This usage file only covers one aspect of kato and we have built a hierarchal structure into kato, which is not natively supported with docopt.
We built a wrapper around our docopt usage, so that we can direct it at the appropriate usage file, if that usage file exists. This is done with a simple directory hierarchy and paired "usage" and "cmd.rb" files. The "usage" file is the docopt usage file and the "cmd.rb" implements the kato functionality for that part of the CLI.
kato's commands are generally structured as "kato
". In the example above, the noun being "config" and the verb being "get", "set" or "del" (delete) etc. It's not a strict rule, but it is a good guideline. Some commands fit a different structure, such as "kato
" or "kato
". We want consistency, but not at the cost of our system administrators having to type long-winded commands for common actions. For instance, we chose "kato start" and "kato stop" over "kato node start" or "kato role start".
Here is a complete list of the current 60 usage files that make up kato's user interface.
Some verbs, such as "kato role add" get their own usage file, because, like with our Ruby code, we want to limit the size of the usage file to something easily consumable. We do not want kato's help output to be too overly verbose. This would be the case if we listed the options of each verb and each verb took a lot of options.
If the user is only trying to use one specific action for a kato command, then they only see the options for that action, unless the total options for a command is quite small. Typing "kato role add --help" or "kato role add -h" or "kato role add help" or "kato help role add" will bring up just enough usage information for this specific command.
For some commands, such as the above "kato config", we have not yet felt the need to break out the verbs into their own usage files. If I type either "kato help config" or "kato help config set", I'll see the same "kato config" usage file output to my terminal.
So what happens if I type "kato help role" if there is a usage file for each verb? We put additional simple usage files higher up in the directory structure. These just list the commands. This could be optimized with a little bit of parsing of lower level usage files and auto-generation, but it has not become a maintenance issue yet. Also, we can write much clearer help text in the higher-level usage files, which gives the user a nice overview of what that area of kato does.
Where we do optimize is at the very top layer usage file. If you type simply "kato help" you will get a usage file that lists all the top level commands that kato supports. This is generated from an ERB template so, unlike the other static usage files, this does not port over to other languages as easily.
kato fetches the name and first line description of any usage file it finds in the first level directory. It then prints them in name-sorted order.
Generating Documentation
Stackato's documentation is traditionally written in a markup language called "Sphinx". From here we generate the HTML documentation that you see at docs.stackato.com or bundled with the Stackato virtual appliance (go to /docs in your Stackato web console).
We wrote a tool to parse the all the usage files and generated kato reference documentation as Sphinx files. This pleased the documentation team immensely. They were able to integrate the kato Sphinx file generation into their own documentation build process. Now the documentation always 100% reflects the usage that kato accepts. This is true even if you are working on a git branch.
We still wanted the documentation team to be responsible for the documentation, so we gave them ownership of the usage files. They are responsible for fixing descriptions or commands and options. They were given clear guidelines about what changes would have what effect and how the docopt usage files were structured. So far, this collaboration around usage files has worked well.
Secondary Validation
Simple flags like --no-color need no extra intervention by the developer. They can simple use the boolean options[:no_color] in their code.
This also applies to enumerated lists of options such as "(apple|banana|orange)". docopt will reject anything not fitting this clear list of possible values.
Where we need secondary validation, is where we accept values, such as node IP addresses.
Many options in kato come up again and again. Therefore, we have plenty of help functions for the validation.
Secondary Parsing
You will notice from the above examples that, for instance, where the option "-n --node
" is given we get options[:node] in our Ruby code. Where we see --no-color, we get options[:no_color]. This is because kato normalizes the arguments that docopt returns. The output for docopt is fine for quick usage, but we wanted a little cleaner and consistent Ruby syntax if we were to use this across so much of our code. We prefer options[:no_color] over options['--no-color'], so our post-docopt options processor gives our Ruby this format.
A Simple Example
Here is the simplest example I could find in kato of a usage file with its relevant cmd.rb file. It will output the list of the log drains known to the Stackato cluster. The output will either be in YAML format, which is the default, or in JSON format.
usage file:
cmd.rb file:
Interactive Shell
docopt.rb gave us a way to parse the usage files to some extent, but we needed to do a little extra work to expose all the details we needed in order to build "kato shell".
"kato shell" is an interactive version of the kato CLI. Simply type "kato" or "kato shell" on the command-line and it will drop you into the shell.
The shell provides tab completion and a way to see which commands are available based on what has been typed so far. It is much easier to hit the tab key to see what sub-commands are available, or typing "--
" to see what options are available, than going back to the docs every time. It also helps with navigating through Stackato's cluster configuration.
Being an interpreted language, Ruby is not very fast when it comes to boot up time. We see over half a second overhead in starting up the kato Ruby process. With the kato shell everything is already loaded, so it makes things snappier when navigating around the commands and administrating your Stackato cluster.
Ruby
Stackato is built on-top of the Cloud Foundry open-source project. Ruby is still a large part of the Cloud Foundry code base, including the Cloud Controller. This was the major factor in choosing Ruby for kato. We wanted to expose some of kato's functionality through the Cloud Controller's REST API. Being able to drop a piece of kato's Ruby code into the Cloud Controller made sense and enabled us to keep things DRY.
Go
The Cloud Controller component of the Cloud Foundry project is still written in Ruby. I do not see this changing anytime soon, though things are moving quickly. We are seeing several other components receive Go love. Go seems to be the goto language for writing distributed systems these days and for good reason.
I was pleased to see a Go version of docopt appear on my radar last week, which is what led to this blog post.
Conclusion
For the past 18 months that we have used docopt, there have been few complaints. Only when we wanted to port a piece of functionality to kato, and the interface was non-standard, have we seen issue. I see this as a benefit. It ensures consistency across the user experience. docopt provides enough features to handle most situations, if approached in the right way.
My hope is that docopt will gain more adoption with newer command-line tools. I find docopt to be one of those clean separations of concerns, such as we have seen with code and view templates or HTML and CSS.
Follow @philwhln
Related Posts
Ben Golub Explains Docker Inc
In July, Ben Golub joined as CEO. This enabled Solomon Hykes, founder and then CEO, to become CTO and truly focus on the technology.
Last week, I met with Ben Golub to find out more about Docker Inc and where they see things going.
Read more...
Alex Polvi Explains CoreOS
A couple of months ago we interviewed Solomon Hykes about Docker, which is a way to build and manage Linux Containers with a lot of nice features. The next question was: if the full-stack can be provided by a Docker image and everything can be Dockerized, what is the minimum OS we need to run Docker images? CoreOS was announced a few weeks ago and seemed to answer this question. For this blog post I interviewed Alex Polvi, the CEO of CoreOS, to find out more.
Read more...