In my current role, we've been using Ansible for our orchestration and
configuration management. Ansible is okay, but after several months wrestling
with its extension API and being frustrated by YAML syntax ideas started
popping into my head for something better.
Chopsticks represents my vision of Pythonic orchestration. It's not an
orchestration framework in itself. It's more the transport layer for other
orchestration systems. It's a remote procedure call (RPC) system that relies on
no agent on the remote host: the agent is built dynamically on the remote host
by feeding code over the SSH tunnel to the system Python.
For example, you can create an SSH Tunnel to a remote host:
Then you can pass a function (any pickleable Python function), to be called on
the remote host. Here I'm just calling the standard time.time() function.
Of course, you might want to do this in parallel on a number of hosts, and
this is also built-in to Chopsticks:
Note that the code for chopsticks.facts does not need to be installed on
the remote hosts. They will load it on demand from the orchestration host.
Effectively, Chopsticks gives you the ability to write Python programs that
straddle a number of machines, all sharing a single codebase.
See the README for a summary of how this works.
SSH tunnels are not the only connection type Chopsticks supports. It
communicates over stdin/stdout pipes, so can work with any system that supports
these without inteference. Such as Docker (this is on Github, but not PyPI
yet):
Why "Chopsticks"?
Chopsticks gives fine control at a distance - like chopsticks do.
Chopsticks vs ...
It's natural to draw comparisons between Chopsticks and various existing tools,
but I would point out that Chopsticks is a library, not an orchestration
framework, and I'd invite you to think whether other tools could benefit from
using and building on it.
Ansible
Perhaps the immediate comparison is with Ansible, because it is frustrations
with this that inspired Chopsticks.
Ansible feels a lot like Bash scripting across hosts, but in a warty YAML
syntax. So first and foremost, I'm attracted to the idea of describing plays in
nice, clean Python code. Python code is also more easily testable, and there
are great documentation tools you can use.
Ansible's remote execution model involves dropping scripts, calling them, and
deleting them. In Ansible 2.1, some of Ansible's support code for Python-based
Ansible plugins gets shipped over SSH as part of a zipped bundle; but this
doesn't extend to your own code extentions. So Chopsticks is more easily and
naturally extensible: write your code how you like and let Chopsticks deal with
getting it running on the remote machine.
Fabric
Fabric is perhaps more similar to Chopsticks - it's a thin framework around
the SSH transport, that allows scripting across hosts in Python syntax.
The big difference between Fabric and Chopsticks is that Fabric will only
execute shell commands on the remote host, not Python callables. Of course you
can drop Python scripts and call them, but then you're back in Ansible
territory.
The difference in concept goes deeper: Fabric tries to be "of SSH", exploiting
all the cool SSH tunnelling features. Chopsticks doesn't care about SSH
specifically; it only cares about Python and pipes. This is what allows it to
work identically with Docker as with remote SSH hosts.
Execnet
As I was sharing Chopsticks on the Twitters, people pointed out the similarity
to execnet, which I had not heard of.
Chopsticks has similarity to execnet, but from what little I've read it works
in a very different way (by shipping selected code fragments), and will not
allow importing arbitrary code from the orchestration host (ie. full import
hooks).
Future of Chopsticks
Chopsticks is open source under the Apache 2 license, and at the time of
writing, is at a very early stage - barely more than a proof-of-concept - but
under very active development.
It currently has support for:
SSH, Docker and subprocess tunnels
Python 2.6-2.7 and 3.3-3.6
Parallel execution
It needs:
Error handling
Documentation
Tests
Send/receive file streams
Higher-level orchestration functions
If Chopsticks looks interesting to you, I'd appreciate your feedback, and I
welcome any pull requests.