At Delphi in Ancient Greece, where Apollo wrestled a great snake [1], many came to visit the oracle known as the Pythia. After breathing in the mind-expanding vapors that wafted through the temple, she would pronounce obscure prophecies, and everyone would spend their Sunday afternoon trying to interpret them, because they didn’t have search engines in those days.
I started working at St James shortly thereafter, and at the time everyone was very excited about Python egg files – convenient bundles of Python libraries that you could install using a new tool called easy_install from the setuptools library. Then (as far as I can tell) setuptools fell into disrepair, and developers discovered you couldn’t easy_uninstall, which was a bit inconvenient. Let’s just say some people shared their frustrations on the interwebs. [3] As it happens, some good people rallied round and forked the project, and made a new library called distribute that remained backwards compatible with setuptools and fixed a number of bugs. Meanwhile, development had also started on the pip library, which could install and uninstall, and the virtualenv library, which could cocoon your development installations from your operating system libraries and each other. To put a cherry on top, the virtualenvwrapper library made managing virtualenvs really easy. (I leave bundling libraries into a Windows installer as an exercise for the reader.)
Oracles promised great things for the future of Python packaging, and prophecised that a new packaging framework, distutils2, would eventually be merged into Python 3. With the best of intentions, I added their links to our internal Developer Guide. However, some things didn’t turn out quite as planned and some things have turned out better than planned:
the plan to include the new, improved distutils2 in Python 3 was abandoned
a number of other packaging projects have tackled the problem from different, but overlapping, angles
distribute is in the process of merging with setuptools
packaging summits were held at PyCon 2013
the Python Packaging Authority (PyPA) forked the existing packaging guide and created the (informally) authoritative Python Packaging User Guide (PPUG) [4]
For the moment, there is nothing wrong with the winning formula of virtualenvwrapper + virtualenv + pip + distribute, especially on Ubuntu 12.04 LTS, our reference platform for development. But in case you’re curious about what the future holds, I took a sniff at what the PyPA had to say. Their key insights are:
egg was a silly name for a package file. The new distribution format will be called wheel.
Packaging is a task that supports the distribution of files. Distribution occurs at multiple phases in a project’s lifecycle, each phase has different requirements: [5]
Development: working with source code in a VCS checkout
Source Distribution: creating and distributing a source archive
Building: creating binary files from a source archive
Binary Distribution: creating and distributing a binary archive
Installation: installing files from a binary archive onto the target system
Execution: importing or otherwise running the installed files
It would be best to decouple these phases, rather than try to make a single tool to perform all of them.
Multiple packaging tools already exist, each tailored to meet different needs/support different distribution phases. Any future solution should be based on a framework that facilitates multiple, cooperating tools.
Library packages should have useful metadata, which is securely distributed. (As a bonus, this would aid packaging of Python libraries across multiple GNU/Linux distributions.)
We should aid the exiled Alcmaeonidae family in overthrowing Hippias.
This will all come to pass in Python 3.4.
For more details, read the evolving and informative Python Packaging User Guide.
[1] It wasn’t too hard because he used the interactive tutorial. [2]
[2] There’s also a text-based tutorial and an opinionated guide that orientates you about things like PEPs, decorators and common gotchas.
[3] Quoting the benevolent dictator of another project, “history suggests that open source communities are prone to hystrionics up front and pragmatism in the long term”.
[4] Links for the Python Packaging Authority: Google Group, Bitbucket repos, GitHub repos
[5] List taken from The Phases of Distribution (PPUG)