The Bodhi2/Taskotron
Fedora Activity Day happened earlier this month! A bunch of us gathered in
Denver for a few days and worked on some of our critical releng & qa
infrastructure. The hackfest was held in a conference room in my
apartment building, which worked out quite nicely for the amount of
people that we had. The hotel was right up the road, and we were able to
walk to a lot of awesome spots, like the 1UP Barcade :).
It was great to have folks from various corners of Fedora together in the same room for a few
days. As it is, we get a lot done remotely, but being able to leverage the
high-bandwidth face-to-face time is extremely valuable, especially when coming
to consensus on difficult architectural decisions. We used Gobby
to collaborate on a long list of action items, and chipped away most of it.
Thankfully, Bodhi has enough layers that we were all able to split up and dive
into different corners of the code without stepping on each others' toes.
Up until now, our releng stack in staging has always been less than useful.
We've never able to do a full build->update->push->run, and have had to rely on
testing certain codepaths in production. Not only that, but Bodhi's
"masher" never really had proper unit tests, so pushing out major changes to
that stack has always been quite unpleasant. Thankfully, Kevin and Dennis worked on our
staging setup and made it so we can actually use it. I made a lot of headway on
porting the Bodhi masher to a new fedmsg-driven
architecture, while writing unit
tests for it along the way. I'm hopeful that we can write tests for every
part of the "push" process, and then optimize the hell out of it :)
While Mathieu and I mainly
focused on back-end hacking, Ralph and Ricky made some fantastic headway on the
front-end. Ralph started working on the revamped New Update Form, which is
already significantly better than the original. The idea here is that the
maintainer should only need to provide the package name(s), and Bodhi will
automatically find the corresponding candidate builds, related bugs, and
eventually it will pull in all candidate deps as well (or tell if you if any
need to be rebuilt). It would also be very convenient to be able to "push this
entire stack again". Ideally, I'd love to make it so that folks maintaining
large stacks, like GNOME, shouldn't need to use a Google doc to coordinate
their mega-updates ;)
Ralph also started revamping the new karma system (check out his screencast here).
We don't have any of the policy in place to enforce it yet, but eventually
we'd like the maintainers to be able to define custom policy constraints
for their updates. For example, you could say "only allow this update to
go to the stable repo once this specific bug or test case has been
confirmed as fixed/passing".
Ricky made lots of improvements to the Release profiles and Updates/Comments/User pages, which are all looking great. He also created a Bodhi news feed on
the front page using the fedmsg datagrepper widget that
Ralph
blogged about recently.
Other front-end improvements include libravatar support for all
users, proper markdown rendering with previews and image support, and of course
a konami code easter-egg ;)
I was going to post a bunch of screenshots here, but Ralph just deployed a
development instance of Bodhi2 that folks can play around with
instead: http://bodhi.dev.fedoraproject.org
(it's a dev instance, so don't expect it to always be up/functional).
Some other corners of Bodhi that have seen improvements recently:
The API. The Bodhi webapp is written in Python using the excellent Pyramid
web framework. On top of that we are using Cornice, which
makes it really easy to build & document standards-compliant web services
with Pyramid. Thanks to colander
validation/deserialization schemas
and our custom ACLs
and validators,
we are able to write dead-simple
views that can safely assume that all of the data we are dealing with is
valid and in the right format, and that the user has the appropriate
permissions. Cornice also has a Sphinx plugin
that auto-generates our API documentation.
So not only do we now have a semi-RESTful standards-compliant self-documenting API, but Ralph also added a support for
rendering almost every service as an RSS feed as well.
Regarding the Bodhi Python
APIs, I've begun porting them to the new python-fedora
OpenIDBaseClient (see bodhi/client.py).
Since a large percentage of the API usage is through the current python-fedora
BodhiClient, I'd like to try our best to maintain compatibility--at least for a
version or two with deprecation warnings if we have to.
I am really looking forward to finally being able to trash our old TurboGears1 FAS visit/identity layer in favor of FedOAuth.
On top of the Python API lies the bodhi-client. I recently ported the
basic functionality over using the click module,
which makes it really easy to write complex command-line tools (see bodhi/cli.py).
Since the current bodhi-client is an absolute mess, this is one area that I'm
actually okay with breaking backwards-compatibility to a certain extent. Having
a proper command structure, similar to the Koji cli, is well worth some cli
flag changes in my opinion.
In a similar fashion, Mathieu implemented a great release
management tool for Bodhi admins. Currently, creating a release and
changing it's pre-release status involves using the TurboGears Python
shell, creating SQLObject instances by hand, editing config files, etc.
This tool will make it dead simple for releng to create new releases and manage
all of the pre-GA state changes that happen along the way.
Performance was another key area of development. The app is fairly snappy
now, but there is still a ton of room for improvement. The pyramid-debugtoolbar
has been amazingly useful for us so far. It let's us analyze every SQL
statement made, it does full-stack profiling, and it lets us execute commands
in every layer of a traceback. Along with that, Ralph added a SQLAlchemy event hook to our
unit tests to ensure that certain changes don't drastically change how many SQL
statements are getting executed. With regard to Masher performance, there is
still a lot of low-hanging fruit there. We saw a drastic boost in mash speed
recently when Kevin discovered that a couple of releng machines weren't using virtio.
This brought the mash time of EPEL5 updates from 45 minutes down to around 15.
In the new version of the masher, updates are grouped by tag and then processed in a
separate threads. Security updates and critical bugfixes will take priority
over enhancements, and there are also things that we can do to make the bits
hit the mirrors faster once we're done mashing.
Another corner of Bodhi that was the topic of discussion was around
notifications. Bodhi currently sends way too much email, most of which I'm
assuming gets ignored. There are a couple of mails that are a bit more
important, like the update announcement emails that get sent to package-announce,
and the updates-testing digest that goes to the test-list.
The consensus that we came to was that we are going to attempt to use FMN to allow
people to configure what messages they want to receive, and where (IRC, email,
etc). This alleviates the need to build an email layer into Bodhi2, and
allows us to focus on publishing fedmsgs only, letting FMN do the rest.
As far as the transition to Bodhi2 goes, we're going to try our best to not
break the world in the process. Ralph mentioned the potential timeline in his
blog post, and
we still have lots of work to do before then. In order to help ease this
transition, I created a wiki page to track the consumers of Bodhi's
APIs, so we can make the appropriate changes to those codebases before
launch. Please feel free to update it with any that I left out.
Overall, it was a very successful FAD. We got
a ton of stuff done, ate a bunch of great food, and had a lot of fun in the
process. I didn't cover everything that we worked on, so checkout the blog posts from threebean, bochecha,
and nirik
for more details on other things that got done. If you're interested in getting
involved with Bodhi2, grab
the code, checkout the open
issues, and hop in #fedora-apps on Freenode.
Also, I'll be giving a presentation at Flock in Prague this year on "Evolving the Fedora Updates process", which will cover the history of pushing updates as well as an in-depth dive into the new bodhi2 stack.