We just had a work-week in London on Kinto our
JSON storage project, and while heading back to France in the Eurostar, I figured
I'd give a summary of where we're at in the project and what's next.
If you are not familiar with Kinto, it's an HTTP API that'll let you store
(not so) arbitrary JSON content. Data is organized in collections, which are
basically a list of timestamped records. As a client, you can query Kinto
to get records filtered by dates or other fields. Kinto also provides neat
features like a diff-based API to grab only the newest data, like how you
would with a git client.
Kinto is the project designed to serve collections of data to Firefox in
real-time, such as configurations.
Kinto stores arbitrary JSON because like some other storage
systems, you don't really have to describe the data you are pushing in it like
in a classical Database (you can if you want Kinto to control inputs).
Kinto stores not so arbitrary JSON because we're using some of the cool features
Postgres has to index in specific fields the JSON mappings - and we are
making the assumption that your data does follow a schema.
We were featured in Hackernews a couple of times already, and as you would
expect, it sparked a lot of comparisons with other systems that exist. Things
like "Are you the Parse killer?" "Why don't you use CouchDB already?" "You
don't have everything Firebase provides!"
But frankly, for a HN thread, most of the feedback was really positive and useful
for us. In particular it gave us a good hindsight on what things should be improved
in the documentation. For instance, making it crystal clear that we are not
building a product with a tight integration with the clients and the server API
and a service we'd provide (our comparison table right now is a mixed bag
of products and server-side frameworks, so that's confusing.)
We're building an HTTP API and we provide a JS client and working on some other
clients -- that's it for now.
That said, we're sure hoping products and services will be built by the community with this "toolkit"
at some point in the future.
But as far as 2016 is concerned, our main goal is to develop Kinto for our needs
in Firefox.
Kinto & Firefox
For our daily work, the goal of Kinto is to be used in Firefox as the go-to
service to grab data continuously from our servers without having to wait for the
next release.
That's the Go Faster umbrella project.
We're going to use Kinto for:
the OneCRL client in Firefox, that is syncing the list of certificate revocations in
Firefox. The plan is to offer the security team the ability to deploy a change
to all our users as fast as possible.
the AMO blocklist client in Firefox, to get an up-to-date list of add-ons that should
be blocked.
Fennec Assets : static files needed by Fennec (Firefox for Android), like font files,
hypenation dictionaries and so on. The goal here is to reduce the APK file size as much
as possible and to limit the amount of data exchanged between mobile phones and
our servers as well.
OneCRL
Mark is driving the development of the client on behalf of the Firefox Security
Team, and besides the obvious benefit of getting certificate revocations changes
on-the-fly as diffs, he started to work with my team, Julien and Franziskus
on a signing protocol that will allow the Kinto client to verify that the data
that was sent by the server was not tampered with.
That work started a few months ago and the work week was a perfect time
to give it a boost.
On the server-side, the signing is done with a micro-service Julien started,
called autograph that will let a Kinto administrator sign the data before
it's pushed into Kinto.
See https://github.com/mozilla-services/autograph
Kinto is interacting with the signer through a specialized plugin, that triggers
the signing whenever some data is changed into Kinto, and that makes
sure the data is published to clients once properly signed.
See https://github.com/mozilla-services/kinto-signer
The storage itself is pretty passive about this, as it just stores signed
hashes of collections and let any client get them back.
The Kinto client can grab that signature and ask Firefox to verify it before
applying data changes. The verification logic uses a custom PKI that Mark and
Franziskus and building on top of NSS in the client.
Obviously, we should do this for all the data Kinto ever sends to Firefox,
so going forward, all our Kinto deployments will integrate by default signatures
capabilities.
AMO Blocklist
The existing AMO blocklist client is already doing most of what we do in Kinto:
it performs daily download of an XML file and applies changes in Firefox.
So what are the benefits of moving to Kinto for this ?
The biggest one will be the signing of the data, since this is not something
the current client has. The second one will be to separate the feature from
the current AMO Django website. Instead of having a dashboard within the Django
admin to push data, the Addons administrator will be able to manage
the whole dataset in a dedicated web admin.
We've built an admin app for Kinto that will be used for this.
See https://github.com/Kinto/kinto-admin
It's a static React-based app, and we're providing now a Kinto Distribution
to let you have a full Kinto service that include among other things that
web admin hooked into an /admin endpoint.
See https://github.com/Kinto/kinto-dist/
The last benefit is mostly a side-benefit, but quite important. Right now,
the daily ping done by Firefox for the AMO blocklist is used for some metrics
work. This is not related to the blocklist feature itself, but happened for historical reasons.
The plan (ongoing dicussion though) is to separate the two features in
Firefox as they should, and make sure we have a dedicated ping mechanism for the metrics.
We'll end up with two clearly separated and identified features, that can
be maintained and evolve separately.
Fennec Assets
Sebastian, from the Fennec team, has been working on the Fennec client
to sync a list of assets into Kinto. The goal is to reduce the size of the Android
package, and download those extra files in the background.
We've built for this a plugin to Kinto, to extend our API so files
could be uploaded and downloaded.
Kinto stores files on disk or on Amazon S3 (or whatever backend you
write the class for) and their metadata in a Kinto collection.
See https://github.com/Kinto/kinto-attachment/
From a client point of view, what you get is a MANIFEST you can browse
and sync locally, of what files are available on the server.
Fennec is going to use this to let admins manage static file lists
that will be made available to the mobile browser, and downloaded
if they really need them.
Next Steps
They are tons and tons of other stuff happening in Kinto right now,
but I wanted to give you an overview of the three major use cases
we currently have for it at Mozilla.
If all goes according to ours plans, these are the Firefox versions
they will land in:
OneCRL: Firefox 47 - June 7th
Fennec: Firefox 48 - July 18th
AMO: Firefox 49 - August 8th
Good times!