Some notes from the Guardian's responsive mobile team on how they've made deploying software changes quick and simple to do
The Guardian's responsive mobile website team deploys its infrastructure and software several times a day. We've invested quite a bit in tools to help us do that efficiently. Here's a rundown of how we do it.
Ship it
We wanted to make deploying software simple and accessible to everyone on the team.
For reference, we currently have around 10 developers on the team, a set of around 12 applications (eg. articles, section front pages, etc) which make up the user interface. Over August 2013 we averaged around 29 deployments per day (this is effectively about ~2.5 deployments of our entire stack) with an average of 5.2 commits per deployment. The general ethos is to keep changes small and incremental, and get them running in production quickly.
Today, anyone can deploy the entire suite of applications running the public face of theguardian.com at any time by typing this simple command at their terminal:
Several things happen at this point:
A request is made to our deployment API which creates a fresh set of hardware for the deployment (we run on AWS). It then boots up the AMIs and runs the install scripts (Puppet) and shuttles the latest software artifacts onto it (basically a single JAR). It then runs a health check and, if successful, takes the old boxes out of the load balancers.
The end result is that 15 minutes after telling the system to deploy, your new feature is running in production in front of millions of people.
Likewise, reconfiguring the platform after updating the configuration is just as quick and as simple:
And if you fancy, you can remove our production stack from the Internet in a snatch, too:
Don't do that, though.
This has eliminated the rigmarole and theatrics of a typical release process we've experienced in other projects and at previous organisations.
Along the way we've learned several things:
The person writing the code is the person responsible for shepherding it into the production environment. This means no throwing the code at other people to deal with, which means learning more about the whole delivery cycle. Developers, for the most part, like this additional responsibility.
It can eat into your day – the hour you spent on Tuesday fixing a broken server, the time you spend nannying the deployment, the constant eye on the graphs hovering above your head, and so on...
Anything more than a simple bug fix must come wrapped in a feature flag and/or A/B test so it can be enabled/disabled outside of a release cycle. It's permissible to break things, but you'll want to un-break them quicker than it takes to notice and roll out a fix or be falling over the person who is deploying after you. Rolling backwards is a worst-case scenario.
Developers must have complete access to the environments in order to aid the deployment. It's no good being told to take responsibility for the platform and not feel in control of it. Having access to any nooks and crannies is essential, so everyone is granted sudoers access on production.
Just because you can deliver things quickly, doesn't mean the rest of the business always wants that. Oddly, on several occasions, we've shipped features that have quietly sat behind flags for some time while some other business function (a review, a conversation, an email...) is carried out.
Visibility (logging, metrics etc.) is essential to help people understand what state the software and infrastructure are in.
I'll expand on visibility in the next section.
Insight
If you give everyone the responsibility of releasing the code, we've learned that you need to invest in tools to give everyone in the team visibility as to what is happening. Essentially, "has the change I've just made worked and have I left the system in a good state for the next deployment?"
Our strategy here is threefold:
Making the system state very obvious and accessible.
Having some thresholds to know when the system state has become upset.
Notifying people when these thresholds are crossed.
System state
Although we have access to thousands of different metrics, it's important to distil general system health down to a handful of key performance indicators (KPIs). More than anything else we care about the performance of the end-user experience – therefore the most critical errors will be user-facing ones. For example, quite often our back-end system dies or slows down without any user-facing consequences – these are not the things you want to wake the whole team up at 3am for.
Obviously the KPIs correspond with the key purpose of the software we've written – "can someone arriving at the Guardian website read the news?" – so the metrics are fairly self-selecting.
Primarily, we use these three things to determine general system health:
Load balancer latency (CloudWatch) – Time elapsed after the load balancer receives a request until it receives the corresponding response. The whole system will struggle if the load balancer can't process incoming requests quickly.
CDN errors from Europe & US (Fastly) – Errors on our CDN (in two key commercial and editorial territories) means errors to the end-user.
CDN hit & miss ratio (Fastly) – A high hit ratio means we i) have visitors, ii) traffic is being kept off the origin servers.
We have several secondary indicators that also help identify serious failures:
Availability of hosts (Pingdom) – Fairly standard external uptime and response time monitoring.
Production deployment health (internal) – Did the last deployment succeed?
Page views p/min (internal) – How many requests p/sec are being received? If this is much lower than average something isn't right.
JavaScript errors (internal) – Caught on the client and shipped back via a beacon. A growing proportion of our application is written in JavaScript so spikes in errors mean users are seeing broken features.
Arriving at these metrics has been a process of understanding what matters over time. It's evolved along with the architecture and experience of past incidents.
Dashboards
Sometimes the only fun part of a project is making a dashboard. Not this project though, but we still needed a dashboard.
Each service I mentioned above (Fastly, CloudWatch, Pingdom, and the several internal tools we use) has an API that lets us extract the information it collects and place it on a communal dashboard. Here's what our dashboard looked like yesterday at 10:30pm.
Aside from the failed production deploy (in red), everything looks calm. We decided some of the information works better as a red or green boolean state block (red bad, green good) and others on a time series where trends and spikes are what we are looking for.
The dashboard is placed prominently around the team's desks in the office and, importantly, is responsively designed and available outside the company network – You will need to check it in the park on a Sunday afternoon on your mobile phone.
Typically, a wobble on a graph is a prompt to pay closer attention to the finer grain information we have available to us (Graphite, Real User Monitoring (RUM), logs, auto-scale notifications etc.)
Thresholds
While everyone is expected to pay attention to the dashboard and other data we have available, not everyone can do this all of the time. We need to eat, sleep, oversights happen etc., so it's helpful if the platform can report it's own ill-health to us when we are otherwise engaged.
The primary metrics (above) each have CloudWatch alarms configured at thresholds we deem useful to know about (a number you have to arrive at over time), and a few of the secondary indicators have higher thresholds against them.
Where feasible, we've attempted to build a system that can survive minor abnormalities without human intervention (eg. bursts in traffic can be countered by auto-scaling, serving stale content is acceptable to mask a temporary back-end outage etc.), so each system is given a bit of leeway (around five minutes) before it tells a human there's a problem. Likewise, the secondary indicators are given even longer to sort themselves out. Pingdom, for example, waits 15 minutes before telling us about a poorly host.
Alerts
Notifications are fairly straightforward. We use PagerDuty to help co-ordinate the team. It works like this:
Everyone on the team that deploys software to production is added to our PagerDuty rota. This creates a sense of responsibility. It's bad form if you have team members who are deploying things and expecting other people to notice it's broken.
Upon receiving any incident, PagerDuty sends the whole team an alert.
The type of alert is left to individual preference – some like SMS, some email, some push notifications – and if the message isn't acknowledged by someone with 30 minutes it's escalated to a phone call to me (from a spooky automated voice).
The system outline above has served us well to date. We've striven to find a balance between keeping ourselves informed of problems and avoiding the classic over-abundance of alert messages which ultimately get marked
as spam and ignored. A typical month sees us receive a couple of alerts.
Thoughts
Releasing code like this is great fun and relatively trouble-free.
It allows us to push changes out to users quickly, but in a controlled manner. Everything from bugs to split tests to experimental code and larger user-facing features gets the same treatment. This means we learn quickly about what works and what doesn't – both in terms of technical and product performance.
It's free from the problems associated with a single, monolithic release. The cause and effect of problems is quite obvious with this release pattern.
It forces the team to think about priorities. On a four week drop of code – probably spanning many commits and features – you can lose the focus on what really matters. The longer the time between deploys, the easier it is to just add that little extra essential feature you think you need. Giving yourself a day or two between changes helps you focus repeatedly on the next-most-useful-thing, and the next, and the next.
Similarly, a dozen minor improvements over two weeks hugely reduces the cognitive load on the rest of the engineering team in reviewing, sanity checking the code.
One by-product of this is the effect on the management of projects. Before, we might sit in a planning session for a hour or two, sketching out what we want to deliver in the near term, writing out a collective to-do list, tracking the cards in a ticketing system, grouping things up at the end of the week and pushing them out. Now that people understand delivery is ad-hoc, planning is typically often a short conversation, a whiteboard diagram, followed up by a short burst of deploys. Bugs get raised and dealt with in minutes, if serious. It's greatly reduced the overheads in our short-term project planning.
One pleasing aspect of continual delivery is the number of itches which get scratched because of the low cost of doing something and releasing it. The annoying bit of code, that refactoring you do in between things, the bug that you care about but hasn't really been prioritised, all actively get added to the project with very little fuss.
Most pleasingly of all is how the team has grasped ownership of the whole software platform and delivery cycle.
Like going to the gym, it's a constant challenge to keep yourself and the team motivated, plugging away, but the rewards are plentiful.
Matt Chadburn
theguardian.com © 2013 Guardian News and Media Limited or its affiliated companies. All rights reserved. | Use of this content is subject to our Terms & Conditions | More Feeds