tl;dr
On Thursday, August 11, Reddit was down and unreachable across all platforms for about 1.5 hours, and slow to respond for an additional 1.5 hours. We apologize for the downtime and want to let you know steps we are taking to prevent it from happening again.
Thank you all for contributions to r/downtimebananas.
Impact
On Aug 11, Reddit was down from 15:24PDT to 16:52PDT, and was degraded from 16:52PDT to 18:19PDT. This affected all official Reddit platforms and the API serving third party applications. The downtime was due to an error during a migration of a critical backend system.
No data was lost.
Cause and Remedy
We use a system called Zookeeper to keep track of most of our servers and their health. We also use an autoscaler system to maintain the required number of servers based on system load.
Part of our infrastructure upgrades included migrating Zookeeper to a new, more modern, infrastructure inside the Amazon cloud. Since autoscaler reads from Zookeeper, we shut it off manually during the migration so it wouldn’t get confused about which servers should be available. It unexpectedly turned back on at 15:23PDT because our package management system noticed a manual change and reverted it. Autoscaler read the partially migrated Zookeeper data and terminated many of our application servers, which serve our website and API, and our caching servers, in 16 seconds.
At 15:24PDT, we noticed servers being shut down, and at 15:47PDT, we set the site to “down mode” while we restored the servers. By 16:42PDT, all servers were restored. However, at that point our new caches were still empty, leading to increased load on our databases, which in turn led to degraded performance. By 18:19PDT, latency returned to normal, and all systems were operating normally.
Prevention
As we modernize our infrastructure, we may continue to perform different types of server migrations. Since this was due to a unique and risky migration that is now complete, we don’t expect this exact combination of failures to occur again. However, we have identified several improvements that will increase our overall tolerance to mistakes that can occur during risky migrations.
Make our autoscaler less aggressive by putting limits to how many servers can be shut down at once.
Improve our migration process by having two engineers pair during risky parts of migrations.
Properly disable package management systems during migrations so they don’t affect systems unexpectedly.
Last Thoughts
We take downtime seriously, and are sorry for any inconvenience that we caused. The silver lining is that in the process of restoring our systems, we completed a big milestone in our operations modernization that will help make development a lot faster and easier at Reddit.
submitted by /u/gooeyblob to /r/announcements
[link] [comments]