RedisToGo Outage Postmortem

Incident summary

On May 13 2015, our Redis provider RedisToGo experienced numerous service interruptions and drastic increase in latency throughout the day. This in turn caused system-wide interruptions on Shippo's end that affected rating and label purchase.

See http://status.redistogo.com/incidents/x1c77zxt14m9 for RedisToGo's incident history. During the downtime, we have migrated over to another provider.

What we are doing about it

• We have immediately switched Redis providers. We have furthermore established a backup service to immediately switch providers in case of new downtimes.
• We are (and have already been) working on a system-wide migration to not depend on external services like RedisToGo for all of our components. This migration is still in progress.