This post on how Zenefits does continuous testing was excerpted from our eBook, Continuous Testing Strategy for CTOs. Download the full guide here.
To see how continuous testing actually looks in action, let’s take a look at Zenefits. Zenefits has been able to scale their customer base to over 10,000 without building an in-house QA team by leveraging a combination of testing automation and Rainforest. Laks Srini, the founder and CTO of Zenefits, cites maintaining quality while scaling rapidly as a major pain point for his team:
“While we were growing very fast and we were deploying multiple times and adding more and more to our products, at the end of the day we’re also enterprise software. The software cannot really break. If people don’t get paid on time, or people don’t get health insurance, that’s really, really bad…”
The “Hub-and-Spoke” Organizational Model
The bigger a company becomes, the more systems people have. Of the challenges of moving quickly at scale, Laks says, “The problem is that it’s pretty hard to build a distributed database when you own all the nodes and the network, but we don’t own all the nodes. These are health insurance providers and payroll companies and benefits providers. Sometimes, there’s no network.”
To combat fragmentation and communication breakdowns, Zenefits has adopted what Laks describes as a “hub-and-spoke” model, with the core Zenefits product as the hub, and the different services rolled into their offerings as the spokes. Each “spoke” has a dedicated development team. “It lets people make decisions in local maxima and move really fast in terms of their product itself,” says Laks. “They can make decisions, they can do stuff independently. There’s no centralized bottleneck.”
Roadblocks to Continuous Integration at Scale
While the hub-and-spoke model has helped keep Zenefits’ teams productive, one of the major challenges to running a large, segmented team is integration. As code from each spoke is completed, it passes through a CI server before being integrated into the “hub.” But as Zenefits’ team grew and deployment increased, it started taking longer and longer for every engineer to get feedback whenever they committed code, even with extensive automated test suites in place. Developers were sometimes waiting for hours for code to pass integration tests. “It’s almost like you haven’t even got continuous testing because the productivity is so low.”
Developing a Scalable Strategy for Continuous Testing
In order to keep Zenefits’ continuous deployment strategy working efficiently, Laks and his team have established centralized policies and testing practices. Developers cannot make a pull request without running a test. Everyone uses a centralized pipeline for CircleCI, that goes through the staging environment. Additionally, Zenefits has a centralized programmable infrastructure system with standardized containers with the monitoring, logging, application framework and firewall baked in.
Zenefits does not have a QA team, and each of its over 200 engineers is expected to write their own tests. However, they do have an infrastructure team that maintains their testing framework and CI infrastructure to keep tests running faster. By combining this strategic, developer-owned approach to test writing with a comprehensive suite of smoke and sanity tests in Rainforest, Laks has been able to implement continuous testing effectively and keep deployment fast at Zenefits.
Leveraging Automation and Human Testing
In the last year, Zenefits grew their engineering team from 35 engineers to over 200, across 3 global locations. With so many people writing code, it was important for Zenefits to implement a plan to ship code frequently to keep risk low. To ensure that there was a human safety net beyond their extensive automated testing suites, Zenefits has leveraged Rainforest for smoke and sanity testing in production.
By automating as much of their QA as possible, Zenefits can prevent QA from slowing down their deployment pace, even when they deploy multiple times per day. Where automation doesn’t fit their testing needs well, such as for new features and highly-visual features, they use Rainforest. “For every automated test, we probably have a Rainforest equivalent as well,” says Laks. Because their webapp is highly visual, Zenefits relies on tests to make sure that the interface is functionally correctly. These UX tests are challenging to execute effectively with automation. With Rainforest, Zenefits can double-check their automated tests results and ensure that the platform is usable for their end users.
Zenefits Saves $2MM Annually with Rainforest QA
Typically a company with one hundred engineers, like Zenefits, would employ a software QA team of up to twenty people. With the average QA salary ranging between $100K and $125K, Zenefits saves upwards of $2MM a year by using Rainforest to supplement their testing automation activities. Laks clarified the impact of Rainforest on Zenefits’ growth: “In terms of ROI, Rainforest has been great. We don’t have anyone, other than the engineers themselves, that specifically does human QA and testing. And that’s thanks to Rainforest. To be able to have peace of mind when we deploy knowing any human can actually log in, get around the website and everything works, is awesome.”
Continuous Testing with Rainforest QA
Continuous testing can help companies that use Agile and CI/CD methodlogies test faster and more effectively. To read more about how companies are adopting innovative QA strategies that help them move faster and build better products, download the free eBook on Continuous Testing for CTOs now.