In 2013, we had the privilege of working with hundreds of architects, operators, and deployers of private cloud storage infrastructure. Understanding their business requirements - and how they plan to evolve their infrastructure - is critical to what we do here at SwiftStack. This past year we observed several trends worth noting: (1) a transition to object based storage from traditional file system technologies to support new application initiatives, (2) a strong desire to shift to open source software away from proprietary and vendor-specific technologies, (3) a preference for using technologies that best address a specific use-case, and (4) the adoption of OpenStack Swift as the standard for object storage.
In this post, I’d like to describe these observations in a bit more detail - and discuss what the trends mean for the storage market in 2014 and beyond.
The Transition to Object Storage
In 2013, adoption of public object storage services continued at an ever-accelerating rate. In April, Amazon announced that 1 trillion objects had been uploaded to S3 in the prior 10 months. For a bit of context, it took 6 years for the first 1 trillion objects to be uploaded. By some estimates, S3 generated several hundred millions of dollars of revenue for Amazon in 2013 - and grew over 100% annually. HP, IBM, Google, Microsoft, Rackspace, and Oracle also grew or launched their public object storage offerings.
Much of this amazing growth is fueled by the numerous Web, mobile, and software as a service (SaaS) applications that depend on object storage. Some of the most popular applications on the Web are powered by object storage services that store and serve data to millions of users every day. SwiftStack users like Disney Interactive and Concur are among the enterprises that use Swift as the storage engine for their high-growth Web and SaaS applications. Wikipedia.org, the 6th most popular site on the Web, uses Swift to store data.
Companies like MercadoLibre, the e-commerce leader in Latin America, are storing more than 1.4 billion images on Swift as part of their most critical business application - their marketplace. For these applications, the performance, scale, and durability of object storage is essential. Traditional file system technologies cannot provide the dramatic increase of scale that these applications require. Arrays do not meet the price per capacity needs either. Object storage is quite simply the natural storage platform for applications being built today.
"Implementing Swift in MercadoLibre was really easy for us and our developers, mainly because the use of APIs to manage the solution make it developer friendly and provide cost savings, and also because of the big community behind it that gives us support, where we can also contribute with our user experience. We have been using OpenStack for three years and really trust the community involved in this project. We are always ready to contribute," said Maximiliano Venesio, Technical Leader from MercadoLibre.com
The Swift solution delivers low latency, enabling the team to display its billions of stored pictures much more rapidly.
"We are an e-commerce platform and we sell our products through pictures so being able to quickly and reliably display pictures is crucial to our business. With Swift, we are able to display pictures 10x faster than before—allowing us to serve our customers better than ever before," said Alejandro Comisario, Manager from the MercadoLibre CloudBuilders team
"OpenStack Swift in particular has gained a lot of traction both in the enterprise and in the service provider space. All these trends support IDC's assertion that 2013 marked a turning point for the OBS (object-based storage) market. IDC estimates that in 2013, OBS solutions accounted for nearly 37% share of the file-and-OBS (FOBS) market in revenue and is forecast to be a $21.7 billion market in 2017." - IDC, October 2013
Companies looking to store vast amounts of archival data are also benefiting from the cost advantages of object storage. EVault, Seagate’s cloud backup division, took the wraps off their Glacier-like storage service, LTS2, that is based on Swift. With LTS2, EVault is also offering a Swift-based storage service at Glacier-like prices, which is made possible by Swift’s highly flexible architecture. All of the major tape vendors are now also promoting object-based technologies to meet the needs of increased scale and lower the cost of long-term archives.
IDC has recognized the effect that OpenStack Swift is having on the file and object storage market:
OpenStack Swift is re-shaping the object storage landscape in a pretty fundamental way - and is a major force in a market that IDC forecasts to grow to more than $21 billion within the next three years.
Why Being Open Matters
OpenStack Swift is open source software. To understand why this matters for storage architects and operators, let’s look back at what happened to the rest of the infrastructure stack. Once dismissed by their proprietary competitors as "immature," open source operating systems, middleware, application frameworks, and databases are now standards in enterprise and Web infrastructure. In fact, the open source model has so completely and fundamentally transformed the infrastructure tier in the data center that not many proprietary infrastructure platform technologies have a sustainable advantage any longer.
That same transformation is now changing the storage tier, one organization at a time. Vendors of proprietary storage technology will claim that open source cannot produce enterprise-grade storage solutions. We have all seen this movie before - when done the right way (open code, open design, open community), open source platforms can and do win. This change is led by storage architects, operators, and deployers at enterprises, Web companies and service providers alike. Today, using open source platforms is an irreversible change in their enterprise architectures.
Using an open source storage engine does not preclude these organizations from working with commercial vendors in the OpenStack ecosystem. In fact, most organizations adopting Swift use complementary products, such as the SwiftStack Controller, which makes it easy to scale and manage their Swift clusters. Jonathan Bryce, Executive Director of the OpenStack Foundation, explained it this way at the last OpenStack Summit:
"Cutting edge technology from an open source project is being used by some of the biggest companies in the world. How do they get a comfort level deploying this technology? Partners and support from the community, like SwiftStack and others in the ecosystem that have helped Enterprises meet their business needs."
All the Wood Behind one Arrow
But to succeed as a project or as a company, focusing on one thing and doing that thing really well is generally a good strategy. In highly competitive markets, it is often the only strategy that can lead to success. Conversely, there is a long list of defunct companies that attempted to do too many things, too soon. David Packard, co-founder of HP, famously said that more companies die of indigestion than starvation. The same can be true for both products and open source projects.
The message we heard from storage architects and operators in 2013 was that they prefer the best technologies to meet specific requirements and use-cases. All-in-one solutions, like a printer/copier/fax/scanner, are by design not optimized to deliver the best results for each use-case. Swift, which was designed from the ground up to do one thing really well - object storage, has this focus. Swift is not block storage, so it is not for running VMs or databases. It is also not a file system and does not try to emulate those interfaces. Rather, Swift was designed specifically for unstructured data, the type of data that is growing the fastest with today’s applications. A big reason it serves so well for unstructured data is its specialization and design for eventual consistency to support massive scalability and geographic distribution of data.
What makes this focus even more powerful for Swift is the immense support behind it. Over 145 developers have contributed to the OpenStack Swift codebase. The Havana release of Swift, which included support for global clusters, had top contributors (by patch count) from 6 different companies: SwiftStack, Rackspace, Red Hat, eNovance, IBM, and United Stack. In the upcoming IceHouse release, Intel, Red Hat, and SwiftStack are all working on storage policies for Swift. Other contributors like Rackspace are making major improvements to replication.
Intel, Box and SwiftStack are working on erasure codes. In short, the incredible ecosystem of who’s-who in technology that has formed around Swift gives the project strong and deep technical backing, i.e. a lot of wood behind one arrow. This ecosystem will also be the reason why Swift will continue to succeed. IDC, in their October 2013 Vendor Assessment of the Worldwide Object-Based Storage Market, agrees:
"In all likelihood, the only survivors in this market may be vendors with robust partner ecosystems and/or vendors with commercial variants of open source platforms."
With this deep and active community, the Swift project has great development velocity as well as real diversity. Swift is much larger than what any single storage company could build. While storage architects and operators acknowledge the value of open source, not all open source projects are equal. Any open source community where >90% of the contributions come from one entity is not much of a community at all. The Swift community, however, gains momentum and strength from so many contributors distributed across many companies and countries. It's this community, in large part, that is driving enterprises to adopt Swift for their production object storage needs.
Swift, the Object Storage Standard
The Swift API has been adopted by or is on the roadmap for most vendors as an object storage interface
- Terri McClure, Senior Analyst, Enterprise Strategy Group
While Amazon, Google, and Microsoft run very large storage clouds, each using its own proprietary technologies, other big names at the top of the list, such as HP Cloud, IBM Softlayer, Internap and Rackspace all run Swift. Rackspace has more than 85PB of raw disk deployed in 6 data centers, with sustained throughput to a single cluster of 60Gbit/sec. In fact, Swift’s adoption is so robust that by several important measures it already is the object storage technology of choice for large clouds. Deployments like these show that Swift is mature, proven in production, and already runs some of the world’s largest storage clouds.
For companies that need to operate private object storage services in their own data centers - for either control, cost, or performance reasons - choices have been limited. Single-vendor object storage technologies have not been a viable option as they were designed for a different use case and locked customers into a closed-source storage platform. With Swift, storage operators have access to a production-ready object storage platform designed for standard hardware to store and manage massive amounts of unstructured data at scale. Recent features like global clusters and new upcoming features, such as storage policies and erasure codes, will enable a new set of use cases and further transform the object storage landscape.
Swift’s momentum is also part of the reason it is the ideal choice as an engine for software-defined storage solutions. Swift has been used successfully in production by large enterprises and the Swift API is increasingly becoming the API of choice. Naturally, the biggest cloud operators have APIs that developers use so applications can write to those storage clouds. EMC, Oracle, and Red Hat are a few of the largest vendors who announced support for the Swift API in 2013. This allows their users to integrate with public clouds that run Swift, and leverage private clouds riding the popularity of Swift in private deployments.
The adoption of the Swift API as an industry standard has been noted by industry analyst firm ESG:
"With current and expected unstructured data growth rates, organizations developing new applications want and need to leverage the benefits of object storage scalability and manageability. But they need standards-based object interfaces so their investment in development is protected" said Terri McClure, Senior Analyst, Enterprise Strategy Group.
"Storage vendors recognize this fact and are responding. The Swift API has been adopted by or is on the roadmap for most vendors as an object storage interface, enabling those vendors to serve public, private, and hybrid cloud deployments without concerns about users making proprietary API investments."
What this means in 2014
With continued growth of Swift clusters already in production, compounded by the rapid adoption of Swift for new deployments, 2014 will further cement Swift as the standard for object storage. The technology will continue to evolve as well, driven by its exceptional developer community and focus. Industry-changing features like storage policies and erasure codes will enable many more use cases and accelerate the adoption of Swift.
At SwiftStack, we’ve also been developing the management layer for Swift, including features like rolling upgrades, LDAP integration, and management of geographic clusters. Coupled with Swift’s large and growing ecosystem, these advances enable a new wave of innovation in the application and infrastructure tier by enterprises and service providers.
Users prefer technologies that best meet their needs, and we based our product here at SwiftStack on Swift for the same reasons. Swift is the engine that already runs the world’s largest storage clouds, its API has been adopted as standard by vendors both established and emerging, and it is the open source storage technology with the most vibrant contributor community. When deciding on the best object storage technology for your cloud in 2014, consider the proven, highly supported and standard choice: Swift.