Many observers, including the authors of this article, believe that Big Data is the new, new thing that will see some companies leapfrog others to become best in class. There are a few skeptics, but readers will find a compelling case and a toolkit for the smart use of Big Data in this article.
Data are now woven into every sector and function in the global economy, and, like other essential factors of production such as hard assets and human capital, much of modern economic activity simply could not take place without them. The use of Big Data — large pools of data that can be brought together and analyzed to discern patterns and make better decisions — will become the basis of competition and growth for individual firms, enhancing productivity and creating significant value for the world economy by reducing waste and increasing the quality of products and services.
Until now, the torrent of data flooding our world has been a phenomenon that probably only excited a few data geeks. But we are now at an inflection point. According to research from the McKinsey Global Institute (MGI) and McKinsey & Company’s Business Technology Office, the sheer volume of data generated, stored, and mined for insights has become economically relevant to businesses, government, and consumers.
The history of previous trends in IT investment and innovation and its impact on competitiveness and productivity strongly suggest that Big Data can have a similar power, namely the ability to transform our lives. The same preconditions that allowed previous waves of IT-enabled innovation to power productivity, i.e., technology innovations followed by the adoption of complementary management innovations, are in place for Big Data, and we expect suppliers of Big Data technology and advanced analytic capabilities to have at least as much ongoing impact on productivity as suppliers of other kinds of technology.
All companies need to take Big Data and its potential to create value seriously if they want to compete. For example, some retailers embracing big data see the potential to increase their operating margins by 60 per cent.
Big Data: A new competitive advantage
The use of Big Data is becoming a crucial way for leading companies to outperform their peers. In most industries, established competitors and new entrants alike will leverage data-driven strategies to innovate, compete, and capture value. Indeed, we found early examples of such use of data in every sector we examined. In healthcare, data pioneers are analyzing the health outcomes of pharmaceuticals when they were widely prescribed, and discovering benefits and risks that were not evident during necessarily more limited clinical trials. Other early adopters of Big Data are using data from sensors embedded in products from children’s toys to industrial goods to determine how these products are actually used in the real world. Such knoiwledge then informs the creation of new service offerings and the design of future products
Big Data will help to create new growth opportunities and entirely new categories of companies, such as those that aggregate and analyse industry data. Many of these will be companies that sit in the middle of large information flows where data about products and services, buyers and suppliers, consumer preferences and intent can be captured and analysed. Forward-thinking leaders across sectors should begin aggressively to build their organisations’ Big Data capabilities.
In addition to the sheer scale of Big Data, the real-time and high-frequency nature of the data are also important. For example, ‘nowcasting,’ the ability to estimate metrics such as consumer confidence, immediately, something which previously could only be done retrospectively, is becoming more extensively used, adding considerable power to prediction. Similarly, the high frequency of data allows users to test theories in near real-time and to a level never before possible.
We studied five domains in depth—healthcare and retail in the United States, the public sector in Europe, and manufacturing and personal location data (the location data generated by the smart mobile devices many of us now carry) globally—and some broadly applicable ways of leveraging big data emerged.
Five ways to leverage Big Data
1. Big Data can unlock significant value by making information transparent. There is still a significant amount of information that is not yet captured in digital form, e.g., data that are on paper, or not made easily accessible and searchable through networks. We found that up to 25 percent of the effort in some knowledge worker workgroups consists of searching for data and then transferring them to another (sometimes virtual) location. This effort represents a significant source of inefficiency.
2. As organisations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days and therefore expose variability and boost performance. In fact, some leading companies are using their ability to collect and analyse big data to conduct controlled experiments to make better management decisions.
3. Big Data allows ever-narrower segmentation of customers and therefore much more precisely tailored products or services.
4. Sophisticated analytics can substantially improve decision-making, minimise risks, and unearth valuable insights that would otherwise remain hidden.
5. Big Data can be used to develop the next generation of products and services. For instance, manufacturers are using data obtained from sensors embedded in products to create innovative after-sales service offerings such as proactive maintenance to avoid failures in new products.
Value created by the use of Big Data
If the U.S. healthcare system were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300bn in value every year. Two-thirds of that would be an 8 percent reduction in U.S. healthcare expenditure. In the developed economies of Europe, government administrators could create more than €100bn ($123bn) in operational efficiency improvements alone by using Big Data – and that’s not including employing advanced analytic tools to reduce fraud and errors and boost the collection of tax revenues.
But it’s not just companies and organisations that stand to gain from the value that Big Data can create. Consumers can also reap highly significant benefits. For instance, users of services enabled by personal-location data can capture $600bn in consumer surplus.
Take smart routing using real-time traffic information, which is one of the most heavily-used applications of personal-location data. As the penetration of smart phones increases, and free navigation applications are included in these devices, the use of smart routing is likely to grow. By 2020, more than 70 percent of mobile phones are expected to have a GPS capability, up from 20 percent in 2010. All told, we estimate that the potential global value of smart routing in the form of time and fuel savings will be about $500bn by 2020. This is equivalent to saving drivers 20bn hours on the road, or 10 to 15 hours every year for each traveller, and about $150bn on fuel consumption.
Some of the most significant potential to generate value from Big Data will come from combining separate pools of data. The U.S. healthcare system, for instance, has four major pools of data – clinical; activity (claims) and cost; pharmaceutical and medical products R&D; and data about patient behaviour and sentiment – each of which is primarily captured and managed by a different constituency. MGI estimates that if U.S. healthcare fully used all the available techniques that can be enabled by Big Data, such as analyzing records of real-world medical treatments, their costs and health outcomes to guide physicians on which treatments provide the best outcomes at the best cost, the annual productivity of the sector could grow by an additional 0.7 per cent. But achieving this boost in productivity will require the combination of data from different sources – often from organizations that have no history of sharing data at scale. Sets of data such as patient records and clinical claims would have to be integrated.
Doing so would create benefits not just for the various industry players but for patients, who would have broader, clearer access to a wider variety of healthcare information, making them more informed. Patients would be able to compare not only the prices of drugs, treatments, and physicians, but also their relative effectiveness, enabling them to choose more effective, better-targeted medicines, potentially even customised to their personal genetic and molecular make-up. To obtain those broad benefits, healthcare consumers may have to accept a slightly different trade-off between their privacy and the benefits that wider pooling of data would bring.
Sensitivities around privacy and data security are just one hurdle that companies and governments need to overcome if the economic benefits of big data are to be realised. One of the most pressing challenges is a significant shortage of people with the skills to analyse big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical training (in statistics or machine learning) and another 1.5m people with the managerial and quantitative skills to be able to frame and interpret analyses effectively enough to base decisions on them.
There are also many technological issues that need to be resolved to make the most of big data. Legacy systems and incompatible standards and formats often prevent the integration of data and the application of the more sophisticated analytics that create value. Ultimately, making use of large digital datasets will require the assembly of a technology stack from storage and computing through analytical and visualisation software applications.
Above all, access to data needs to broaden. Increasingly, companies will need to access data from third parties, e.g., business partners or customers, and integrate them with their own. A vital competency for data-driven organizations in the future will be the ability to create compelling value propositions for others, including consumers, suppliers and potentially even competitors, to share data. If it looks unlikely that data sharing will occur despite the potential for societal benefits (a market failure), legislators may then have to step in.
As long as companies and governments understand the power of Big Data to deliver higher productivity, better value for consumers, and the next wave of growth in the global economy, there should be a strong enough incentive for them to act robustly to overcome the barriers to its use. By doing so they will unleash avenues to new competitiveness among companies, higher efficiency in the public sector that will enable better services, even in constrained fiscal times, and enable firms and even whole economies to be more productive.
Big data is a big deal
The era of Big Data could yield new management principles. In the early days of professionalized corporate management, leaders discovered that minimum efficient scale was a key determinant of competitive success. Likewise, future competitive benefits are likely to accrue to companies that can not only capture more and better data but also use that data effectively at scale. We hope that by reflecting on such issues and the five questions that follow, executives will be better able to recognize how big data could upend assumptions behind their strategies, as well as the speed and scope of the change that’s now under way.
1. What happens in a world of radical transparency, with data widely available?
As information becomes more readily accessible across sectors, it can threaten companies that have relied on proprietary data as a competitive asset. The real-estate industry, for example, trades on information asymmetries such as privileged access to transaction data and tightly held knowledge of the bid and ask behaviour of buyers. Acquiring both requires a significant expense and effort. In recent years, however, online specialists in real-estate data and analytics have started to bypass agents, permitting buyers and sellers to exchange perspectives on the value of properties and creating parallel sources for real-estate data.
Cost and pricing data are becoming more accessible across a spectrum of industries. Another swipe at proprietary information is the assembly by some companies of readily available satellite imagery that, when processed and analyzed, contains clues about competitors’ physical facilities. These satellite sleuths glean insights into expansion plans or business constraints as revealed by facility capacity, shipping movements, and the like.
One big challenge is the fact that the mountains of data many companies are amassing often lurk in departmental “silos,” such as R&D, engineering, manufacturing, or service operations—impeding timely exploitation. Information hoarding within business units also can be a problem: many financial institutions, for example, suffer from their own failure to share data among diverse lines of business, such as financial markets, money management, and lending. Often, this prevents these companies from forming a coherent view of individual customers or understanding links among financial markets.
Some manufacturers are attempting to pry open these departmental enclaves: they are integrating data from multiple systems, inviting collaboration among formerly walled-off functional units, and even seeking information from external suppliers and customers to co-create products. In advanced-manufacturing sectors such as automotive, for example, suppliers from around the world make thousands of components. More integrated data platforms now allow companies and their supply chain partners to collaborate during the design phase—a crucial determinant of final manufacturing costs.
2. If you could test all of your decisions, how would that change the way you compete?
Big Data ushers in the possibility of a fundamentally different type of decision making. Using controlled experiments, companies can test hypotheses and analyze results to guide investment decisions and operational changes. In effect, experimentation can help managers distinguish causation from mere correlation, thus reducing the variability of outcomes while improving financial and product performance.
Robust experimentation can take many forms. Leading online companies, for example, are continuous testers. In some cases, they allocate a set portion of their Web page views to conduct experiments that reveal the factors that drive higher user engagement or sales gains. Companies selling physical goods also use experiments to aid decisions, but Big Data can push this approach to a new level. McDonald’s, for example, has equipped some stores with devices that gather operational data as they track customer interactions, traffic in stores, and ordering patterns. Researchers can model the impact of variations in menus, restaurant designs, and training, among other things, on productivity and sales.
Where such controlled experiments aren’t feasible, companies can use “natural” experiments to identify the sources of variability in performance. One government organization, for instance, collected data on multiple groups of employees doing similar work at different sites. Simply making the data available spurred lagging workers to improve their performance.
3. How would your business change if you used Big Data for widespread, real-time customization?
Customer-facing companies have long used data to segment and target customers. Big Data permits a major step beyond what until recently was considered state of the art, by making real-time personalization possible. A next-generation retailer will be able to track the behavior of individual customers from Internet click streams, update their preferences, and model their likely behavior in real time. They will then be able to recognize when customers are nearing a purchase decision and nudge the transaction to completion by bundling preferred products, offered with reward program benefits. This real-time targeting, which would also leverage data from the retailer’s rewards program, will increase purchases of higher-margin products by its most valuable customers.
Retailing is an obvious industry for data-driven customization because the volume and quality of data available from Internet purchases, social-network conversations, and, more recently, location-specific smart phone interactions have mushroomed. But other sectors, too, can benefit from new applications of data, along with the growing sophistication of analytical tools for dividing customers into more revealing microsegments.
4. How can Big Data augment or even replace management?
Big data expands the possible domains of application for algorithms and machine-mediated analysis. At some manufacturers, for example, algorithms analyze sensor data from production lines, creating self-regulating processes that cut waste, avoid costly (and sometimes dangerous) human interventions, and ultimately lift output. In advanced, “digital” oil fields, instruments constantly read data on wellhead conditions, pipelines, and mechanical systems. That information is analyzed by clusters of computers, which feed their results to real-time operations centers that adjust oil flows to optimize production and minimize downtimes. One major oil company has cut operating and staffing costs by 10 to 25 percent, while increasing production by 5 percent.
Products ranging from copiers to jet engines can now generate data streams that track their usage. Manufacturers can analyze the incoming data and, in some cases, automatically remedy software glitches or dispatch service representatives for repairs. Some enterprise computer hardware vendors are gathering and analyzing such data to schedule preemptive repairs before failures disrupt customers’ operations. The data can also be used to implement product changes that prevent future problems or to provide customer-use inputs that inform next-generation offerings.
The bottom line is improved performance, better risk management, and the ability to unearth insights that would otherwise remain hidden. As the price of sensors, communications devices, and analytic software continues to fall, more and more companies will be joining this managerial revolution.
5. Could you create a new business model based on data?
Big Data is spawning new categories of companies that embrace information-driven business models. Many of these businesses play intermediary roles in value chains where they find themselves generating valuable “exhaust data” produced by business transactions. One transport company, for example, recognized that in the course of doing business, it was collecting vast amounts of information on global product shipments. Sensing opportunity, it created a unit that sells the data to supplement business and economic forecasts.
Another global company learned so much from analyzing its own data as part of a manufacturing turnaround that it decided to create a business to do similar work for other firms. Now the company aggregates shop-floor and supply-chain data for a number of manufacturing customers and sells software tools to improve their performance. This service business now outperforms the company’s manufacturing business.