Companies that effectively create and implement big data strategies stand to gain a competitive advantage. Readers will learn how to build that advantage from these authors.
Five years ago, most companies collected data that were a part of their daily transactions and stored them in a database. This data was used primarily to keep track of operations or forecast needs. Today, both the sources and volume of data collected have exploded. It is now possible to collect click-stream data about every potential customer interaction with your web site. Marketers can also collect information about every conversation people are having about their brand. These sources of data have created modern-day treasure troves that can be mined to glean insights into products, services and customers. While this is conceptually possible, it requires the implementation of new processes, technology and governance mechanisms that are collectively being referred to as big data. Today, bid data is big business.
We define “big data” as a capability that allows companies to extract value from large volumes of data. Like any capability, it requires investments in technologies, processes and governance. The research firm IDC forecasts that the big data services and technology market will grow in value from $3.2 billion in 2010 to $16.9 billion in 20151.
Big data framework
In our work we have developed a framework (Figure 1) to understand big data strategies and the techniques used with each strategy. The first dimension that we consider is labeled business objective. When developing big data capabilities, companies try to measure or experiment. When measuring, organizations know exactly what they are looking for and look to see what the values of the measures are. When the objective is to experiment, companies treat questions as a hypothesis and use scientific methods to verify them.
The second dimension that we consider is labeled data type. In their normal course of functioning, companies collect data on their operations (e.g., sales) and capture it in their database that has a structure or schema. We call this transactional data. In other instances, companies deal with data that come from sources other than transactions and are typically unstructured (e.g., social media data). This combination results in four quadrants, each representing a different strategy: performance management, data exploration, social analytics, and decision science. We explain each strategy in the next section. (Figure 2 provides a list of popular big data techniques and vendors).
Figure1: Big Data Framework
Figure 2: Popular Big Data Techniques and Vendors
Transactional
Data
Technique
Vendor
Business Intelligence (BI)/Online Analytical Processing (OLAP):
users interactively analyze multidimensional data
users can roll-up, drill-down, and slice data
BI tools provide dashboard and report capabilities
Leading BI Tools:
Microsoft SQL Server Analysis and Reporting Services
SAP BusinessObjects
Oracle Business Intelligence
IBM Cognos/SPSS
SAS
Microstrategy
QlikTech
TIBCO Spotfire
Cluster Analysis:
segment objects (e.g., users) into groups based on similar properties or attributes
Data Mining:
process to discover and extract new patterns in large data sets
Predictive Modeling:
a model is created to best predict the probability of an outcome
SQL:
a computer language that manages (e.g., query, insert, delete, extract) data from a relational database
A/B Testing:
A method of testing in which a control group is compared to test groups to determine if there is an improvement based on the test condition
Often used in website design to test for higher conversion rates
Non-transactional
Social Data
Crowdsourcing:
A process for collecting data from a large community or distributed group of people
Idea submission is a common crowdsourcing activity
Leading Social Listening Tools:
Radian6
Attensity
Visible Technologies
Converseon
HootSuite
NodeXL network graphs
HP Autonomy
Oracle Endeca
IBM Watson
Textual Analysis:
Computer algorithms that analyze natural language
Topics can be extracted from text along with their linkages
Sentiment Analysis:
A form of textual analysis that determines a positive, negative, or neutral reaction
Often used in marketing brand campaigns
Network analysis:
A methodology to analyze the relationship among nodes (e.g., people)
On social media platforms, it can be used to create the social graph of follower and friends’ connections among users
Four Big Data strategies
1. Performance management
Performance management involves understanding the meaning of big data in company databases using pre-determined queries and multidimensional analysis. The data used for this analysis are transactional, for example, years of customer purchasing activity, and inventory levels and turnover. Managers can ask questions such as which are the most profitable customer segments and get answers in real-time that can be used to help make short-term business decisions and longer term plans.
Most business intelligence tools today provide a dashboard capability. The user, often the manager or analyst, can choose which queries to run, and can filter and rank the report output by certain dimensions (e.g., region) as well as drill down/up on the data. Multiple types of reports and graphs make it easy for managers to look at trends. A big benefit for report developers is that they can interact with different aspects of business data including HR, marketing, sales, customer service, and manufacturing data, and get multiple perspectives of how the business is doing.
BizTech, a leading information technology services firm in the Mid-Atlantic region, is hoping to use business intelligence to help it grow sales. Founded in 2001 by Tom Connolly, BizTech’s 2011 revenues were approximately $14M. Tom believes that significant improvement in measuring and reporting performance could help take BizTech to the next level of growth. In particular, BizTech plans to use Oracle’s CRM-OD (On Demand) business intelligence application to improve its opportunity-management process that involves generating, reviewing, and acting on leads. The company’s sales representatives and consultants will be able to generate new pipeline reports, including summaries by practice, regions, and sales representatives. These reports will be actively reviewed in weekly practice meetings, which will promote specific pipeline targets. In addition, learning from these reports can be tied directly to sales representatives’ skills development, coaching, and recruitment strategy.
The good news is the functionality and ease-of-use of business intelligence tools has improved greatly over the past several years. If designed and implemented effectively, these tools give managers a window into a vast amount of business transactions that can help with their everyday decision-making. The main challenge is to ensure that the quality and completeness of transactions entered into the system or the result will be “garbage in, garbage out.” Also, to guarantee a complete picture of the business, multiple databases across functions have to be integrated.
2. Data Exploration
Data exploration makes heavy use of statistics to experiment and get answers to questions that managers might not have thought of previously. This approach leverages predictive modeling techniques to predict user behavior based on their previous business transactions and preferences. Cluster analysis can be used to segment customers into groups based on similar attributes that may not have been on analysts’ radar screens. Once these groups are discovered, managers can perform targeted actions such as customizing marketing messages, upgrading service, and cross/up-selling to each unique group. Another popular use case is to predict what group of users may “drop out.” Armed with this information, managers can proactively devise strategies to retain this user segment and lower the churn rate.
With an increased emphasis on digital, inbound marketing, organizations want to attract prospects to their website with engaging, robust, and targeted content. Running experiments, organizations can test two webs sites, each containing different content such as white papers and demos, events such as webinars, and landing pages and lead form designs. The results of these experiments can help predict which combination of these variables twill lead to the highest conversion rate of site visitors to qualified leads, and qualified leads to customers.
The large retailer Target used data mining techniques to predict the buying habits of clusters of customers that were going through a major life event.2 Predicting customers who are going through big life changes such as pregnancy, marriage, and divorce, is important to retailers since these customers are most likely to be flexible and change their buying habits, making them ideal targets for advertisers. Target was able to identify roughly 25 products, such as unscented lotion and vitamin supplements, that when analyzed together, helped determine a “pregnancy prediction” score. Target then sent promotions focused on baby-related products to women based on their pregnancy prediction score. The result: sales of Target’s Mom and Baby products sharply increased soon after the launch their new advertising campaigns.
The rise in robust statistical/analytical techniques can lead to fast, direct results for data exploring organizations. The big challenge is the lack of qualified statisticians with expertise in the latest business analytical techniques. Another challenge is around data privacy/policy issues. Organizations need to think through the most effective way to use the results of their data mining techniques to improve the customer experience, and not make customers feel that retailers are “spying” on them. For example, Target had to adjust how it communicated this promotion to women who were most likely pregnant, once it had learned that the initial advertising had made some of them upset. 2. As a result, Target made sure to include advertisements that were not baby-related so the baby ads would look random.
3. Social analytics
Social analytics measure the vast amount of non-transactional data that exists today. Much of this data exist on social media platforms, such as conversations and reviews on Facebook, Twitter, and Yelp. Social analytics measure three broad categories: awareness, engagement, and word-of-mouth or reach.3 Awareness looks at the exposure or mentions of social content and often involves metrics such as the number of video views and the number of followers or community members. Engagement measures the level of activity and interaction among platform members, such as the frequency of user-generated content. More recently, mobile applications and platforms such as Foursquare provide organizations with location-based data that can measure brand awareness and engagement, including the number and frequency of check-ins, with active users rewarded with badges. Finally, reach measures the extent to which content is disseminated to other users across social platforms. Reach can be measured with variables such as the number of retweets on Twitter and shared likes on Facebook.
Social metrics are critical since they help inform managers of the success of their external and internal social digital campaigns and activities. For example, marketing campaigns involving contests and promotions on Facebook can be assessed through the number of consumer ideas submitted and the community comments related to those ideas. If the metrics indicate poor results, managers can pivot and make changes. For example, low Facebook engagement may mean more interesting and interactive content needs to be created.
With recent advancements in social measurement techniques, we can now calculate one’s “digital footprint” in the social media world. Companies like PeerIndex and Klout can measure a digital user’s social influence. A Klout score ranges from 1 to 100, based on their algorithm involving number of followers, re-tweets, the influence of the followers themselves, and other variables. Marketers are using social metrics to identify “influencers,” those well-followed individuals who are discussing their particular brand and can serve as a brand advocate. Using Klout’s services, Virgin America identified 120 individuals with high Klout scores and offered them a free flight to promote their new Toronto route.4 These individuals were not obligated to write about their experience. But, between these 120 individuals and another 144 engaged influencers, the campaign resulted in a total of 4,600 tweets, 7.4M impressions, and coverage in top news outlets. Thus, the campaign created a high brand awareness of the new airline route.
Social analyzers need a clear understanding of what they are measuring. For example, a viral video that has been viewed 10M times is a good indicator of high awareness, but it is not necessarily a good measure of engagement and interaction. Furthermore, social metrics consist of intermediate, non-financial measures. To determine a business impact, analysts often need to collect web traffic and business metrics, in addition to social metrics, and then look for correlations. In the case of viral videos, analysts need to determine if, after viewing the YouTube videos, there is traffic to the company web site followed by eventual product purchases.
4. Decision Science
Decision science involves experiments and analysis of non-transactional data, such as consumer-generated product ideas and product reviews, to improve the decision-making process. Unlike social analyzers who focus on social analytics to measure known objectives, decision scientists explore social big data as a way to conduct “field research” and to test hypotheses. Crowdsourcing, including idea generation and polling, enables companies to pose questions to the community about its products and brands. Decision scientists, in conjunction with community feedback, determine the value, validity, feasibility and fit of these ideas and eventually report on if/how they plan to put these ideas in action. For example, the My Starbucks Idea program enables consumers to share, vote, and submit ideas regarding Starbuck’s products, customer experience, and community involvement. Over 100,000 ideas have been collected to date. Starbucks has an “Ideas in Action” section to discuss where ideas sit in the review process.
Many of the techniques used by decision scientists involve listening tools that perform text and sentiment analysis. By leveraging these tools, companies can measure specific topics of interest around its products, as well as who is saying what about these topics. For example, before a new product is launched, marketers can measure how consumers feel about price, the impact that demographics may have on sentiment, and how price sentiment changes over time. Managers can then adjust prices based on these tests.
In 2009, Whirlpool, the largest manufacturer of home appliances, wanted to discover what their customers and consumers were saying about their products and services on social media platforms.5 They used Attensity360 for continuous monitoring and analysis of conversations across popular channels such as Facebook, Twitter, and Youtube, review and blogger sites, and mainstream news. Attensity’s text analytics findings were incorporated into Whirlpool’s decision models to accurately predict customer churn, loyalty, and satisfaction. This process enabled the company to listen, respond, and measure on a scale unobtainable by manual methods. The results revealed that Whirlpool improved its understanding of its overall business. There was increased satisfaction, faster responsiveness, and overall, more satisfied experiences with customers. The company also incorporated customer feedback to improve its product development and planning process.
While technology has helped companies scale the listening process involving social big data, the accuracy of listening tools is nowhere near perfect. Manual work is needed to “train” these technologies on company- and industry-specific keywords with regard to textual and sentiment analysis. Another good practice is to initially do parallel manual and listening tool analysis to understand the accuracy of the tool and determine ways to improve its effectiveness.
With respect to future trends in the big data field, the following practices are starting to emerge:
1. Integrating multiple big data strategies.
While a company can be effective with a single big data strategy, the most effective companies leveraging big data today are combining strategies. For example, one financial institution is leveraging both Social Analytics (non-transactional, social data) and Performance Management (business intelligence using transactional data) strategies to guide its customer service. The bank traditionally determined its “top” customers based on metrics such as number and balance of accounts; these were the customers who received premium service. Now, the bank is planning to incorporate social metrics into the equation. Those online customers who are very active with respect to mentioning, engaging with, and promoting the bank on social channels will also be considered for high-level service programs. The financial institution believes this is a much more balanced way to segment its most influential customers for customer service.
2. Build a Big Data capability.
We define a big data capability as the roles, technologies, processes, and culture needed to support big data initiatives. Perhaps the most critical of these are the roles, and in particular, the expertise and experience needed to devise and implement big data strategies. As mentioned earlier, multiple roles are needed: statisticians who are skilled in the latest statistical techniques; analysts and decision scientists who understand business measurement and experimentation and who can be the broker between statisticians and business managers; the IT group who provides guidance on selecting big data technologies/techniques and who integrates business intelligence tools with transactional systems such as CRM and Web analytical tools; and business managers and knowledge workers who own the business process and have to be comfortable with the new “language” of big data and social analytics. In addition, some leading companies have created specific group structures focused on big data analytics, and social content strategy and integration.
3. Be proactive and create a Big Data policy.
Companies need to keep up with policies and guidelines surrounding the use of big data, especially with non-transactional, social data that is often created and accessed outside company walls. Leading companies often have social media policies and certificate programs/training regarding social data use. Big data policies should also address issues regarding compliance, privacy, and security. Leading organizations clearly communicate and are honest in telling customers and consumers how they are using personal data, such as demographic information and past purchases. A rule of thumb that organizations should follow is to always think about the customer/consumer/employee experience and their personal benefits from big data projects. Big data projects that create a negative experience with users, despite the company benefits, should be redesigned.
With the cost of data capture and acquisition decreasing at a rapid rate, the real value of Big Data will be in its use. Companies that effectively create and implement Big Data strategies – such as those described above — stand to gain a competitive advantage. Big data strategies need to take into account both transactional and non-transactional data. In addition, the focus needs to extend beyond using Big Data to answer known questions, to experiment and discover trends that could help managers consider decisions and opportunities they could never have imagined before.
References
1. Vesset, D., Morris, H.D., Little, G., Borovick, L., Feldman, S., Eastwood, M., Woo, B., Villars, R.L., Bozman, J.S., Olofson, C.W., Conway, S., & Yezhkova, N. IDC market analysis: Worldwide big data technology and services, IDC #233485, March 2012.
2. Duhigg, C. How companies learn your secrets, The New York Times, 2/16/2012.
3. Hoffman, D.L., & Fodor, M. (2010). Can you measure the ROI of your social media marketing?, MIT Sloan Management Review, 52(1), 41-49.
4. Schaefer, M.W. (2012). Return on influence, McGraw-Hill, pp 5-6.
5. IDC Customer Spotlight: Whirlpool corporation’s digital detectives: Attensity provides the lens, March 2011.