According to John Maynard Keynes, the famous macro-economist , when choosing between the alternatives we often fall back on our motive or sentiment or chance, because mathematical basis is inadequate in human decision making under conditions of uncertainty. This does not bode well and is in contrast to predictive analytics where mathematics and statistics play a key role in anticipating future outcomes and establishes certain accuracy of certainty.
Predicting market performance from statistical models and even automating such models with sophisticated algorithms have been common practice in large enterprises. Nevertheless, failings of such models and automation with no human oversight is also evident from the recent Nasdaq Data Crash much of which seem to be attributed to Big Data, i.e. the explosion in data and complexity of the systems created to support big data which is beyond the understanding of a single person.
Neil MacDonald of Gartner Group makes the observation, "You get under the covers and high frequency trading algorithms are beyond understanding. Sub-millisecond trades taking place, tens of thousands per second, and when that fails, it fails spectacularly. That is what you are seeing manifested in Nasdaq". With big data, the knowledge gap appears to be widening!
Knowing that it is simply not part of human nature to make totally detached decisions about anything, never mind about choices which will affect them personally, perhaps there is a need for a balance between automation and human oversight / intuition supported by established system and data governance processes.
In this context, understanding the psychology (i.e. cognitive and emotional make-up of human mind) behind human instinct is useful. It may come as a surprise to note that human bias or intuition is already present in the final predictive statistics. This is because Machine Learning still requires humans to make key decisions, such as which scenario to simulate or which algorithms to use, which introduces bias into the process.
In addition, it is ultimately a human that interprets the models until the models provide (i.e. what the individual believes to be) reliable and (sometimes) acceptable output. For example, a churn prediction model based on Social Network Analysis using Spreading Activation technique (barrowed from cognitive psychology) provides accurate prediction of the transfer function in exerting influence among contacts than algorithms using Decision Trees. Similarly, Regression Models predict propensity for churn better than Decision Trees in some circumstances.
Similarly, gut instinct prevails when executives make decisions. Any decision is influenced by a range of factors, some rational, some non-rational, some explicit others implicit. These factors clearly carry different weights in the mind of each decision-maker. Each is derived from external information of various types which in itself may come through various filters such as opinions provided by subordinates, consultants and journalists. This information then goes through a further interior filter of personal experience, controllability, consequence, timeliness and decision maker’s cognitive and emotional make-up, before being assimilated and weighed up to make a decision.
A large component of the interior filter is the decision-maker's perception of value.
This sort of decision-making or judgement making is not irrational. But it is often made without going through the apparent rational step by step processes which management decision-makers are expected to follow. That is where instinct or intuition comes into play.
In their book, “Competing on Analytics”, Thomas Davenport and Jeanne Harris say, “Areas of decision making that were once well suited for intuition accumulate data and analytical rigour over time, and intuition becomes suboptimal ……In a few years, firms that do not employ extensive analytics in acquisition (decisions) will be considered irresponsible”. Perhaps discovery analytics could help to improve intuition by closing the knowledge gap. Best of all, it combines predictive modelling and human intuition in enabling data-endorsed decisions! 
Want to learn more on Data Discovery and how it can improve the decision making of your business? Click the banner below to request a whitepaper on Data Discovery: A New Approach to Analytics
Sundara Raman is a Senior Communications Industry Consultant at Teradata ANZ. He has 30 years of experience in the telecommunications industry that spans fixed line, mobile, broadband and Pay TV sectors. At Teradata, Sundara specialises in Business Value Consulting and business intelligence solutions for communication service providers. You can also connect with Sundara on Linkedin.
Tweet
Related Posts
20/20 Vision: What can we Learn from Kodak and eBay for Big Data Analytics?
Why the Telcos need to Revisit their Big Data Strategy Now
Cost versus Value Metrics: You are What you Measure
Consumer Data – The ultimate weapon for agile retention management
‘Single View’ to ‘Social View’: A paradigm shift in Customer and Product Management