This is not a “thought piece”. This is catharsis.
I studied social science research methods at University. When I went to University, it was in the very earliest days of the World Wide Web (actual quote from Information Systems lecturer: “I don’t think it will be as successful as WAIS or Gopher”).
Back in those days before online civilisation, surveying people to find out what they thought and felt was complex and expensive.
Complex because finding the right questions to ask people to solicit the information you are trying to uncover without falling prey to bias or misinterpretation is really difficult, and the reality is that often what people say isn’t what they feel or mean anyway.
It was expensive because issuing a survey involved paper and postage, telephones, or people meeting in person. Survey researchers were a common feature. I remember working on a project at the London School of Economics soon after graduating in the mid-1990s to buy a very expensive and complicated piece of machinery that could automatically scan and process paper survey forms. Just imagine that.
Today, with the ubiquity of the World Wide Web, software as a service and tools like Survey Monkey or Google Forms, the cost of administering surveys has dropped to almost zero.
However, the complexity of designing surveys remains as mind-boggling as ever. And this, I reckon, is the single biggest reason why we are now bombarded daily, if not hourly, by the most mindless question in modern life. “How likely are you to recommend us?”.
Last week, I was in a hurry to produce a small piece of graphics for a task I needed to do. Before Adobe Illustrator would allow me to DO MY WORK, it asked me if I was likely to recommend Adobe Illustrator to others.
I now put it on record that despite my frequent use of Adobe Illustrator, I cannot begin to imagine why you, dear reader, or anyone else on this planet would use a product that prioritizes the manufacturer’s vanity and neediness over its users’ work. No, Adobe, I positively anti-recommend your product to anyone who will listen. Use a piece of paper and a ruler instead. It won’t demand you answer stupid questions at inopportune moments.
Last week, in my local Halfords Autocentre, whilst dropping off our car for a service, I was confronted with this at the reception desk:
I’m a man of a certain age. I should be protected from things that make my blood pressure rise. This is a health and safety violation.
There are so many things wrong with this little notice. Ok, so it’s not technically an NPS question, but it’s of the ilk. And a presumably Head Office-mandated attempt to game the scores as they have KPIs about customer satisfaction to hit and it’s judged more acceptable to prime the audience to give 5 scores than actually looking to receive feedback that might impact their ratings.
There are two ironies in this example. Firstly I have often recommended the local Halfords to others (despite, it seems, them having a rather murky reputation nationally) because I’ve always found the local branch good to deal with. And secondly, I didn’t get an email to recommend them. I wonder if that’s because they recommended some quite expensive work that needs to be done on the car soon and have correlated that outcome to poor feedback scores? I wouldn’t put it past them.
I could go on with examples, because they are just so frequent, but these are the two that have sparked this rant.
Why do I have such a problem with NPS? Well, because it’s a silver bullet to the complexity problem of survey design. By “complexity problem” I mean the hard work necessary to design questions that solicit meaningful data and by “silver bullet” I mean a mythological weapon that doesn’t exist because werewolves don’t exist either.
Net Promotor Score was originally concocted as a single metric that could reliably be used to predict whether companies would grow or not. It was launched into the world in this article from the Harvard Business Review in 2003 by its progenitors from the strategic consulting firm Bain & Co.
It asks people their likelihood to recommend a product or services on a 10-point Likert scale, and then some jiggery-pokery ignores anything that isn’t rated as a 9 or a 10.
What is particularly fascinating is that for something that is so well adopted, and has such an air of scientific rigour, there is very little independent verification that it is in any way useful.
This paper from 2022 concludes that in healthcare:
“The literature suggests that many of the proposed benefits of using NPS are not supported by research. NPS may not be sufficient as a stand‐alone metric and may be better used in conjunction with a larger survey. NPS may be more suited for use in certain healthcare settings, for example, where patients have a choice of provider. Staff attitudes towards the use of NPS for patient surveying are mixed. More research is needed to validate the use of NPS as a primary metric of patient experience.”
This MIT Sloane piece notes:
“we have not seen any rigorous studies that would prove to academics’ satisfaction that NPS is superior to other metrics”
And there is very little else to be found on the web when you look for evidence of the efficacy of the tool.
But even more problematic is that NPS isn’t even used by organisations for the purpose it was originally intended. Go back to the original HBR article and you will find it quite clear that the inventors believed it was a predictor of company growth. Yet NPS is used almost exclusively as a measure of customer satisfaction.
Let’s just put that into context. It’s a bit like asking the question:
“What is your favourite football team?”
and then extrapolating from the answers, which appear to be about towns and cities, that the responses tell you the most popular towns and cities.
But not only that, because we are being asked so often, and by such KPI-hungry questioners, to answer the question about favourite football teams, we are regularly told that unless we answer the question “Manchester City”, all other responses will be discarded, so you might as well just say Manchester City and if you like another football team can you let the person at reception know?
And breathe…
If you truly care about your customers, you should care enough to spend some time thinking about what to ask them. If you want to know how likely they are to recommend you to someone else, why not set up a recommendation service that allows them TO ACTUALLY RECOMMEND YOU. (Don’t get me started on companies that do this and then ask NPS questions too).
NPS was invented in the days before ubiquitous internet. It was invented when recommending something to someone wasn’t merely a click away on a social network. It was invented when AirBnB and TripAdvisor and Uber were not even ideas yet.
And even then it’s not clear that it was in any way an effective measurement of company growth.
If you care about your customers, you should take the money that your company would once have put into the administration of social surveying, and instead spend it on people who can design good surveys for you that can then be administered quickly and cheaply on the internet.
If you care about your customers, you should think about when you ask them for their opinions and design out asking needy questions at points when they might well have something better to do, like actually use your product or service.
I know that this is probably a futile battle. But for me NPS signifies so much that is wrong with the use of data in organisations. Thank you for letting me rant.