(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
This one is fun because I have a double conflict of interest: I’ve been paid (at different times) both by Google and by Microsoft.
Here’s the story:
Microsoft, September 2012:
An independent research company, Answers Research based in San Diego, CA, conducted a study using a representative online sample of nearly 1000 people, ages 18 and older from across the US. The participants were chosen from a random survey panel and were required to have used a major search engine in the past month. Participants were not aware that Microsoft was involved.
In the test, participants were shown the main web search results pane of both Bing and Google for 10 search queries of their choice. Bing and Google search results were shown side-by-side on one page for easy comparison – with all branding removed from both search engines. The test did not include ads or content in other parts of the page such as Bing’s Snapshot and Social Search panes and Google’s Knowledge Graph. For each search, the participant was asked which search engine provided the best results – “Left side search engine”, “Right side search engine”, or “Draw.” After each participant performed 10 searches, their votes were totaled to determine the winner (Bing, Google or Draw, in the case of a tie).
When the results were tallied, the outcome was clear – people chose Bing web search results over Google nearly 2:1 in the blind comparison tests. Specifically, of the nearly 1000 participants: 57.4% chose Bing more often, 30.2% chose Google more often; 12.4% resulted in a draw.
A group at Yale Law School, October 2013:
In advertisements associated with its ―Bing It On‖ campaign, Microsoft claimed that ―people preferred Bing web search results nearly 2:1 over Google in blind comparison tests.‖ We tested Microsoft‘s claims by way of a randomized experiment involving U.S.-based MTurk subjects and conducted on Microsoft‘s own www.bingiton.com website. We found that (i) a statistically-significant majority of participants preferred Google search results to Bing search results (53% to 41%); and (ii) participants were significantly less likely to prefer Bing results when randomly assigned to use popular search terms or self-selected search terms instead of the search terms Microsoft recommends test-takers employ on its website.
And then the punchline:
Our findings suggest that each of these implicit claims is likely false and might provide the basis for a viable Lanham Act claim by Google.
It seems a bit much to claim false advertising because a 2013 survey on Mechanical Turk differs from a 2012 survey on a random sample of Americans. Also, and possibly more important than the populations being studied, the experimental conditions were different. In Microsoft’s original study, it appears that the purpose of the study was somewhat concealed, whereas the Yale study went directly through the Bing website. So, lots of differences from two very different studies performed on two very different populations under two very different conditions, a year apart.
The authors of the 2013 paper argue that the Turk sample is better than the U.S. random sample because it better captures internet users. That’s a reasonable point but I think I’d prefer to go with the random sample and just reweight. In any case, it hardly seems like deceptive advertising for Microsoft to make a claim about Americans, based on a random sample of Americans.
Just imagine the results had gone the other way around. Suppose Microsoft had done a MTurk study and the Yale group turned around and found a much different result using a third-party random sample. I can only assume that they’d be saying that Microsoft was doing illegal advertising by representing the results of a convenience sample as if it were real.
But maybe I’m not really putting on my “lawyer hat” here. I was thinking that, because the authors of this 2013 paper wrote that Microsoft could be sued for false advertising, that they believe that Microsoft did illegally false advertising. But the threat is more powerful than the execution, right? You only have to threaten to sue, maybe that’s enough? It seems a bit creepy to me, along the lines of throwing mud at something on the hope that it might stick. (Microsoft’s own response is here.)
That said, I personally prefer Google to Bing. My impression that Google is a bit more tailored to academic users, and Bing doesn’t really work for me.
But I’m with the Bing guys in the above debate. I’m no fan of their search engine, but their study seems much more professional to me. Really it seems like no contest. Ultimately, Bing wins for some searches and for some users, and Google wins for other searches and for other users. For all I know, they’re using very similar algorithms, just with different weights. Or maybe when you type something into Bing, it just googles it. Whatever.
The statistical message here is that in the presence of large and systematic variation, different experimental conditions will give you different average treatment effects. If I had to pick just one of the experiments discussed here, I’d choose Microsoft’s, hands down. But the variation is the interesting story. To their credit, the Yale team did explore the variation a bit, before they went into the legal mumbo-jumbo.
A fight that benefits both sides
Often we say that, in a fight, both sides lose. In this case, though, maybe both sides win.
From Microsoft’s side, more attention is drawn to their Bing-is-preferred-to-Google study, and the attack is so weak that it lends credence to Microsoft’s pro-Bing claims. (Just consider: if Microsoft had just released this study on its own, not in the context of a ham-handed attack, maybe I’d be posting on all the reasons not to take the Bing claims seriously.) Bing is the forgotten and distant #2 in searches, so I assume they’ll be happy with anything that keeps them in the news.
And, from the Yale team’s side, I can’t imagine that these people are really so fired up angry about Microsoft’s purportedly illegal misrepresentation. Rather, they did a cute little project (the authors are a professor and 5 students) and thought it would be fun to get some press. No harm in that: I do fun projects myself from time to time and am not averse to seeing my name in the newspaper. And they might even get some legal business from this, I suppose, for example if Google or some legal entrepreneur wanted to actually try out their suggested lawsuit.
I do feel a little bit bad about contributing to the publicity. But I’ll do it, for the statistical content.
P.S. I tried to play “bingiton” myself but when I clicked on the website (“Bing It On is a side-by-side Bing versus Google search-off challenge. . . .”), I got switched directly to www.bing.com. I guess bingiton is no more?
P.P.S. I purposely did not post the names of the Microsoft psychologist or the Yale law professor here, because I’d like to avoid personalizing the conflict. Microsoft vs. Yale seems better off than Mr. A vs. Mr. B. (If you care, all the names and links are in this news article by Will Oremus.)
The post Bing is preferred to Google by people who aren’t like me appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
The post Bing is preferred to Google by people who aren’t like me appeared first on All About Statistics.