Do you remember last year when Bing aggressively marketed their “Bing It On” challenge? Back then, Bing also made the claim that searchers prefered Bing in the blind test 2:1. Now, a recent study from the “Freakonomics” blog harshly disputes Bing’s claims as well as strongly critiquing their marketing of the study as possibly constituting deceptive advertising.

Law and economics professor Ian Ayers explained the study and says the claims made by Bing came from a small sample of only 1,000 participants, too few to be a reliable sample of the population. He attempted to replicate the study at Yale Law School where he teaches:

I set up a similar-sized experiment using Microsoft’s own BingItOn.com site to see which search engine users prefer. We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were “ties”)…

The thing is the results of this study still aren’t too bad for Bing. Yes, it shows that Bing’s claims may have been disingenuous, and Google could very possibly still be the more favored search engine. However, Bing still came out with a very respectable number of people favoring their search engine. This could be a sign that Bing could very feasibly grow their market share further than they have. They currently take 18 percent of the market, compared to Google’s static 67 percent.

Of course, Microsoft has responded to the study already. They issued a formal statement from Matt Wallaert, behavioral scientist at Bing, which read:

The professor’s analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100% accurate and we’re glad to see we’ve nudged Google into improving their results. Bing it On is intended to be a lightweight way to challenge peoples’ assumptions about which search engine actually provides the best results. Given our share gains, it’s clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service.

There was also a much more extensive response to the Ayers study in the form of a post on the Bing Blog.

And of course, Matt Cutts from Google couldn’t help but share his two cents on Google+:

Freakonomics looked into Microsoft’s “Bing It On” challenge. From the blog post: “tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results. …. The upshot: Several of Microsoft’s claims are a little fishy. Or to put the conclusion more formally, we think that Google has a colorable deceptive advertising claim.”

I have to admit that I never bothered to debunk the Bing It On challenge, because the flaws (small sample size; bias in query selection; stripping out features of Google like geolocation, personalization, and Knowledge Graph; wording of the site; selective rematches) were pretty obvious.

So yes, Bing’s study had a fair amount of problems, and their critics seem poised to take action about deceptive marketing, but there is an upshot for Bing. The search engine still performed very well in the results and could gain a fair amount of searchers in the future. But, it also confirms something Bing may have been more worried about. As Greg Sterling points out, “the Google brand and not necessarily search quality is now what sustains Google’s dominance in search.”