New York has cracked down on fake Yelp reviews, and new research from Harvard Business School suggests that such vigilance might be necessary. According to a study (pdf) titled "Fake It Till You Make It," nearly 16% of all Yelp restaurant reviews in the metropolitan Boston area are fake. Not surprisingly, the phony reviews also tend to be "more extreme" (either positive or negative). The study has findings about why restaurants might write or procure both positive and negative fraudulent reviews.
Through an analysis of Yelp's filtering system, the team comes to the following conclusions:
First, roughly 16 percent of restaurant reviews on Yelp are identified as fraudulent, and tend to be more extreme (favorable or unfavorable) than other reviews. Second, a restaurant is more likely to commit review fraud when its reputation is weak, i.e., when it has few reviews, or it has recently received bad reviews. Third, chain restaurants — which benefit less from Yelp — are also less likely to commit review fraud. Fourth, when restaurants face increased competition, they become more likely to leave unfavorable reviews for competitors. Taken in aggregate, these findings highlight the extent of review fraud and suggest that a business's decision to commit review fraud respond to competition and reputation incentives rather than simply the restaurant's ethics.
The conclusions were drawn from a quantitative analysis of posted restaurant reviews (filtered and unfiltered). The analysis covers "every Yelp [restaurant] review" in the Boston area "that was written from the founding of Yelp in 2004 through 2012, other than the roughly 1% of reviews that violate Yelp's terms of service (for example, reviews that contain offensive, or discriminatory language.)" The study analyzes 316,415 reviews of 3,625 restaurants. Yelp filtered reviews might be an "imperfect indicator of fake reviews," but the analysis attempts to "empirically identify the circumstances under which fraud is prevalent." The study works under the assumption that "the proportion of fake reviews is strictly smaller among the reviews Yelp publishes, than the reviews Yelp filters," and offers mathematical analysis to support that theory.
The study asks: "Does review fraud respond to economic incentives, or is it driven mainly by a small number of unethical restaurants that are intent on gaming the system regardless of the situation?" It finds that restaurants with fewer reviews (i.e. "worse" reputations) are "more likely to engage in fraud," which "motivates an additional explanation for the often-observed concentration of high ratings earlier in a business' life-cycle." According to the study, some new restaurants are creating fake positive reviews to drum up early support.
Similarly, the study finds that after receiving negative reviews, restaurants are more likely to engage in "positive review fraud." There was no link found between negative review fraud and a restaurant's reputation (as indicated by the number and quality of Yelp reviews). The study suggests that fake negative reviews are "presumably left by a business' competitors." Chain restaurants, the study concludes, are less likely to commit review fraud because their reputations "offline" are so solidified.
While the study does seem to make sense on a sort of intuitive logic level, there seems to be a missing piece in the fake review landscape: individual Yelpers with an ax to grind. The study says: "consider what a fake review might look like: fake reviews may consist of a business leaving favorable reviews for itself, or unfavorable reviews for its competitors." The study's aim is to analyze the broadest trends, but isn't it a leap to assume that fake reviewing (especially negative fake reviewing) is done by competitors as opposed to disgruntled employees or unscrupulous Yelp addicts? The phenomenon of Yelp extortion is nothing new, and doesn't seem to be accounted for in the study.
The study puts its faith in the Yelp filtering system's ability to identify which reviews are fake, and which are real. But given that CEO Jeremy Stoppelman has claimed that Yelp is in an "arms race" regarding fake reviews and the Atlantic has reported on the Yelp filter's several slip-ups, the study's trust might not be entirely deserved. However, a New York Attorney General's report cites research firm Gartner who "projects that by 2014, between 10% and 15% of social media reviews will be fake." The statistic is slightly lower than the Harvard study's, but the data on fraudulent restaurant reviews makes it clear: Yelp still has a lot of work to do.