A few months ago, in a paper that eventually appeared at WSDM 2012, John, Michael and I (Giorgos) presented a finding that attracted a fair bit of attention: using data we collected from Groupon and Yelp we observed that subsequent to Groupon daily deals Yelp merchant ratings, on average, declined. Our finding can be pretty much summarized in one figure.
The figure displays the running-average Yelp rating (reset on day zero) of about five thousand merchants who ran Groupon deals centered around their deal offer dates. You can immediately observe a sharp discontinuity at offset zero coinciding with the day the deals were offered. Our original plot, limited by the data available to us at the time, only went out to offset 180. With more data becoming available we plotted Yelp ratings further out in the future. A second dip around offset 180 initially puzzled us. Looking deeper into it we realized that a large fraction of the deals we were studying had a six-month expiry date suggesting consumers were rushing to use their purchased coupons in the last minute.
Naturally, these observations raised more questions than they could answer. Why were ratings declining? Was it truly something to do with Groupon? Both online (e.g., see Rocky Agrawal's blog post), and at various places where I presented our work people came up with various legitimate hypotheses that could explain the decline, some of them unrelated to Groupon.
Here's the list I compiled:
1. Intrinsic decline: Prior work (e.g., Godes and Silva) shows that review scores fall over time.
2. Critical reviewers: Groupon users are more critical than their peers.
3. Bad, unprepared, or discriminatory businesses: Merchants who feel compelled to offer a Groupon are in trouble, unprepared to handle the influx of Groupon customers, or discriminatory against Groupon bearing customers.
4. Experimentation: Groupon users are trying new things out when using coupons and hence are more prone to disappointment.
To the above we added a hypothesis of our own:
5. Artificial reviews: Reviews mentioning Groupon are less likely to be fake, hence, the observed decline is due, at least in part, to a previously inflated Yelp rating.
In a new paper to be presented at EC 2012, we consider the above hypotheses by building empirical models for our data. We find varying degrees of support for all of them except, interestingly, for the second one: Groupon users are not by nature that much more critical than their peers -- that is, unless they mention Groupon in their reviews. In fact, they are more moderate, providing fewer 1- and 5-star ratings. Furthermore, we find that they are more active on Yelp, they write better and longer reviews that are less likely to be filtered, and they have more Yelp friends. This suggests to us that, if anything, merchants running daily deal offers should be treating their Groupon customers with special care.
Even though the focus of this work is fairly specific, I think it has broader implications. First, it demonstrates the side-effects marketing efforts can have when considered as part of the greater online ecosystem; and, second, that in fact, these effects can be accurately measured and accounted for. Groupon, in a recent NPR program discussing our work, commented: "It's been documented that anytime an influx of customers visits a business their online reviews tend to see a decrease in quality. This effect holds true for online as well as traditional advertising such as print, broadcast, radio." I find their comment particularly insightful. The greater question to be answered now is not the isolated effectiveness of Groupon (and other daily deal providers) as a marketing mechanism but how it fares in comparison to other customer acquisition methods. I hope that our techniques can be applied in answering this question.
Thank you to everyone who's come to my talks, asked questions, and shared your thoughts on the subject. Hopefully, our new paper demonstrates that I was listening!
14 comments:
Part of the drop may be due to Groupon inducing people to visit a business selling something they otherwise wouldn't be interested in. E.g., I might check out a Greek restaurant even though I'm indifferent to Greek food because of a good deal and curiosity. Given the same experience as someone who likes the cuisine, I might then give a lower rating.
I think an alternate explanation would be that people might go a bit out of their comfort zone in order to save money. For instance, someone might not like Indian food that much, but they see a 50% off coupon for an Indian place, and go for it, They would have a much harder time giving a 5 star rating compared to a burger joint. This might be what you're trying to say with the "experimentation."
The other reason I thought of is that there are 2 classes of people that leave ratings: people that really like or dislike the place, and people that rate because they've been there independent of liking or not. Personally, I'm more in the former camp. It's possible that the groupon population is selected for being in the second category rather than the first.
Anons #1 & #2: What we mean by "experimentation" (point 4 above) is indeed what you describe: the coupon discount encouraging consumers to try out businesses that they otherwise wouldn't. Unsurprisingly, we find that Groupon does indeed encourages experimentation. However, we have not determined whether more experimentation leads to worse ratings. People might be pleasantly surprised by a new discovery as often as they are disappointed. It's something we are looking into.
Anon #2: We also considered the possibility that Groupon reviewers are more extreme in their ratings (point 2). It turns out, that in fact they are on average more moderate.
Hi George, a really nice and convincing body of work! I find the evidence you have presented that Groupon users are in general unbiased, in fact better socially connected and more influential than non-Groupon reviewers, an immensely compelling contribution. I do realize that my statement might be slightly over-stating your result :-).
Two aspects that I feel are important here relate to the quality of service offered by businesses once they know that their Groupon deal is 'on'.
I wonder with regards to distinguishing the quality of service offered by businesses to customers, is there a difference between businesses where the customer presents the Groupon prior to the service being rendered vis-a-vis businesses where the Groupon is presented as payment after the service has been provided? I can also see that presenting a Groupon as payment could in many cases elicit an unpleasant reaction from the service provider which in turn could compel the user to leave a lower than deserved rating. Evaluating this, of course, is not trivial.
The other aspect is that since businesses are aware that their Groupon deal is on and hence expect a significant fraction of customers are going to be non-profitable Groupon users, the quality of service for both Groupon and non-Groupon customers might take a back seat. Is there a comparison you could draw between the reviews provided by these two sets of customers in the first few days after the deal is run? It seems from your plots that both sets of reviews would have a similar negative trend. If indeed true, this would once again substantiate the unbiasedness of Groupon users argument you have made.
Hi Anon #4. Thanks for your good words and feedback.
Regarding your first point, this would be something interesting to look into but we haven't yet. Maybe the user reviews can in some cases reveal whether the Groupon was redeemed before or after services were rendered. Or, maybe there are businesses where it's customary to pay up front (like fast-food places), and business where you pay later (like restaurants) and therefore we can guess when the Groupon was presented. I wonder if the data can tell us something here. I will look into it.
Regarding your second point, reviews that do not mention Groupon also suffer a drop post Groupon-deal but it's smaller than the drop we see in Groupon-mentioning reviews. For some numbers, you might want to look at Table VI of our paper, specifically the row labelled "During Groupon deal". These are the changes in ratings during a Groupon deal after controlling for Groupon users. There is a caveat to be aware of here: maybe some consumers who use Groupon do not mention it explicitly in their reviews, so, in part, this could be an artifact of our methodology for identifying Groupon users (i.e., those who mention "Groupon" in their reviews). We are working on evaluating the accuracy of this heuristic.
Thanks again for sharing your thoughts!
In couple instance in bay area it seems there is discrimination. The groupon has to be presented before teh service and the groupon tag is on the order. Every person involved in the service s aware that this person isn't paying full price. also the stigma that this person is shopping for lowest cost and hence may not be a good tipper.
Hi, I have two comments. First, you write "The figure displays the running-average Yelp rating (reset on day zero)..." I take this to mean that the thing being plotted is the average rating over all time, and from day zero onward you're plotting the average rating over only ratings from day zero onward. If this is the case then the discontinuity you see could very well be due to the reset mechanism. Consider this situation: the _instantaneous_ rating scores are falling over time. Since you're taking the average over all time, the scores before day 0 are artificially higher than what raters are rating _on that day_, because they also include data from the past, when ratings were higher. On day 0, you throw out all those higher ratings, so the running average reverts to the instantaneous daily average, hence the jump. If this were the case, you would get a jump in the graph no matter which day you pick as day 0. It would be much more convincing to me if you plotted the figure, for example, as a 10-day windowed average (taking the average of only ratings from the 10-day window surrounding the current date instead of the running average). The figure would be spikier, but there would be no "0-day bias".
Second, it would be nice to compare the data in your paper with data from merchants who did not participate in Groupon deals (this is the concept of a "control group") to discern how much of the effects you're seeing are specific to Groupon/Livingsocial.
I would be very interested to see the analysis from these two points.
The 180 day drop is *after* the 180 days. It's people going in with a Groupon that just expired and being denied.
What is the quantity of the dip? Is it 30% of a star or what? Hard to tell from the chart (you cut the y-axis off quite a bit as well, it should at least start at 0)
Good points, Anonymous April 8, 2012 3:10 PM.
I also think regular customers still posting reviews then an abrupt shift needs a significant increase in the number of reviews.
hi Anon 3:10PM, if you we look at the main graph in their new EC paper they do plot the raw daily ratings data (i.e, no running average of any type) along with the information presented here. The drop in rating is rather obvious even in the raw data, so the running average is not the problem. I personally found the raw data to be particularly illuminating, however, I understand George's reason to present the uncluttered graph in the blog post. About your second statement once again the main graph shows the huge jump in the number of ratings post-groupon and the associated decline after groupon deal. For each company we can compare before and after to see the difference. So this already seems to answer your question.
I think that the Groupon purchaser comes into the place of business expecting that they are not going to get the service and quality that a non Groupon customer would get, kind of like with a chip on their shoulder. I know, I have run sales with Groupon and other social sites. The Groupon customer wants the deal and then they want more! And in most service situations they resent leaving a tip much less leaving one based on the full price of their service. Then they bash you on Yelp for "suggesting" that they tip.
Many congratulations to the authors on earning the first plenary talk of the day at EC, well deserved recognition for what can only be regarded as extremely influential work! I for one, root for, and look forward to more good news to come in a few weeks.
Groupon deals have gone down for sometime because of its unattractive deals.People go in for the regular shopping they acquire satisfaction over it.Groupon should be publishing more real time deals often.
free groupon clone
Post a Comment