Thursday, March 29, 2012

The Groupon effect on Yelp ratings (Guest Post)

Giorgos Zervas discusses our recent paper (just accepted to EC).

A few months ago, in a paper that eventually appeared at WSDM 2012, John, Michael and I (Giorgos) presented a finding that attracted a fair bit of attention: using data we collected from Groupon and Yelp we observed that subsequent to Groupon daily deals Yelp merchant ratings, on average, declined. Our finding can be pretty much summarized in one figure.



The figure displays the running-average Yelp rating (reset on day zero) of about five thousand merchants who ran Groupon deals centered around their deal offer dates. You can immediately observe a sharp discontinuity at offset zero coinciding with the day the deals were offered. Our original plot, limited by the data available to us at the time, only went out to offset 180. With more data becoming available we plotted Yelp ratings further out in the future. A second dip around offset 180 initially puzzled us. Looking deeper into it we realized that a large fraction of the deals we were studying had a six-month expiry date suggesting consumers were rushing to use their purchased coupons in the last minute.

Naturally, these observations raised more questions than they could answer. Why were ratings declining? Was it truly something to do with Groupon? Both online (e.g., see Rocky Agrawal's blog post), and at various places where I presented our work people came up with various legitimate hypotheses that could explain the decline, some of them unrelated to Groupon.

Here's the list I compiled:

1. Intrinsic decline: Prior work (e.g., Godes and Silva) shows that review scores fall over time.

2. Critical reviewers: Groupon users are more critical than their peers.

3. Bad, unprepared, or discriminatory businesses: Merchants who feel compelled to offer a Groupon are in trouble, unprepared to handle the influx of Groupon customers, or discriminatory against Groupon bearing customers.

4. Experimentation: Groupon users are trying new things out when using coupons and hence are more prone to disappointment.

To the above we added a hypothesis of our own:

5. Artificial reviews: Reviews mentioning Groupon are less likely to be fake, hence, the observed decline is due, at least in part, to a previously inflated Yelp rating.

In a new paper to be presented at EC 2012, we consider the above hypotheses by building empirical models for our data. We find varying degrees of support for all of them except, interestingly, for the second one: Groupon users are not by nature that much more critical than their peers -- that is, unless they mention Groupon in their reviews. In fact, they are more moderate, providing fewer 1- and 5-star ratings. Furthermore, we find that they are more active on Yelp, they write better and longer reviews that are less likely to be filtered, and they have more Yelp friends. This suggests to us that, if anything, merchants running daily deal offers should be treating their Groupon customers with special care.

Even though the focus of this work is fairly specific, I think it has broader implications. First, it demonstrates the side-effects marketing efforts can have when considered as part of the greater online ecosystem; and, second, that in fact, these effects can be accurately measured and accounted for. Groupon, in a recent NPR program discussing our work, commented: "It's been documented that anytime an influx of customers visits a business their online reviews tend to see a decrease in quality. This effect holds true for online as well as traditional advertising such as print, broadcast, radio." I find their comment particularly insightful. The greater question to be answered now is not the isolated effectiveness of Groupon (and other daily deal providers) as a marketing mechanism but how it fares in comparison to other customer acquisition methods. I hope that our techniques can be applied in answering this question.

Thank you to everyone who's come to my talks, asked questions, and shared your thoughts on the subject. Hopefully, our new paper demonstrates that I was listening!

Sunday, March 25, 2012

Aside on the Politics of Teaching

Paul Krugman says:

"OK, I wasn’t going to post anything else, but as Robert Farley says, this WaPo piece on underworked college teachers is outrageous."


I'll agree.



Saturday, March 24, 2012

Best Written Paper

Daniel Lemire pointed to an article on bad writing in science (here if you care to see, not CS-specific), which got me to thinking:  do we (in whatever subcommunity you think of yourself being in) value good writing?  Should we?

One question is what qualifies as good writing in science.  I'm not sure there's any consensus here -- although that's true for writing more generally as well.  While colorful word choice and usage can garner some attention* (and, generally, wouldn't hurt), unlike what some people may think, good writing in science is not a vocabulary exercise.  I find that two key features cover most of what I mean by good writing:

1)  Be clear.
2)  Tell a story.

The two actually tend to go hand in hand.  It helps clarify things for the reader if you have a natural progression -- a story to tell -- and if in advance you find and lay out your story -- what is it that is important that you want to convey to your reader -- your writing will naturally become more clear.  The simple exercise I try to follow (when I'm writing well) is

1)  look at each paragraph
2)  make sure I can state to myself what the point of that paragraph is
3)  make sure that point fits in with the story I'm trying to tell
4)  make sure that point is what is coming out from what I wrote

For really good papers, my co-authors and I do the same thing at a more detailed level, looking at each sentence within a paragraph to that it assists in making the point of of the paragraph.  And yes, this all is easier to do with co-authors;  it's much harder to look critically at your own writing at this level.  

I like to think good writing matters.  I am happy to say that I have received reviews saying that my submission was well written, so clearly some subset of people notice.  (Of course this happens much more often with co-authored papers -- thank you co-authors!)  I certainly think I notice good writing in a paper.

I'd like to believe that good writing increases the chances that one's paper is accepted... but I admit, my perspective both as a reviewer and an author is that the effect is probably small.  Bad writing can kill a paper, but my impression is that good writing is just a minor delta over passable writing in conference reviews.  I can understand the perspective.  Conferences are about getting good, interesting work out in a very timely fashion.  The level of writing, one might expect, is secondary.  (Hence the paper reviews saying "Well written, but not deep/novel/exciting enough...")

There is one caveat to the above.  I believe that trying to write clearly forces a researcher to refine and think more carefully about their own work, giving one more insight, and thereby making the work better.  It's said that you haven't really learned something until you've had to teach it to someone else.  The very act of having to "teach" your work, through writing it well, can improve your work.  I've often had interesting ideas arise in the act of writing a paper, when confronted with what felt like holes that needed to be filled.

I also think the value in the writing is longer term.  I believe that well written papers are more likely to pass the test of time, although I admit to having no particular evidence to back that conception.  Similarly, I believe that writing well is good for one's standing in the community longer-term:  if your papers are more pleasurable to read, more people will read and remember them.  Of course ideas are primary, but innovative ideas can have a larger impact based on the quality of the writing explaining them.    

Do you find writing quality matters?  Do you think it should?  We give "Best Paper" awards at (some) conferences.  Perhaps there should be a specific award for "Best Written Paper", to encourage quality writing.  Of course, then we'd have to find people to judge writing for such an award, and we all know scientists can't write.

* Some authors like to choose a catchy title, or an amusing acronym for their system.  Nothing wrong with that, but I think the emphasis on these hooks is easily overstated.  

Sunday, March 11, 2012

A Brief Comment on Conference Software

I'm currently juggling serving on two PCs.  One uses HotCRP.  One uses EDAS. 

In my humble opinion, HotCRP hugely dominates EDAS.   Perhaps EDAS isn't actually so bad, but by comparison, it's just painful. 

Housing Day Midterm

As a sign I just must be getting old, I don't get an issue some students have with me.

Since I've been professing, I hold the midterm for my class the Thursday before spring break.  It's where the exam fits best, and I'm well aware that the Thursday before spring break would unofficially become a holiday if the midterm wasn't there.  This is, already, bad enough for students, in that they now can't leave for spring break until Thursday afternoon.  

A few years ago, however, a new tradition sprung up around Harvard.  Housing Day is when freshman (who live, generally speaking, in Harvard Yard) are told which of the 12 Houses (or big Harvard dorms) they will live in the next 3 years.  Housing Day has become a celebration day, apparently with parties all day for upperclassmen to welcome the new students to their House.  (A Crimson article explains further.)

And, of course, Housing Day falls on the Thursday before spring break. 

The first year or two I heard rumblings, and they've grown louder every year.  When students hear about the midterm, they ask, "Don't you know that's Housing Day?" 

I'm afraid I consider my class "grandfathered in".  I started teaching it well before Housing Day became important;  I don't see any reason for me to change my scheduling to accommodate it.  I'm not sure whether I would be more careful to avoid Housing Day if I were teaching a new class, where I could consider it a constraint from day one. 

But, I admit, I probably wouldn't. 

Tuesday, March 06, 2012

Pointers, Not Papers

Having just crammed everything we could of a 10 page paper into 5 pages for a workshop paper, I admit that besides (once again) being annoyed by having to go through this process, I'm not clear for the reasons behind it. 

I understand reasons for page limits.  Some authors would not self-edit unless forced to do so, and a little regulation arguably makes life better for everyone -- especially the reviewers, but also other readers, and even the authors who wouldn't otherwise take care to edit their text.  In some (rare, I think, these days) cases, there might actually be sound logistical publishing reasons for a page limit.  Also, one might argue that, as a matter of fairness, authors should be obliged to have a somewhat tight rather than loose upper bound on what they can write. 

But 5 double-spaced pages?  (Or, for those who use LNCS format, 12 pages, which is roughly the same?)  Page 1 is a basic intro, Page 2 gives relevant background, such as prior work, definitions and terminology, Page 3 (for my papers, generally) provides theoretical results (little or no proofs, of course, for any subject with meat -- at best a high-level sketch and discussion), Page 4 and whatever room I can take from Page 5 for experiments (including plots!) and analysis, and what's left of Page 5 for a brief conclusion and the references.  Anyone who writes papers can tell there's going to be plenty of stuff you have to cut to make this work.  Some of what is cut is just the textured, nuanced commentary that would help the reader;  some of it is actually important.  (Leading to my irrational anger when I get reviewer comments of the form:  "Why didn't you talk about XXX?" when my answer is WE DID, but we had to take it out to meet his arbitrary page limit.)  

Papers these days, I guess, are often just advertisements for the work up on the arxiv.*  Is this the product of a reasoned argument on the right page limit?  I'd like to hear the arguments.**

*  I note some conferences have gone the other way, allowing longer papers since it's all electronic and cost of paper is not really a concern.  See EC this year...

** Right now the only one I know of is that workshop papers for things like HOT papers should be 5-6 pages, so when you submit the 10-14 page version to the regular conference there will be enough "new" material to justify another publication.  I don't find this a good argument, personally.    

Monday, March 05, 2012

Making a Midterm is No Fun

I think one of most challenging parts of teaching a course is setting up exams. 

There are a lot of constraints:

You want to cover, as reasonably as possible, all the material.
You want to target somewhere around the median of the class;  not too hard, not too easy. 
You don't want there to be any severe typos or other problems with problem;  you can't have a problem without a solution or is otherwise confusing.  
There are a limited number of good problems readily available;  making up new problems is challenging.  (Again, you don't want errors...)
You have to make it all do-able in 90 minutes.  
And so on....

I look forward to reaching a point where I don't think it's worth giving exams to students.  (I don't in my grad classes.)  For my undergrad class, I still think it's necessary. 

Saturday, March 03, 2012

Inaccessible

The major reason for my lack of blog posts recently is that I was cut off from the Internet for a week.*  Surprisingly, the experience was not so bad.  I actually enjoyed being offline, and even the pain of coming back was not as bad as I had thought -- it only took about a day to catch up with e-mail.  Much e-mail was junk that needed no attention, a good bit was time-sensitive stuff that simply moved on happily without me, and a smallish fraction was actually important and time-consuming.  Of course, there was an inherent bias, since many people knew I'd be away.  I can't remember the last time I was off the Internet for a week -- but it has been a long time.  There are plenty of days that I go without watching TV, or even driving my car, but a day without the Internet is in itself a fairly rare event.

On the trip back I used GogoInflight and simultaneously had a great and frustrating experience.  Since I had a stopover, I went with a 24-hour package.  My first short flight, on a relatively small plane, had service and all went very well.  Internet on a plane is indeed useful (particularly when you have a week of e-mail to catch up on).  However, my much longer second leg on a much bigger plane didn't have the service, making the purchase something of a waste.  I don't blame Gogo, I blame American Airlines...

And for those with stopovers in Dallas, always remember to bring your CAT5 cable.  Somehow, the terminal there I frequent has free WIRED Internet access, but only if you know to bring your own cable to connect your laptop.  On the plus side, I've never had problem getting a seat at the Internet table, since nobody else seems to travel with the required cord.  It makes me happy that Boston is one of those friendly airports with free wireless.

The other reason for the lack of blog posts has been the need to catch up.  When were those reviews due again?....  What's that next conference deadline?.... When's my midterm?....  Back to it.

*OK, technically, not true.  However, cruise ship rates for Internet access are a fairly ridiculous 65 cents a minute (with smallish discounts for bulk time).  It didn't seem worthwhile.