Sandel on Colbert

Taking a class from Michael Sandel was one of the highlights of my college experience.  (You can sit on the lectures online, here.)  Here's his latest appearance on Colbert from yesterday. (And a link to his new book...)

A Collaborative Atmosphere

I was excited to hear that "my student"* Justin Thaler, working with Salil Vadhan and his student Jon Ullman, had a paper accepted to ICALP (Faster Algorithms for Privately Releasing Marginals).  It's a great example of something that's been going very nicely here at Harvard:  students (+advisors) crossing boundaries and collaborating together on problems. 

I know it's a common refrain that people don't realize that computer science research is, generally speaking, a highly collaborative enterprise, but it's certainly worth repeating.  Most projects aren't done by one person coding or mathematically proving alone, but working as part of a team.  Graduate students should know that and embrace it.  There are huge advantages of this approach.  The natural one to point out is the great synergies in developing and working through ideas from multiple people with different skill sets and points of view.  You also learn about new things that can turn out to be useful to you in unpredictable ways by working on projects outside your standard box.  But perhaps less often mentioned is that working with others is generally just much more fun, and I think it's easier to produce good and great research when you're having fun doing it.

I feel like that's stating the obvious, but thinking back to my time in graduate school, maybe it's not.  Collaboration in the theory group certainly wasn't frowned upon, but it wasn't entirely actively promoted, either.  Micah Adler and John Byers worked very successfully together on several papers early in their graduate student careers, and I remember working on some problems with multiple friendly older students.  But I also recall, while not what I would call a competitive atmosphere, the sense that you really had to prove yourself "on your own", especially in putting together a thesis, leading to at least some cases where "credit" became a issue to some people.  The theory group was pretty social, so overall the joys of working together probably won out overall.  To me today feels like a non-trivial delta from that (here at Harvard at least, but more widely as well), though perhaps to the students aiming to graduate and get jobs it doesn't feel that way -- I can imagine, depending on the setting, that the pressure to do something great on your own is still primary. 

Maybe the most encouraging thing I can try to say, which really clicked solidly in my head this recruiting season, is that one thing I look for in a candidate is whether they can work well with other people -- not just their advisor.  I think that's important, both for the candidate's future success and for CS at Harvard.  I don't think there's a formula I have in mind to weight "individual success" vs. "group success", but I'd be more skeptical of a candidate that didn't have signs of both.  Further, I recognize that the talent of the individual can shine through on group projects -- while the ability of a candidate to work within groups can't really shine through on individual projects.

 
*Increasingly a misnomer, as Justin's well beyond being "my student" in any meaningful sense at this point, but it's still the most convenient reference.

Thanks, Yale Daily News!

Harvard's CS50 class is nicely lauded -- by a column in the Yale Daily News. 

First, the Harvard-centric stuff:

The good news is that there’s an easy first step that Yale computer science could take to start addressing this issue. It pains me to say it, but we could learn a thing or two from a certain institution in Cambridge. One thing that Harvard has absolutely gotten right is its introductory CS50 class that teaches students of all majors the practical scripting and Web programming skills they need to apply tech to their other interests. My sister — who chose Harvard over Yale partially because of classes like CS50 — started the semester knowing almost nothing about programming and finished with a job offer from a tech startup. She might not even end up being a computer science major, but the class gave her a solid set of skills that she’s already putting to work.

But there's also a nice part in the beginning:

Code is the lingua franca of the 21st century. Whether it’s putting together a website for an advocacy campaign, writing a script to analyze some economic data or creating an app to help kids learn math, programming fluency has become a required skill for anyone looking to have an impact on the world.

I phrase it a bit differently, but try to give the same message when I explain what the Harvard CS department's goals are.  We think Harvard students are future leaders;  code is the "lingua franca" of the century;  surely, we want our leaders to understand the basics of coding.  

Things Worth Reading

A blog post describing the disturbing official version of the facts in the UC-Davis pepper spray incident.  This should be a link to the actual report.    

A post by an undergraduate discussing the importance of math/algorithms in computer science in a creative writing sort of way.  But the most interesting things are the comments, which seem split between those agreeing that such conceptual skills are important, and others who seem to say the real skills are in running the business and hiring the people with these skills.  

At Freedom to Tinker, Andrew Appel discusses ACM/IEEE copyright policies and what to do about them here, here, and here (with more to come). 

Lots of other people besides me are "uncomfortable" with the new gmail interaface.  Here's one site for people to moan and complain;  I've seen others. 

Living with New Interfaces

After weeks of asking Google mail to revert to the old look, I find I no longer have that option.  Similarly, I'm being forced to write this post using the new blogger interface. 

I suppose I'll be open-minded and try them, since Google is not giving me a choice.  I realize I'm not the only one to complain about that by any means.  Perhaps I'll quickly adjust, and after a few days I'll agree with Google that the new systems are better.

But perhaps not.  I think software builders fail to recognize the level at which people grow accustomed to their tools.  Changing the interface is forcing me to change the way I've worked for multiple years.  If I don't like having to make that change, it's a fine opportunity for me to change the product I use.  As long as I'm being forced to learn something new to get done familiar tasks, maybe it's time I learn something really new. 

Google's done the math and decided this is best for them.  They've figured that most of us will adjust to and even like the changes after a short period of time.  We'll see.  But if anyone's decided this is a good time to jump ship or thinks they have better options, let us all know in the comments.  

In the News this Week...

Our large data sets work on MIC is discussed in Nature Biotechnology.  (Funnily, none of the us authors of the paper knew this was coming out.  One of us heard from a colleague that it was there.)  I'm unused to this format where other people discuss your work openly and without your explicit participation.  It's a different culture.  I've heard people call for this style for various online publishing systems -- allowing comments and discussions to follow papers -- and I can see how new social norms might have to be established in various scientific subdisciplines to handle this type of scientific discourse.       

Also, our Groupon-Yelp work is mentioned in Huffington Post.  And, perhaps more importantly, ycombinator.  According to Blogger's stats the ycombinator link sent many many thousands of page views over to Giorgos's post here from two weeks ago, which is a large multiple over the usual traffic.

Late entry:  the great growth of Harvard CS is lauded in this Crimson article from today.

Congratulations to Radhika Nagpal

I'm very, very happy to report that Radhika Nagpal has been offered tenure at Harvard.

Radhika works in several areas, including biologically inspired multi-agent systems, which covers such areas as modular and swarm robotics;  and computational models for biological systems, such as collective behavior and multicellular morphogenesis.  (That's a mouthful.)

If you want to see her work in action, you have to go to her lab's Youtube channel.  Check out robots laying sandbags, or demonstrations of the very very tiny Kilobots acting in concert.  Or check out cell division in a larval fruit fly.  She also the faculty lead on the Harvard side for the Harvard-MIT robot soccer team.  Here's an older but still cool news article on the team. 

We're all excited by the excellent news.  Congratulations Radhika!

Talk at Hearst Mining....

Giorgos is on a west coast tour this week. Today he's speaking at the Berkeley EconCS seminar on our work on Groupon and Yelp.  12:30, 410 Hearst Mining.  Info here.  If you're around, go to the talk and tell him I sent you. 

The Groupon effect on Yelp ratings (Guest Post)

Giorgos Zervas discusses our recent paper (just accepted to EC).

A few months ago, in a paper that eventually appeared at WSDM 2012, John, Michael and I (Giorgos) presented a finding that attracted a fair bit of attention: using data we collected from Groupon and Yelp we observed that subsequent to Groupon daily deals Yelp merchant ratings, on average, declined. Our finding can be pretty much summarized in one figure.

The figure displays the running-average Yelp rating (reset on day zero) of about five thousand merchants who ran Groupon deals centered around their deal offer dates. You can immediately observe a sharp discontinuity at offset zero coinciding with the day the deals were offered. Our original plot, limited by the data available to us at the time, only went out to offset 180. With more data becoming available we plotted Yelp ratings further out in the future. A second dip around offset 180 initially puzzled us. Looking deeper into it we realized that a large fraction of the deals we were studying had a six-month expiry date suggesting consumers were rushing to use their purchased coupons in the last minute.

Naturally, these observations raised more questions than they could answer. Why were ratings declining? Was it truly something to do with Groupon? Both online (e.g., see Rocky Agrawal's blog post), and at various places where I presented our work people came up with various legitimate hypotheses that could explain the decline, some of them unrelated to Groupon.

Here's the list I compiled:

1. Intrinsic decline: Prior work (e.g., Godes and Silva) shows that review scores fall over time.

2. Critical reviewers: Groupon users are more critical than their peers.

3. Bad, unprepared, or discriminatory businesses: Merchants who feel compelled to offer a Groupon are in trouble, unprepared to handle the influx of Groupon customers, or discriminatory against Groupon bearing customers.

4. Experimentation: Groupon users are trying new things out when using coupons and hence are more prone to disappointment.

To the above we added a hypothesis of our own:

5. Artificial reviews: Reviews mentioning Groupon are less likely to be fake, hence, the observed decline is due, at least in part, to a previously inflated Yelp rating.

In a new paper to be presented at EC 2012, we consider the above hypotheses by building empirical models for our data. We find varying degrees of support for all of them except, interestingly, for the second one: Groupon users are not by nature that much more critical than their peers -- that is, unless they mention Groupon in their reviews. In fact, they are more moderate, providing fewer 1- and 5-star ratings. Furthermore, we find that they are more active on Yelp, they write better and longer reviews that are less likely to be filtered, and they have more Yelp friends. This suggests to us that, if anything, merchants running daily deal offers should be treating their Groupon customers with special care.

Even though the focus of this work is fairly specific, I think it has broader implications. First, it demonstrates the side-effects marketing efforts can have when considered as part of the greater online ecosystem; and, second, that in fact, these effects can be accurately measured and accounted for. Groupon, in a recent NPR program discussing our work, commented: "It's been documented that anytime an influx of customers visits a business their online reviews tend to see a decrease in quality. This effect holds true for online as well as traditional advertising such as print, broadcast, radio." I find their comment particularly insightful. The greater question to be answered now is not the isolated effectiveness of Groupon (and other daily deal providers) as a marketing mechanism but how it fares in comparison to other customer acquisition methods. I hope that our techniques can be applied in answering this question.

Thank you to everyone who's come to my talks, asked questions, and shared your thoughts on the subject. Hopefully, our new paper demonstrates that I was listening!

Aside on the Politics of Teaching

Paul Krugman says:

"OK, I wasn’t going to post anything else, but as Robert Farley says, this WaPo piece on underworked college teachers is outrageous."


I'll agree.

Best Written Paper

Daniel Lemire pointed to an article on bad writing in science (here if you care to see, not CS-specific), which got me to thinking:  do we (in whatever subcommunity you think of yourself being in) value good writing?  Should we?

One question is what qualifies as good writing in science.  I'm not sure there's any consensus here -- although that's true for writing more generally as well.  While colorful word choice and usage can garner some attention* (and, generally, wouldn't hurt), unlike what some people may think, good writing in science is not a vocabulary exercise.  I find that two key features cover most of what I mean by good writing:

1)  Be clear.
2)  Tell a story.

The two actually tend to go hand in hand.  It helps clarify things for the reader if you have a natural progression -- a story to tell -- and if in advance you find and lay out your story -- what is it that is important that you want to convey to your reader -- your writing will naturally become more clear.  The simple exercise I try to follow (when I'm writing well) is

1)  look at each paragraph
2)  make sure I can state to myself what the point of that paragraph is
3)  make sure that point fits in with the story I'm trying to tell
4)  make sure that point is what is coming out from what I wrote

For really good papers, my co-authors and I do the same thing at a more detailed level, looking at each sentence within a paragraph to that it assists in making the point of of the paragraph.  And yes, this all is easier to do with co-authors;  it's much harder to look critically at your own writing at this level.  

I like to think good writing matters.  I am happy to say that I have received reviews saying that my submission was well written, so clearly some subset of people notice.  (Of course this happens much more often with co-authored papers -- thank you co-authors!)  I certainly think I notice good writing in a paper.

I'd like to believe that good writing increases the chances that one's paper is accepted... but I admit, my perspective both as a reviewer and an author is that the effect is probably small.  Bad writing can kill a paper, but my impression is that good writing is just a minor delta over passable writing in conference reviews.  I can understand the perspective.  Conferences are about getting good, interesting work out in a very timely fashion.  The level of writing, one might expect, is secondary.  (Hence the paper reviews saying "Well written, but not deep/novel/exciting enough...")

There is one caveat to the above.  I believe that trying to write clearly forces a researcher to refine and think more carefully about their own work, giving one more insight, and thereby making the work better.  It's said that you haven't really learned something until you've had to teach it to someone else.  The very act of having to "teach" your work, through writing it well, can improve your work.  I've often had interesting ideas arise in the act of writing a paper, when confronted with what felt like holes that needed to be filled.

I also think the value in the writing is longer term.  I believe that well written papers are more likely to pass the test of time, although I admit to having no particular evidence to back that conception.  Similarly, I believe that writing well is good for one's standing in the community longer-term:  if your papers are more pleasurable to read, more people will read and remember them.  Of course ideas are primary, but innovative ideas can have a larger impact based on the quality of the writing explaining them.    

Do you find writing quality matters?  Do you think it should?  We give "Best Paper" awards at (some) conferences.  Perhaps there should be a specific award for "Best Written Paper", to encourage quality writing.  Of course, then we'd have to find people to judge writing for such an award, and we all know scientists can't write.

* Some authors like to choose a catchy title, or an amusing acronym for their system.  Nothing wrong with that, but I think the emphasis on these hooks is easily overstated.  

A Brief Comment on Conference Software

I'm currently juggling serving on two PCs.  One uses HotCRP.  One uses EDAS. 

In my humble opinion, HotCRP hugely dominates EDAS.   Perhaps EDAS isn't actually so bad, but by comparison, it's just painful. 

Housing Day Midterm

As a sign I just must be getting old, I don't get an issue some students have with me.

Since I've been professing, I hold the midterm for my class the Thursday before spring break.  It's where the exam fits best, and I'm well aware that the Thursday before spring break would unofficially become a holiday if the midterm wasn't there.  This is, already, bad enough for students, in that they now can't leave for spring break until Thursday afternoon.  

A few years ago, however, a new tradition sprung up around Harvard.  Housing Day is when freshman (who live, generally speaking, in Harvard Yard) are told which of the 12 Houses (or big Harvard dorms) they will live in the next 3 years.  Housing Day has become a celebration day, apparently with parties all day for upperclassmen to welcome the new students to their House.  (A Crimson article explains further.)

And, of course, Housing Day falls on the Thursday before spring break. 

The first year or two I heard rumblings, and they've grown louder every year.  When students hear about the midterm, they ask, "Don't you know that's Housing Day?" 

I'm afraid I consider my class "grandfathered in".  I started teaching it well before Housing Day became important;  I don't see any reason for me to change my scheduling to accommodate it.  I'm not sure whether I would be more careful to avoid Housing Day if I were teaching a new class, where I could consider it a constraint from day one. 

But, I admit, I probably wouldn't. 

Pointers, Not Papers

Having just crammed everything we could of a 10 page paper into 5 pages for a workshop paper, I admit that besides (once again) being annoyed by having to go through this process, I'm not clear for the reasons behind it. 

I understand reasons for page limits.  Some authors would not self-edit unless forced to do so, and a little regulation arguably makes life better for everyone -- especially the reviewers, but also other readers, and even the authors who wouldn't otherwise take care to edit their text.  In some (rare, I think, these days) cases, there might actually be sound logistical publishing reasons for a page limit.  Also, one might argue that, as a matter of fairness, authors should be obliged to have a somewhat tight rather than loose upper bound on what they can write. 

But 5 double-spaced pages?  (Or, for those who use LNCS format, 12 pages, which is roughly the same?)  Page 1 is a basic intro, Page 2 gives relevant background, such as prior work, definitions and terminology, Page 3 (for my papers, generally) provides theoretical results (little or no proofs, of course, for any subject with meat -- at best a high-level sketch and discussion), Page 4 and whatever room I can take from Page 5 for experiments (including plots!) and analysis, and what's left of Page 5 for a brief conclusion and the references.  Anyone who writes papers can tell there's going to be plenty of stuff you have to cut to make this work.  Some of what is cut is just the textured, nuanced commentary that would help the reader;  some of it is actually important.  (Leading to my irrational anger when I get reviewer comments of the form:  "Why didn't you talk about XXX?" when my answer is WE DID, but we had to take it out to meet his arbitrary page limit.)  

Papers these days, I guess, are often just advertisements for the work up on the arxiv.*  Is this the product of a reasoned argument on the right page limit?  I'd like to hear the arguments.**

*  I note some conferences have gone the other way, allowing longer papers since it's all electronic and cost of paper is not really a concern.  See EC this year...

** Right now the only one I know of is that workshop papers for things like HOT papers should be 5-6 pages, so when you submit the 10-14 page version to the regular conference there will be enough "new" material to justify another publication.  I don't find this a good argument, personally.    

Making a Midterm is No Fun

I think one of most challenging parts of teaching a course is setting up exams. 

There are a lot of constraints:

You want to cover, as reasonably as possible, all the material.
You want to target somewhere around the median of the class;  not too hard, not too easy. 
You don't want there to be any severe typos or other problems with problem;  you can't have a problem without a solution or is otherwise confusing.  
There are a limited number of good problems readily available;  making up new problems is challenging.  (Again, you don't want errors...)
You have to make it all do-able in 90 minutes.  
And so on....

I look forward to reaching a point where I don't think it's worth giving exams to students.  (I don't in my grad classes.)  For my undergrad class, I still think it's necessary. 

Inaccessible

The major reason for my lack of blog posts recently is that I was cut off from the Internet for a week.*  Surprisingly, the experience was not so bad.  I actually enjoyed being offline, and even the pain of coming back was not as bad as I had thought -- it only took about a day to catch up with e-mail.  Much e-mail was junk that needed no attention, a good bit was time-sensitive stuff that simply moved on happily without me, and a smallish fraction was actually important and time-consuming.  Of course, there was an inherent bias, since many people knew I'd be away.  I can't remember the last time I was off the Internet for a week -- but it has been a long time.  There are plenty of days that I go without watching TV, or even driving my car, but a day without the Internet is in itself a fairly rare event.

On the trip back I used GogoInflight and simultaneously had a great and frustrating experience.  Since I had a stopover, I went with a 24-hour package.  My first short flight, on a relatively small plane, had service and all went very well.  Internet on a plane is indeed useful (particularly when you have a week of e-mail to catch up on).  However, my much longer second leg on a much bigger plane didn't have the service, making the purchase something of a waste.  I don't blame Gogo, I blame American Airlines...

And for those with stopovers in Dallas, always remember to bring your CAT5 cable.  Somehow, the terminal there I frequent has free WIRED Internet access, but only if you know to bring your own cable to connect your laptop.  On the plus side, I've never had problem getting a seat at the Internet table, since nobody else seems to travel with the required cord.  It makes me happy that Boston is one of those friendly airports with free wireless.

The other reason for the lack of blog posts has been the need to catch up.  When were those reviews due again?....  What's that next conference deadline?.... When's my midterm?....  Back to it.

*OK, technically, not true.  However, cruise ship rates for Internet access are a fairly ridiculous 65 cents a minute (with smallish discounts for bulk time).  It didn't seem worthwhile. 

Technology Diffusion (Guest Post from Zhenming Liu)

My student, Zhenming Liu, writes about some of his recent work:

Sharon Goldberg and I recently submitted a paper about technology diffusion in the context of communication networks. There has been lots of work in this area in the networking literature; while networking researchers and practitioners usually know what technology they want to deploy in the network (e.g. Secure BGP, QoS, IPv6, etc.), they often don't know how to get the economically-motivated nodes in the network to adopt it. What sort of utility models could capture an economically-motivated node's decision to adopt a communication technology? Some assumptions Sharon has found in the literature (and used in her own work) include:

• A node's utility depends on the number of other nodes it can communicate with using the new technology. This is sometimes known as Metcalfe's Law.
• Two nodes can communicate with each other only if there is a path between them such that every node on the path has deployed the new technology.

Thus, unlike existing works on viral marketing in social networks -- where a node's utility function depends only on its immediate neighbors-- in our setting, node utility is non-local. We considered the following question:

Given a communication network $G(V, E)$, we assume that node $u$ activates (i.e., deploys the new technology) if it is adjacent to a connected component of active nodes in $G$ of size exceeding node $u$’s threshold $\theta(u)$. Our goal is to find an efficient algorithm that determines the smallest seedset of early adopter nodes, that once activated, causes a cascade that eventually causes all other nodes in the networks to activate as well.

We designed an approximation algorithm that returns a seedset which is at most $O(r\ell \log |V|)$ larger than the optimal seedset, where $r$ is the graph diameter and each node's threshold can take on one of at most $\ell$ possible values.

1. We seem to have the first tractable model that considers cascade effects with non-local influence in a graph; in the social network literature, the influences are local, while the networking literature considers non-local influences, but thus far has not developed any approximation algorithms.
2. Our algorithm substantially deviates from the greedy strategies we see in almost all influence maximization papers for social networks today. After reading lots of papers in this area and writing some ourselves (e.g. this and this), my colleagues and I started to suspect that a “computational dichotomy” exists among these problems: either the problem is deadly hard or greedy algorithms already work well. Our new result demonstrates that this is not the case, at least when influence is non-local.
3. Finally, our randomized algorithm makes use of a linear program, which is quite surprising because the problem seems highly non-linear; we managed to linearize the process by first carefully restricting our search space, while paying the price of only constant approximation loss.

I hope this paper will lead to more work on diffusion processes with non-local influences; studying other relevant models, or developing more efficient algorithms would both be interesting.

Advantages of a Bigger Class

I'm several weeks into the super-sized class this semester.  Mostly, I find it's going fine -- indeed, there are some advantages, I'm finding, to the larger class size.

1)  Student participation.  While I'm sure the fraction of students attending is probably about the same, with a bigger class, the room actually feels full.  And, when I ask a question, there are more students who will answer.  I think the interactive aspect of the class is much higher this year.  (Related bonus:  sounds like more laughing when I tell a joke.)
2)  More creativity.  This is mostly due to my teaching staff.  With a larger teaching staff, we've optimized our resources a bit (e.g.: not every TA teaches a section every week, freeing them to do other things;  one-person jobs like prepping section notes have to be done less frequently per person).  So now we have special post-homework-go-over-the-problems sections, a new Web site, and Piazza.  Thanks team.

The advantages, so far, outweigh the downsides.
1)  Grading.  I historically grade the first programming assignment on my own, and help grade the midterm/final.  I haven't had to do any of that yet.  I'm not looking forward to it...
2)  Student interest.  More students want to meet with me, have lunch, and so on than in previous years.  Normally, I would put this in the positive column.  But since the semester started I've been soooo busy that I just end up feeling guilty telling students yes... but later in the semester.

I'm curious to see if the increase in students will hold for next year.  But if it does I think I can find the positive.

Using Piazza

My TAs forced me to "get with the program" and use Piazza as our mechanism for student questions.  In the past I've simply used a central e-mail address that got forwarded to all the TAs (and me);  questions sent there would be answered by a TA (or me) and we all would see the reply.

Piazza offers some additional bulletin-board+ functionality on top of that framework -- things like being able to type things in Latex, easily see what questions are left open, etc.  It seems useful enough;  it's definitely better than past bulletin-board-style offerings I've seen.  Honestly, though, for me the primary benefit is that it lessens the amount entering my mail spool -- it's more organized having these questions all in one place, with formatting.  I'll have to see what the students think during and at the end of the semester. 

At the moment I'd recommend it, which is not something I thought I'd say -- like I said, I've eschewed such tools in the past.  On the other hand, I'm not clear on their eventual business model -- I don't think I'd pay for using it as a service.  Perhaps the plan is just to get us so used to it that the idea of not using it is painful enough that we decide to pay.  We'll see if I reach that point over the semester.

A Systems Complaint

I think of myself as a person who works both in systems and theory.  And I've found that's often a challenging position.  I've been vocal before that I think theory conferences, to their detriment, downplay experimental work -- most algorithms papers don't actually implement algorithms.  Today it's time for a complaint about the systems culture.

I think many systems people appreciate good and useful theoretical results.  However, they have I think some unusual expectations abut what is good and useful, in that it has to be something they can absorb in the space of a few pages*.  (Really, in the space of the paper you submit, but given that you have to actually produce some implementation results for a systems paper, you really only have a couple of pages to get your theory across.)  For example, I was fairly disturbed when I got back reviews for a paper (some time back now...) sent to a systems conference with significant theoretical content that included comments like (edited to protect the conference):

While the program chairs were eventually able to locate reviewers who knew enough about the previous work in the area to understand and verify your work, this indicates to me that the XXXX audience will have a difficult time understanding this paper.   (The average XXXX attendee will not be familiar with Celebrated20YearOldTheoryResult, much less the more recent work you build on.)

Another review stated:

It is reasonable (though a bit of a stretch) to assume a XXXX reader is familiar with MAJOR20+YEAROLDAREAOFTHEORY and Celebrated20YearOldTheoryResult. It is less reasonable to assume the reader is familiar with [major paper cited several times in our submission], and not at all reasonable to assume the reader is familiar with [other work under submission and on the arxiv], an unpublished manuscript by the authors. Yet I found that this paper was not sufficiently self-contained to understand it without referring to those prior works.

Of course, this was a challenging paper, and in fact in the paper we were quite clear that we were building on lots of previous work.  We tended to state things in the form of, "It is known that X holds [citation], and using that we do..."  So if they didn't believe us, they could go look it up (though, admittedly, it's not easy stuff).  We had expected that they would take our word on background material;  as the reviews state, they did verify we were accurate in our use of the prior work, and that we had clearly cited it. 

But apparently (and I didn't get the memo on this), "self-containedness" is considered important for systems conferences.  Which, interestingly, goes against most work in theory, where you're often trying to build on a substantial chain of previous work.

I find this worrisome for work spanning the systems/theory divide, and will plan to fight this attitude on whatever systems PCs I am on.  I think it's great that systems people are interested in and like to use theory.  But if they want to apply something more complex than, say, a Bloom filter (a truly wonderful data structure, easily described completely in 2-3 paragraphs), the community might have to accept that, yes, to fully grok the interesting stuff in our paper, they'll either have to accept our word for what's in the past papers, or find and trust reviewers who can vouch for what's in past papers, or go read the papers themselves.  Not everything can be adequately spelled out in complete detail starting from a blank slate in a 10 page paper.  

Perhaps one could argue that this was an isolated incident, and certainly I know there are (plenty of) people in systems who do not think this way.  But I do think it's representative of (a non-trivial part) of the culture.  

I find the idea that a paper that involves theory that goes beyond what you, or even your community, has yet to experience is automatically out of scope a disturbing framework, and I find the expectation that I should be able to get you up to speed on a complex area in a few pages so that the paper is "self-contained" unreasonable.  

*I've heard some people more disparagingly say that you have to make them think they've understood it, not actually understand it.