My post today is live-blogging the NSDI PC Meeting -- with a delay for security purposes, of course.
My take on the reviews (and from past experience) is that the NSDI PC is a very, very tough committee. People are looking for exciting and novel ideas, with clear and detailed experiments demonstrating real-world benefits (which usually means comparing against a real implementation from previous work). It's hard to get all that into one paper -- and to get everything so that the reviewers are all happy. And once in a while you can run into a reviewer like me, who expects your "good idea" to also have a suitable mathematical formulation when that makes sense. (If you're claiming to optimize something, I -- and others -- want a clear notion of what you're trying to optimize, and why your idea should help optimize it.)
So it's not surprising that, 4th paper in from the top, we've already hit our first paper where we're deferring our decision instead of accepting, and we're already getting some detailed discussions on whether a paper is good enough to accept. We'll have to speed up to make dinner....
As if to underscore that I did not have a great set of papers, I have just a few that I reviewed in our "DiscussFirst" pile, which takes us through lunch. Good thing I can keep busy with blog entries. And I have a review to write for another conference...
My submission appears (tentatively) accepted. Hooray! For this PC, we're not kicking people out of the room for conflicts -- people are just supposed to be keeping their mouths shut on papers where they have a conflict. For PC members with papers, however, you get kicked out of the room. So I've just spent a tense 15 minutes or so outside, but I'm happy to see the news once I'm back in. (More on this paper in another post....) Overall, I'd say (as expected) PC papers had no special treatment -- they were as harshly judged as all the other papers.
We're now having an interesting discussion about "experience" papers -- what do you learn after building/running a system after several years? A lot of people really think that having experience papers is a good idea, but there's some discussion of the bar -- what makes such papers interesting, and how interesting should they be? (Good anecdotes, but with quantifiable data to support the lessons.)
We're now about in the middle of the papers we're meant to discuss. Things here could go either way. Lots of technical discussion. As an aside, I can give my point of view on what are "hot topics". Data centers seems to be a big topic. There seemed to be a number of papers about scheduling/optimizing/choosing the right configuration in cloud computing systems -- how that could be done without making the programmer explicitly figuring out what configuration to use (but just give hints, or have the tools figure it out automatically). There's a significant amount of EE-focused papers -- essentially, trying to gains with some detailed, explicit look at the wireless signal, for example.
Headed to the end, more or less on schedule. Now we're assigning shepherds to all of the papers.
Sorry to say, I won't be revealing any information on specific papers -- everyone will find out when the "official" messages go out, or from their own connections on the PC...
I think the PC chairs have done a good job pushing us to keep on schedule; I think the discussions have been detailed and interesting. I think the committee is perhaps overly harsh (a target of 30 papers for about 175 submissions, or 17-18% acceptance; we ended up with 29). But I think we did a good job overall, and have earned our post-meeting dinner out.
Subscribe to:
Post Comments (Atom)
10 comments:
"PC papers had no special treatment -- they were as harshly judged as all the other papers. "
HAHAHA!!!
I didn't know what NSDI was before this. Thanks for the post.
"Overall, I'd say (as expected) PC papers had no special treatment -- they were as harshly judged as all the other papers."
Is that so? Could you tell us how many PC members submitted a paper and how many of these papers got accepted?
Anon #3: I will leave it to the PC chairs to decide if they'll release that data. I don't wish to introduce any issues related to confidentiality; I don't think it's my place as a PC member to provide that data.
I would like to point out that that data would be insufficient to make a judgment as to whether PC papers had special treatment. It would stand to reason that the PC members for NSDI have a much higher acceptance rate at conferences (on which they are not PC members) than average; I'm not aware of any source of information that would allow us to calculate that background number. (So I'm just pointing out that PC members could have 10% of submissions, but 20% of acceptances, and that might not only be perfectly reasonable -- it might be evidence that PC members were held to a higher bar!)
FWIW, by my count the acceptance rate was about same for both groups (within a couple percent +/-).
I'm always fascinated at the skepticism engendered by statements that pc papers are not treated with special deference. Is it that people assume Michael is lying or was fooled at the PC meeting?
Dear Michael,
I am not Anon #1 or #3, and have learnt a lot from your papers, and follow your blog quite regularly, not just for the technical stuff, but also for the discussion on state of science.
It would be too easy to just dismiss off the skepticism some of show towards unbiased reviews. However, do you truly believe that there could be so much disgruntlment without any smoke.
I was a volunteering at a few conferences, and just happend to be a fly on the wall, and was totally disgusted as to how some of the best paper awards are given out. Do you really want everybody to believe that there are no biases. If you think so, I will have to politely disagree.
If you are aware of flaws in the system that could lead to such disgruntlement, please continue to blog about it, many of us are grateful for it. Something like Sigcomm open reviews is really refreshing and I hope more conferences adopt this. Perhaps allow comments on those reviews too.
Anyways, I have reached a point where, I dont care about conference paper acceptances, being outside US helps. Arxiv gets you a fair audience now, and journals papers are facilitate a better (hardly perfect) dialogue.
My apologies for the rant :).
Not kicking conflicts out of the room is unusual, in my experience. So the conflicts don't get to defend the paper, but they do get to observe who is arguing against the paper and report back. I don't think that's a good idea.
Also in my experience, most PCs hold PC papers to a slightly higher bar than non-PC papers, in order to avoid the perception of conflict of interest. There are arguments against this, of course ("why penalize PC members? That only discourages being on the PC"), but it does hopefully reduce the amount of critique claiming "that piece of crap only got in because so-and-so's name is on it."
NSDI is one of the conferences which I deliberately miss because I don't think papers are treated fairly. I came to this decision after listening to how certain PCs went about 'preparing' for their papers' acceptance. I hear these accounts from the PCs themselves.
There was this PC who went about promoting the idea behind a paper (which he is an author of) to other PCs before the discussions. An analogy would be members of congress / senate going around before the official vote on a bill to secure the required number of votes to get the bill passed. During the discussions, there will certainly be multiple pros and cons for each paper. If one is disgusted enough or has agreed to champion the paper, the appropriate pro / con can be emphasized. It becomes a matter of how loud you want to shout.
It's all politics folks. For those outside of 'the circle', don't feel too bad about it. What you can do is to de-emphasize these conference acceptances when considering new hires for instance. For those in 'the circle', you ought to be ashamed of yourselves.
Anon of Jan 6:
At the risk of feeding trolls:
I wouldn't want to belittle what might be your experience (although your writing suggests that you're claiming only indirect experience), but I have not seen this at NSDI or SIGCOMM PC meetings. On the contrary, when (as is natural) I ask other PC members what they might be up to, as a rule they go out of their way to NOT discuss papers they have under submission at the conference -- or any conference where I am a PC member.
(I'm anon 6)
Yes, it's my word against PC members'. But there's no need to trust what I say, nor to resort to calling each other names, there's a simple way to resolve all this: PC members and chairs must not be authors of submitted papers.
This used to be the rule, unfortunately things kept degenerating to the point where now even PCs with conflicts can still be in the same room when the corresponding papers are being discussed. Is this because we have a severe shortage of folks who can serve on the committees?
Post a Comment