I am absolutely, positively, not liveblogging from the SIGCOMM PC, as that is, I am rightly told, a bad idea. But these are my impressions after the fact. (I'm told notifications have gone out.)
This was a big PC -- 50 people. They did get a great space for the meeting that worked very well -- it didn't seem crowded, we could actually hear each other, etc. It did seem like a very large PC -- at the PC meeting, if I've calculated right, each PC member would have on average 7-8 papers to discuss, which seems low for spending a day-plus at a PC meeting. I think the chairs wanted 50 in order to get a wider range of expertise on the PC. (External PC reviewers are very rare, as compared to theory conferences.) On the plus-side, it's an interesting meeting, and I enjoyed listening the the various discussions. (And, of course, I got some other work done in the times my papers weren't being discussed.) 50 PC members, however, just seems too big. I would suggest that 40 would have left the workload manageable, still left the committee with enough expertise, and made the PC more interesting for everyone there.
I'm afraid to say that, unsurprisingly, as a PC we were moving far too slowly, and were, I think, far too negative. First day, by dinner time, we still had over a dozen papers to discuss -- and all decisions up to that point were supposed to be considered tentative, to be finalized in day 2, which meant plenty more discussion if there was time. Also, at the end of the first day, had fewer than 20 papers in the Accept/Accept If Room categories (albeit several were in the "Tabled" category, which generally meant more PC members were looking at it or the reviewers were discussing some finer points); we could, ostensibly, accept about 36 papers.
Matt Welsh wrote an excellent and humorous post on the Psychology of Program Committees, which to some extent applies. Overall, though, I'm not clear what the problem is. I think the PC is quite negative; on the other hand, as I've mentioned, I also think the papers I had to review were pretty bad, so I guess I'm part of the problem. In terms of timing, though, I think I'm part of the solution -- as was Matt Welsh. (Early on, asked about a paper that had already been summarized, Matt briefly said, "I liked it. It's not perfect, but it's very good, everyone rated it high, let's accept." And we quickly moved on...)
I wonder if there needs to be an attitude change for SIGCOMM (and related) PCs -- a thought that I know has been expressed before. Papers on the borderline tend to get killed. There were certainly a couple of papers I would have pushed for that I thought would be reasonable accepts, but was conflicted out of engaging in discussions. (Perhaps that's why I liked them, and one can interpret individually whether you think that means strong conflict rules are a good or bad idea.) I'm not sure if starting a meeting by saying, "We're GOING to accept X papers, so find the best ones!" is a possible approach, or even a good idea. But maybe it's worth trying.
Tuesday we started early and tried to speed up to get through everything. Overall I think we did a better job Tuesday, generally moving faster with fewer slowdowns, and we were up against a deadline as people had planes to catch. I've said before that, overall, I like the conflict rules used by the SIGCOMM PC. That being said, with time running out, I think a more flexible approach that doesn't require people to physically leave the room should be acceptable (as I've also said in the past, for theory conferences). I certainly think it's very desirable that conflicts leave the room, but it does take time, and that has to be traded off against other concerns.
In the end, after various reconsiderations and re-animations (some papers, indeed, seem to rise from the dead), we accepted 33 papers; announcements should now be out to authors, even though the PC still gets some time to finalize reviews. We finished right on schedule.
Subscribe to:
Post Comments (Atom)
3 comments:
Someone should split a PC this size into two, then give them the same set of papers, have them meet in parallel, and then merge the resulting lists and publish how different they were - as an attempt to "do science' on the operations of PCs
I've often thought one of the fundamental issues with CS-oriented conferences is that PC members view themselves as gatekeepers, keeping bad work out rather than prospectors, trying to figure out how to get interesting work in. Typically in other disciplines, that ferocity is saved for journal submissions. As Matt had noted, it is quite rare for PC members to champion a paper, much more likely for someone to smite a paper.
I agree with clog, that is a good idea.
Looking at this year's accepted papers, I cannot help but think there is a slant towards a different clan. I expected this when the chairs were announced, and chose to stay away from this conference. If I had decided to submit, whether my paper gets accepted or not, it doesn't mean anything to me.
I thought about the issue of fairness, and I think the best thing to do is to have PCs who have nothing to lose. Unfortunately, this type of people don't exist. I suppose the next closest group are the tenured faculty members, but I don't see too many of them in conferences. Perhaps they understood the insignificance of these conferences, compared to what the rest of the world is doing, a long time ago :)
Post a Comment