Guest Post by Dan Spielman.
The list of papers accepted to FOCS 2009 is now available at: http://www.cs.yale.edu/focs09/
As Mike did after the STOC 2009 PC Meeting, I'd like to provide a short summary of what happened at the FOCS 2009 Meeting.
We arrived at the meeting having at least three reviews for each paper. To save time for discussions of papers during the meeting, we voted to reject many papers in the two weeks before the meeting. We also voted to accept a small number that had overwhelmingly strong reviews. I would like to have heard more about these papers, but it would not have been a productive use of time.
After our preliminary decisions we had 123 papers to discuss over 2 days. This gave us only a few minutes to discuss each paper. In spite of the short timeframes, the discussions of papers were remarkably insightful, intelligent, and witty. Things were said that I will remember always. Unfortunately, confidentiality prevents me from sharing them with you. As chair, my job was to prematurely halt these discussions so that we could vote and move on to the next paper.
As was done in many PCs before, I asked committee members with conflicts of interest to leave the room. Committee members were deemed to have a conflict of interest with authors who were their advisor or advisee, who were at the same institution, and with whom they were close friends. My test for friendship was "would you be proud if this paper was accepted?" As many observed, these are not perfect tests for COIs, but they are a first-order approximation. This policy did have some drawbacks, as it often meant that the person most expert in the area of a paper was excluded from the discussion. To ameliorate this problem, I solicited extra reviews of papers for which this was the case. I am thankful that so many in our community responded to my urgent requests for reviews.
I am VERY happy that we took this approach to handling COIs. I know that I would have had a difficult time being unbiased when discussing papers with which I had a COI. One advantage of this approach that I did not see discussed in the comments on Mike's blog is that it preserves the committee members' reputations for integrity. We accepted many papers with which I had a COI, and I am glad that I will not be suspected of tilting the process in favor of my friends.
Deciding which papers to accept is a very difficult process. With insufficient time and consideration, we were forced to make decisions that are very important to many people. We did the best we could, but I am sure we made mistakes. At least there is no paper on which a majority of the committee believed we made a mistake.
I hope you all will join us for the 50th IEEE FOCS.
15 comments:
So, Dan, when are you planning to post all those wonderful 2-page summaries you got from the authors?
You are going to post them, aren't you?
The authors never agreed to have the two-page summaries posted. According to the CFP, they will be treated as confidential.
that's a shame. from what I heard, the 2-page summaries did not really have the desired effect, and I'd love to validate that claim against the actual summaries.
I was happy to see among the accepted papers a paper that a majority of another conference program (wrongly) claimed was a mistake to accept. It's nice to see different committees do error correction on each others choices.
Also, a question to anyone who knows concerning the following paper:
Convergence of Local Dynamics to Balanced Outcomes in Exchange Networks
Yossi Azar, Benjamin Birnbaum, L. Elisa Celis, Nikhil R. Devanur and Yuval Peres.
The paper appears to answer an open question of Kleinberg and Tardos in their bargaining paper. I have heard (through the grape vine) that the results in this latter paper were actually already known by economists, although I have never seen this publicly announced by the authors. Does anyone know anything about that? Thoughts on what to do if you find out results you prove and publish are already known?
Since FOCS PC introduced this new burden on the authors of producing 2 pages summaries, I would like to hear some explanations from Dan about their value and their use.
In fact, I refereed several papers but no 2 pages summaries were provided to me!
For one of the papers I reviewed, I looked at the summary first, provided an initial opinion on that, and then later was sent the full paper and reviewed it. So they were actually used, but even in that case it wasn't clear to me that they were actually useful.
I'd like to thank Dan for guest posting and sharing his thoughts on the PC process. I'd like to remind commenters that his doing so was not obligatory, while his duty to keep various information confidential is obligatory. I hope future PC chairs will continue to provide feedback openly to the community about the process in a similar fashion. (Dan is of course invited to share more thoughts on this blog -- on this or other issues -- if he likes, as are others in the community.)
If the 2-page summary is an important issue, perhaps it will be the subject of another blog post.
Anon #4: I find your phrasing odd -- "I was happy to see among the accepted papers a paper that a majority of another conference program (wrongly) claimed was a mistake to accept." There may be a variety of reasons a paper is not accepted. In particular, papers are often improved by feedback obtained when rejected, in which case a paper that was rejected may naturally get in at a future conference.
I'd point you to the excellent guest post by Aaron Sterling,
http://mybiasedcoin.blogspot.com/2009/05/life-after-rejection-guest-post.html
Michael,
I don't think the issue is whether anything is obligatory or not. The 2-page idea was an experiment to make the reviewing process easier. As such, we'd like to know whether the experiment was successful or not, and how this conclusion was reached, and what lessons we can draw from the future.
I agree that this doesn't require the 2-page submissions to be made public. But it does set up the expectation that the efficacy will evaluated by *someone*, and of course the most efficacious way would be merely to make them public. Obviously, we'd only want to see the 2-pagers for accepted papers.
If in fact authors were promised that their 2-pagers would be given confidentiality indefinitely, (i.e beyond the review process), this in my view was a mistake. Although I don't recall anyone ever saying this explicitly, or in the CFP.
Regarding guest post of A Sterling: I did not find it excellent or insightful at all.
I believe the two page summaries also had a pedagogic effect of forcing authors to write a decent introduction. This can be measured by the writing quality of the accepted papers
My own two-page summary was quite bad so I do not want it released, but writing it did help me write a better introduction for the final version. The additional marginal cost was zero.
I think I've heard a variety of different opinions regarding the 2-page summary "experiment". Really, the primary person Dan has to convince (if he wants to) is the next program chair, who can continue and revise the experiment if he (and the PC) wish.
I should have finished my guest blog post by mentioning that I was leaving for a well-deserved vacation.
Now, that I'm back, I'd like to clear up three points.
1. We never told the authors that we would post their 2-page summaries, so it would be unreasonable for us to post them.
2. Many sub-reviewers of papers were not sent the 2-page summaries. This was a bug in EasyChair that we did not anticipate (and could not test beforehand).
3. We will be discussing the utility of the 2-page summaries, and I'll address it in a later guest blog post. Unfortunately, we didn't perform any of the obvious randomized trials.
But, before I do any more blogging, I still have work to do on FOCS 09.
Post a Comment