tag:blogger.com,1999:blog-8890204.post5854741366106181779..comments2024-03-10T05:26:42.148-04:00Comments on My Biased Coin: Discussing STOC 2013 PC with Joan FeigenbaumMichael Mitzenmacherhttp://www.blogger.com/profile/06738274256402616703noreply@blogger.comBlogger43125tag:blogger.com,1999:blog-8890204.post-72857510013545319272013-03-13T14:51:09.602-04:002013-03-13T14:51:09.602-04:00Hi Sanjeev,
About the magic size 20, it is of cou...Hi Sanjeev,<br /><br />About the magic size 20, it is of course not magic, but it is a balance based on experience.<br /><br />As discussed by many above, the cost of a global view is an additive cost per PC member. The total global work load is thus proportional to the PC size, so for this cost, the work load on the community is minimized by a small PC.<br /><br />Joan realized this, which is why she for a larger PC, introduced a small EC, where only the EC had the global view.<br /><br />The large PC was focussed on the refereeing, which parallellizes perfectly since the total refereeing load only depends on the number of submissions and the number of reviews per submission (Joan cut this to an initial 2, which is something that could also be done with a normal PC).<br /><br />One prize payed was that for selecting papers from areas in different fields, the EC relied on reports from different PC members. In my opinion, this makes it much harder to make fair selection across areas. In a traditional PC, besides the simple comparison of like papers, you have PC members handling papers from different areas, and who are all involved in the global outcome.<br /><br />Direct comparisons is also an argument for fewer PC members taking many papers (with k papers you make k^2/2 direct comparisons).<br /><br />All in all, from a total work load and direct comparison perspective, we are better off with a small PC, but this has to be balanced with how much work a single person can take.<br /><br />To match the regular load of 20 PC members, in the new system with 10 EC and 50 local PC members, for each regular PC, we now have to be<br />on 0.5 EC and 2.5 local PC. <br /><br />Best, MikkelMikkel Thoruphttps://www.blogger.com/profile/10495805784088145688noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-32186520853644517662013-03-13T01:39:17.964-04:002013-03-13T01:39:17.964-04:00I enjoyed reading about this experiment. I have to...I enjoyed reading about this experiment. I have told SIGACT chairs before that there is no reason why traditional PCs have to stay at a size of 20. (The main reason as far as I can tell is to keep down the budget for PC meeting.) Sigact has a big reserve fund! Use it to subsidize the PC meeting and raise the size to 30. That would again reduce load per PC member to a more manageable 30-35.<br /><br />I'll also send this suggestion to Paul Beame (new SIGACT chair) privately.<br /><br />On a different note, I think experimenting with the product (to use Joan's term) is more important. Sanjeev Aroranoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-14165910110745711372013-03-12T10:17:31.808-04:002013-03-12T10:17:31.808-04:00Alex: Thanks for your comments. I agree with you ...Alex: Thanks for your comments. I agree with you that the PC Chair should explain the entire process that will be followed in making accept/reject decisions when the invitations to serve on the PC are issued; I certainly would have done so had I known exactly what the process would be. This time, we were making much of this "experimental" process up as we went along, and we really didn't know exactly what we were going to do until we did it. If the community continues to experiment with two-tiered PCs, I expect future chairs to benefit from our experience, to determine the entire process and schedule before getting started, and to make potential PC members aware of them.Joan Feigenbaumnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-74582163759976155992013-03-06T04:53:39.715-05:002013-03-06T04:53:39.715-05:001) I spent probably half as much time as on the SO...1) I spent probably half as much time as on the SODA 2013 PC I just had been on before. <br /><br />Also, it was a much more enjoyable job as most of the papers I got to look at I truly wanted to read. <br /><br />Needless to say, this resulted in me spending more time on reading the submissions and ability to make much more knowledgeable recommendation on them.<br /><br />2) It was definitely less stressful and more manageable. So, indeed, I should be willing to do it more often. <br /><br />3) Yes, being able to submit was a factor in my decision (although, I ended up not submitting anything). I don't like this aspect in traditional PCs when you have to decide half-a-year in advance if there is something you will want to submit or not. (And if you have some co-authors this becomes even more complicated.) <br /><br />I did not feel awkward when reviewing PC member's submission. (I do not see much difference between reviewing PC member's submission and evaluating submission by my other colleagues.)<br />Also, the fact that I had time (and expertise) to make a fairly knowledgeable assessment - and thus stand by my decision - for most of the submissions I was handling, was a factor here too.<br /><br />Finally, one point I would like to make is that - similarly, to others commenting here - I really felt that the process behind making the final accept/reject decisions by the EC committee was very opaque to me. Of course, I could feel that our reviews were taken into account and there was a chance for us to discuss some of these preliminary decisions. But, overall, the thought process of the EC - especially, behind deciding between papers from different areas - was not disclosed to the PC.<br /><br />I am not saying that this is necessarily bad, but I would feel better if the invitation to the PC was more upfront about the fairly limited role of PC members in the whole selection process. <br /><br />However, I must say that I am happy with the final program of STOC (at least, regarding the papers I have some knowledge about). <br /><br /><br />Aleksander Madrynoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-4600980397232830192013-03-04T23:39:07.727-05:002013-03-04T23:39:07.727-05:00I completely agree with Anonymous on March 1, 2013...I completely agree with Anonymous on March 1, 2013 at 11:47 PM: Accepting and more importantly Rejecting<br />papers based on only two reviews was <br />a failure of this two-tiered PC. Indeed this was my first experience of a CS theory conferenc with only two reports.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-65635675725117281792013-03-04T13:38:32.711-05:002013-03-04T13:38:32.711-05:00Russell: Apology accepted.
Anonymous on March 1, ...Russell: Apology accepted.<br /><br />Anonymous on March 1, 2013 at 11:47 PM: Nine reviews per conference submission is just about the most asinine thing I've ever heard about a CS conference! We simply do not need nine reviews in order to make sound accept/reject decisions and to give constructive feedback to authors. Perhaps systems people don't mind wasting their time on this type of overkill, but wasting STOC and FOCS people's time in that manner would be criminal.<br /><br />Anonymous on March 4, 2013 at 9:24 AM: In fact, after Jeff E. took the lead and, in his comment, answered the four questions I mentioned in my original post, I emailed the entire PC and asked them either to blog their answers to those four questions or to email me their answers. I plan to summarize the results and distribute the summary in some medium or other. However, I must warn everyone that the majority of the PC members have not answered. Somehow, this feels analogous to the fact that many PC members did not write thorough reviews. There seem to be some things that STOC PC members feel they must do and some that they don't. Fortunately for all of us, they do seem to feel obligated to put together a high-quality STOC program by whatever means necessary!Joan Feigenbaumnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-43182077251990722432013-03-04T09:24:22.399-05:002013-03-04T09:24:22.399-05:00While I do find this open discussion useful and in...While I do find this open discussion useful and interesting, I think it would be good to collect some harder data. Joan could run a poll among the members of the PC and EC and then release the numbers (maybe separately for PC and EC). For instances, we could ask a question like the following.<br /><br />Do you think STOC should adopt a two-tier PC approach in the future:<br /><br />a) no<br />b) yes, possibly with minor changes from this year<br />c) yes, but with major changes<br /><br />This would give us some quantitative feedback complementing the more qualitative feedback. In particular since, as Joan pointed out, the feedback received so far might suffer from a selection bias.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-65609107726403098252013-03-04T00:22:30.573-05:002013-03-04T00:22:30.573-05:00Just as one more data point: this was the first ti...Just as one more data point: this was the first time I got a STOC/FOCS review from a reviewer who showed no understanding of the field, and barely any understanding of the results of the paper. The review was close to nonsensical. The other review was extremely high quality and very useful. <br /><br />The obvious explanation is that there was only one expert on the PC qualified to review the paper (checking the list of PC members confirms this). I do understand the usefulness of making a first round of quick decisions after two reviews. However, I think that once a paper makes the cut and is eventually accepted, it should receive an additional review -- a great review and a bad one felt like less feedback than I get from other STOC/FOCS committees. <br /><br />Also, it seems that even though reviewers could ask the opinion of outside experts, some felt disinclined to do so, even when they were ill prepared to judge a paper.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-12399899609374650402013-03-03T20:04:15.394-05:002013-03-03T20:04:15.394-05:00So, just to be clear Russell, while you have your ...So, just to be clear Russell, while you have your opinion, it seems far from universal. Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-10110528289701299582013-03-03T03:12:01.582-05:002013-03-03T03:12:01.582-05:00I apologize to Joan for using loaded language. I ...I apologize to Joan for using loaded language. I only meant that, for me, the answer to the question ``Does the new process produce good results with less work and stress?'' was no. Thus, the hypothesis the experiment was designed to test (if all the other runs were to be like mine) was not confirmed, and hence the experiment ``failed''. I didn't mean to imply that performing the experiment was not worthwhile, nor that it was not correctly designed, only that the outcome did not verify the hypothesis. <br /><br />RussellAnonymoushttps://www.blogger.com/profile/18394636887484222063noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-63066911897313546782013-03-02T18:18:03.288-05:002013-03-02T18:18:03.288-05:00Joan's framing of the different tasks a PC per...Joan's framing of the different tasks a PC performs and how one might refactor them is incredibly useful, and is an excellent basis for continued discussion. I don't see how this experiment is a failure at all. It's an experiment, and it has generated observations and thoughts for the next experiment. Of course there's something both hilarious and ironic about theoreticians arguing about the nature of an experiment :). Suresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-29682432904919878392013-03-02T14:07:52.500-05:002013-03-02T14:07:52.500-05:00Concerning participating in the PC, I participated...Concerning participating in the PC, I participated for the sake of STOC. I was skeptical about 2-tier, and thought it very important that such an experiment was not run by only the people believing in it. I also wanted to help STOC'13 as much as possible, giving the 2-tier system the best chances. I told Joan up-front about my skepticism, and that I would write my opinion about the experiment.<br /><br />My expectations were even worse, and I think Joan did a surprisingly good job.Mikkel Thoruphttps://www.blogger.com/profile/10495805784088145688noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-20899245965140792872013-03-02T13:42:01.101-05:002013-03-02T13:42:01.101-05:00I agree with Joan that it was a important experime...I agree with Joan that it was a important experiment, and that it is not a failure to perform such experiments.<br /><br />My fundamental objection is that we had only 10 EC members with the global view, relying on different local PC members for reviews for different areas. <br /><br />This it to be contrasted with 20 traditional PC members being involved both globally and directly in the papers.<br /><br />As Joan pointed out, having 50 local PC members work globally would defeat the purpose of reduced work load for local PC members.<br /><br />I don't think more thorough reports are going to have any significant effect on acceptance/rejections. The significance of a paper should be explained in the abstract and introduction of a paper. The body of the paper (and possibly appendices) serves mostly to justify the claims made in abstract and intro. If there are some particularly interesting technical details, then the intro should at et least point to them, so the referee knows to check them out. I normally only read further details if the claims made surprise me, and very often this is when I end up finding a bug in the submission. <br /><br />I myself had a paper rejected based on two reviews. While I disagree with the conclusion of the referees (otherwise I would not have submitted), I do not think it would have made any difference if these particular referees had read and reviewed my paper in any more detail.<br /><br />Conference refereeing is extremely taxing for the community, making it very hard to get reviewers for the proper journal refereeing, which is where the detailed refereeing is really needed.<br /><br />Finally, getting a good program is not hard based on the submissions (this time with the extra PC submissions that would otherwise have gone to FOCS'13). The hard part is to try to be as fair as possible in the rejections.<br /><br />Let me also say that I am proud of our community. I have had experience with systems conferences (both accepts and rejects), and while they may generate more reviews, the quality of those reviews in terms of understanding has typically been abysmal. I don't think we have much to learn from them.<br />Mikkel Thoruphttps://www.blogger.com/profile/10495805784088145688noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-86128714418651909232013-03-02T11:44:57.942-05:002013-03-02T11:44:57.942-05:00This is the continuation of my previous post.
Glo...This is the continuation of my previous post.<br /><br /><b>Global view</b>: As I said in an earlier post, it is impossible for a PC member to have both an extremely light work load and a global view. The main value proposition to potential PC members whom I invited to serve on this committee was the chance to handle few papers but to do a very thorough job on each one. Almost everyone whom I invited accepted the invitation, and many expressed delight that they'd be handling few submissions and could do a great job on each one. It never occurred to me to warn them that they would not have a "global view"; I thought that that was obvious. Recall that PC members who wanted a broader view than just their assigned submissions were welcome to read and comment on other submissions. Some of them did so; perhaps that's a disjoint set from people now regretting that they didn't have a global view!<br /><br />Anyway, this is another example of valuable information that we've gained from this experiment: Even though they didn't think about it when they accepted the invitation to serve, some people care a lot about "having a global view." Another way in which the experiment was a success!<br /><br />Note that all of the so-called "subreferees" that traditional PCs enlist for detailed reviews also don't have a global view. (BTW, I think that we should simply call these people "reviewers" or "outside experts" or something else more accurate. We don't do full-fledged "refereeing" for conferences, and so there is no "referee" to which they people are sub'd.) <br /><br />There are several important functions that must be performed in order to put together the program. They include:<br /><br />1. Detailed (written) reviewing.<br />2. Discussion of individual submissions (especial those about which reviewers disagree).<br />3. Discussion/comparison of comparable submissions to determine which are better than others.<br />4. Overall balance, apportionment of space to various technical areas, and attempts to compare, for lack of a better word, incomparable submissions. Here is where one needs the vaunted "global view."<br /><br />Somehow, we need to assign these functions to PC members and outside experts. Traditionally, we have a monolithic PC that takes on all of 1 - 4 and delegates some of 1. I maintain that there's no reason that that division of labor is sacrosanct, and I think that the STOC 2013 experiment, which produced a high-quality program, is evidence that I'm right. I think we should continue to experiment with a variety of divisions of labor.Joan Feigenbaumnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-28175265744147748172013-03-02T11:44:00.801-05:002013-03-02T11:44:00.801-05:00Thanks again to all who have commented on the STOC...Thanks again to all who have commented on the STOC 2013 experiment. There are three themes in these comments that I'd like to respond to. I'll address two here and one in my next message.<br /><br /><b>"Failure":</b> I am taken aback by Russell's and other's use of this extremely derogatory word. As I explained in my original post, the goals of this experiment were (unavoidably, given how the experiment was undertaken) extremely modest and extremely vague. To the extent that I can sum them up briefly, they were to answer the question "Can we produce a high-quality STOC program with a different committee process from the one we've traditionally used"? and "Which changes to the process should we consider"? We now have one (multidimensional) data point with which to start answering those questions. If the answer to the first question is "no," then the community has the option of simply reverting to the traditional process. That doesn't make the experiment a "failure."<br /><br />Are some of you using "the experiment was a failure" to mean "I did not enjoy participating in this process"? If so, I think that's nonsensical. One thing we can learn from experimentation is what type of process potential committee members prefer participating in. Eliciting that information is a sign that the experiment was a success, not that it was failure.<br /><br />(BTW, unsurprisingly, the PC members who really liked this alternative process have, for the most part, been emailing their views to me privately rather than blogging about them. I feel sort of like a professor reading course evaluations -- it's the dissatisfied students who feel the deepest need to make their views known. :=))<br /><br /><b>Early accept/reject decisions based on two reviews:</b> No decisions were made based only on numerical scores; the EC read the reviews carefully and in many cases asked for additional input from PC members and outside experts before making any decisions. Moreover, <b>all PC members were empowered to prevent early decisions</b>. Approximately a week and a half after the first round of scores and reviews were due, I sent email to the entire PC telling them (1) tentative early decisions had been made, and PC members had 12 days to raise objections and concerns before those early decisions were finalized, and (2) they were (as I had told them earlier) welcome to comment on any submissions they wished to comment on, regardless of whether those submissions had been assigned to them (except, of course, for those with which they have CoIs). In fact, some PC members <b>did</b> raise objections and concerns during those 12 days, and some decisions were postponed for exactly that reason. If there are PC members who now think that particular submissions got short shrift, we need to ask why those PC members did not raise these concerns during the deliberation process when they were asked for their input. Perhaps that problem can be avoided in the future by (as I also suggested in my original post) setting a firm schedule at the beginning and making sure that everyone is aware of it.<br /><br />That being said, I certainly agree (and said in my original post) that the quality of reviews in our community has always been inadequate and was inadequate this time as well. I hoped that the markedly light load on each PC member would result in (almost) uniformly thorough reviews, but it did not. Rather than focus on whether we should demand two or three reviews on each submission, I think we should figure out how to get <b>good</b> reviews on submissions to theory submissions. Does anyone have any ideas?<br /><br />Joan Feigenbaumnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-72563380695557186012013-03-02T11:03:52.580-05:002013-03-02T11:03:52.580-05:00While I'm making an effort not to respond, I d...While I'm making an effort not to respond, I do have to clear something up for the last anonymous.<br /><br />Papers were initially assigned 2 reviewers. After the first stage, they were rejected only if they got two very low (clear reject) scores. Even if these papers had gotten a 3rd review, they were never going to make the top list. They didn't have one positive score where another review might give them two positive scores. They had two (very) negative scores; they weren't getting accepted.<br /><br />Now, I do think that the papers left after the 3rd round should have consistently obtained a 3rd (or even 4th) review; that's a different matter, and I do think Joan discussed that if she did it again she would have had additional reviewers pre-assigned for papers in the 2nd round. <br /><br />I think there's nothing wrong (and a lot right) with a system that does 2 initial reviews and cuts a number of papers off at that point, and then focuses on the remaining X%, for some reasonable X and cutoff method.Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-24943931130461447082013-03-01T23:52:02.403-05:002013-03-01T23:52:02.403-05:00(Same commenter as previous comment) I would like ...(Same commenter as previous comment) I would like to emphasize that I don't at all blame Joan for the negative aspects I described above. She obviously worked extremely hard and repeatedly urged the committee members to do good reviewing, but this didn't always stick. <br /><br />I think it's good that STOC did this experiment, but my sense is that having many papers receive only two reviews should be avoided in the future. The real issue is how to increase the overall cultural norms of reviewing quality in our field.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-18995549295451142342013-03-01T23:47:56.707-05:002013-03-01T23:47:56.707-05:00I was a member of the outer PC. (I prefer not to ...I was a member of the outer PC. (I prefer not to identify myself for reasons that will be obvious from the contents of this comment.) <br /><br />I am afraid that my experience is largely consistent with that of "Anonymous March 1, 2013 at 5:49 PM". A significant fraction of the papers I dealt with received at least one very superficial and low-quality review. Such reviews almost always corresponded to mediocre (though usually not really low) scores. I think that in the normal three-review process one such review, if matched by two positive and more thorough reviews, would not torpedo a paper. This year, with only two reviews, even in cases where the other review was significantly more positive and more thorough, the paper rarely survived to receive a third review.<br /><br />Like Russell and Mikkel, I felt that I did not get a 'global' view of the submissions pool, so the experience of being on the PC was somehow unsatisfying (while certainly less time-consuming than it would have been to manage 40+ papers under the usual process).<br /><br />Apart from other aspects of the process, I think the fact that so many papers received only two reviews is a significant drawback of this STOC as opposed to previous conferences. This feels to me like a step in the wrong direction -- a colleague recently told me over lunch how his submission to a top systems conference received *nine* detailed reviews. I was ashamed to say that our flagship conferences in theory are experimenting with two reviews, down from three, and that those reviews are sometimes very superficial.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-28796774951701156382013-03-01T17:49:05.643-05:002013-03-01T17:49:05.643-05:00With most high-quality conferences, I accept that ...With most high-quality conferences, I accept that random choices (after filtering) lead to a good program (albeit not optimal), and I don't mind (that much) even if my own deserving papers get rejected. <br /><br />That is how I feel about the normal reviewing structure. But in the case, I was left unsatisfied by a 2-review rejection of one of my submissions: One review was by an admitted nonexpert, and hence probably had little chance of receiving the strongest score. The other review was one of the lowest-quality reviews I have ever received. An outright rejection at this point is disappointing. <br /><br />In my opinion, if this process is continued, the first-round rejections should be augmented to account for the quality of reviews. A paper should not be rejected based on first-round scores if the scores themselves are meaningless.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-64473445508245621112013-03-01T02:39:25.640-05:002013-03-01T02:39:25.640-05:00This post is an utter shame, and this experiment i...This post is an utter shame, and this experiment is an utter failure.<br /><br />As few (or no?) external reviewers are summoned, while the committee is not representative enough, especially when the COI is taken into account, the review process is terribly bad. Many submissions were reviewed by totally irrelevant and incompetent reviewers, though they're real experts in their own area.<br /><br />As a result, we might expect the lowest quality in general since the inception of STOC. It will surprise nobody if FOCS'13 turns out to contain better papers than STOC. WHAT A SHAME!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-79087832567976107252013-02-28T18:38:08.786-05:002013-02-28T18:38:08.786-05:00As someone posted earlier, the quality of reviews ...As someone posted earlier, the quality of reviews is affected by the number of submissions relative to the expected number. In other words, if you get less than you expected, you do a better job, and more than you expected, you do worse. The time required here was less than a normal PC, but more than I expected, and more than I budgeted for. If I did it again, I would budget more time, and maybe it would be less unpleasant and I would do a better job. <br /><br />Maybe the actual sub-referee policy was less strict than what I thought. The papers were in my general areas, but for only a few would I (or someone I work with) be a suitable sub-referee. In particular, what I worried about was being ignorant of related work that was not cited in the<br />submission. Anonymoushttps://www.blogger.com/profile/18394636887484222063noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-81864946503840297732013-02-28T17:15:58.996-05:002013-02-28T17:15:58.996-05:00This is second part of my comment..
PRAISE
I thi...This is second part of my comment..<br /><br />PRAISE<br /><br />I think Joan was successful reducing the work load per PC member. One of the interesting new ideas was to have an initial round with only two reviewers per paper, thus trying to handle all the easy cases with minimal resources. I do think such an adaptive approach is a good idea. Many of us procastinate, doing things as close to deadline as possible. Having an initial round has two advantages: (1) spreading out the work, getting some done early for the initial round, and (2) minimizing the work on easy cases. I later got called in on other papers needing more reviews.<br /><br />About PC members submitting. I have no strong opinion. Obviously this meant that we got some extra submissions to this STOC. Next FOCS will be missing the submissions that it normally get from the preceding STOC PC.<br /><br />Also, I was quite happy with the electronic PC. Have tried electronic PCs both at SODA and at one previous STOC.<br /><br />All in all, this time I felt like a glorified referee, with very little feel for the overall program. With more effort, I could have have helped much more with the global program, but getting a global view would imply spending the same time as on a normal PC, and having 50 PC members do that would hugely increase the total work load for the community. Moreover, since I knew the EC would make all the final decisions, I didn't feel very motivated getting too involved outside my batch. That being said, I was totally happy with all decisions made<br />on the papers I handled.Mikkel Thoruphttps://www.blogger.com/profile/10495805784088145688noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-86766424802887309332013-02-28T17:14:52.222-05:002013-02-28T17:14:52.222-05:00This was my 10th STOC/FOCS PC. I think Joan did an...This was my 10th STOC/FOCS PC. I think Joan did an excellent job and introduced many good ideas, but I do think that the 2-tier system is a fundamental mistake that we will suffer from in the long run if it is continued.<br /><br />CRITICISM<br /><br />I will go straight to what I see as the main problem, and return with some praise in the end.<br /><br />The easy job of a PC is to compare like papers on similar subjects. The hard part is to compare accross areas, attempting some uniform accept level. This is where STOC/FOCS traditionally do much much better than journals.<br /><br />What makes cross-area comparision work with a traditional PC is that many of the more senior members have worked in several areas. Handling papers from multiple areas they offer direct<br />cross-comparisons. Here, by a direct comparison of two papers, I mean a comparison done by a PC member who is responsible for both papers.<br /><br />One of the ideas in the 2-tiered EC/PC was to minimize the number of cross-comparisons. It was attempted to let PC members handle like (same area)<br />papers, so most of the direct comparisons where not across areas.<br /><br />On top of that, note that with the same total reading load, we get only half as many direct comparisons if we double the PC size, giving each PC member only half the load; namely (2k choose 2) versus 2*(k choose 2).<br /><br />The result is that we have an EC doing cross comparisons indirectly, relying on the scores from different PC members for different fields. This means that we rely on PC members from different fields to have a uniform sense of quality. This is pretty much the problem with journals, where standards are much more fluctuating across fields.<br /><br />I do not think the problem is big yet, because thanks to the work of past STOC/FOCS PCs, we all have a pretty good feeling for STOC/FOCS standards. I am not saying that these are even remotely perfect, and<br />there will always be lots of papers in the gray zone, but compared with the journal situation, I think STOC/FOCS has a far more uniform accept level (the fact that conference papers are not refereed for<br />correctness is a totally different issue).<br /><br />Without direct cross-comparisons, subfields get to evaluate themselves, and establish their own standards.<br /><br />So stepping back: the hard part of selecting a program is to select across subflields. Normally, we have a PC of 20 members doing this in a discussion involving many direct cross comparisons. Instead this was done by a hardworking EC with only 10 members relying on different PC members<br />for reviews on different fields.<br /> <br />Joan was very focussed on getting good direct comparisons of like papers, but I think that has been a main emphasis of any PC chair I have served under... while it is the easier thing to do, it is also<br />the one that is most embarrassing to get wrong. As mentioned above, with more PC members handling fewer papers, we simply had much fewer direct<br />comparisons than usual.<br /><br /><br />Because of character limit, praise and conclusion follows in second comment.<br />Mikkel Thoruphttps://www.blogger.com/profile/10495805784088145688noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-16923762172356389332013-02-28T10:23:38.122-05:002013-02-28T10:23:38.122-05:00My answers to Joan's questions:
1. I spent si...My answers to Joan's questions:<br /><br />1. I spent significantly less time overall than when I'm on a traditional PC, and I spent more time per paper. The time spent per paper was in between (a) the amount of time I spend on a paper that I assign to a subreviewer when I am on a traditional PC; (b) the amount of time I spend when I am a subreviewer for someone else serving on a traditional PC. The time spent per paper was much closer to (b) than (a).<br /><br />2. I'd be willing to do it more often than serving on a traditional PC, because the workload is so much lighter. On the other hand, given a choice between serving on one traditional PC or two PCs like this one, I would greatly prefer to serve on one traditional PC (even if it was somewhat more than twice the workload) for the reason Russell explained: "After serving on a normal PC, I have a vastly improved vision of the state of ToC as a whole." This new PC format doesn't afford the same benefit to its members.<br /><br />3. Once again, Russell's answer sums up my view: "This year, being able to submit papers was not a big issue, although I took advantage of it. In steady state, I expect it would become very important, since I would expect to be asked to be on a STOC/FOCS committee three times as often as with a committee 1/3 the size."<br /><br />4. For this conference, none of papers assigned to me had an author on the PC. I've served on other PCs for conferences that allowed PC-authored submissions and didn't find it excessively awkward.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-78491388017965935732013-02-28T03:42:59.820-05:002013-02-28T03:42:59.820-05:00Reasonable objections by well-regarded members of ...Reasonable objections by well-regarded members of the community: "Noise."<br /><br />Confirmation bias: No one risked outright rejection by submitting in the wrong format, thereby validating our choice.Anonymousnoreply@blogger.com