tag:blogger.com,1999:blog-8890204.post2646254982962162684..comments2024-03-10T05:26:42.148-04:00Comments on My Biased Coin: Conferences and CorrectnessMichael Mitzenmacherhttp://www.blogger.com/profile/06738274256402616703noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-8890204.post-75375105138663767082008-03-17T06:30:00.000-04:002008-03-17T06:30:00.000-04:00I think it makes sense that PCs are careful with p...<I>I think it makes sense that PCs are careful with papers for which reviewers are doubt. The worse that can happen is that the paper is rejected even though it was right, and the author resubmits a better version, which is hardly a bad thing.</I><BR/><BR/>I could not agree more. If, during a PC meeting, a referee claims that a paper contains a serious mistake or that (s)he cannot see why some result is correct, then the other reviewers for that paper should make some effort to check that claim. In case the issue cannot be cleared up for lack of sufficient detail in the submission, I think that the author should be asked for full details of the proof. If the PC cannot convince itself that the result "smells right", then the paper should not be selected for presentation at the conference.<BR/><BR/>If the claimed result turns out to be correct, the author will publish it somewhere else, after having spent more time and care in writing up its proof so that an expert can vouch for its correctness. Publishing a better conference paper will benefit both the author and the research community.Luca Acetohttps://www.blogger.com/profile/01092671728833265127noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-79454829035427179302008-03-16T03:23:00.000-04:002008-03-16T03:23:00.000-04:00With my hypothetical example I wanted to point out...With my hypothetical example I wanted to point out that this "giving the benefit of the doubt" policy can suffer from diagonalization.<BR/><BR/>Do you give the benefit of the doubt to a paper that contradicts another paper to which the benefit of the doubt was previously given? I guess the answer has to be yes, but imagine having two published, and possibly incorrect, papers claiming results at odds with each other. <BR/><BR/>It's enough to kill whatever subfields they are about, because who wants to use results from those papers (at the risk of basing his own work on incorrect results), or, even worse, come up with a third proof?<BR/><BR/>I think it makes sense that PCs are careful with papers for which reviewers are doubt. The worse that can happen is that the paper is rejected even though it was right, and the author resubmits a better version, which is hardly a bad thing.Lucahttps://www.blogger.com/profile/17835412240486594185noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-34018252929302980222008-03-15T14:51:00.000-04:002008-03-15T14:51:00.000-04:00Paul, I'm certainly not suggesting that PCs abdica...Paul, <BR/><BR/>I'm certainly not suggesting that PCs abdicate their role in trying to assure correctness of what gets published (I hope that was clear in my post). Surely it should keep out things that seem clearly wrong. But when one reviewer says they doubt the argument (a situation which comes up not infrequently), and the question doesn't seem to have a simple resolution, what does a PC do?Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-60886923362778421482008-03-14T16:24:00.000-04:002008-03-14T16:24:00.000-04:00I disagree that PCs have no responsibility w.r.t. ...I disagree that PCs have no responsibility w.r.t. correctness. By that standard the Complexity PC this year might have accepted both the paper claiming a proof that P != NP and the paper claiming an efficient (linear time?) algorithm for SAT since, obviously, the problems are of sufficient importance! <BR/><BR/>Once you accept this level of filtering, where do you draw the line? Certainly, the arguments had better be plausible. The more suspicious they are of correctness, the more careful a PC should be, and the same probably should apply based on how big the claims are. <BR/>That's why we often allow appendices or longer versions than will appear to let the PC have a chance to verify claims if they want to.<BR/><BR/>Final responsibility lies with the authors but out of fairness the PC ought to make some effort to check claims.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-26753756876282291732008-03-14T07:46:00.000-04:002008-03-14T07:46:00.000-04:00Seems to me there should be an onus on the author ...Seems to me there should be an onus on the author to write a clear proof to begin with. Do we really want to accept papers where the proof is incomprehensible, regardless of whether the theorem is ultimately correct or not?<BR/><BR/>On another note, people seem to assume that journals are <EM>the answer</EM>. Come on, you've never seen journal papers with bugs?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-20103495728792958942008-03-14T01:01:00.000-04:002008-03-14T01:01:00.000-04:00My simple suggestion is to allow papers to appear...My simple <A HREF="http://infoweekly.blogspot.com/2008/03/correctness-conference-ranking-etc.html" REL="nofollow">suggestion </A> is to allow papers to appear in conferences after being accepted (past round-1) to a journal. Of course, we shouldn't <B>require</B> this for most papers, as that would destroy the rather wonderful dynamic environment that we have now. But in difficult cases, I think doing this covers a crack in the system.Mihaihttps://www.blogger.com/profile/11599372864611039927noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-6840949495622492592008-03-14T00:22:00.000-04:002008-03-14T00:22:00.000-04:00But my argument is that conferences (and PCs) are ...<I>But my argument is that conferences (and PCs) are not designed to accurately tell if papers are completely correct. </I><BR/><BR/>Problem is that we act as if they were. Just try to publish a correct and complete proof of a result that has been incorrectly claimed in the past (or claimed without anything approaching a complete proof). I can think of several surgical procedures that would be considered a more pleasant experience.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8890204.post-14827604698370181972008-03-13T22:58:00.000-04:002008-03-13T22:58:00.000-04:00I think the decision to make in these cases would ...I think the decision to make in these cases would be clearer if the purpose of the conference is clear -- is the PC looking for novelty and plausibility (something to spark discussion) or breadth ad correctness (something to give a snapshot of the "state of knowledge"), or something else?<BR/><BR/>An interesting but ultimately flawed result, if presented, may lead to a real solution.<BR/><BR/>What about having, in the proceedings (or somewhere else) a response/critique format like they have in some statistics journals? A reviewer who really takes issue with some bug can have a formal rider on the paper in the proceedings (but not a presentation, clearly). That would require them to de-anonymize themselves of course.asarwatehttps://www.blogger.com/profile/07587164580336503000noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-36917704885691904022008-03-13T21:14:00.000-04:002008-03-13T21:14:00.000-04:00Arvind -- since trickiness is in the eye of the be...Arvind -- since trickiness is in the eye of the beholder (reviewer), I think you're saying <I> everyone </I> needs to submit a video. People can have different opinions on whether that's a good idea or not, but I'd personally be against it.Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-52158635472820801552008-03-13T21:11:00.000-04:002008-03-13T21:11:00.000-04:00Luca -- hey, were we on the same PC where the scen...Luca -- hey, were we on the same PC where the scenario you're describing occurred? :) <BR/><BR/>Personally, I think having both papers published would benefit the community more than just the first -- at least then it might attract enough attention to the issue to have people resolve the problem, instead of just letting it sit there. And if the first paper was wrong, how else does one get a correct proof out?Michael Mitzenmacherhttps://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-70189651509379233752008-03-13T21:09:00.000-04:002008-03-13T21:09:00.000-04:00First, I don't think it's unfair. The authors migh...First, I don't think it's unfair. The authors might even be happy to work with the reviewers to explain the proof; after all, if the paper gets published, this is something they will need to do many times over.<BR/><BR/>But the solution to me is clear: allow authors to submit a video explaining proofs that they think are tricky. The reviewers are of course not required to look at them, but it gives the PC the moral ground to reject a suspicious looking paper if there is no accompanying video. Authors submitting a video slightly decrease the odds of getting rejected because of a complicated proof.<BR/><BR/>I believe a similar idea has been tossed around before, and there were objections of a fundamental nature to any sort of non-paper submission. I believe those don't apply here since the video doesn't make or break the paper. In any case, it's no use pretending that we live in some sort of idealized world where participants don't have material manifestations, or that anything human needs to be taken out of the process.Arvind Narayananhttps://www.blogger.com/profile/02495762505427759752noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-16674851027203206862008-03-13T20:46:00.000-04:002008-03-13T20:46:00.000-04:00Another scenario to consider: suppose a paper with...Another scenario to consider: suppose a paper with a serious bug, which looked suspicious to the reviewers, gets into a conference because the PC thinks it should have the "benefit of the doubt." At the next conference, a paper is submitted claiming to point out the error and to give a correct proof. <BR/><BR/>The new paper, however, does not give a concrete counterexample, but rather an explanation of why there is a gap; and the new proof is very complicated. The new paper looks suspicious to some reviewers. Now do you give the benefit of the doubt to the new paper?Lucahttps://www.blogger.com/profile/17835412240486594185noreply@blogger.comtag:blogger.com,1999:blog-8890204.post-11706921435915637242008-03-13T19:11:00.000-04:002008-03-13T19:11:00.000-04:00On the other end are information theory conference...On the other end are information theory conferences -- ISIT is huge and has a large acceptance rate, but the point of the conference is to essentially issue an advertisement for some journal paper that you have submitted or will be shortly submitting.<BR/><BR/>The result of this is that journal papers are everything, and while it's nice to send papers to conferences, it's not the important part of your CV, even as a student.<BR/><BR/>I think you can't address the issue of PCs guaranteeing correctness without addressing the reason for extreme selectivity at theaconference. If the PC's role is largely curatorial, is having a lot of papers in FOCS/STOC/SODA an indication of how trendy your research is?asarwatehttps://www.blogger.com/profile/07587164580336503000noreply@blogger.com