I introduced another change for the PC meeting for STOC this year. My experience is that theory conferences are somewhat "loose" about conflicts of interest. What I mean is that while PC members can't submit papers, and it's expected that you don't review papers of current (or recent) students/advisors, other than that conflicts of interest aren't usually treated as a big deal. As a contrast, it's standard for the networking PCs I've been on to be at least as rigorous as an NSF panel; for example, if a paper has ANYONE at your institution as an author, you have no electronic access to the reviews and you're expected to leave the room during the discussion (and similarly for other "standard" conflicts of interest, including any collaborators from say the last two years). In my experience, on theory PCs, people don't leave the room just because they have a conflict. It might be expected they'd not comment, although it's not always clear to me this expectation is followed.
Indeed, I remember my first SIGCOMM PC -- the first hour or two I kept wondering why people kept coming and going in and out of the room every time a new paper was discussed. It seemed like a lot of people were off to get coffee when they weren't a reviewer on the paper. Someone explained to me that these were the people with conflicts, and I was shocked at what was considered a conflict. (I figure anyone from UCSD or Microsoft currently has to leave the room for at least 1/4 of the papers on any networking PC.)
Interestingly, when I mention this difference in "style" to people, networking people seemed shocked (and maybe a little horrified) by how theory PCs handle conflicts, and theory people seem shocked by how networking PCs handle conflicts.
[Aside: it's a bit interesting to think about further this in light of the comments on my previous post on sending scores, where many people note that many authors can get "inside information" on their paper from PC members after the fact, even though the meeting is supposed to be "confidential". This post won't cover that further -- it was actually written before the other post!]
It's a bit hard to imagine utilizing such a strict conflict of interest polity for a theory PC. The community's a bit smaller, we're highly collaborative, and the papers can be so specialized that if you stick hard and fast to conflict rules you might not have enough PC members who are really suitable to judge a specific paper. Another negative is that it does cost time -- there's a switching cost when people enter and leave the room.
But I asked the PC for a straw vote, and a large majority of those with an opinion thought it was a good idea to have people with conflicts of interest leave the room, primarily for the obvious reason that it's a lot easier to have an open discussion when you're not worried what some people might hear. (Not surprisingly, younger PC members on average seemed to think this was a bigger concern.) I should be clear, however, that this idea was quite controversial; many PC members expressed a very clear and vocal dislike for a policy that had people leaving the room regularly. Indeed, it was definitely the most controversial decision I made as the PC chair.
In order to be practical, I made it clear that I understood there would have to be exceptions, as needed, to deal with special cases, since I didn't really plan for this policy from the beginning (including when making paper assignments, although I did ask PC members to mark conflicts with the software when ranking papers they wanted -- this did not seem to be treated uniformly and universally by the PC, again because it's probably not standard in theory conferences). But the policy I planned to implement was essentially the following: if you had any standard conflict on a paper, and there wasn't a very good overriding reason for you to be in the room (e.g, you were an original reviewer, or you had a specific expertise the committee needed to evaluate the paper), you should leave.
In the next post, I'll talk about how that worked.
But before that, I think this is a matter the community (or, to the point, conference steering committees) should address. My personal take over many, many PCs is that while the strict conflict interpretation used in networking conferences may not be completely workable for a STOC/FOCS type conference, if I had to choose, it's clearly better than the "We'll assume everybody will behave properly" approach taken at most theory conferences. (Many theory conferences take a strict line on PC members not submitting, but then do essentially nothing about other possible conflicts -- that just doesn't seem right.) I think something in between these two extremes are possible, where exceptions based on needed expertise is possible, and that's what I tried to implement, but it could use some more careful thinking through. What do you all think?
Subscribe to:
Post Comments (Atom)
25 comments:
"... if you had any standard conflict on a paper, and there wasn't a very good overriding reason for you to be in the room (e.g, you were an original reviewer, or you had a specific expertise the committee needed to evaluate the paper), you should leave."
But what if you were the advisor of an author--then obviously you have expertise in their area unless you/they totally switched. Sounds like one could always stay in the room in this case (which is a BIG conflict of interest).
Anon #1--
Please, don't be silly. The advisor relationship creates such a conflict that the person really should not take part in the discussion in any way -- it overrides any other good reason to be involved. There's essentially universal agreement on that point.
I think that the policy of leaving the room for conflicts is a good one. On the other hand, I think not allowing the PC to submit papers is a -bad- policy. This sometimes delays papers for years, when a paper has multiple authors, who sit sequentially on FOCS/STOC PCs. Why don't we have a strict conflict of interest policy when the PC reviews papers, but allow PC members to submit?
What about conflicts for subreviewers? They obviously aren't in the room for discussions, but they do "comment" quite a lot.
I agree that forbidding committee members from submitting is too restrictive. However, i've seen extreme cases where committee members had five (!!!) or more papers in the same conference and i'm referring to a well known mid-level conference. Maybe a good policy is to restrict the number of submissions to one and be extra careful about conflicts of interests.
In any case, way to go Michael!
Thank you Michael for addressing this! I have always been a little shocked at how loose the conflict rules are for the theory community even in terms of reviewing. I have been offered, for reviewing, papers written by friends (who are at the same institution) and very recent coauthors, and when I said I had a conflict, I was told it wasn't a problem. I am also aware of cases where close colleagues of one of the authors reviews the paper, and later discloses themselves -- the reviewing is not double-blind, mind you, so it is not like they did not know who the authors were from the start. And then from MIP, we know that well-connected people can always find out their confidential scores. It is really time we addressed this.
I agree with anon 10:54 and anon 12:11, and like in particular anon 12:11's proposal. PC members should be able to submit, but perhaps only 1 or 2 papers.
If reviewing were simply an objective determination of correctness then conflicts of interest wouldn't matter much. In reality reviewing is necessarily very subjective, so it's unreasonable to treat theorists as selfless emotionless calculating machines.
I therefore support measures to ameliorate improper influences, including:
* evict PC members with conflicts of interests from the room unless there's a good reason not to
* Prohibition on PC members or subreviewers saying anything whatsoever to authors about how their paper was reviewed for 10 years after the conference, including revealing scores unofficially or revealing that they were a reviewer.
* double-blind reviewing
I'm surprised by this
`leave the room' talk since
I thought most PC meetings
were done all-electronic.
Is there really a reason to
meet face-to-face. And if people
keep having to leave the room
then there is not that much
face-to-face anyway.
"Prohibition on PC members or subreviewers saying anything whatsoever to authors about how their paper was reviewed for 10 years after the conference, including revealing scores unofficially or revealing that they were a reviewer."
People have a right to know what was said. They have no right to know who said it.
At the risk of exposing my anonymity to Michael:
I was one of the PC members who voted for having people leave the room, but after the meeting I changed my mind. I thought there was too much overhead with people coming and going; people from big institutions (Weizmann, MIT) were out of the room too much of the time; and we lost expertise.
(At the time I voted, I also thought that leaving the room was only going to be for adviser/advisee relationships.)
just curious about the following:
do the names of the subreviewers get revealed to the entire PC? or only to the PC member that contacted that subreviewer?
that greatly influences my ability to give a harsh review (even if i genuinely think the paper deserves it). i'm not talking about being cruel, but if i am the only one that said something like "this result is a simple extension of that other result", while no other subreferee noticed it, then i can fully imagine my authors' best friend in the PC letting the authors know who "killed" the paper (even if the PC member had nothing to do with the reviewing process).
The PC needs to know who the sub-referees are. One chooses sub-referees based on expertise. Often this means that they are either (1) previous collaborators of the authors or (2) competitors of the authors. The key is for potential sources of conflict to be out in the open. The committee should be cognizant of sub-referee conflicts as well as their own, and weigh the value of the information.
The committee does not take anonymous opinions; however sub-referees often make clear that they are concerned about information leakage and PC members don't let that get back to authors. We see plenty of critical comments and confidentiality is maintained. I don't think that this issue is a concern.
There is another way to deal with conflicts in the PC meeting than having people leave the room. That is that PC members with conflicts on a paper simply declare their conflict and are expected to keep silent unless their specific expertise is needed. This is the standard that I applied when I was FOCS PC chair and would have preferred to the "leave the room" scenario. (With the software we used, if you declared a conflict at the start then by default you couldn't even see the scores (even the average score) or reviews on a paper - something that I found absurd. It would make sense for PC member submissions but not here.)
I was one who originally voted against the "leave the room" requirement. It worked out better than I had expected because committee discussions of individual papers with conflicts did not drag on for very long periods of time (something that has happened with virtually every other PC I have been on, independent of whether there are conflicts) but it sometimes left me with a bit disjointed feeling of what was going on and I didn't always end up back in the room in time.
The usual conflicts enforced in the past have been: 1. same institution. 2. advisor/advisee. Michael added 3. recent collaborator to the list (paralleling the NSF conflicts rule). Recognizing such kinds of conflicts, which I would broaden to include "good friend of", is the only aspect of conflicts that I think has not been well-handled by past PCs. However, again, I would just want these to be declared rather than a cause to require someone leave the room.
Warren: I for one dislike double-blind reviewing. To me as an author it feels like a form of censorship or gag rule: I am prevented from making preprints of my work or from telling anyone else about it because to do so might break the blinding.
As for PC member submissions: it's allowed for some conferences I've been on the PC of (WADS and Graph Drawing, although I think WADS has a rule that the paper has to include at least one non-PC co-author) and seems to work ok. For Graph Drawing, it's important to allow PC submissions because the community is small enough that otherwise there would be a big reduction in total submissions. I don't think this is such an issue for STOC/FOCS/SODA, though.
When do we get to see the list of accepted papers?
11011110 wrote:
I for one dislike double-blind reviewing. To me as an author it feels like a form of censorship or gag rule: I am prevented from making preprints of my work or from telling anyone else about it because to do so might break the blinding.
I talked to a friend in a AI about blinding and pre-prints some months ago. Apparently his community does allow pre-prints, especially for people on the job market. They're a risk of name leakage from pre-prints but that's not a catastrophe. After all the we have 100% name leakage and the sky hasn't fallen yet!
Suppose submissions were double-blind but pre-prints were allowed. Most of the time most of the PC would be in the dark. Isn't it better to get 90% effective double-blinding than 0%?
This is a bit off-topic so we should probably discuss it elsewhere.
Re: Suppose submissions were double-blind but pre-prints were allowed. Most of the time most of the PC would be in the dark. Isn't it better to get 90% effective double-blinding than 0%?
Then we fall into a problem of having a double standard: the set of well-known people who publish preprints gets treated one way, the set of people who choose to maintain their anonymity or who are not well enough for their preprints to be recognized gets treated differently. To some extent that's a problem anyway with double-blind submission: it's not really as double-blind as it purports to be because one can often guess the authors of a submission.
But for true double blind in which one is forced to remain anonymous (or have one's submission rejected on procedural grounds) there's a big cost in its prevention of the open exchange of information, and I'm not convinced that cost is worth the benefit (appearance of greater impartiality, potentially some actual greater impartiality).
it's not really as double-blind as it purports to be because one can often guess the authors of a submission.
People often make this unsupported claim, but the data does not back this up. When people have been asked to guess the accuracy is rather low. For people interested on this I suggest reading the SIGMOD report where studies on this were published.
People often make this unsupported claim, but the data does not back this up. When people have been asked to guess the accuracy is rather low. For people interested on this I suggest reading the SIGMOD report where studies on this were published.
I suspect this varies quite a lot from field to field (or even between subfields). Not only do standard practices vary between communities, but the size of the communities vary enormously.
I've done quite a bit of reviewing for Crypto, which is double-blind, and it's not very effective:
First, for high-profile authors or breakthrough results, I usually know perfectly well who it is (through the grapevine, preprints, talks, etc.).
Second, for many other papers, I am not quite certain about the authors, but I can make a good guess. For example, I'm often pretty sure about the identity of one of the authors, or that one of them is a student of someone specific, or that one of them is part of a small group who sometimes write papers separately and sometimes in collaboration.
Third, Crypto doesn't anonymize self-citations (which some social science journals do!). This can give a big clue.
The net effect is that of course I can't always specify the full author list with complete confidence, but I can usually get some pretty reliable partial information. The availability of this information is highly correlated with the quality of the paper: if I can't guess anything at all about the authors, it's usually a mediocre paper on a less exciting topic by people I've never heard of.
Maybe I am a fanatic, but I think that in the context of conference reviews, conflict-of-interest rules should only be applied in the most extreme cases.
Specifically, one should be barred from seeing the discussion of their own papers, papers of their current students (or current advisor), and close family members, but that's pretty much it.
My reasoning is simple: the PC's only job is to select a good program, and to do that it must evaluate a very large number of papers in a very short period of time. To do a good work, it really cannot afford to give up the opinions of people who know about the subject matter.
I think that the comparison with NSF is wrong. NSF is government, and there is "real money" involved, so regulation (and even over-regulation) is appropriate. In contrast, the PC is ephemeral and really cannot do much damage, so making strict rules is counterproductive.
My experience on committees has been that trusting people to behave properly is usually the right thing to do, and the vast majority of people will indeed do the right thing. It is the PC chair's job to watch out for exceptions, but I think that making strict rules "just in case something goes wrong" will just end up getting in the way and making the PC job harder than what it needs to be.
Regarding the argument about the ability to provide a frank review (by Anonymous from Feb-3 @ 2:28): From a PC member I would expect to just get over it: PC members made a commitment to provide good reviews when they agreed to be on the committee. Things may not be as black and white for subreviewers, but they can communicate their concern to the PC member who asked for their help (and, e.g., ask that their name will be withheld). Either way, in the choice between allowing more opinions on the paper and making some people more comfortable with what they wrote, I would choose the former.
I note that another source of information leaks when using double-blind reviewing is the people who excuse themselves due to conflicts of interest. So not only do the MIT faculty know (from conversations with their colleagues, etc.) which other papers are from MIT, but if the MIT people leave the room, everyone else on the PC knows where the paper being discussed is from.
If one knows which university a paper's from, I'm sure it becomes much easier to guess the authors.
I find it interesting that the conflict-of-interest policies vary so much from one community to the other. I suppose the networking community can be more strict in defining conflicts of interest because the community is larger, meaning it is still possible to do a thorough job reviewing a paper even if folks with conflicts are excluded from reviewing and from participating in the PC-meeting discussion. I don‘t know the history of how the policies got started, and whether there was ever a time the networking community had laxer policies, but would be very interested to know.
At least for some of the conferences (e.g., SIGCOMM), I know there has been over the years a perception that the community is “cliquish,” dominated by a “mafia.” We can quibble about whether the perception is accurate or not, whether or not it is improving, whether it is inevitable, etc., but in any case even the perception matters. If someone with a close relationship, like belonging to the same institution or being a recent collaborator, plays a role in accepting a paper, it may undermine the credibility of the decisions the PC makes. Even having the “conflicted” person in the room can mean that some PC members (particularly junior folks) may feel uncomfortable speaking freely, as they may worry that their words may go straight back to the authors who will soon be asked to write letters for their tenure cases.
Now, that’s not to say that the policy doesn’t lead to odd extremes. This is particularly true with “institutional conflicts,” given the recent internationalization of industrial research labs. Nowadays, folks from Bell Labs or Microsoft Research could unwittingly have a conflict with someone they’ve never heard of, who works half a world away. That may be taking things a bit too far, though it really depends on whether we are talking about *actual* conflicts or the *perception* of conflict. It is also true with papers that have a long list of authors from different institutions, or with PC members who collaborate extensively. Sometimes it seems that the majority of the PC is sitting outside of the meeting room, and who knows if they are using that time to plot a coup! ;)
Another interesting policy question is whether the PC members, and particularly the PC chairs, can submit to the conference, and whether there should be limits placed on the number of submissions. The policies sometimes vary from year to year, for example if PC chairs may impose restrictions on themselves.
I have mixed feelings about these issues. Part of me thinks the PC chairs should decline to submit, a policy my co-chair and I followed when we chaired SIGCOMM’04. That said, what it really meant was just that I removed my name from papers where I was heavily involved in the work, rather than asking my collaborators to decline to submit. Now, one could argue that some purpose was served by that – if for nothing else, perhaps it reduced the *perception* of bias – but I’m not it really achieves that, either. This past year for NSDI’09, my co-chair and I did decide to allow PC-chair submissions, and handle all of them out of band from the normal reviewing process. I suppose that’s a compromise solution, though admittedly it is imperfect. This all gets even more complicated if you try to handle PC-chair authored papers at a conference that anonymizes the submissions!
As for restricting the PC members themselves… I think disallowing submissions is too extreme, and too many good reviewers would decline to serve on the PC---ultimately to the detriment of the conference. Restricting to (say) two submissions seems like a reasonable policy, though when I’ve floated that idea to potential PC members, I’ve heard a lot of pushback. They say that have to “choose their favorites” amongst their papers, and leave their names off others---something junior faculty are particularly loathe to do. Especially when someone targets one or two main conferences a year, and the acceptance rates are pretty low, folks are loathe to have these kinds of (admittedly somewhat arbitrary) restrictions imposed on them. So, again, I have mixed feelings. I do think it looks bad when the program of a competitive conference is seemingly overloaded with papers by the members of the PC---or by one or two particular people on the PC. Even if the selection process was not explicitly biased, these kinds of things lead to *perceptions* of bias, or at least of insiderness, and lead to cynical attitudes about a conference, all of which can be unhealthy for the community.
So, what to do? I’m not sure, but here are a few thoughts. One is to have “shadow PCs,” something that has become relatively common for some of the networking conferences. (See reports on the shadow PC she ran for SIGCOMM at http://www2.net.in.tum.de/~anja/feldmann/papers/shadow.pdf and http://ccr.sigcomm.org/online/?q=node/331 for details.) This exposes more people to the process of selecting papers, and provides a window into the natural randomness from one PC to the next. Another is to make a greater effort to convey the value structure of the conference, so new entrants have a better chance at casting their paper in the right direction. Another is to find ways to accept more papers, or have multiple venues of comparable status (something the theory community seems to have done already, but perhaps networking has not), so folks don’t get overly invested in getting one of a really small number of papers slots at a particular venue. Or, have workshops co-located with the main conference to foster the community and create opportunities for publishing new work and technical discussions. And so on. That’s not to say that reviewing policies aren’t also an important part of the solution, but perhaps there is a broader issue here about how to have a strong research community where the best research ideas have a serious shot of being heard, whoever the authors.
Jen makes many valuable points. One bit I'd like to add in the systems/networking community is that while PC member papers are allowed, they are frequently held to a higher standard than other papers to avoid the perception that the fix is in. I've been on many PCs where there was a pile of borderline papers at the end of the day fighting for the last few slots, and PC papers tended to be excluded there for that reason. This is a slight bias against PC papers, but a reasonable one in my view.
http://MeAmI.org/clayproblem.pdf
It may be the language variables are cornered and in fact the structures of permissable references only allow publications of certain theorems by individuals with certain names that satisfy values which fit other reference values like a puzzle piece. What is inherently wrong about this is the potential exists someone could not publish a paper simply because of their name! If there name does not satisfy the mesh required by the context of the theorem and associated variables established by this hierarchial (academic disservices) the spread and inclusion of free thought and fairness will be changed.--M.M.Musatov
Post a Comment