CONFLICT OF INTEREST GUIDELINES ============================================================================= A program committee member (including the chair of the committee) is considered to have a conflict of interest on a submission that has an author in any of the following categories: 1. the person themselves; 2. a past or current student or academic adviser; 3. a supervisor or employee in the same line of authority within the past five years; 4. a member of the same organization (e.g., company, university, government agency, etc.) within the past five years; 5. a co-author of a paper appearing in publication within the past five years; 6. someone with whom there has been a financial relationship (e.g., grants, contracts, consultancies, equity investments, stock options, etc.) within the past five years; 7. someone with whom acceptance or rejection would further the personal goals of the reviewer (e.g., a competitor); 8. a member of the same family or anyone considered a close personal friend; or 9. someone about whom, for whatever reason, their work cannot be evaluated objectively.
These guidelines are roughly the same (with minor variations) as what I've come to expect from other networking conferences. I feel I have to point out the remarkable difference between how conflicts are treated in the networking world and the theory world. In the theory community #1 is a standard conflict; #2 and #3 are also pretty standard although, in my experience, definitely not universally applied; and after that conflicts are generally, in my experience, up to the individual PC member to declare if they happen to feel like it.
There's been debate on this blog about the subject before, and I certainly don't mind there being more. I maintain that the theory community is far too lax in its handling of conflicts. We can certainly reasonably argue whether the true impact of conflicts in actual decisions in theory conferences is negligible or substantial -- a matter of appearance or a matter of substance. I can say that, in terms of appearance, people from the networking side (and other communities) are shocked by the lax approach adopted by the theory community.
Thanks to Muriel and Tim for allowing me to post from their document.
14 comments:
The difference is that in TCS we have a less subjective standard because a theorem is either true or it isn't. Subjectiveness comes into play is only, for example, in the tradeoffs between technical difficulty and conceptual development.
In Networks a lot of the work is incredibly subjective. I've seen seemingly complicated and arcane proposals in search of an application be widely accepted (e.g. MPLS, multicast) while simple elegant algorithms be generally rejected (e.g. TCP Vegas) for what are seemingly purely social reasons. This is why the conflict of interest rules in Networks needs to be much stricter.
Anon #1: I think you're fundamentally mistaken in your assessment of how subjective and not subjective decisions in TCS are.
I was just a PC member for FUN. The papers I read all had correct theorems. But they weren't going to go to STOC or FOCS. So just because theorem are either true or false does not mean we aren't using subjective standards regularly.
We do have standards for technical difficulty and conceptual development for our various conferences. And on the level of "the community" we tend, I think, to do a good job in establishing the tradeoffs (though this is subject to debate). At the level of individual papers? I don't see any reason to believe we're better than networking. In particular, your point about the complicated and arcane being accepted over the simple and elegant seems humorous to me, given how many people (myself included) have said in the past that FOCS/STOC seems to have a predilection toward mathematical complexity over mathematically simple (but useful) solutions.
I don't think that, for our community, (5) is compatible with having papers reviewed by people who know what's going on.
You're a PC member for fun? Masochist!
(I'll show myself out.)
Luca -- I'd actually agree with you that the "co-authored a paper within the last 5 years" rule is far too strict for the theory community. That doesn't mean, though, that we shouldn't have some sort of similar, more appropriate rule in our community. And exceptions can always be made if a particular expertise is absolutely required.
Andrew -- I first typed it in as fun, then on proofreading changed it to FUN. Your joke's funny in either case but I'm glad I made the correction...
This is Anon #1.
Just to make it clear, no claim was made that there are no subjective components of TCS decisions. The claim is that they are one order of magnitude lower than those in networks.
At the level of individual papers? I don't see any reason to believe we're better than networking.
It's not about being better or worse. We have one big objective filter that goes ahead of everything else: "is the theorem true?". We also have, for better or for worse, a second criterion which is also relatively objective "is this among the top x% most difficult papers submitted to the conference?", only in third do we have subjective measures such as "is this is hot", "is this worth telling other people about", "is this written by my friend, coauthor, colleague", etc.
your point about the complicated and arcane being accepted over the simple and elegant seems humorous to me, given how many people (myself included) have said in the past that FOCS/STOC seems to have a predilection toward mathematical complexity over mathematically simple (but useful) solutions.
Point taken, bad example. I still think the conclusion holds. My comment was not meant to suggest that we don't need better conflict of interest rules. After all I've been a long standing proponent of double blind submissions. The point is to explain why Networks is seemingly so paranoid about COI as compared to TCS.
p.s. FUN is such an odd beast that it is dubious anything of general relevance about TCS can be deduced from it.
We use sub-reviewers extensively in place of having a formal two-tier PC. It is appropriate that the conflict-of-interest rules you choose should apply to sub-reviewers as well since their reviews play a major role in whether a paper is accepted or rejected.
I recently served as a sub-reviewer for CCC and their review form included the following language. The key aspect is disclosure rather than outright prohibition. (I would attach some of your list as additional examples of the kinds of conflicts that must be disclosed.) I feel that people are much more likely to be honest about the real conflicts that they have if they can give it in this narrative form:
The PC-only part should also contain a conflict-of-interest statement. This is an attempt to address concerns that were raised in the community in recent years. Please be cooperative with the experiment.
The statement should indicate:
(i) any relationship of the authors to you and of the submission to your work that may be viewed as a conflict of interest by someone, and
(ii) your assessment of your bias due to the conflict of interest in the evaluation of the submission.
Below are some sample statements. Note that these are just meant as illustrations. There are undoubtedly other bases for (i) and other combinations for (ii) that apply.
o The author is a friend and former co-author of mine but I feel I can be fully objective.
o The first author is a former student of mine, which may bias my opinion somewhat but not by much.
o Due to my personal relationship/feeling towards the second author, I feel that I cannot offer an unbiased evaluation. Still, I think I can be trusted for the following factual information.
o I had disagreements with one of the authors in the past and fear that my opinion may be significantly influenced by that. Nevertheless, it may be helpful for me to point out the following.
o I have a competing submission to another conference and describe the relationship with the CCC submission below. I tried to be objective but there may be an unconscious bias in my evaluation.
In case there is no conflict of interest, simply write "none".
Paul,
I think it would be an interesting discussion whether disclosure or prohibition (or some mix of the two) is most effective. However, avoiding that discussion for now, I would certainly agree that a formal disclosure process would already be a significant positive change for the theory community, and I would support its regular use.
I'd like to note that, as a subreviewer, I've been including such narratives regarding my conflicts in my comments to the PC as a matter of course for several years. It's been unsolicited, but I've felt it appropriate (actually, imperative) to include it anyway. One of my standards is, "A possible conflict of interest is that I have also submitted a paper to this conference, which I naturally believe is much more worthy of acceptance than this paper."
"A possible conflict of interest is that I have also submitted a paper to this conference, which I naturally believe is much more worthy of acceptance than this paper."
Why "naturally"? Do you think a priori that your submissions are always the best ones?
I hope this is just an inaccurate use of words...
Anon #9: Sigh. I hate having to explain to the humor-impaired, but:
"That's a joke... I say, that's a joke, son."
"A possible conflict of interest is that I have also submitted a paper to this conference, which I naturally believe is much more worthy of acceptance than this paper."
This is a good statement. It could be automatically determined. Given the current job situation -- one job for every 20 graduates, maybe? -- students have every incentive to shoot down their competitors. Students should probably be disallowed from writing reviews.
Anon #11: You're getting the joke much better than #9.
Do I think that just because I have a paper submitted to a conference that my papers are automatically the best, or that I'm more inclined to give a negative review to another paper? No, actually, I don't. But whether I'm biased or not is really for the PC to decide, not me; and I think it's a clear conflict of interest that I should report when reviewing.
And I might as well point out the obvious conflict in what I think is an amusing way.
As a proposed subreviewer for major theory publications I have previously turned down requests to review papers by close colleagues with whom I was in active collaboration. I have been surprised by the surprised reaction I have received from the main reviewer. This always strikes me as somewhat worrying.
I agree with Anon #1 that networking papers seem to be more subjective...
And here I'll offer an extreme view of the situation, which will hopefully indicate why conflict of interest rules may be hopeless for networking. Even though it's extreme, I hope it's not brushed off completely, because of the special, real-world circumstances that networking is subject to (as opposed to theory).
In networking, there are many different sub areas that are very well established. Well established here means that protocols, equipment, and everything else network related has been out there for a while. Furthermore, any disruption can cause significant changes to one's livelihood.
Here's an example. Suppose reviewer A works in the industry and makes his living building equipment X. Along comes a young naive PhD student who proposes equipment Y, that is so brilliant that it knocks the socks off of X. Now A, whose livelihood depends on making X, wonders if he will accept his destiny and let the paper through, or delay his job search by rejecting it and he can go home tonight and put food on the table for his family. What do you think he will do?
Now bring into focus the general consensus that networking papers have become more airy-fairy in the recent past. Couple that with the last point raised and I wonder more and more: are the papers that are being accepted those that do not raise any controversy, they don't impact anyone's lives? Being 'benign', they're easier pills to swallow?
And I think the crux of the matter is: who governs the PC committee? Looking at all the politicians (and yes, the more political ones tend to become faculties and move on to PCs for networking), I highly doubt that they can control themselves. Imagine young assistant professors who need tenure get onto a PC. Since they have the power to accept or reject papers, what do you think they will do? And since the rest of the committee have above-average relations with these young urns, what do you think they will do? And since networking is more subjective than say theory, what do you think will happen?
Post a Comment