Back to STOC stuff, continuing my past post on conflicts. While the theory of how to handle conflicts is one thing, the practice is another. How did things work in practice?
For those who say it's mildly annoying, I admit I agree. People having to leave the room is obviously less appealing than people not having to leave the room -- I'm not going to argue that. But really, I think the annoyance is minor, at worst. In most cases, zero to two people had to leave the room. People seemed to pay attention and get up and leave when they had to. (Arguably, people pay more attention to what's coming up when there are conflicts, which is actually a plus.) We could have been better about remembering to call people back in promptly. With practice, I don't think that would be a big deal either.
I didn't feel that we lost out on needed expertise because of the policy, but again, the implementation was more flexible than in networking conferences I've been involved in.
One PC member insisted on leaving the room for a discussion and vote because of a conflict, but we insisted they stay to answer our questions first!
Overall, I did not find it overly disruptive. And I think such a policy greatly reduces or even avoids worst-case scenarios where a PC member -- intentionally or unintentionally -- biases the outcome of a paper where they have a conflict. I've been on many PCs over the years, and I've seen it happen more than once. No, I'm not saying such happenings are rampant -- I think they're rare -- but I think they could be rarer still. To me, the annoyance involved is a small price to pay for that. Also, I think even if you disregard the possibility of someone intentionally pushing a paper where they have a conflict (which, actually, does happen), on the whole people underestimate the power of unconscious and unintentional bias.
Paul Beame (in comments to the previous post) says that people should be allowed to stay in the room, but remain silent and declare conflicts as they arise; and similarly, that software shouldn't block people from seeing reviews/discussions on conflict papers. I think he offers a consistent alternative, but I (as he knows :) ) disagree. His approach seems designed to avoid any possible manipulation of decisions by PC members that may have a conflict, which of course is all to the good. In practice, in my experience, that silence rule is not generally maintained. (For a variety of reasons, many of them laced with good intentions. It's hard for experts to stay quiet, particularly when a colleague/student/etc. is involved.) He ignores the issue that simply having the person in the room may stifle some discussion in the PC meeting. (Junior people can be and are often intimidated, to various degrees, by senior people in such a setting. We can argue about whether they should be, but in practice, they can be.) And finally (and most importantly), he's dismissing the issue of information in reviews or discussions getting back to the authors. (Yes, discussions are confidential -- as are the parts of the reviews labelled "to the committee" instead of "to the author" -- but as we've read in the previous comments, there can be a fair amount of leaking of information after the fact.) My thought is that when you say you have a conflict, that precisely means you SHOULD NOT see the reviews or hear the discussion for the paper -- the networking standard -- unless there's an important overriding reason for you to do so. That prevents you from intentionally or unintentionally leaking supposedly confidential information.
And to be clear, I think the "we need the expertise" argument is overhyped. In cases where it's needed, I would agree exceptions need to be made. In most cases, it's not needed (there are other people in the room capable of making the judgment), but people don't want to be left out of the decision process where they are also an expert. There's a difference.
I have no regrets about implementing things this way. Others disagree. Overall, I think it's a topic worthy of more community discussion. In particular, the use of conflicted subreviewers that also came up in comments on the previous post really deserves some attention. I think a more consistent and thought-through standard should apply. Our community may have it's own standard, but as a community I think more discussion including ackowledgment and understanding of the pros and cons of the various possible approaches would be useful.
Subscribe to:
Post Comments (Atom)
54 comments:
And to be clear, I think the "we need the expertise" argument is overhyped.
I disagree. Even when there are other people who are capable of evaluating the paper, I believe that giving up the opinion of a knowledgeable person only because she is a colleague of the authors is a losing proposition.
I think such a policy greatly reduces or even avoids worst-case scenarios where a PC member -- intentionally or unintentionally -- biases the outcome of a paper where they have a conflict.
I strongly disagree. What you call a conflict of interest, others may call a legitimate opinion regarding the research direction. Fact is, people have agendas, and some people will try to push their agendas on the PC. This has nothing to do with working in the same institution with the authors, and trying to solve it using conflict-of-interest rules will never work. The only way to deal with agenda-driven people on the committee is to be aware of this possibility, keep an eye out for it, and try to neutralize it in the (rare) cases where it comes up.
Yes, discussions are confidential -- as are the parts of the reviews labelled "to the committee" instead of "to the author" -- but as we've read in the previous comments, there can be a fair amount of leaking of information after the fact.
People are naturally chatty. This is another fact of life, and we all need to live with it. Again, as long as there are no egregious violations of secrecy, I think that we should err on the side of having more information available for the PC, not less.
Shai -- I think it's clear we just disagree. But I respect your opinion and the clear and polite way you presented your side. (It's true -- people who sign their names do give better arguments on blogs! :) )
I am all for a completely inclusive meeting, not excluding *anyone* for purported conflicts -- after all, we are a community of civilized individuals. If someone is ostensibly biased in their views, I am confident that that information will propagate and that person won't get to sit on future PCs (if the chair does a decent job of politely inquiring about their choices' past committee performances).
Having said that, I agree completely with Michael about the "expertise" argument being overhyped. If a certain committee member's expertise is required to save/kill a particular paper, then one the following is true:
(1) that field is too marginal (not enough experts on the PC or in the large pool of sub-reviewers)
(2) [completeness] the authors did not do a good job of underscoring the importance of their paper to the larger theory community
(3) [soundness] the rest of the PC/sub-reviewers/review process did not figure out that the paper does not really add value to the conference.
In any of these situations, if the "expertise" was required, the PC + Reviewers or the authors didn't do their job properly.
There is something way off in PCP's comments about expertise. These comments seem to misunderstand both the operation of PCs and the diversity of our field.
* No PC of the manageable size we use can possibly cover the breadth of the field with even 2 reviewers for every important research area, let alone the preferred 3 or 4 per paper. A very important area may not generate a large volume of submissions.
* Expertise on the PC is essential. Sub-refereeing only goes so far: A sub-referee might see a couple of papers, but does not see the comparison papers on other topics and cannot weigh their relative value or importance. Given limited space, assessing relative value is the the really hard job of program committees and they need "in house" expertise for this job.
By the way, a point I should have made clearly in my post -- it's not clear to me how we as a community logically maintain the ideas that
1) PC members shouldn't be able to submit to conferences (even though this is done in many other CS communities) because of "appearances"
and
2) we should not have a strict policy for conflicts (e.g., in my mind we should essentially ignore conflicts) within a committee meeting, because we're all adults and it will work out fine.
Hi Michael,
I am a bit puzzled by your last comment. In the PC I participated or chaired there was usually a distinction between soft conflict of interest and hard conflict of interest. The latter was usually interpreted in the narrowest possible terms (being the author, family relation to the author, the author is current student, etc.) In such cases, the PC member was blocked from the discussion from the beginning and through the PC meeting. On the other hand, soft conflict of interest (e.g., being from the same institute) was handled with much more flexibility, and participation depended on how valuable your contribution was. So regarding your last comment, do you really not see a difference between being the author and being in the same institute? Why is it inconsistent to draw a distinction between various levels of conflict of interest? Somehow I believe I would feel more uncomfortable criticizing a work with the author attending than with a college attending …
Regarding the discussion, I fully agree with Shai's comments. Particularly, technical rules such as `coming from the same institute' do not start to cover true biases (for example, I have close friends that are not in my institute and with whom I hardly collaborated). So if we assume that everyone is malicious then the entire process is doomed. I think the role of a PC chair is to select his/her PC well and then to put some trust in it. After all, the person that has the largest potential of biasing the decisions (intentionally or unintentionally) is the PC chair. The collective wisdom is always greater...
Omer --
Several conference simply don't allow PC members to submit. I don't see why -- it's easily handled by current software, and PC members can be excused from the room during discussions. The main reason I've heard against having PC members submit is that it would somehow give the appearance we're not a "serious" conference. Yet we allow all sorts of other conflicts without question. That's inconsistent to me. Why is having PC members submit so bad, but it's OK for someone to be in on discussions of a colleague's paper? Seems odd to me.
If you're suggesting that having "loose rules" regarding soft conflicts is OK, while also allowing PC members to submit with stricter rules, I'd agree that's a consistent position. Again, not a position I'd necessarily agree with, but at least a consistent position.
As for your second paragraph, you seem to be saying, "I can't design a system that covers all possible biases. So I'll put my head in the sand and ignore them and assume they aren't there, or if they are there they'll be so transparent to everyone they won't actually be a problem."
I heard things like that a fair bit. (One thing I heard is, "It really doesn't matter. We're all adults.") I really think this is fundamentally naive, based on my experience. More kindly, I'm glad to hear people have had such good experiences they think this well of human nature, to the point where what I find to be the relatively minor requirement that people with obvious nominal conflicts leave the room others find unnecessary.
Not at all, what I'm saying is that it is better to put the trust and responsibility in people (asking them to be open about their biases and to be honest about their COIs), than to put it in a set of rules that to me simply seem misguided and ineffective.
PS. Regarding good experiences, I'd like to add that (contrary to what blog readers may think), we have a community to be proud of. It is not that we do not have any jerks and large egos in our community, and we are not free of politics. But this is nothing compared with other areas of academia. My non-cs colleagues here at Weizmann would switch with us in a heart beat …
Omer --
1) I'd agree with your last comment. I like the CS community. I too am often surprised to hear of stories of bad behavior of other communities -- we're remarkably free of that.
2) Regarding the previous comment, I wouldn't say the rules solve every problem. But they do solve problems that your "trust everyone to behave" attitude just doesn't. I don't see why you wouldn't have to rules PLUS the standard that people be open about any other conflicts. In particular, you don't mention how to handle it when people don't live up to this trust (which I've seen, and my rules can potentially avoid), and you don't discuss the other points I've brought up -- unconscious bias, the potential for intimidation (purposeful or no), and leakage of information post-meeting, all of which my suggested rules address. You are welcome to correct me, but your argument so far is "I prefer to trust everyone and it will be OK". You've stated you think my proposed rules are misguided and ineffective but haven't provided reasons behind that. So we can take this offline, but I'm not convinced of anything yet.
The thing is that in my opinion your rules come at a heavy price (as I said, I fully agree with Shai). In any case, as optimistic as I am, I did not really believe I will convince you … :-) Taking it offline sounds good to me (I don't think I am much of a blog person after all).
Omer --
I've also heard the opinion that the price is high. My experience suggests otherwise. However, I am also cognizant that for theory it makes sense that there might be cases where committee needs require that the default (people have to leave the room for a conflict) needs to be dismissed, as I have said, and implemented it that way in practice.
I worry people here are conflating cases where a PC MEMBER who has a conflict would LIKE to express an opinion -- which I imagine is relatively frequent -- and cases where the PC COMMITTEE NEEDS information from someone who has a conflict -- which in my experience is relatively rare, and can be handled as an exceptional case. In my mind, the first type of case should not override these simple rules for dealing with conflicts.
If we took a straw poll of the community, I wonder how many people would support allowing PC members to submit (say) one paper? Assuming significant support for this, how would we as a community go about getting the rules changed?
JK --
That's a great question. I'm honestly not sure. I think as a starting point, someone could bring it up at a conference business meeting -- that's a reasonable place to have a straw poll. I think, though, that it's something the steering committees of conferences would have to be comfortable with before a PC Chair could suggest trying it.
I would hope the arguments in favor would be clear -- easier to get people serve on committees, potentially bigger committees, avoids committee members having to wait several months (or more) to get results to the best conference. The biggest argument I can think in favor is that it's actually a benefit to our graduate students -- I had potential PC members turn me down because they had students submitting work, and I had PC members say that they wish that they had turned it down because their work with a student had turned into a potential paper after they had agreed to serve...
Dear Michael,
I was referred to this posting by somebody and although I only read it casually and partially it is my impression that you fail to distinguish between what you are allowed to do as an individual (e.g., have a blog, express personal opinions on anything, suggest various reforms of various institutes, etc)
and what you are allowed to do when you are **entrusted**
by the community to serve as a PC chair (rather than as a reformer).
[And yes, I wish to stress the words "entrusted" and "PC chair"
(and remind you that a "PC" is entrusted to determine the scientific program of the conference according to predetermined principles
and that a "chair" is merely the person that orchestrates the work
of a committee according to predetermined procedures).]
Personally, I consider the COI policies that you introduced
to be both ridiculous and harmful. I believe I'm free to say
this opinion openly (and pay the price of annoying you).
However, if these were the policies in effect for ages
and I were the PC chair, I would not have allowed myself
to abolish them (definitely not without a fair and comprehensive
discussion of some wide body (i.e., wider than the current PC,
which by your post only held "a straw vote" in which "a large majority of those with an opinion thought ...").
Oded (Goldreich)
P.S.: I usually do not read blogs, so I'll not be seeing
responses to my posting. Anyone wishing to communicate to me,
may just send me email.
Adding to what has been said.
Publishing in FOCS/STOC has regrettably become the yard stick according to which promotions are decided and positions are awarded.
We must be extra careful, in this climate, not to test out
new experimental theories in the name of `ethics' without proper reflection, and risk the possible loss of expertise and deeper understanding of submissions.
And to the heart of the matter,
true conflict of interest occurs when people compete for the same
resources: jobs, letters of reference which include comparisons, awards.
Such competition has little correlation with formal associations of the kind outlined above, e.g. co-authorships, same institute, etc. The best we
can do is to institute an atmosphere of trust and honesty in which PC members are willing to listen to each other and evaluate each work on its merit putting aside their `suspicion' capes.
Disallowing expression of opinions and forcing people to leave the room suggests the opposite spirit.
Let me add another p.s.
My main concern is actually a feeling that the quality of the work of the PC has deteriorate throughout the years. The first question, of course, is whether my impression is right. Assuming so, then one may seek to understand the roots of this phenomenon, and then to try to devise ways of redeeming the situation.
Needless to say, these questions
(assuming my basic impression is right) should be considered carefully by adequate bodies.
My own impression is that the PCs are not only overloaded by work, but rather they lost sight of the reason we care and invest so much effort in these conferences, which have been reduced to "weight lifting" competitions. The original vision, to remind all, was to have a forum for exachange of ideas...
Oded
Oded --
Might instead part of the reason for the decline in quality be that PCs are a bit too chummy sometimes with the authors, so "less good" papers with name recognition/lots of friends on the PC are more likely to get in?
I'm not saying this is the case. However, I do think part of the reason that stronger CoI rules were put in place in other areas/conferences was the impression or the appearance that this was happening. (I can't speak to the reality.) You haven't seemed to entertain that possibility.
Oded --
Of course you're allowed to express your opinion! The only thing that annoys me is that you've chosen to express this opinion without actually engaging in any actual argument to any of my points, but instead just state you disagree. I'm happy to listen to counterarguments.
Your argument seems to be that I should have kept things the way things have always been. I'm not sure what this standard is -- it seems to me from talking to people there have been slightly different standards on what exactly has always been -- conflicts have been handled differently from conference to conference. Of course, your argument also suggests that we should keep having paper proceedings for eternity, we shouldn't bother adding abstracts when we announce papers, we should never have moved to an on-line system, etc., so I'm not having trouble taking that argument too seriously.
Perhaps you object that I made the decision unilaterally? First, as far as I know, I had the right and responsibility to if I thought it best. Second, I didn't. As I said, I asked the committee, and besides being my opinion, it was the opinion of a substantial majority. Since we're the ones deciding the papers, it seems reasonable to me we set the rules.
Michael.
I really don't want to get into details; this will be way too time-consuming. Overall, I did read some of your text (but not the entire text and all responses etc, as this would be too time consuming... again...), and do not agree with it. I do not think that the current COI policies (at their variety) are the source of trouble (i.e., the one that concerns me) nor the sourse of any existing trouble.
My point was and is that you should have not introduced that radical change from past practices
(and a "strew vote" is not a proper replacement for an actual elaboatret discussion, a discussion you want to hold now (whereas you should have conducted it with your PC and beyond months ago!!!!)). My views are based on my understanding of the rule of a PC chair, which includes what this person is allowed to do and the care to be exercised in not using this position in order to promote various personal agendas and controversial reforms. That is,m you do not have the right to do what you think is best, but rather you have a right to exercise your judgement *within the boudaries* of what is agreed and accepted in our community. I claim that you went outside these boundaries (which is indeed a question of judgement, as discussed in next paragraph...)
It is easy to ridicule one's position by taking an argument to the extreme (ad absordum). In contrast, a candid discussion should not refer to such tricks.
E.g., common sense will tell you that introducing e-submissions
(which was, btw, done carefully and gradually...) in response to a new technological possibility isdifferent from introducing a new COI policy without any real reason.
The fact that other disciplines do it is no reason.
I really hate using this media. If you write a position paper on COI or anything else andsend it to me, I will try to undertake writing an answer in the form of a position paper, provided I get to it...
Btw, I was thinking of writing a position paper on my main concern (see my 2nd posting -- the "additional p.s.") for more than a year... Maybe I'll get to it some day.
Oded
Having people, who are from the same institution as the authors of the paper under discussion, leave the room surely does not address all possible sources of a COI; but if this is a "radical" change from past procedure (as OG says), then that is pretty scary. What were PCs doing in the past? No wonder people think that theory is "chummy".
To annonymous #21.
(Last time I'll read this blog...
It is a pain as I feared...)
I don't think that mild levels of dishonesty and clumsiness or whatever human failures are eliminated by fomal rules, but rather by creating a certyain atmophere (as indicated by anonymous #16 or so). Having an expert leave the room (just for a formally ridiculus reason of coming from the same institute) reduces the quality of the PC signaificantly, because you are likely left with 1-2 experts (at best) and a handful of others that are qualified to comment (also in good cases -- nowadays!!!). So the variance of opinions (viewed as independent random variables with mean at the paper's quality and variance inversely proportional to expertise and judgement) dteriorates....
If we are not willing to trust the PC on honesty, then we are indeed in trouble. But the same holds for more importnat procedures like hiring etc (where you also trust the hiring committee and the letter writers etc etc).
I could go on, but as I said I really hate this media and suffer fromn it. I'll be happy to continue by email.
Oded
Michael, to try to swing things to the more positive side, let me thank you for being the first FOCS/STOC PC chair to discuss so candidly your expereinces, opinions, and recommendations. I find it interesting and informative, and think such discussion can only help the community move forward.
And THANKS for all your hard work!
Anonymous 23: Thank you for the kind comment. You really bring up what (to me) is an important point. People can agree with me or disagree with me -- but I think, as a community, it's important that we all know (especially students!) how things work, and that we discuss how things are and how they might be better. One reason I took the job was that I thought it would be worthwhile to blog about it and let people see a bit of what was going on through my eyes. I appreciate that you saw that and chose to comment on it.
People can agree with me or disagree with me -- but I think, as a community, it's important that we all know (especially students!) how things work, and that we discuss how things are and how they might be better.
I have to say that one of the things that surprised me the most when moving to TCS was the lack of such dialogue. Mathematicians (including many of the most brilliant ones) devote a non-negligible portion of their time to house-keeping tasks (discussing direction of the field, cleaning proofs of known results, writing surveys, etc).
Lance's blog and now yours have become the natural fora where TCS people can discuss important issues to the community, and we all the better for it. Keep up the good work!
I understand the need for
`artistic expression' via blogging. However, one gets the feeling you have used this forum instead of allowing discussion, of what policies should and should not be followed, with colleagues and PC members. Blogging does not equal transparency. In fact, it may give one a false feeling of being in the right. Very dangerous indeed.
Anon 26:
I have to admit, because you're writing a little cryptically, I don't know what you're talking about.
The blog is certainly open for discussion. I haven't censored any comments from those who disagree with me; indeed, I've often tried to respond to them, which in a number of cases seems to be more courtesy than they've given me.
I expect the discussion to continue outside the blog where the community finds it's warranted.
Obviously, I have also used this as a forum, to express my opinions. It's my blog, that's one thing it's there for. But that could and hopefully should be the beginning of a dialogue. Feel free to participate -- here, or elsewhere -- and expect me to do the same.
I think the calls to continue the discussion in email are very disappointing, since that is a
much less transparent medium.
I think the calls to continue the discussion in email are very disappointing, since that is a
much less transparent medium.
That exchange was a bit on the light weight side. It is not possible to write "I really don't want to get into details; this will be way too time-consuming" and expect to be taken seriously.
We are all busy people: think Michael with a young one at home, a renowned conference to run (STOC) and another to review (SIGCOMM). Yet he replied to every question and engaged in the debate.
That exchange was about whether it was acceptable of Michael to make such significant changes without any serious debate, even among his own PC. Oded refused to get into the details of why the suggested policy is wrong, and this is completely fair as this was not the issue he raised (by the way, I don't think there is too much competition to Oded in terms of service to the community). He did suggest a more serious forum for debate – position papers.
Anon #30: If the issue then is just the procedural one, then I disagree with Oded on the extent of the change implemented (see my description of its working in practice), I disagree that I did not have the right make this policy as PC chair without further input, and I disagree that there was no "serious debate". In the end, though, as I think I've made clear, I made the decision, and I stand by it.
I think having clear rules regarding CoI is indeed important and will help the whole process. In that sense I agree with Michael's proposal.
Saying that all other communities within CS have it for no reason or that we are all civilized adults who know how to behave is among the weakest sort of arguments I have heard. If true, it could be applied to the whole society at large and therefore no need for laws and rules.
I have heard far too many complaints from people about these things in our community. If you haven't, perhaps you are on the other end of the stick.
Something needs to be done to make things more transparent and make more trust.
Thanks to Michael for his work.
In this blog the claim that "we are all civilized adults" was only made by Michael. It is not an unusual tactic to make the weakest arguments in the name of your opponents and then to crush it. People complaint because they think wrong decisions are made. I have heard the claim that being on a PC is a lot of work only to make people unhappy. The right way to improve the decision making is far less superficial.
Dear Michael,
My impression was that I was quite blunt in my initial posting,
but it seems that I was not blunt enough. So let me try to be
even more blunt now.
I consider the abuse of power entrusted in a person
to be one of the worse cases of unethical behavior.
(I am aware that some people think that COI policy
of the type that you introduced are meant to prevent
such abuses when they take the form of a biased evaluation
of papers by PC members (whicd I'd actually call "dishonesty").
However, I disagree with the effectiveness of formal rules
in coping with these cases, and believe that other ways are
much more effective (and have no harmful side effects).
But this was not the focus of my initial posting.)
The focus of my initial posting was the abuse of power
entrusted on a chairperson of a committee. I wrote
I wish to stress the words "entrusted" and "PC chair"
(and remind you that a "PC" is entrusted to determine the scientific
program of the conference according to predetermined principles
and that a "chair" is merely the person that orchestrates the work
of a committee according to predetermined procedures).
I was referring to the description of the events
as appearing in your own blog (of Feb 1st).
This description mentions no extensive discussion,
but rather a strew vote (taking place before any general discussion)
and a chairperson processing the conflicting opinions
(again without mention of a general discussion),
and ruling according to his understanding
(i.e., you explicitly say that YOU made the decision).
Furthermore, it was stated that there was a minority that opposed the motion,
and in my opinion the combination of improper procedure and a controversial reform
suffices to make one require that the decision be revoked and made void.
(Bear in mind that we are not talking about some minor technicality,
but rather about a significant change of the common practices of PCs in STOC.)
The key paragraph is (partially) reproduced here.
"But I asked the PC for a straw vote, and a large majority of those
with an opinion thought ... I should be clear, however, that this idea
was quite controversial; many PC members expressed a very clear
and vocal dislike for a policy ... Indeed, it was definitely the most
controversial decision I made as the PC chair."
My (reasonable) assumption was and is that the above provides
a good description of the process in which the decision was made.
I find this process highly inappropriate and believe that the
chairperson's actions constituting a clear case of abuse of power.
I don't recall whether you describe the minority's reaction to your decision
(once made) and whether the entire PC was made aware of the controversy.
In my opinion,the minority should have demanded that the decision be revoked
and everybody on the PC should have supported this demand (regardless of his/her
opinion about the specific issue (i.e., your COI policy); just out of understanding
that such controversial decisions cannot be taken by a strew vote and are clearly
not up to the chairperson). If I were on the PC, then I would have resigned
immediately if this demand would not be granted.
Oded
Oded --
Right. Then we agree to disagree.
A bit off-topic perhaps, but I disagree at some level with Oded's description of the relations between the PC and its chair:
... a "PC" is entrusted to determine the scientific program of the conference according to predetermined principles and that a "chair" is merely the person that orchestrates the work of a committee according to predetermined procedures
I tend to think of the chair as the one who is entrusted with determining the scientific program, and the PC as a body whose job is to assist the chair. This is clearly how the review process is perceived "from the outside": the chair is the one appointed by the steering committee, and the chair is also the editor of the resulting proceedings.
Beyond just play of words, I think that the chair has more leverage in setting rules and procedures than what's implied by Oded's description. Just as I believe that PC members should generally be trusted with the review process and not saddled with unnecessary rules, I also believe that chairs should generally be trusted with making the rules (and not saddled with unnecessary meta-rules about what procedures they can or cannot establish). The PC members should be consulted and the rules must be made clear everyone, but I don't see how making "reasonable" rules by the chair can be considered an abuse of power.
As to what is a "reasonable" rule, the test is actually very simple: A rule is "reasonable" is the PC accepts it. If Oded was serving on Michael's committee, then maybe the leave-the-room rule would have generated a heated debate and eventually rejected. From Michael's description, it seems that the committee accepted this rule without much fuss. In my book, this makes this rule reasonable.
It is amusing to see a discussion about transparency and public debate end with "the PC chair makes these decisions and the hell with everybody else".
Anon 37: How odd that you quote that, since, actually, I don't think anyone said anything like that, or even close.
Indeed, you did not say the hell with everybody else. Apologies for rephrasing.
Eventually you need to decide if it is merely a procedural issue or a substantial one.
I don't think we've heard yet from the PC
(or if we did I missed it)
so...
I was on the PC, and I'd like to
thank Michael for doing a thorough and
conscientious job.
Leslie Goldberg.
For the sake of the community, I post my answer to Shai,
who wrote me email reproducing his posting.
I was hopining that people will write me email
rather than post on the blog...
Oded
ENCL: EXTRACTS FROM MY EMAIL TO SHAI
>> From shai.halevi@gmail.com Wed Feb 11 07:58:48 2009
>> Oded, you wanted to get replies by email, so here is one..
[I was hoping that people will write to me rather than post...]
>> A bit off-topic perhaps, but I disagree at some level with Oded's
>> description of the relations between the PC and its chair:
>>
>> ... a "PC" is entrusted to determine the scientific program of the
>> conference according to predetermined principles and that a "chair"
>> is merely the person that orchestrates the work of a committee
>> according to predetermined procedures
>>
>> I tend to think of the chair as the one who is entrusted with
>> determining the scientific program, and the PC as a body whose job is to
>> assist the chair. This is clearly how the review process is perceived
>> "from the outside": the chair is the one appointed by the steering
>> committee, and the chair is also the editor of the resulting
>> proceedings.
I strongly disagree with you and am quite surprised that you think so.
I am aware that some people "from the outside" say this at times,
but I'm surprised that a clever person being in "sound mind (mood)"
thinks so.
A chair (of a PC or any other committee) is merely the person
that runs the committee. This may include choosing the committee,
but from that point on all the decisions are of the committee.
The model you describe is a different one, which when existing
is referred to an advisory board to the decision-maker.
This is NOT how PC are understood in TCS
(neither in FOCS/STOC nor in TCC or RANDOM or CC,
to cite confrences in which i am quite involved).
>> Beyond just play of words, I think that the chair has more leverage in
>> setting rules and procedures than what's implied by Oded's description.
Indeed, the chair has a ruling power in two aspects
(1) reminding all what is the procedure in place
and insisting that it be followed.
(2) making various low-level and technical decision
invlved in running the committee.
The chair has no madate to introduce major changes in
the common procedures, let alone without proper consultation
of the PC and in light of strong opposition of some of its members.
Note that doing so is also a breach of confidence wrt these members,
who agreed to served under one set of rules and practices...
Furthermore, when the chair acts within the domain of (1) and (2) above,
he/she ***should be very careful NOT to use these powers in order to
promote his/her own views***.
The fact that MM violated this basic principle
(and thus has abused his chairing powers)
is what made me react to his posting...
>> Just as I believe that PC members should generally be trusted with the
>> review process and not saddled with unnecessary rules, I also believe
>> that chairs should generally be trusted with making the rules (and not
>> saddled with unnecessary meta-rules about what procedures they can or
>> cannot establish).
a) The chairs do not have a mandate to make rules (except technical ones),
they only have a mandate to implement the common rules.
b) I distinguish between informal meta-rules of proper behavior
and formal rules that will always fail to capture this evasive notion.
The principles I used in my reasoning are not stated anywhere
(i.e., not in TCS documents at least). They are just common sense.
>> The PC members should be consulted and the rules must
>> be made clear everyone, but I don't see how making "reasonable" rules by
>> the chair can be considered an abuse of power.
The rule made by MM is, in my opinion,
not "reasonable" at a sufficient level to allow him to DICTATE IT
(e.g., he ack's that it was controversial in the PC).
Thus, in my opinion, he did abuse his powers both in the procedure
he employed (for deciding on this rule -- see my previous post)
and in the fact that he followed the rule in face of objection.
>> As to what is a "reasonable" rule, the test is actually very simple: A
>> rule is "reasonable" is the PC accepts it. If Oded was serving on
>> Michael's committee, then maybe the leave-the-room rule would have
>> generated a heated debate and eventually rejected. From Michael's
>> description, it seems that the committee accepted this rule without much
>> fuss. In my book, this makes this rule reasonable.
I reject this definition of "reasonable" when meant to describe cases
in which the chair can decide on action by him/herself.
I wish to distinguish two cases re arguing for the legitimacy
of activating the new COI rule in the last PC.
1) If the basis for legitimacy is a majority vote (as hinted above),
then this can serve as a basis only after a general and extensive
discussion of the committee. This legitmacy (for a controversial act)
cannot be based on a strew vote, let alone one taking place before
any general discussion.
2) If the basis for legitimacy is the chair's ruling,
then I claim that the chair has no such mandate in such matters
and thus the decision is illegitimate.
Furthermore, in my opinon such a change of rules should have
been discussed at the very beginning of the PC work.
Oded
I was on the PC and as such feel conflicted about writing.
There is an unwritten rule that what goes on amongst the PC is private.
However, since the chair and other PC members seem comfortable with
disclosures about the COI issue,
I would like to clarify some of Shai's impressions.
The COI rules that were suggested did generate strong opposition by at least one committee member. However, there was no open debate
regarding this issue, as Michael felt that it was not a subject for
discussion but rather for a straw vote made to the chair.
One committee member chose not to attend the PC meeting since they
realized they would have had to spend a substantial amount of time out of the room, if they
followed the rules faithfully. Luckily, other PC members `didn't feel we lost out on needed expertise because of the policy' (I quote an earlier post).
I personally do not agree with this COI policy, nor do I find it necessary. Most of the reasons for this opinion have already been articulated by others in earlier posting.
I would like to add one point which has not been raised before. Regardless of the merit (or lack thereof) of this COI policy,
I think it is IMPOSSIBLE TO IMPLEMENT in our highly collaborative field, IF senior members of the community are to continue to be a part of future PC's. To give you an example, as someone who has been publishing at these conferences for about 27 years, to follow the rules proposed strictly, I would have had to exit the room between 30-50 times.
One may take the position that senior people are not essential for the make up of a good PC. Certainly, according to these rules, they are likely to have conflict of interest with almost `everyone' whose work they understand.
Does this mean they lack have the ability to exercise sound and `impartial' judgment??
I have great respect for the young people in this field.
Many are technically brilliant beyond anything my generation was capable of. As we like to say, they can `digest stones'.
But, are they less biased and more `impartial' than senior people?
I strongly doubt it.
Oded and Shafi: it bewilders me that you both appear to be fighting so strongly for the right to commit conflicts of interest. Why do you think that to be more important than the rights of the submitters to have their papers discussed free from such conflicts?
This is, by the way, not the first time I've seen people leave the room for conflicts of interest in a STOC PC meeting, so I don't think Michael was setting any kind of exciting new precedent here.
I did not sent any paper to stoc this year, but i would definitely prefer a PC that errs on the side of too restrictive COI.
In my opinion, I see 'a recent coauthor' not a reason for COI, but would mine if it is decided that way.
I see a clear COI when people are or were in the same institution for some time.
11011110, what bewilders me is that you think that working in the same institution as the authors is a real conflict of interests. Even more so that you seem to believe that this rule is somehow relevant to "the rights of the submitters."
A point that I was trying to convey in this entire argument (unsuccessfully, I think), is that relying on common sense and informal "standard procedures" typically works better than setting rules that must always be obeyed. The latter should only be invoked if there is a serious failure of some sort
Lest we forget, the only reason that society cares to pay our salaries is that we somehow advance science. In my world, this goal is best served when the flow of information is as free as we can make it. Hence my strong bias against setting strict rules that limit information flow, unless there is a proven need.
11011110:
I am not sure what in my post made you think that I am fighting to commit COI ???
All I said was that I don't agree with the COI policies DEFINED HERE,
and believe they will lead to loss of expertise/information which may result ultimately in less knowledge/fairness than existed before.
Also, the implication you seem to make is that one is fighting to join stoc/focs pc's in order to twist the outcome --
a kind of outrageous proposition.
Being on a committee is a lot of work with the dubious return to the PC members of having people suspect you were responsible for rejecting their paper, or at least did not appreciate it enough to accept it. From that point of view, Michael certainly has made the life of his PC very easy. No one we know can suspect us of any influence ! The question remains however not whats right for us but whats right for the program.
"Being at the same institution = COI" seems unfair to researchers from large institutions. Your paper misses more expertise in the discussions than someone from a smaller institution.
My main concern about these new COI policy is that it creates the impression of eliminating most COIs, both to people already in the community and to incoming students.
First, when people believe COI is already eliminated (even if only to large extent), they might be less careful when getting somebody's input, and other situation (e.g. disclosing a conflict when writing a review).
In fact, many of the COIs occur for a different reason, which is likely to be unknown to the PC, and thus honest disclosure of them seems a better policy than strict rules. For example, suppose a borderline submission builds heavily on your own work in a recent STOC paper. Clearly you are the expert, but of course you are likely to gain from this citation (think e.g. about the talk basically describing your own work).
Second, I often felt that its much easier for a PC member to "kill" a submission they dislike, then to "promote" one they do like. A large number of the accepted papers actually lie close to the borderline, and its just easier to emphasize their downside (or perhaps it is only human that negative feelings are stronger). If I understand correctly, the new COI policy is aimed at eliminating positive bias, but is useless against negative bias.
Indeed, in my experience, (1) the hardest yet decisions are regarding the borderline cases, which tend to be a large fraction of the accepted papers, and (2) these papers are borderline exactly because there is some aspect in which they are not so great.
Finally, I am not convinced the situation regarding COI is any better in other CS communities (with double blind submissions etc.), despite the better impression received by a superficial inspection (e.g. by students submitting a paper for the first time). Furthermore, in most papers in such areas, the main contribution is "conceptual" (sorry for using this controversial word, I cannot find a better term off hand) in the sense that almost any reviewer on the PC can understand and evaluate the paper. Thus the expertise argument mentioned in previous comments may apply to TCS more than in some other communities.
One really last comment: Although I disagree with some of the arguments and actions), I admire Michael for openly discussing his PC work, which invites criticism. But this is probably the price a blogger must pay to gain the controversy required to have a highly read blog.
Is it necessary for the PC chair to appoint a sub to "chair", to allow the PC chair to step out during discussion of submissions from:
The same institution (Harvard)?
Or of previous students, advisor, collaborators?
Or from research labs and organizations that granted the PC chair research funds (or can potentially do this)?
Clearly, the last argument applies not only to the PC chair, but I don't think such COI is usually "disclosed" to the rest of the PC to be aware of.
Anon 49:
As Chair, I indeed appointed someone to run the meeting while I stepped out for conflicts related to other submissions from Harvard or for advisers and students. That is, I held myself to the same rules I held others.
I admit the issue of "funding sources" I had not considered a conflict, as it had not occurred to me before; it certainly merits further discussion.
"unfair to researchers from large institutions. "
Sounds like a white man
complains that he is
discrimated against because of race.
Come on, do we already have too many papers from those institutions?
Is it time to get some alternative
view?
Michael you did a good job!
Anon 47:
You write:
"...seems unfair to researchers from large institutions."
It shouldn't be. As I've stated (but has seemed to have been ignored) multiple times before, I understand the issue of coverage; a paper should get reviewed by a minimum number of qualified reviewers, or (in my mind) the conflict rule would have to be waived as an exceptional case. It would just mean that when the paper has authors from a large institution, you have to find reviewers not from that institution. That's not impossible. And, I argue, that challenge of finding reviewers is better than the committee trying to determine on-the-fly if there are any current department politics affecting the review of the paper (positively or negatively) -- or even just determining if there's the standard unconscious bias that we tend to favor our colleagues.
Anon #48: Thank you for you comment; you raise a number of points that, should the community continue to discuss this issue, should surely be considered.
Two quick points. First, adding default rules (and remember, my proposal what that the default is you should leave the room -- unless there's a very good to exceptional reason not to) is not meant to prevent disclosure of other possible biases. I agree that, for example, it makes sense under any set of rules for someone to say, "I wrote a previous paper on this topic, and XXX", both to provide context, and to let people know if you may have any bias.
Also, while the rules don't handle all types of negative bias (or, of course, all positive ones), the same institution rule could stop some negative bias cases, for example! (Same with recent co-authors! Or students/advisers!)
Shai --
One can argue whether "same institution" should be a conflict of interest. I think it's a quite reasonable default, but I do understand there are other points of view.
One additional reason for it is the issue of "leakage" of information after the PC that I've brought up. That is, even if you wish to make the argument that same-institution conflicts don't potentially yield significant unfair or inappropriate decisions, there's also the matter of whether it's appropriate for people at the same institution to have complete access to discussions of a paper from the same institution, given that people will talk. Your previous comments suggest you feel this isn't a big deal, but I think many others might disagree.
Post a Comment