Tuesday, July 31, 2007

The Reviewers' Pledges

The state of journal reviewing in CS and EE journals is generally pretty dismal. It often takes an unforgivably long time to get reviews back, and often reviews offer less in the way the feedback and constructive criticism than one might hope.

Here I'll focus on aspects related to timing. (I'm unconvinced that reviews that take longer to get back are any better in terms of feedback. Often, just the opposite -- the reviewer realizes the paper has been on their desk too long and just writes a quick paragraph to be done with it.) Let me suggest a set of basic pledges -- rules that everyone should follow. Feel free to suggest your own additions....
  1. When asked to review a paper, I will respond as quickly as possible, even if the answer is no.
  2. I will review at least 1 journal paper for every journal paper I submit. (This does not mean "I will give my graduate students 1 paper to review for every journal paper I submit." Though graduate students should also do reviewing...)
  3. I will skim through a paper I plan to review within one month of receiving it, so that if it is a clear reject for any reason, or there is an important question that needs to be raised, I can promptly write a short note to the editor (and thereby hopefully avoid a more detailed review).
  4. I pledge to review papers within six months of receiving them. I acknowledge that six months is an upper bound, not a target deadline.
  5. I accept that there is reviewing karma, and I should not be surprised if editors pass my papers on to slow reviewers if I am a slow reviewer.

9 comments:

Aaron said...

These are, obviously, great rules to try to live by. As someone who typically publishes more often in physics journals, however, I'm a bit spoiled with relatively fast reviews. And, as such, I strive to fulfill my own reviewing obligations in a timely fashion (I try to get things done within a month of receipt). The fact that the APS journals have an automated system that bugs you every month is also a great help (but also a great annoyance; at least they let you set "away" status on the website so that they won't send you new papers to review during that time).

This topic makes me wonder whether the state of reviewing in CS and ECE is at least partially due to the emphasis on conference proceedings rather than journal publications. In my experience, conference reviews are short and often unhelpful. Is this because people in CS and ECE generally aren't schooled (acculturated) in how to write a good review?

gilad said...

It seems that if you want more than one person to review your paper (well, if someone wants each paper reviewed by more than one person), you should review more than one paper for each paper you submit (or have your students review more than one paper).
The truth of the matter is that different people have different styles of thinking, and I suspect that reviewing is easier for some than for others just like publishing is. Perhaps reviewing articles (combined, say, with writing reviews of topics) should give people some academic credit, just like publication. The ability to write good reviews and have a wide knowledge of the field is an important contribution to the community, and a great asset for a teacher.

Michael Mitzenmacher said...

Aaron - that's an interesting point about how conference reviews may have spoiled our "reviewing culture" in CS. I think conference reviews have grown increasingly less useful over the years for major CS conferences -- enough that people are starting to pay attention to the issue. But I think our whole culture could use a tune-up. (1 month! I'm asking editors to send you my submissions from now on...)

Gilad -- most CS papers have more than one author. I'm making the handwaving estimate of approximately 2-3 authors per paper on average, and 2-3 reviewers on average. Though I agree that 1 review per paper should be a lower bound...

In general, it's an interesting question how much "service", such as reviewing, is credited by our community. This is something I've been mulling over for a future post.

Paul Beame said...

I think conference reviews have grown increasingly less useful over the years for major CS conferences

If you focus on theory conferences I think that the exact opposite is true. Until electronic review systems were introduced in the mid-90's, there were virtually no conference reviews for major theory conferences, just conference decisions. Moreover, my impression is that reviews communicated to authors are now more detailed than they ever have been for major theory conferences.

Michael Mitzenmacher said...

Paul --

The difference in impression may be that I didn't start publishing until the early-mid 1990's, when electronic systems started. I recall getting fairly detailed reviews at that time. These days, it's not surprising for me to get 3 conference reviews, with 1 being empty, one being 1-2 sentences long, and one actually being somewhat useful.

Anonymous said...

I support most of the pledges, but I do have issue with pledge #1. I think that the issue here can be better resolved by imposing, say, a one week timeout. If the response is negative or none, the editor can assume that the review has been declined.

Unfortunately, often the editors do the opposite: if the review is not declined, they assume this implies acceptance. Sometimes, declining can be quite time consuming in itself, if you have to login somewhere and answer a bunch of questions.

I can see that this is a way to get reviews, but I dont think the reviews obtained in this manner are particularly good.

Piotr

Michael Mitzenmacher said...

Piotr --

I'm also not always thrilled about having to respond to a review, but consider my own point of view as an editor. I might set aside 1/2 a day every two weeks or 1 day a month to deal with editor duties, specifically finding reviewers. If I send you a note and don't hear from you, you're delaying the reviewing process on that paper by as much as a month. If you send me a quick no while it's still fresh in my mind, that month might be avoided.

Anonymous said...

Two comments, one specific and one general.

First, as Piotr indicates, sometimes the review interface is part of the problem. I just today went to try to review a paper and the system forced me to spend 5 minutes filling out a biography first.

Second, I don't know about the rest of you, but I have 15-20 journal publications and get asked to do, I would say, about 4 times as many reviews. I try to say yes most of the time, but this ratio is ridiculous.

Anonymous said...

Michael,

I do sympathise with your point of view - whenever I ask someone for a review, and he/she responds immediately, it makes me a very happy man... However, this is the best case scenario. Often people are busy with other stuff, and relegate dealing with reviews etc to less busy times of the week. Given that you seem to be doing the same, I think you can appreciate this perspective.

Of course, this can cause a delay. However, I think that, in your example, the proper accounting is that the reviewer is responsible for 1 week of the delay, while the editor is responsible for the remainder (i.e., 3 weeks).

Piotr