Back to STOC stuff, continuing my past post on conflicts. While the theory of how to handle conflicts is one thing, the practice is another. How did things work in practice?
For those who say it's mildly annoying, I admit I agree. People having to leave the room is obviously less appealing than people not having to leave the room -- I'm not going to argue that. But really, I think the annoyance is minor, at worst. In most cases, zero to two people had to leave the room. People seemed to pay attention and get up and leave when they had to. (Arguably, people pay more attention to what's coming up when there are conflicts, which is actually a plus.) We could have been better about remembering to call people back in promptly. With practice, I don't think that would be a big deal either.
I didn't feel that we lost out on needed expertise because of the policy, but again, the implementation was more flexible than in networking conferences I've been involved in.
One PC member insisted on leaving the room for a discussion and vote because of a conflict, but we insisted they stay to answer our questions first!
Overall, I did not find it overly disruptive. And I think such a policy greatly reduces or even avoids worst-case scenarios where a PC member -- intentionally or unintentionally -- biases the outcome of a paper where they have a conflict. I've been on many PCs over the years, and I've seen it happen more than once. No, I'm not saying such happenings are rampant -- I think they're rare -- but I think they could be rarer still. To me, the annoyance involved is a small price to pay for that. Also, I think even if you disregard the possibility of someone intentionally pushing a paper where they have a conflict (which, actually, does happen), on the whole people underestimate the power of unconscious and unintentional bias.
Paul Beame (in comments to the previous post) says that people should be allowed to stay in the room, but remain silent and declare conflicts as they arise; and similarly, that software shouldn't block people from seeing reviews/discussions on conflict papers. I think he offers a consistent alternative, but I (as he knows :) ) disagree. His approach seems designed to avoid any possible manipulation of decisions by PC members that may have a conflict, which of course is all to the good. In practice, in my experience, that silence rule is not generally maintained. (For a variety of reasons, many of them laced with good intentions. It's hard for experts to stay quiet, particularly when a colleague/student/etc. is involved.) He ignores the issue that simply having the person in the room may stifle some discussion in the PC meeting. (Junior people can be and are often intimidated, to various degrees, by senior people in such a setting. We can argue about whether they should be, but in practice, they can be.) And finally (and most importantly), he's dismissing the issue of information in reviews or discussions getting back to the authors. (Yes, discussions are confidential -- as are the parts of the reviews labelled "to the committee" instead of "to the author" -- but as we've read in the previous comments, there can be a fair amount of leaking of information after the fact.) My thought is that when you say you have a conflict, that precisely means you SHOULD NOT see the reviews or hear the discussion for the paper -- the networking standard -- unless there's an important overriding reason for you to do so. That prevents you from intentionally or unintentionally leaking supposedly confidential information.
And to be clear, I think the "we need the expertise" argument is overhyped. In cases where it's needed, I would agree exceptions need to be made. In most cases, it's not needed (there are other people in the room capable of making the judgment), but people don't want to be left out of the decision process where they are also an expert. There's a difference.
I have no regrets about implementing things this way. Others disagree. Overall, I think it's a topic worthy of more community discussion. In particular, the use of conflicted subreviewers that also came up in comments on the previous post really deserves some attention. I think a more consistent and thought-through standard should apply. Our community may have it's own standard, but as a community I think more discussion including ackowledgment and understanding of the pros and cons of the various possible approaches would be useful.
Friday, February 06, 2009
STOC Links
Here they are.
Titles and authors: http://www.umiacs.umd.edu/conferences/stoc2009/titles.shtml
Abstracts also (html): http://www.umiacs.umd.edu/conferences/stoc2009/abstracts.shtml
Abstracts (raw text): http://www.umiacs.umd.edu/conferences/stoc2009/abstracts-raw.txt
Titles and authors: http://www.umiacs.umd.edu/
Abstracts also (html): http://www.umiacs.umd.edu/
Abstracts (raw text): http://www.umiacs.umd.edu/
Thursday, February 05, 2009
Enrollments, 2009
CS-124, Algorithms and Data Structures
2008 : 36 students
2009: 83 students
That 83 will probably deviate by a few after the last add/drops shake out, but it's a somewhat surprising doubling+ of the class size. That's what we get for introducing a new, dynamic instructor for the intro CS course last year, who seems to have boosted CS enrollments across the board (taking effect this year). I was expecting 60-65.
By the way, 2008 was the lowest enrollment I've had, by far, although the previous few years had been in the 40's. My biggest year, back in the boom, was 90.
I'm double-teaching this semester, and my graduate "Algorithms at the End of the Wire" class has 23, which is pretty standard. (I think the biggest year it had more than 40... and given this enrollment boom, that's what I'll expect NEXT time I offer it.) It might lose one or two but that should hold steady. I admit I'm pleased to look around the room and see a lot of the top undergraduates I know taking the course (which was one of the reasons I arranged to double-teach this semester; I wanted them in the class before they all go and graduate...)
2008 : 36 students
2009: 83 students
That 83 will probably deviate by a few after the last add/drops shake out, but it's a somewhat surprising doubling+ of the class size. That's what we get for introducing a new, dynamic instructor for the intro CS course last year, who seems to have boosted CS enrollments across the board (taking effect this year). I was expecting 60-65.
By the way, 2008 was the lowest enrollment I've had, by far, although the previous few years had been in the 40's. My biggest year, back in the boom, was 90.
I'm double-teaching this semester, and my graduate "Algorithms at the End of the Wire" class has 23, which is pretty standard. (I think the biggest year it had more than 40... and given this enrollment boom, that's what I'll expect NEXT time I offer it.) It might lose one or two but that should hold steady. I admit I'm pleased to look around the room and see a lot of the top undergraduates I know taking the course (which was one of the reasons I arranged to double-teach this semester; I wanted them in the class before they all go and graduate...)
Wednesday, February 04, 2009
Rule Stupidity
Note: Update 1 Below
I thought I'd give my readers a STOC-break and discuss beginning-of-the-semester frustration with a stupid Harvard rule.
Setup: Math 123 (2nd semester algebra) and CS 124 (my algorithms/data structures course) are at the same time this year. Both are important prereq courses for many other courses. There are a small number of joint Math/CS majors, who, naturally, really want to take both. (They're both only offered spring semester.)
Now, since my class is recorded and put online for the Harvard Extension School -- usually now within 24 hours -- and Harvard students are given access to all these videos, this wouldn't seem to be a problem. The students take both classes and just watch the videos for mine. Is it ideal? Of course not. But it's better than making them wait a year to take one of these important classes. I do have Teaching Assistants and sections and office hours and TA office hours and all class notes online and such so they do have resources besides the recording. There are other details -- arranging final exams, for instance -- but I (and the math professor) have expressed our willingness to deal with that.
The administration says no.
Apparently, the rules on the books for "simultaneous enrollments" (taking two classes at the same time) won't allow for recorded lectures, and despite my plea for an exception in this clearly suitable case, the rule is, apparently, the rule.
Except (as I've also informed the rule-enforcing-body) it's really not. Because in this case the students will simply sign up for an independent study with me, CS91r, and have the content of their independent study be the class CS 124. This is less than ideal for everybody -- the students don't get proper credit for the course they're taking, I don't get the TA-resources associated with the students, and there's more paperwork for everyone. But it seems clear to me (and, apparently, the students, who asked me to do this) that this is in the students' best interest, so that's what I'll do.
But still, the rule-enforcing body says, the rule's the rule, and they'll enforce it.
I'm sending them all a link to this post so they can defend themselves if they choose. Feel free to comment with your own stories of stupid rule enforcement at your university...
Update 1:
I'm happy to say that I've had further conversations with the powers-that-be, and I think we've come up with a mutually acceptable interpretation of the rules that will allow the students to register for the class. I have to thank the powers-that-be for working with me, and on reflection, it undoubtedly wasn't because I got angry about it, but simply because I was consistent in asking if there was some kind of solution. It's still got to pass through a vote of a committee, but I'm optimistic, and I certainly feel they listened to my concerns -- which doesn't always happen with a bureaucracy!
I thought I'd give my readers a STOC-break and discuss beginning-of-the-semester frustration with a stupid Harvard rule.
Setup: Math 123 (2nd semester algebra) and CS 124 (my algorithms/data structures course) are at the same time this year. Both are important prereq courses for many other courses. There are a small number of joint Math/CS majors, who, naturally, really want to take both. (They're both only offered spring semester.)
Now, since my class is recorded and put online for the Harvard Extension School -- usually now within 24 hours -- and Harvard students are given access to all these videos, this wouldn't seem to be a problem. The students take both classes and just watch the videos for mine. Is it ideal? Of course not. But it's better than making them wait a year to take one of these important classes. I do have Teaching Assistants and sections and office hours and TA office hours and all class notes online and such so they do have resources besides the recording. There are other details -- arranging final exams, for instance -- but I (and the math professor) have expressed our willingness to deal with that.
The administration says no.
Apparently, the rules on the books for "simultaneous enrollments" (taking two classes at the same time) won't allow for recorded lectures, and despite my plea for an exception in this clearly suitable case, the rule is, apparently, the rule.
Except (as I've also informed the rule-enforcing-body) it's really not. Because in this case the students will simply sign up for an independent study with me, CS91r, and have the content of their independent study be the class CS 124. This is less than ideal for everybody -- the students don't get proper credit for the course they're taking, I don't get the TA-resources associated with the students, and there's more paperwork for everyone. But it seems clear to me (and, apparently, the students, who asked me to do this) that this is in the students' best interest, so that's what I'll do.
But still, the rule-enforcing body says, the rule's the rule, and they'll enforce it.
I'm sending them all a link to this post so they can defend themselves if they choose. Feel free to comment with your own stories of stupid rule enforcement at your university...
Update 1:
I'm happy to say that I've had further conversations with the powers-that-be, and I think we've come up with a mutually acceptable interpretation of the rules that will allow the students to register for the class. I have to thank the powers-that-be for working with me, and on reflection, it undoubtedly wasn't because I got angry about it, but simply because I was consistent in asking if there was some kind of solution. It's still got to pass through a vote of a committee, but I'm optimistic, and I certainly feel they listened to my concerns -- which doesn't always happen with a bureaucracy!
Tuesday, February 03, 2009
STOC PC Meeting : Part III: (Conflicts of Interest)
I introduced another change for the PC meeting for STOC this year. My experience is that theory conferences are somewhat "loose" about conflicts of interest. What I mean is that while PC members can't submit papers, and it's expected that you don't review papers of current (or recent) students/advisors, other than that conflicts of interest aren't usually treated as a big deal. As a contrast, it's standard for the networking PCs I've been on to be at least as rigorous as an NSF panel; for example, if a paper has ANYONE at your institution as an author, you have no electronic access to the reviews and you're expected to leave the room during the discussion (and similarly for other "standard" conflicts of interest, including any collaborators from say the last two years). In my experience, on theory PCs, people don't leave the room just because they have a conflict. It might be expected they'd not comment, although it's not always clear to me this expectation is followed.
Indeed, I remember my first SIGCOMM PC -- the first hour or two I kept wondering why people kept coming and going in and out of the room every time a new paper was discussed. It seemed like a lot of people were off to get coffee when they weren't a reviewer on the paper. Someone explained to me that these were the people with conflicts, and I was shocked at what was considered a conflict. (I figure anyone from UCSD or Microsoft currently has to leave the room for at least 1/4 of the papers on any networking PC.)
Interestingly, when I mention this difference in "style" to people, networking people seemed shocked (and maybe a little horrified) by how theory PCs handle conflicts, and theory people seem shocked by how networking PCs handle conflicts.
[Aside: it's a bit interesting to think about further this in light of the comments on my previous post on sending scores, where many people note that many authors can get "inside information" on their paper from PC members after the fact, even though the meeting is supposed to be "confidential". This post won't cover that further -- it was actually written before the other post!]
It's a bit hard to imagine utilizing such a strict conflict of interest polity for a theory PC. The community's a bit smaller, we're highly collaborative, and the papers can be so specialized that if you stick hard and fast to conflict rules you might not have enough PC members who are really suitable to judge a specific paper. Another negative is that it does cost time -- there's a switching cost when people enter and leave the room.
But I asked the PC for a straw vote, and a large majority of those with an opinion thought it was a good idea to have people with conflicts of interest leave the room, primarily for the obvious reason that it's a lot easier to have an open discussion when you're not worried what some people might hear. (Not surprisingly, younger PC members on average seemed to think this was a bigger concern.) I should be clear, however, that this idea was quite controversial; many PC members expressed a very clear and vocal dislike for a policy that had people leaving the room regularly. Indeed, it was definitely the most controversial decision I made as the PC chair.
In order to be practical, I made it clear that I understood there would have to be exceptions, as needed, to deal with special cases, since I didn't really plan for this policy from the beginning (including when making paper assignments, although I did ask PC members to mark conflicts with the software when ranking papers they wanted -- this did not seem to be treated uniformly and universally by the PC, again because it's probably not standard in theory conferences). But the policy I planned to implement was essentially the following: if you had any standard conflict on a paper, and there wasn't a very good overriding reason for you to be in the room (e.g, you were an original reviewer, or you had a specific expertise the committee needed to evaluate the paper), you should leave.
In the next post, I'll talk about how that worked.
But before that, I think this is a matter the community (or, to the point, conference steering committees) should address. My personal take over many, many PCs is that while the strict conflict interpretation used in networking conferences may not be completely workable for a STOC/FOCS type conference, if I had to choose, it's clearly better than the "We'll assume everybody will behave properly" approach taken at most theory conferences. (Many theory conferences take a strict line on PC members not submitting, but then do essentially nothing about other possible conflicts -- that just doesn't seem right.) I think something in between these two extremes are possible, where exceptions based on needed expertise is possible, and that's what I tried to implement, but it could use some more careful thinking through. What do you all think?
Indeed, I remember my first SIGCOMM PC -- the first hour or two I kept wondering why people kept coming and going in and out of the room every time a new paper was discussed. It seemed like a lot of people were off to get coffee when they weren't a reviewer on the paper. Someone explained to me that these were the people with conflicts, and I was shocked at what was considered a conflict. (I figure anyone from UCSD or Microsoft currently has to leave the room for at least 1/4 of the papers on any networking PC.)
Interestingly, when I mention this difference in "style" to people, networking people seemed shocked (and maybe a little horrified) by how theory PCs handle conflicts, and theory people seem shocked by how networking PCs handle conflicts.
[Aside: it's a bit interesting to think about further this in light of the comments on my previous post on sending scores, where many people note that many authors can get "inside information" on their paper from PC members after the fact, even though the meeting is supposed to be "confidential". This post won't cover that further -- it was actually written before the other post!]
It's a bit hard to imagine utilizing such a strict conflict of interest polity for a theory PC. The community's a bit smaller, we're highly collaborative, and the papers can be so specialized that if you stick hard and fast to conflict rules you might not have enough PC members who are really suitable to judge a specific paper. Another negative is that it does cost time -- there's a switching cost when people enter and leave the room.
But I asked the PC for a straw vote, and a large majority of those with an opinion thought it was a good idea to have people with conflicts of interest leave the room, primarily for the obvious reason that it's a lot easier to have an open discussion when you're not worried what some people might hear. (Not surprisingly, younger PC members on average seemed to think this was a bigger concern.) I should be clear, however, that this idea was quite controversial; many PC members expressed a very clear and vocal dislike for a policy that had people leaving the room regularly. Indeed, it was definitely the most controversial decision I made as the PC chair.
In order to be practical, I made it clear that I understood there would have to be exceptions, as needed, to deal with special cases, since I didn't really plan for this policy from the beginning (including when making paper assignments, although I did ask PC members to mark conflicts with the software when ranking papers they wanted -- this did not seem to be treated uniformly and universally by the PC, again because it's probably not standard in theory conferences). But the policy I planned to implement was essentially the following: if you had any standard conflict on a paper, and there wasn't a very good overriding reason for you to be in the room (e.g, you were an original reviewer, or you had a specific expertise the committee needed to evaluate the paper), you should leave.
In the next post, I'll talk about how that worked.
But before that, I think this is a matter the community (or, to the point, conference steering committees) should address. My personal take over many, many PCs is that while the strict conflict interpretation used in networking conferences may not be completely workable for a STOC/FOCS type conference, if I had to choose, it's clearly better than the "We'll assume everybody will behave properly" approach taken at most theory conferences. (Many theory conferences take a strict line on PC members not submitting, but then do essentially nothing about other possible conflicts -- that just doesn't seem right.) I think something in between these two extremes are possible, where exceptions based on needed expertise is possible, and that's what I tried to implement, but it could use some more careful thinking through. What do you all think?
Monday, February 02, 2009
STOC PC Meeting : Interlude
First, I'd like to remind everybody that the day before STOC this year, instead of tutorials or other activities, we'll be having a special event for Les Valiant's 60th birthday on Saturday, May 30th. There will be talks by a number of people to commemorate his tremendous body of work. The organizers are currently trying to finalize arrangements so that the event is free to everyone, but if needed there might be a nominal registration fee to pay for coffee and such. Confirmed speakers include:
Jin-Yi Cai
Steve Cook
Vitaly Feldman
Mark Jerrum
Mike Kearns
Michael Rabin
Rocco Servedio
Paul Valiant
Vijay Vazirani
Avi Wigderson
We hope you will plan on attending.
Second, I sent out the accept/reject notices. Comments coming later this week.
Jin-Yi Cai
Steve Cook
Vitaly Feldman
Mark Jerrum
Mike Kearns
Michael Rabin
Rocco Servedio
Paul Valiant
Vijay Vazirani
Avi Wigderson
We hope you will plan on attending.
Second, I sent out the accept/reject notices. Comments coming later this week.
Sunday, February 01, 2009
STOC PC Meeting : Part II
For this PC, I asked reviewers to use a 5 point scale, corresponding to
Overall, I think it worked well. One plus it that I think the scale makes it very easy to find the bottom half of the papers (easy rejects) and top 10-15% of the papers (easy accepts), so that less time needs to be spent discussing those papers.
On the other hand, on day 2, we were left with a bunch of papers with scores with about a 3 average. This makes sense -- since we accepted about 25% of the papers, papers with about a 3 average were, by definition, borderline. In short, a grade of 3 could mean, "I like the paper but it's a reject" or "I like the paper and it's an accept."
One solution might be to tweak those percentages (an experiment worth trying) to better match the acceptance boundary. But, at the end of the day, I think the fact of it is that borderline papers are hard -- that's why we still have face-to-face PC meetings. No matter what voting system you use, these papers are the hardest to deal with. When you get down to these papers, the real question is, "Do you want to accept the paper or not?" I think a mechanism in the review software to allow a second round of voting -- corresponding to the question, "Conditioned on this being one of the X papers left we have to decide on, do you think we should accept or reject?" would be useful and would have saved us some time. In practice, we just did that verbally (approximately) in the meeting (as part of the discussion).
I think there are other advantages of this 5 point scale. When a PC member isn't following the scale -- say assigning much less than 1/2 of their papers scores of 1, or much more than 20% scores of 4 and 5 -- it's essentially immediately apparent to everyone. That's more transparent than the 10 point scale. (One can always use software that "re-calibrates" individual's scores to some sort of baseline -- that also works, but I think is much less transparent.)
To me it's just clear the 5-point scale approach must be better. At the end of the day, we have to make a binary decision on each paper. This scale gets us most of the way there, while giving enough room to distinguish the best papers and papers that need more discussion. I would use it again as a chair, and I prefer it as a PC member as well.
There's one downside to this scale -- which I'd appreciate comments on. Do we send the scores with the reviews, or not? It can be disheartening to get back scores of 1. On the other hand, it's always annoying when your paper is rejected, and I think scores provide useful feedback to the authors. (If you got all 1's and 2's, perhaps you should consider a conference other than FOCS for re-submission.) Several PC members said we shouldn't send the scores with the comments. I think we should -- of course, I'm used to getting scores of this form back from networking conferences. What do you think?
- 1: Bottom 1/2 of submissions.
- 2: Top 1/2 but not top 1/3 of submissions.
- 3: Top 1/3 but not top 1/5 of submissions.
- 4: Top 1/5 but not top 1/10 of submissions.
- 5: Top 1/10 of submissions.
Overall, I think it worked well. One plus it that I think the scale makes it very easy to find the bottom half of the papers (easy rejects) and top 10-15% of the papers (easy accepts), so that less time needs to be spent discussing those papers.
On the other hand, on day 2, we were left with a bunch of papers with scores with about a 3 average. This makes sense -- since we accepted about 25% of the papers, papers with about a 3 average were, by definition, borderline. In short, a grade of 3 could mean, "I like the paper but it's a reject" or "I like the paper and it's an accept."
One solution might be to tweak those percentages (an experiment worth trying) to better match the acceptance boundary. But, at the end of the day, I think the fact of it is that borderline papers are hard -- that's why we still have face-to-face PC meetings. No matter what voting system you use, these papers are the hardest to deal with. When you get down to these papers, the real question is, "Do you want to accept the paper or not?" I think a mechanism in the review software to allow a second round of voting -- corresponding to the question, "Conditioned on this being one of the X papers left we have to decide on, do you think we should accept or reject?" would be useful and would have saved us some time. In practice, we just did that verbally (approximately) in the meeting (as part of the discussion).
I think there are other advantages of this 5 point scale. When a PC member isn't following the scale -- say assigning much less than 1/2 of their papers scores of 1, or much more than 20% scores of 4 and 5 -- it's essentially immediately apparent to everyone. That's more transparent than the 10 point scale. (One can always use software that "re-calibrates" individual's scores to some sort of baseline -- that also works, but I think is much less transparent.)
To me it's just clear the 5-point scale approach must be better. At the end of the day, we have to make a binary decision on each paper. This scale gets us most of the way there, while giving enough room to distinguish the best papers and papers that need more discussion. I would use it again as a chair, and I prefer it as a PC member as well.
There's one downside to this scale -- which I'd appreciate comments on. Do we send the scores with the reviews, or not? It can be disheartening to get back scores of 1. On the other hand, it's always annoying when your paper is rejected, and I think scores provide useful feedback to the authors. (If you got all 1's and 2's, perhaps you should consider a conference other than FOCS for re-submission.) Several PC members said we shouldn't send the scores with the comments. I think we should -- of course, I'm used to getting scores of this form back from networking conferences. What do you think?
Friday, January 30, 2009
STOC PC Meeting : Part I
Over the next week, I'll present some thoughts related to the STOC PC meeting, which wrapped up earlier today. (Please don't send mail to ask about your paper(s). Messages will be sent out sometime next week.)
Today's post will be about basic logistics, which you wouldn't normally think about, unless you suddenly found yourself running the meeting. We started with roughly 150-160 papers left to consider. At 5 minutes on average per paper, you're already at about 13 hours. And if you slip to 6 minutes on average per paper, well, now you're at 15 to 16 hours, which when you consider overhead time (computer setup, lunch, the occasional rest break), you see the difference between that 5 and 6 sharply when you know people have flights to catch at the end of day 2. I think the hardest part of the meeting, from my end, was simply time management.
We started at 9 am on day 1, with plans to break at 6. It became clear that we were falling behind schedule, so we changed the dinner reservation and kept going until 7. We still seemed behind -- and with most of the easier accept and reject decisions behind us -- so we agreed to meet at 8:30 am on day 2. The mumblings and rumblings over dinner of "We're never going to finish on time" were worrisome, and I stayed up late looking over and organizing the remaining papers, so we could go through a bunch very quickly on day 2. I was definitely aided in this goal by the rest of the committee, who had clearly also done additional prepping post-dinner and were ready when they came in the door. In the first hour we made up our deficit in the schedule, and finished our decisions around 2:30, actually leaving us a comfortable amount of time to talk about other organizing issues before people started heading to the airport for their flights around 3.
While it all ended up OK, perhaps I could have done better day 1.
Also, the lesson for future PC members -- be ready to talk when your papers come up, and be prepared to make your points quickly. The seconds add up! I thank the committee for both putting in the extra long hours and for being well prepared, so we could finish on time. And everyone who submitted papers should be thanking them as well.
Another piece of "chair advice" I was told that I can't over-emphasize enough -- have someone whose job it is take notes and record the status of papers as you go. During the meeting I vaguely asked one of the PC members to keep track of things. At the end of the first night, trying to prepare for day 2, I would have been lost without his guide as to where everything stood. As chair (particularly as a single chair -- we should have co-chairs...) I found it difficult to keep things running, enter things into the system, and track things so I could go back and really see where everything was; the notetaker essentially did the tracking job for me, making the whole meeting run much more smoothly, and allowing me to get some sleep before the second day.
And speaking of sleep, I'm very tired, and need at least a day without thinking of anything related to STOC.
Today's post will be about basic logistics, which you wouldn't normally think about, unless you suddenly found yourself running the meeting. We started with roughly 150-160 papers left to consider. At 5 minutes on average per paper, you're already at about 13 hours. And if you slip to 6 minutes on average per paper, well, now you're at 15 to 16 hours, which when you consider overhead time (computer setup, lunch, the occasional rest break), you see the difference between that 5 and 6 sharply when you know people have flights to catch at the end of day 2. I think the hardest part of the meeting, from my end, was simply time management.
We started at 9 am on day 1, with plans to break at 6. It became clear that we were falling behind schedule, so we changed the dinner reservation and kept going until 7. We still seemed behind -- and with most of the easier accept and reject decisions behind us -- so we agreed to meet at 8:30 am on day 2. The mumblings and rumblings over dinner of "We're never going to finish on time" were worrisome, and I stayed up late looking over and organizing the remaining papers, so we could go through a bunch very quickly on day 2. I was definitely aided in this goal by the rest of the committee, who had clearly also done additional prepping post-dinner and were ready when they came in the door. In the first hour we made up our deficit in the schedule, and finished our decisions around 2:30, actually leaving us a comfortable amount of time to talk about other organizing issues before people started heading to the airport for their flights around 3.
While it all ended up OK, perhaps I could have done better day 1.
Also, the lesson for future PC members -- be ready to talk when your papers come up, and be prepared to make your points quickly. The seconds add up! I thank the committee for both putting in the extra long hours and for being well prepared, so we could finish on time. And everyone who submitted papers should be thanking them as well.
Another piece of "chair advice" I was told that I can't over-emphasize enough -- have someone whose job it is take notes and record the status of papers as you go. During the meeting I vaguely asked one of the PC members to keep track of things. At the end of the first night, trying to prepare for day 2, I would have been lost without his guide as to where everything stood. As chair (particularly as a single chair -- we should have co-chairs...) I found it difficult to keep things running, enter things into the system, and track things so I could go back and really see where everything was; the notetaker essentially did the tracking job for me, making the whole meeting run much more smoothly, and allowing me to get some sleep before the second day.
And speaking of sleep, I'm very tired, and need at least a day without thinking of anything related to STOC.
Wednesday, January 28, 2009
Graduate Student Folders Part II
We're getting to the point where we have to make our decisions on graduate student admissions. In the theory group at Harvard we got a sizable number of folders that we've cut down to about 25. We'll probably cutting those down to about 10 acceptances.
There's clearly an abundance of good people, who could become solid TCS researchers. And as we went over in my last post on this, it's difficult to choose based on an application who will become successful.
My guilt in having to choose from among these many talented people is assuaged mostly by my concern for the people who we do admit to graduate school, especially in TCS. Where will the jobs be when these people graduate? I've heard the arguments that CS is still expanding, well maybe not in America but elsewhere in the world, and that PhDs can find other work besides being a professor. I've seen that most of the other students I knew as a graduate student in Berkeley have ended up with good positions -- mostly professors -- albeit some after a number of years of postdocs. But still, at the end of the day, when I do the math, it just doesn't add up. I expect to be in my position for at least 30 years (barring the current curse of Harvard CS faculty to be moved into administration); how many professors (or scientists at research labs) can I produce over my lifetime?
The financial meltdown has only raised my concerns that in CS generally (and theory in particular) we've been living through a bubble, and we'll all be shocked when that bubble pops, though we shouldn't be.
I'm not so entirely pessimistic -- CS PhDs, for as far as I can see, should have no trouble getting pretty good jobs. But how many will get the jobs they think they're getting the PhD for? (Note the internal mental bias here -- professors remember mostly the students who did go on to get faculty/research jobs...) We'll see.
There's clearly an abundance of good people, who could become solid TCS researchers. And as we went over in my last post on this, it's difficult to choose based on an application who will become successful.
My guilt in having to choose from among these many talented people is assuaged mostly by my concern for the people who we do admit to graduate school, especially in TCS. Where will the jobs be when these people graduate? I've heard the arguments that CS is still expanding, well maybe not in America but elsewhere in the world, and that PhDs can find other work besides being a professor. I've seen that most of the other students I knew as a graduate student in Berkeley have ended up with good positions -- mostly professors -- albeit some after a number of years of postdocs. But still, at the end of the day, when I do the math, it just doesn't add up. I expect to be in my position for at least 30 years (barring the current curse of Harvard CS faculty to be moved into administration); how many professors (or scientists at research labs) can I produce over my lifetime?
The financial meltdown has only raised my concerns that in CS generally (and theory in particular) we've been living through a bubble, and we'll all be shocked when that bubble pops, though we shouldn't be.
I'm not so entirely pessimistic -- CS PhDs, for as far as I can see, should have no trouble getting pretty good jobs. But how many will get the jobs they think they're getting the PhD for? (Note the internal mental bias here -- professors remember mostly the students who did go on to get faculty/research jobs...) We'll see.
Tuesday, January 27, 2009
SIGCOMM PC starting... STOC PC ending...
I guess there's another week until the "official deadline" for SIGCOMM, but the deadline for registration of abstracts/titles has already happened, and I was asked to "bid" on papers I was interested in. It's hard to tell from the limited information, but it seems like there are plenty of interesting papers. Mostly I was looking for papers covering algorithmic engineering or just plain algorithmic problems, the econ/CS interface, and coding (network coding seems to be getting bigger), but there were certainly papers outside those areas that caught my eye. Out of the 300-400 submissions, there were more than 50 that looked interesting enough for me to request them; I was only asked to choose at least 30 to start.
Of course, since the whole process is double-blind, I don't know who submitted the papers, and hopefully the authors won't be able to figure out which reviews I end up writing.
This would all be great fun, but there's of course STOC to finish up. We've rejected about half the papers so far (we're still in the pre-meeting stage). Of course, that's the "easy" part. Now things get harder.
Of course, since the whole process is double-blind, I don't know who submitted the papers, and hopefully the authors won't be able to figure out which reviews I end up writing.
This would all be great fun, but there's of course STOC to finish up. We've rejected about half the papers so far (we're still in the pre-meeting stage). Of course, that's the "easy" part. Now things get harder.
Sunday, January 25, 2009
More on Algorithms and Implementation
In the comments on my last post on algorithms and implementation, someone asked where to publish when you sit in the middle, and how can a graduate student succeed by being in the middle. The comments seemed to suggest that publishing was hard and succeeding as a graduate student this way was largely impossible.
Sadly, and disconcertingly, I find myself largely agreeing. I think the theory community has developed to the point where, almost universally, if you want to earn a reputation as a top theorist in graduate school, you have to publish FOCS/STOC/SODA papers. And really this can't be done by focusing on practical issues. More practical papers can be published in other venues -- WWW, INFOCOM, even SPAA and PODC -- but to the theory community these papers don't carry as much weight. (Note: yes, obviously there are RARE exceptions.)
Strangely, I still think there's a reasonable job market for practically-minded theorists, both in research labs, and in smaller universities, where they'd much prefer a theorist who would interact with other groups (like networking or AI) because they really aren't big enough to have theorists who don't have "outside interests". And I think evidence of being able to work with practitioners helps most any theorist looking for a job -- you just can't build your reputation on it. But it can be hard to get that balance of theory and practice "right" as a graduate student.
I see a fair amount of people who are theoretical but practically minded simply going into other fields, either as a graduate student or after. You can be a "theoretical" networking person or AI person and succeed, and a number of graduate students who could be theorists figure out they'll be better off in these areas. You can start out as a FOCS/STOC/SODA person and then branch out to other areas. Go look at Karger or Kleinberg's DBLP pages and see how much more they've been doing in the recent past outside the confines of FOCS/STOC/SODA compared to earlier in their careers. (Or look at John Byers and Soumen Chakrabarti, who worked primarily in theory as graduate students, but switched to other areas.)
Perhaps this would make a good conference panel someday. (Offhand I'd recommend getting Mikkel Thorup and Muthu Muthukrishnan to participate -- they seem to me like role models for people who want to sit in the middle -- though I can think of many others as well...) Heck, come to think of it, I'm the STOC chair, maybe it would make a good panel coming up real soon. Please let me know if you'd like a panel on this issue -- something like "How to Succeed doing Practical Theory?" -- at STOC.
Sadly, and disconcertingly, I find myself largely agreeing. I think the theory community has developed to the point where, almost universally, if you want to earn a reputation as a top theorist in graduate school, you have to publish FOCS/STOC/SODA papers. And really this can't be done by focusing on practical issues. More practical papers can be published in other venues -- WWW, INFOCOM, even SPAA and PODC -- but to the theory community these papers don't carry as much weight. (Note: yes, obviously there are RARE exceptions.)
Strangely, I still think there's a reasonable job market for practically-minded theorists, both in research labs, and in smaller universities, where they'd much prefer a theorist who would interact with other groups (like networking or AI) because they really aren't big enough to have theorists who don't have "outside interests". And I think evidence of being able to work with practitioners helps most any theorist looking for a job -- you just can't build your reputation on it. But it can be hard to get that balance of theory and practice "right" as a graduate student.
I see a fair amount of people who are theoretical but practically minded simply going into other fields, either as a graduate student or after. You can be a "theoretical" networking person or AI person and succeed, and a number of graduate students who could be theorists figure out they'll be better off in these areas. You can start out as a FOCS/STOC/SODA person and then branch out to other areas. Go look at Karger or Kleinberg's DBLP pages and see how much more they've been doing in the recent past outside the confines of FOCS/STOC/SODA compared to earlier in their careers. (Or look at John Byers and Soumen Chakrabarti, who worked primarily in theory as graduate students, but switched to other areas.)
Perhaps this would make a good conference panel someday. (Offhand I'd recommend getting Mikkel Thorup and Muthu Muthukrishnan to participate -- they seem to me like role models for people who want to sit in the middle -- though I can think of many others as well...) Heck, come to think of it, I'm the STOC chair, maybe it would make a good panel coming up real soon. Please let me know if you'd like a panel on this issue -- something like "How to Succeed doing Practical Theory?" -- at STOC.
Friday, January 23, 2009
Algorithms and Implementation
I was leafing (again) through George Varghese's text, Network Algorithmics
, and this caught my eye:
, and this caught my eye:In summary, the uncritical use of standard algorithms can miss implementation breakthroughs because of inappropriate measures (e.g., for packets filters such as BPF, the insertion of a new classifier can afford to take more time than search), inappropriate models (e.g., ignoring the effects of cache lines in software or parallelism in hardware), and inappropriate analysis (e.g., order-of-complexity results that hide constant factors crucial in ensuring wire speed forwarding).These words succinctly express feelings I commonly have. Standard algorithms theory often fails to live up to its promise in practice, untethered as it has become from applications. At the same time, a properly designed algorithm can yield immense breakthroughs in performance, so implementors cannot do without them. I believe we need more people willing to sit in the middle, but perhaps first we need more creative ways to bring theory and practice, or theorists and practitioners, together.
Thus another purpose of this book is to persuade implementors that insight into algorithms and the use of fundamental algorithmic techniques such as divide-and-conquer and randomization is important to master.
Thursday, January 22, 2009
Georgia Tech visit
I had the pleasure of visiting Georgia Tech Wednesday, giving a theory seminar, in this case my survey talk on Deletion Channels.
A fun day filled with conversations about research, kids, and other goings-on. Highlights include getting to catch up with my academic older sister Dana Randall and younger brother Eric Vigoda. And this trip, as it seems with every trip I've taken to Georgia Tech, my host Vijay Vazirani exceeded expectations with his choice of dinner location. (And by now, I have high expectations for meals in Atlanta!)
The department has really grown into a theory powerhouse (and has a very strong networking group as well). For some time it's been on my list of schools undergraduates should think about for graduate school (but might not have realized they should be thinking about). The visit was a strong reminder of why.
A fun day filled with conversations about research, kids, and other goings-on. Highlights include getting to catch up with my academic older sister Dana Randall and younger brother Eric Vigoda. And this trip, as it seems with every trip I've taken to Georgia Tech, my host Vijay Vazirani exceeded expectations with his choice of dinner location. (And by now, I have high expectations for meals in Atlanta!)
The department has really grown into a theory powerhouse (and has a very strong networking group as well). For some time it's been on my list of schools undergraduates should think about for graduate school (but might not have realized they should be thinking about). The visit was a strong reminder of why.
Saturday, January 17, 2009
Summer Funding
I should have blogged about this before, but didn't notice over break that Yale had "agreed to pay7.6 million to resolve allegations" regarding how it was handling research grants. My understanding is that this primarily related to summer funding -- the government apparently objects to professors saying they're working 100% on their research over the summer and then doing other stuff. (You know, pesky things like writing letters of recommendation or preparing for next semester's classes.) Anyone with more info, please speak up or link in the comments.
Now, of course, this all just boils down to accounting. The university (nominally) pays 9 months of salary, but strangely, we seem to be expected to also be working on our government-funded research during the academic year as well. One would think there might be some understanding that there's some appropriate blurring of time spent on various tasks. But Harvard, responsive as always when it sees another school getting fined, is changing its process to avoid the same sort of investigation. I don't quite get how this works, but apparently my June summer funding for '09 will be paid out over the spring semester, and the rest of my summer funding will be paid out over the fall semester. I'm assuming that when I eventually have to fill out forms accounting for my time they'll be modified so that research time during the year is counted some way as well.
As one colleague said to me, the new system is stupid, but so was the old system, and sometime down the road, some bureaucrat will decide this is wrong, and we'll switch to a different stupid system. What I find most odd about the new system is it seems at least part of the time I'm getting paid from a government grant for work I will only do in the future. That's OK with the government? I imagine if for whatever reason I later decided not to work over June, they'd pull the money back out of future paychecks. I also suppose this provides some understanding of why grant overhead rates, which I generally complain about, are so ridiculously high; if there's really this much paperwork (and computer accounting software setup) involved in grant management, maybe the overhead really is necessary.
Now, of course, this all just boils down to accounting. The university (nominally) pays 9 months of salary, but strangely, we seem to be expected to also be working on our government-funded research during the academic year as well. One would think there might be some understanding that there's some appropriate blurring of time spent on various tasks. But Harvard, responsive as always when it sees another school getting fined, is changing its process to avoid the same sort of investigation. I don't quite get how this works, but apparently my June summer funding for '09 will be paid out over the spring semester, and the rest of my summer funding will be paid out over the fall semester. I'm assuming that when I eventually have to fill out forms accounting for my time they'll be modified so that research time during the year is counted some way as well.
As one colleague said to me, the new system is stupid, but so was the old system, and sometime down the road, some bureaucrat will decide this is wrong, and we'll switch to a different stupid system. What I find most odd about the new system is it seems at least part of the time I'm getting paid from a government grant for work I will only do in the future. That's OK with the government? I imagine if for whatever reason I later decided not to work over June, they'd pull the money back out of future paychecks. I also suppose this provides some understanding of why grant overhead rates, which I generally complain about, are so ridiculously high; if there's really this much paperwork (and computer accounting software setup) involved in grant management, maybe the overhead really is necessary.
Wednesday, January 14, 2009
Graduate Student Folders
When I'm not looking over STOC papers the next few weeks, I'll also be looking over graduate student admissions folders. (At Harvard, we're small enough that the theory folks just split up all the folders and then meet to discuss them.) Even though I've been doing this for a while, I'd love to hear from all of you -- what should I -- or we as a community -- be looking for in selecting graduate students?
I think that on the theory side we tend to look for pure brainpower over other skills. Arguably moreso in theory than in other subjects, raw intelligence matters most. (I'm not saying I agree with that argument; I'm just saying it's an argument.) I think we also know that grades don't necessarily provide the right information we need to judge this, however, and so we look for evidence that one has the ability to do research in the form of actual research projects accomplished or letters from faculty members we know (and believe) who suggest that there's intellectual power and creativity there.
I do think most faculty do keep in mind the other skills that can make for a successful student. Communication skills, the ability to work well with others, a sense of what they want to accomplish, the mindset to deal with the one-step-forward-two-steps-back nature of research -- all of these also come into play in decision-making. These can be much harder to judge, however, unless they're clearly marked by their absence in some way.
I wonder if the competitive landscape for graduate slots at the top schools causes us to miss out on some promising people. Over at the FemaleScienceProfessor blog, she discusses why she'd rather take the motivated B student over a lot of A students for undergraduate research projects. I'm curious, having no evidence one way or another, do people think we're doing a good job getting B undergraduates who could become great researchers into the pipeline early on enough so they can construct a good application for graduate school? Do we accept enough of these students when they apply?
I think that on the theory side we tend to look for pure brainpower over other skills. Arguably moreso in theory than in other subjects, raw intelligence matters most. (I'm not saying I agree with that argument; I'm just saying it's an argument.) I think we also know that grades don't necessarily provide the right information we need to judge this, however, and so we look for evidence that one has the ability to do research in the form of actual research projects accomplished or letters from faculty members we know (and believe) who suggest that there's intellectual power and creativity there.
I do think most faculty do keep in mind the other skills that can make for a successful student. Communication skills, the ability to work well with others, a sense of what they want to accomplish, the mindset to deal with the one-step-forward-two-steps-back nature of research -- all of these also come into play in decision-making. These can be much harder to judge, however, unless they're clearly marked by their absence in some way.
I wonder if the competitive landscape for graduate slots at the top schools causes us to miss out on some promising people. Over at the FemaleScienceProfessor blog, she discusses why she'd rather take the motivated B student over a lot of A students for undergraduate research projects. I'm curious, having no evidence one way or another, do people think we're doing a good job getting B undergraduates who could become great researchers into the pipeline early on enough so they can construct a good application for graduate school? Do we accept enough of these students when they apply?
Monday, January 12, 2009
Deletion Channel Paper
There's a new paper with bounds on the capacity of the deletion channel on the arxiv by Fertonani and Duman. The main results are some new upper bounds which generally are better than the upper bounds found in a paper by Diggavi, Mitzenmacher, and Pfister, which was the first look at the problem in a long while.
The bounds achieved are very nice. They use a simpler approach than our earlier paper, essentially dividing the input or output into fixed sized blocks and numerically calculating the capacity per block. With clever manipulation, they can extend the bounds in various ways -- the nicest result (in my mind) being that, in the limit as the deletion probability d goes to 1, the capacity is upper bounded by 0.49(1-d). [Our previous result gave a bound of 0.79(1-d), and some of my work on lower bounds showed that the capacity is at least 0.11(1-d).]
It seems to me their work raises some interesting questions. One question is the following: at some point, they have to rely on numerical calculations to determine their upper bounds, using the Blahut-Arimoto algorithm to calculate the capacity of a fixed finite channel. Their approach is limited by this calculation; for instance, if one "chops" the input into blocks of 10 bits, then there are 1024 possible input sequences and 2048 possible output sequences for the deletion channel, and essentially they have to compute the capacity of this fixed channel in a "brute force" way, which requires keeping track of the probability that each input results in each output. Bigger blocks means better bounds but bigger calculations. (Technically one can find ways to avoid keeping the entire 1024 by 2048 probability matrix, since it's fairly sparse, but the calculations still grow big very fast.)
Is there any hope for an approximate version of the Blahut-Arimoto algorithm that would give provable upper bounds on the capacity but could take smaller time/space? It would seem there might be other uses for an approximate expectation-maximization procedure, although perhaps generally one wants such good approximations that there's no real benefit in practice. Any thoughts?
The bounds achieved are very nice. They use a simpler approach than our earlier paper, essentially dividing the input or output into fixed sized blocks and numerically calculating the capacity per block. With clever manipulation, they can extend the bounds in various ways -- the nicest result (in my mind) being that, in the limit as the deletion probability d goes to 1, the capacity is upper bounded by 0.49(1-d). [Our previous result gave a bound of 0.79(1-d), and some of my work on lower bounds showed that the capacity is at least 0.11(1-d).]
It seems to me their work raises some interesting questions. One question is the following: at some point, they have to rely on numerical calculations to determine their upper bounds, using the Blahut-Arimoto algorithm to calculate the capacity of a fixed finite channel. Their approach is limited by this calculation; for instance, if one "chops" the input into blocks of 10 bits, then there are 1024 possible input sequences and 2048 possible output sequences for the deletion channel, and essentially they have to compute the capacity of this fixed channel in a "brute force" way, which requires keeping track of the probability that each input results in each output. Bigger blocks means better bounds but bigger calculations. (Technically one can find ways to avoid keeping the entire 1024 by 2048 probability matrix, since it's fairly sparse, but the calculations still grow big very fast.)
Is there any hope for an approximate version of the Blahut-Arimoto algorithm that would give provable upper bounds on the capacity but could take smaller time/space? It would seem there might be other uses for an approximate expectation-maximization procedure, although perhaps generally one wants such good approximations that there's no real benefit in practice. Any thoughts?
Sunday, January 11, 2009
Co-Chairs
Although it's far from over, I'm already clear that one of my recommendations for the future of FOCS/STOC conferences is to start having co-chairs instead of a single chair.
The reasons for having co-chairs that I can see include:
1) Chairing involves a lot of administrative work, often bursts for short intervals. Being able to share that kind of work seems to have a huge upside.
2) Although I don't think it's come up for STOC yet, having a co-chair watch your back can help avoid silly mistakes.
3) Particularly for FOCS/STOC, having co-chairs from two different subareas (one from a more algorithmic side and one from the complexity side) would seem to offer better coverage.
Many conferences in other areas of comparable size to FOCS/STOC have co-chairs (e.g., NSDI, SIGCOMM that I know of) and in my experience as a PC member it's always gone well.
What are the possible downsides of co-chairs?
1) Whenever two people are co-in-charge, they have to work together well. I hope it wouldn't be too hard to find people who could capably co-chair for these conferences.
2) I imagine it can be hard finding people willing to chair; now you'd have to find 2! Although I actually think it might make it easier to get people to accept if they can co-chair -- it will seem like less work, and if one person is more experienced, you could have a somewhat less experienced person co-chair.
The reasons for having co-chairs that I can see include:
1) Chairing involves a lot of administrative work, often bursts for short intervals. Being able to share that kind of work seems to have a huge upside.
2) Although I don't think it's come up for STOC yet, having a co-chair watch your back can help avoid silly mistakes.
3) Particularly for FOCS/STOC, having co-chairs from two different subareas (one from a more algorithmic side and one from the complexity side) would seem to offer better coverage.
Many conferences in other areas of comparable size to FOCS/STOC have co-chairs (e.g., NSDI, SIGCOMM that I know of) and in my experience as a PC member it's always gone well.
What are the possible downsides of co-chairs?
1) Whenever two people are co-in-charge, they have to work together well. I hope it wouldn't be too hard to find people who could capably co-chair for these conferences.
2) I imagine it can be hard finding people willing to chair; now you'd have to find 2! Although I actually think it might make it easier to get people to accept if they can co-chair -- it will seem like less work, and if one person is more experienced, you could have a somewhat less experienced person co-chair.
Thursday, January 08, 2009
So, how's STOC going?
I suggested that I'd blog about my experiences as STOC PC chair, so I thought I'd give an update.
We're nearing the point where all the "first round" reviews are supposed to be in. All papers had 3 PC members assigned to them, and my goal was to have 3 reviews for each paper a few weeks before the PC meeting. So far, it seems to be going well; most PC members have been putting in reviews. A few who seem to be working with an offline scorecard haven't uploaded what they've completed -- which makes things a bit more difficult for me, naturally :( -- and I expect some papers won't have 3 reviews by the deadline for various reasons, the most likely being that a subreviewer hasn't finished. Subreviewers, please get things in! (And thank you.)
After that point, I hope to reject a slew of papers outright. Recall that I'm trying a new scoring scale. 1 = bottom 50%, 5 = top 10 %, other scores in between. Anything with 3 1's seems to be an easy reject. Arguably, if there's no score of 3 or above in the 3 reviews, a paper should be rejected. To be clear I won't make those decisions unilaterally, but will set up votes for the PC.
The other work at this point is finding more reviewers for papers with widely disparate scores (where the problem does not seem to be "this paper has a bug" that one reviewer noticed but others didn't, but rather a wide disagreement on the quality/importance of the result) or for papers that seem like they'll be on the borderline. Here I'm looking for people both on and off the PC -- so don't be too surprised if you get an e-mail asking for a quick review -- with the emphasis on trying to find relevant experts in the area of the paper.
How many papers will have both a 1 and a 5 score? (Or 2/5s, or 1/4s?) Seems like a statistic for the business meeting roundup. But so far, the answer seems to be more than you might think...
We're nearing the point where all the "first round" reviews are supposed to be in. All papers had 3 PC members assigned to them, and my goal was to have 3 reviews for each paper a few weeks before the PC meeting. So far, it seems to be going well; most PC members have been putting in reviews. A few who seem to be working with an offline scorecard haven't uploaded what they've completed -- which makes things a bit more difficult for me, naturally :( -- and I expect some papers won't have 3 reviews by the deadline for various reasons, the most likely being that a subreviewer hasn't finished. Subreviewers, please get things in! (And thank you.)
After that point, I hope to reject a slew of papers outright. Recall that I'm trying a new scoring scale. 1 = bottom 50%, 5 = top 10 %, other scores in between. Anything with 3 1's seems to be an easy reject. Arguably, if there's no score of 3 or above in the 3 reviews, a paper should be rejected. To be clear I won't make those decisions unilaterally, but will set up votes for the PC.
The other work at this point is finding more reviewers for papers with widely disparate scores (where the problem does not seem to be "this paper has a bug" that one reviewer noticed but others didn't, but rather a wide disagreement on the quality/importance of the result) or for papers that seem like they'll be on the borderline. Here I'm looking for people both on and off the PC -- so don't be too surprised if you get an e-mail asking for a quick review -- with the emphasis on trying to find relevant experts in the area of the paper.
How many papers will have both a 1 and a 5 score? (Or 2/5s, or 1/4s?) Seems like a statistic for the business meeting roundup. But so far, the answer seems to be more than you might think...
Tuesday, January 06, 2009
Preparing for Class
It's that time of the year where it's time to get my next class ready. I could use your help.
For various odd reasons, in the coming semester I'll be teaching both my undergraduate Algorithms (and Data Structures) class [it's not parenthesized like that in the course catalog, but it probably should be], and my graduate "big data" class, dubbed Algorithms at the End of the Wire. For the graduate class, I cover a variety of topics -- last time, the main areas were search engine algorithms, compression, stream-based algorithms and data structures, and coding (network coding and otherwise), and essentially each class the students are supposed to come in having read a couple of papers. One of the goals of the class is to present algorithmic theory and practice, so I emphasize trying to find practical papers that really make use of underlying theory, or theory papers that really try to impact how a practical problem is thought about. Another goal of the class is to introduce students to the ways of research, by having them read both ancient classics (like, say, Kleinberg's HITS paper) and more recent (e.g. the last year or two) papers.
I'd be interested in any suggestions for good papers for students to read -- particularly recent papers. Don't feel tied to the topics above -- I should be trying to keep up with good papers bridging the algorithm/theory divide in all areas!
For various odd reasons, in the coming semester I'll be teaching both my undergraduate Algorithms (and Data Structures) class [it's not parenthesized like that in the course catalog, but it probably should be], and my graduate "big data" class, dubbed Algorithms at the End of the Wire. For the graduate class, I cover a variety of topics -- last time, the main areas were search engine algorithms, compression, stream-based algorithms and data structures, and coding (network coding and otherwise), and essentially each class the students are supposed to come in having read a couple of papers. One of the goals of the class is to present algorithmic theory and practice, so I emphasize trying to find practical papers that really make use of underlying theory, or theory papers that really try to impact how a practical problem is thought about. Another goal of the class is to introduce students to the ways of research, by having them read both ancient classics (like, say, Kleinberg's HITS paper) and more recent (e.g. the last year or two) papers.
I'd be interested in any suggestions for good papers for students to read -- particularly recent papers. Don't feel tied to the topics above -- I should be trying to keep up with good papers bridging the algorithm/theory divide in all areas!
Friday, January 02, 2009
Consistency in Scoring
Having just finished serving on the NSDI PC committee, and being hard at work on the STOC committee, one issue that has been on my mind is consistency in scoring papers. Ideally, when submitting to a conference, it shouldn't matter WHO reads your paper; your score should be intrinsic to your work. Of course, we don't expect -- or even necessarily want -- the complete ideal; there will be differences of opinion that occur for natural reasons. Arguably, for example, "visionary" papers are hard to judge and tend to lead to differences of opinion. So you can take solace that if you get widely varying reviews and your paper is rejected, your paper was probably just too visionary. The more cynical among us might instead think that large variances in opinion have to do with how close the reviewer is to the subfield of the paper. In theory, my take is that referees within a subarea generally give higher marks to papers in that subarea. (At NSDI, I found myself with a different impression; if anything, I thought referees working in a subarea tended to be a bit harsher towards papers in that subarea!)
My impression historically has been that there's more variance on the theory side of the world than on the systems side, particularly for the big conferences like FOCS/STOC/SODA. I was amazed, on the whole, at the consistency of NSDI reviews for papers I was on. STOC reviews are coming in, and I'm looking forward to seeing how consistent things are there, but my early impression is that there's not the same consistency, and it's interesting to hypothesize why. [Perhaps, after it's all over, I'll attempt to gather some real quantitative data on the issue, to see if my impressions match reality.]
I'll throw out a straw man for people to discuss (and attack). In theory, all (reasonable) papers start with the same basic grounding -- you have proofs of some theorems. After that point, though, things get a bit hard to judge. How "nice" are your theorems, in terms of technique/mathematical beauty/novelty? How "big" is your improvement over previous work -- for example, is improving an O(log n) competitive ratio to O(log n/log log n) "important" enough? How practical is your solution (hard to say, when nobody actually implements algorithms or gives performance results...)? A lot of this seems subjective, so naturally there are differing opinions.
In networking or other more practical fields, the same issues -- technique/beauty/novelty, quantity of improvement, practicality -- all come into play. But there's a lot more focus on the bottom line of "Could this lead to a substantial improvement for an important real-world problem." Perhaps this consistency in the criterion for what is a good paper leads to more consistency in reviews?
My impression historically has been that there's more variance on the theory side of the world than on the systems side, particularly for the big conferences like FOCS/STOC/SODA. I was amazed, on the whole, at the consistency of NSDI reviews for papers I was on. STOC reviews are coming in, and I'm looking forward to seeing how consistent things are there, but my early impression is that there's not the same consistency, and it's interesting to hypothesize why. [Perhaps, after it's all over, I'll attempt to gather some real quantitative data on the issue, to see if my impressions match reality.]
I'll throw out a straw man for people to discuss (and attack). In theory, all (reasonable) papers start with the same basic grounding -- you have proofs of some theorems. After that point, though, things get a bit hard to judge. How "nice" are your theorems, in terms of technique/mathematical beauty/novelty? How "big" is your improvement over previous work -- for example, is improving an O(log n) competitive ratio to O(log n/log log n) "important" enough? How practical is your solution (hard to say, when nobody actually implements algorithms or gives performance results...)? A lot of this seems subjective, so naturally there are differing opinions.
In networking or other more practical fields, the same issues -- technique/beauty/novelty, quantity of improvement, practicality -- all come into play. But there's a lot more focus on the bottom line of "Could this lead to a substantial improvement for an important real-world problem." Perhaps this consistency in the criterion for what is a good paper leads to more consistency in reviews?
Subscribe to:
Posts (Atom)