The chairs of the SIGCOMM PC were asked to submit a note to CCR on the Commitee Process, available here. It's an interesting read. (Overall, I've found the systems community a bit ahead of the theory community in thinking about and reviewing its conference processes.) They also make some anonymized data available -- which looks safe to me, but if anyone thinks that's a bad idea, I think it's worth speaking up about it.
Some highlights:
1: "We do see a problem with requesting only 14-page submissions, and want to use this document to convey a position we frequently heard from TPC members as well: short papers could be an interesting addition to future CFPs. In a conference with such a low acceptance rate as that of Sigcomm, “small contributions” naturally find themselves at a disadvantage. The issue is that the description and evaluation of an ingenious but small idea certainly takes less than 14 pages. The same is true for work that improves on prior work or presents tools with important practical use but small technical contribution."
Short papers are, in my mind, also a potential mechanism for mixed theory/practice papers to get into SIGCOMM. One issue that arises is that theory papers aren't written in the way systems papers are. In a theory paper, if we're describing an algorithm or data structure, our goal is to make it as GENERAL as possible -- here's a neat idea, here are some places it might be useful, please find others. SIGCOMM pappers are designed almost the opposite way -- to show an algorithm or data structure is useful requires looking at a very specific application and experimentally detailing exactly what performance gains arise in the context of that specific application. I will push for short papers to include interesting algorithms and data structures that might be GENERALLY useful to lots of people and applications but may not yet have been shown to offer groundbreaking performance for a specific application. (As pointed out in the PC meeting, my work is often in that space, so I have a bias in this argument...) Suggestions for making this workable are desired.
2: "ACM SIGCOMM defines a conflict of interest between an author and a reviewer if the two have worked together in the past two years. Students and advisors are considered “conflicted for life” and of course any institutional or private relationship between the author and the reviewer instantly qualifies as a conflict."
I just had to point out the "of course" there, after some of the inane commentary regarding my introducing much weaker conflict of interest policies and goals for STOC. (See here and here.) [When I've talked about the theory community's standard conflict practices to systems people, most politely shake their heads and mumble, "Well, if that's the way it's done..." The more open people privately make clear their opinion our approach opens the door for all sorts of systematic biases and seems remarkably backward. And I'm phrasing that politely.]
3: "For a few years Sigcomm has used a two-tier TPC: “light” members are asked to do 10-15 reviews and are not invited to the meeting; “heavy” members are enrolled for 20-25 reviews plus mandatory participation in the TPC meeting. The split is mostly motivated by keeping the TPC meeting of manageable size. This year we introduced a third tier, called “senior” for lack of a better name. The role we intended for senior TPC members was a mix of conflict solver, an additional voice during the TPC meeting, helping hands for last minute reviews, and possibly carriers of a different perspective that could put the submitted research under a more positive light."
Compare with the theory discussion taking place here on the roles of junior/senior PC members in theory conferences. Also, compare with the theory conference practice of using 100+ subreviewers with PC members responsible for 45+ reviews. I'm not claiming this approach is better; but it is food for thought.
Subscribe to:
Post Comments (Atom)
13 comments:
"I will push for short papers to include interesting algorithms and data structures that might be GENERALLY useful to lots of people and applications but may not yet have been shown to offer groundbreaking performance for a specific application"Just a small question. Why do you think such a paper can not go to some theory/algorithm conference? Is there a theory venue that accept short papers?
Anon #1: There are two aspects.
1) If I think my algorithm is actually practical and useful, I want to get it in front of people who might actually use it. That is, I'd rather it go to a systems/networking conference.
2) If I think my algorithm is actually practical and useful, it's not clear it will get into a good theory conference. Because often the focus will be on clever handling of cases or specific details, and not on "mathematical elegance". People who will actually implement the algorithm may care about the former more than the latter.
Also, "short" here just means less than 14 SIGCOMM pages; more like, say, a 10 page SODA-style paper would be "short" by SIGCOMM standards...
1. The notion of short papers died at SODA after such papers proved uncompetitive.
2. It would be very interesting to compare the co-authorship and co-institution graph in theory vs SIGCOMM. I would bet that there are huge structural differences in the two graphs. I would also suspect that SIGCOMM co-authorship and co-institution conflicts (outside of the adviser/advisee relationships) would tend to correlate with each other much more than in theory.
Also, "short" here just means less than 14 SIGCOMM pages; more like, say, a 10 page SODA-style paper would be "short" by SIGCOMM standards...
Thanks for the clarification. When you mentioned short, I was thinking more on the line of 5-6 pages. I guess 5-6 is more appropriate for workshops instead of conferences. I think 10 is a good number, but a 10-page paper (using sigcomm latex style) is not that short. May be we need to call it something else.
-Sushant
"If I think my algorithm is actually practical and useful, I want to get it in front of people who might actually use it"
I might have missed something, but in this case, why not write a full 14 page sigcomm paper showing why the algorithm is useful in the context of a specific application?
Wouldn't that have more impact?
My impression, at least in theory, is that page limits stated in the CFP are upper bounds -- if you manage to make a substantial contribution in 3 pages, then great. Are short papers sent to SIGCOMM frowned upon? (Shouldn't the PC make its decisions solely based on the quality of the work?)
@Anonymous: Yes, but it's rare that a paper < 12 pages makes it in. The range of 12-14 is typical, with probably 70+% of the papers at 14.
@Michael: The sigcomm program has experimented in the past with different categories of papers, such as "position papers." In general, they haven't stuck around successfully.
Anon 5: There are (at least) three main reasons to avoid the specific application. First, it actually can HIDE the utility of the data structure/algorithm for more general problems. Second, it may be that viewed from the point of view of any specific application, it's not so great. Maybe, for example, there's a hash table design that saves a factor of 2 in space. That's not a big deal for many applications (just buy twice as much memory!), but leveraged over many applications, and if it's a simple enough idea, it's something worth having everyone know. Third, the work involved in getting things to work for a specific application can be huge. For example, if there's an insistence I test a data structure/algorithm on "real" (as opposed to simulated) data, it may be that since I'm not at Google/AT&T/Microsoft/etc. I don't have access to appropriate data/infrastructure. That doesn't mean I don't have a worthwhile idea. Even if I do have that sort of access, if it's truly a "general" contribution, it could be 10x times the work for minimal actual gain (the main actual gain being I'm now satisfying the bar of SIGCOMM).
Anon 6: To clarify David's comment, there's a reason why almost all papers are the full 14 (or otherwise 12+). The "proof" SIGCOMM is looking for is generally based on real-world experiments. There are always large number of parameters and variations to consider when suggesting a novel idea for a real-world setting; if you're not using all the pages, someone can quite reasonably suggest that there were other parameters, variations, details etc. that should have been checked. Indeed, one reason to have a page limit is to force authors to try to think carefully about which experiments they really want to include!
David: I'm aware of the potential problems. I, personally, would like to see an avenue for SIGCOMM (or other similar conferences) for algorithmic ideas that may not rise to the level of full systems, and this seems a possible way; I imagine there might be other uses for it as well. Perhaps it won't work out -- but perhaps it's worth a try!
Its an interesting idea. Indeed, as David mentions, SIGCOMM has previously experimented with other submission formats -- to its credit I think. That challenge is that it is far easier to add a category to the CFP than to actually get such papers accepted. The core of the problem is that while there is some rough community consensus about what we expect of a traditional SIGCOMM submission (i.e., as you mention, proof by experiment :-) no such similar expectations exist for the short paper category. Indeed, the first year that position papers were added as a category, the SIGCOMM PC was hard pressed to accept any and the debate around a handful of such submissions literally consumed hours at the PC meeting (I know this because I was conflicted on several and thus spent a small child's lifetime in the hallway). The trick is coming up with an easy to evaluate and clearly separable definition of what a strong short paper looks like. This is hard without historical context (as with many communities, SIGCOMM seems to have a Justice Potter Stewart-like definition for what an appropriate paper is). I fear absent such shared expectation that the category will either languish through underuse or will end up becoming an outlet for certain longer submissions at the PC meeting (i.e., "reject to short").
I wonder what people think about potential conflict in the situation where submission X presents a significant improvement, generalization, simplification, or application of previous work Y. Are the authors of Y in conflict of interest?
The authors of Y might be vested in the specific technique, problem or research program? On the other hand, they are probably the best experts to judge the paper, no?
And following a previous comment, are theory papers different than systems in this respect? For instance, if a small group claims a networking paradigm competing with a system from major university M, should authors from M be reviewing my paper? In most likelihood, each of the two systems wins in some aspects, and overall evaluation could be very subjective judgment...
Should reveiwers start writing a disclaimer "this submission is related to my work hence I might be positively/negatively biased"???
Short papers create many problems, perhaps as many as they solve. As Stefan mentioned, short papers can end up being consolation prizes at conferences. The short paper becomes a compromise between two opposed PC members (one for and one against acceptance). This certainly happens at SigMetrics and has happened at USENIX ATC. In addition, short papers cause problems in terms of citations and value of contribution. I've seen people list short SigMetrics papers on their CVs as if they were full papers (i.e., they don't say "extended abstract" or "short paper") in the listing. You then get into a guessing game about the author: are they being intentionally misleading, or are they just clueless? Neither is good. This leads me to think short papers just aren't worth it.
The main problem, as Stefan and others, seems to be that the quality of the short papers seem to be low. If that is the only problem then why not have a better quality control by only accepting the really good papers. In addition, there does not have to be a minimum number of papers to be accepted for these kind of papers. In this way only the best contributions can be let in. This would give the theory guys and others a nice outlet to publish those occasional good contributions.
Regardless, SIGCOMM conference is a joke. Lovely to see the 'old boys' pat each other on the back for a job well done!
Post a Comment