Monday, June 28, 2004

On gaming program committees






1. Our PCs are small







  1. I don't think that simplicity in itself is viewed
    negatively by program PCs. In fact if someone has
    a simple proof/algorithm/result on a well studied
    open problem it is definitely a plus. The flip side
    is that if the problem studied is not known
    to members of the PC or to the wider community of
    reviewers it becomes difficult to judge. One has to
    spend time thinking about the problem first to appreciate the difficulty of coming up with a simple
    proof. Given the short review cycle of conferences,
    very few people are going to think about the problem
    by themselves.


  2. exactly: consider the following two situations:

    1. You have a trivial proof of a claim that took a year to discover
    2. You have a trivial proof of a claim that took 5 minutes to discover

    The only way to distinguish between 1 and 2 is if the reviewer is able to prove the claim themselves. For a paper not in one's area, this is not easy.

    However, although our normative standards might prefer 1 to 2, it is not clear why this should be the case.

  3. Suresh, I don't think we are on the same page
    as you imply by "exactly". It is not the duration
    of the time it took to obtain a simple proof that
    matters. What matters is whether the problem in
    question has been studied before by enough people
    and whether there is an implied interest in the problem.
    For something new or not well known problem a simple
    solution might be viewed with caution which is not
    necessarily a bad thing.

  4. ok fair enough. clearly if a new problem has what appears to be a simple proof, one has the right to question the value of the result IF there are no other mitigating factors (the problem itself is interesting, has applications, etc etc).

    Of course this doesn't apply to the problem references in Lance's original post, since that paper got a best student paper award from FOCS.

  5. Could you be more specific about the "many other conferences" that give better feedback than theory? I just finished my reviews for OSDI, and while I believe I did a good job, I know that there are plenty of issues I may have missed or just not covered in enough detail. Maybe I haven't been around enough yet, or maybe both OSDI and theory conferences enough to see the differences. If there's a community that's doing much better, though, I'd like to hear about it.

    -David Molnar

  6. I think not even the criterion in the comment above is always fair: even if the proof can be rediscovered by the reviewer in short time, it might have taken a year of research to discover the right, true statement to prove, and only given that it is easy to find a proof.

    IMHO reviewers should primarily judge the results, whether they're interesting, new, unexpected, useful etc. etc., and the proofs should just be viewed as means to an end, to certify the truth of the proven statement. From this point of view, which unfortunately is not taken by large parts of the theory community, the simpler a proof is, the better (given, obviously, that the result is of value by itself.)

  7. I can't say I agree with what Jan argues: a proof is not just a means to an end: it often provides new ideas to add to one's toolkit. We need go no further than Erdos's "book proofs" to appreciate this idea....

  8. Of course, I won't deny that proofs have value besides being a certificate to the truth of the proven statement. What I wanted to say is only that for the purpose of reviewing a paper for a conference, these extra values should be considered secondary, and the primary emphasis should be on the results themselves, not the proofs.


Disqus for The Geomblog