As I go through the reviews from my latest NSF proposal, there are general high-level comments related to my past work. This leads to an interesting and perhaps-not-often-enough discussed issue -- how should the past influence present funding decisions? Mikkel Thorup recently tried to open up discussion on this theme with a viewpoint piece in the Communications of the ACM. (Sorry it's behind a firewall; if someone has a direct link please post in comments.)
To be clear, I'm not sure what the current NSF policy is on "history", as I haven't served on an NSF panel for a while. In the past I recall hearing that NSF panels were not supposed to take PI history into account in evaluating the proposals, although it seemed to be implied that the program manager would take that into consideration if needed. Another country I do reviews for does take history into account in an a fairly direct way -- there are explicit questions regarding whether the applicant has demonstrated the qualifications necessary to carry out the proposed research project. That's a fairly broad phrasing, that at least in my mind opens the way to discussing the PIs past work. So in that case there is some weight put on past performance.
I admit that I think past performance is a perfectly reasonable criterion for judging funding proposals. It helps to think of extreme cases. For example, if Les Valiant wrote a proposal and I somehow was on the panel and didn't understand or appreciate it, history (and/or basic probability) would overwhelmingly suggest that the fault is with me. Or even if one would assume or argue that the proposal wasn't well written, wouldn't the risk/reward calculation based on the track record argue strongly for funding? At the other extreme, one must make sure that new faculty with limited track records obtain enough funding to have the chance to ignite, either through special programs (like CAREER grants) or with some understanding that new faculty should be given an appropriate share of the funding pie.
In the end, my mindset is that there is a difference between writing a compelling sounding proposal and being able to deliver the research results. Looking at a researcher's past helps calibrate what the final outcome will be. Not taking it into account seems inherently wrong, under standard success metrics. However, it's hard to assign a number as to how much it should affect the overall (and arguably quite subjective) scoring of proposals in any given panel that decides grants. I don't envy people at the NSF the job of trying to untangle this messy situation, particularly in settings where they feel others (say, up in the direction of Congress) are out to second-guess their decisions.
In terms of my current reviews, I thought the panel took into account my past history in mild and reasonable ways that I appreciated. Unsurprisingly my proposal dealt with problems at the intersection of algorithms, networking, and information theory, and I discussed that I had a (successful) history of interacting with all of these communities. The reviewers acknowledged this history and noted that my background, including previous research in hashing, coding theory, and so on would likely be relevant to solving the problems I proposed. I don't know how much it played into the final decision, but I was glad they agreed with the argument that I had the right background. I should note, though, that in the past I've written proposals where I was trying to "branch out" into areas that were newer to me (they did not -- gasp -- involve hashing, or coding theory....), and I believe I experienced the same effect in reverse. So there's certainly an argument the other way...
The other way the past seemed to be taken into account was with regard to some of the "broader impacts". This blog, for instance, seems to be seen as a "positive" in terms of community outreach. Similarly, the fact that I have produced a body of educational materials over the years (class notes*, surveys, a textbook, assignments, etc.) was noted. Again, I think history should be taken into account somehow here. It's easy to say you're going to write a textbook or large-scale survey in a proposal, but it's another thing to have a record of delivering.
How much should past performance count in reviewing proposals? And can we actually influence the funding agencies to take past performance into account in ways we as a community find suitable? I'm curious to see if people have thoughts they'll share in the comments.
* I'm off teaching the Computer Science 124, Algorithms and Data Structures, now for sabbatical and potentially for a while. If anyone teaching this wants my materials, though, let me know; I've often passed them along to faculty starting to teach that type of course.
Subscribe to:
Post Comments (Atom)
2 comments:
I live in a country whose funding body takes recent history under consideration and allows for partial funding.
My proposal which was well funded had the comment (paraphrasing) "project seems at first overly ambitious and broad but applicant has a record of delivering. rank: top category".
This is a mini-version of your Les Valiant example above. It is meant to show that lower tiers can also benefit from looking at the historical record instead of focusing in grantsmithing alone.
The NSF review criteria specifically include "4. How well qualified is the individual, team, or organization to conduct the proposed activities?", so your track record is an important consideration.
Post a Comment