By Shaun Hendy
The MacDiarmid Institute’s Kate McGrath created a bit of a stir on Friday with a blog post that took a critical look at the Marsden Fund’s decision to invite 15% fewer proposals into the second round of its funding process than it did last year. The Marsden Fund runs a two-round process: applicants submit a one-page expression of interest in February, with about a fifth of these then invited by expert panels to submit a more comprehensive proposal in May.In 2014 20% of first round applicants were invited to submit a second-round proposal. This year this figure fell to just 17%. Below, I’ve plotted the fraction selected to submit second round proposals since 1998. After a spike in 2003, the fraction selected at the second round has returned to the levels seen in fifteen years ago. So, while not unprecedented, a 17% throughput at the first round is something not seen for a while.
The advantage of running a two-round system is that it can reduce the burden placed on the sector in the writing of comprehensive proposals. If we had a one-round system, everyone who wanted to apply would have to write a comprehensive proposal that was detailed enough for peer review. The more proposals that get through to the second round, the more work there is for the researchers that write them and the selection panels that evaluate them. The Marsden Fund Council chose to reduce the number of proposals selected for the second round this year because they wanted to reduce the workload for the sector.McGrath is conscious of this issue, having blogged about compliance costs in late 2013: “The compliance costs of everyone applying for everything is costing our country real money that could be used producing real outcomes from the scientists utilising their time to ply their trade; scientific research rather than applying for funding that invariably they will never get.” In other words, the amount of effort we spend writing proposals should not be allowed overwhelm the effort we put in to our research.ExpectationsA slightly different way to look at this is to compute the expected value of writing a second-round proposal i.e. what is the expected pay-off in funding from submitting a second-round proposal? You can compute this by multiplying the likelihood that a second-round proposal will be funded by the average amount awarded. I’ve plotted this quantity below from 2001 to 2014, with a fantasy figure for 2015 based on the Marsden Fund’s indication that this year they will fund 90-95 proposals.
By reducing the number of proposals at the second round, the Marsden Fund Council can increase the expected value of second-round proposals to the applicants. A standard second-round proposal (as opposed to a fast-start) this year is expected to return $315,000-$319,000 compared to $287,000 in 2014. As there is less money available overall this year, if the Marsden Council had okayed a similar proportion of second-round proposals to what it did in 2014, then expected value of a second-round proposal would have fallen to around $250,000. This would have been well below the long-run average of $290,000 for the expected value of a second-round proposal.However, in her recent post, McGrath says that is now “more difficult to get your foot in the door of arguably the most prestigious grants in the country”. Her concern is that panels might not be expert enough to make good decisions at the first cut, so that the country might be missing out on funding the best science. The assumption that McGrath makes is that decisions made at the second round, when panels have access to expert peer review, will be of significantly higher quality than those at the first round.Picking winnersYet recent work by Te Pūnaha Matatini investigator Adam Jaffe suggests that Marsden panels are not very effective at predicting the eventual success of the second round proposals they view. Jaffe’s findings (not yet published, but which were commissioned by the Marsden Fund Council) are broadly consistent with the findings of many similar studies in other countries: panels and peer reviewers are not very good at picking winners (although see this recent article).These findings do not to say that panels aren’t doing their best. Rather, the evidence suggests it is simply an almost impossible challenge to predict the outcomes of cutting-edge scientific research. It is also far from clear that letting more proposals through to the second round would improve matters. In my opinion, when we know that panels have such a difficult job to do, it is better to acknowledge their limited effectiveness and to reduce the burden on the system accordingly rather than to double down as McGrath suggests.One-round system?One of the key disadvantages of a two-round process is that, all things being equal, it will have a much lower success rate than a one-round process. More people will be prepared to chance their arm in round one if they only need to complete a short application. Filling out a single-round application for an Australian Research Council Discovery Grant, for instance, is considerably more onerous than the preparation of even a second-round Marsden proposal. The Australians are rewarded for the extra work involved with a success rate of 18%.Despite this, I still prefer our system. If the ability of expert panels to predict funding success is poor, then better to have a system that minimises transaction costs. And as I noted in a blog post in March, I think we also need to acknowledge the increased research intensity in universities that has driven the oversubscription of the Marsden Fund.I think the Marsden Fund Council is well aware of the trade-offs described above and has chosen a sensible course. In fact the Council should be commended for commissioning work to investigate the effectiveness of its decision-making process. The real fix for the system – increasing the Marsden Fund to match the increased level of research activity in our universities – is not within the Council’s control.