Which matters more? Excellence or impact?

The Aotearoa New Zealand research, science and innovation (RSI) system is one that purportedly values and rewards research excellence and research impact equally.

Despite this, the current system favours an approach that focuses on excellent research that has impact vs impactful research that is excellent.

It is a subtle difference but let me explain how that plays out in practice and why there is an inherent bias. For me, this is an important discussion to have as it has flow-on impacts for industry-led research – which is mostly impact-led.   

The road to funding

According to MBIE “the Endeavour Fund plays a unique role in the science system through an open, contestable process with a focus on both research excellence and a broad range of impacts”. The indicative investment in the 2024 investment round is $57 million per year (projects range 2-5 years).  

This makes it a key, if not the key, research investment mechanism for research organisations in Aotearoa.  It is highly competitive, with most Universities, Crown Research Institutes and Independent Research Associations competing for this pool of funding. As a result, the success rate is very low (usually below 20%). The application process is onerous and it is run in set rounds, with only one round per year. Missing out on funding can have crippling effects for many organisations. The two main criteria for assessment are excellence and impact. However, excellence is assessed first and then only the top ranking submissions are progressed to the next step where they are assessed for impact.  

Essentially, this means that research excellence is given priority in practice. There could be an amazingly impactful research proposal that misses out because it rates less highly in terms of excellence. It won’t even get considered. The fact is that it can’t progress, no matter how impactful it is, even if it is still pretty decent in terms of excellence.  

I am not at all advocating that we support projects with low research excellence. What I am asking is, why are we OK with a process that allows this to happen? Are we happy with a process that potentially knocks out the most impactful science before it can even be considered? 

Playing the scenario out!

Let’s say that we have two projects. Project A is high in research excellence (let’s say it would be assessed as being a 5/5 for research excellence) and slightly less high in research impact (let’s say it would be assessed as being a 4/5 for research impact).  

On the other hand, Project B is less high in research excellence (let’s say it would be assessed as being a 4/5 for research excellence) and high in research impact (let’s say it would be assessed as being a 5/5 for research impact).  

In the current assessment process, they are both rating 9/10. However, project B has a higher chance of not being considered for funding as it is more likely to be knocked out of the first stage of assessment because it has rated 4/5 for research excellence. 

This is not fair. It is also not sensible.  

An alternative approach is to ensure that all proposals are assessed for both impact and excellence or that impact is the first hurdle. Surely, all research must be high impact? Is there any point doing the most excellent research in the world if it has no impact? Conversely, I can see many reasons why it would be important to do the most impactful research in the world, despite it being less “excellent” (noting that excellence is generally described as “the best people, a rigorous approach and optimum results”.  

How have we got it so wrong? Is this a sensible way to do things? I can see why it is done this way… it leads to less resources being required for project assessment. However, is that the best outcome for how our major RSI funding is allocated? 

This is only one example amongst many where impact is touted as being equivalent but is actually subservient to excellence.  

In a 2021 report titled “Excellence in Research” it was pointed out that MBIE uses a definition of “Science excellence”, from the  National Statement of Science Investment 2015-2025 (NSSI) “that differs from international practice in excluding non-academic impact.” This report also states the critical point that “Research excellence and impact (seen as the second pillar, alongside excellence, of a healthy RSI sector) are assessed separately”. Unfortunately, this document only focused on discussing the “concept of research excellence: what it is, how it can be assessed and why it matters”. It did not address research impact, despite noting “excellence and impact are closely linked: in mission-led research, for instance, we want both excellence and impact and these are likely to be correlated”. 

In a 2019 position paper titled “The Impact of Research” MBIE introduced a definition of research impact as “A change to the economy, society or environment, beyond contribution to knowledge and skills in research organisations”. Surely, that is the hurdle whereby we should be first assessing where our national research funding is directed. At the very least, it shows why research impact should be considered equal to academic measures of research worthiness for funding.  

So where to from here?

In my summation of these two reports; the definitions used in Aotearoa for “research excellence” and “research impact”; and the practical ways that excellence and impact are assessed, there is one very concerning outcome.  We are favouring measures of research excellence that largely assess the value of research to the academic community over measures of research impact that assess the value of the research to the whole community. 

The National Statement of Science Investment is due for review. I hope that the system is revised to either assess impact and excellence simultaneously or is amended to favour impact. After all, what actually is the point of research without impact… even if it is performed by the best researchers, in the best facilities, with the best methodologies and technologies, and then published in the best journals?