Could maths be the answer to ‘glaring weakness’ of the REF?

Mathematicians are calling for a change in Research Excellence Framework assessments to reduce potential bias.

In a paper published this week in the Royal Society Open Science, Robert MacKay and Sarah Parker from the University of Warwick, along with Ralph Kenna and Robert Low from Coventry University, use maths to try to remove bias and averaging from major assessments.

The academics now hope the algorithm they have created will transform the way funding and interview panels make decisions.

The new technique, known as calibrate with confidence, asks assessors to give their score for each item and also to rate their level of confidence in providing that score. Each time a grant proposal, exam paper or job application is assessed, it is then possible to remove bias compared to simply using averages.

When used on a previous assessment of 44 funding bids from 2015, the formula showed a dramatic change in the ranking of the results compared to using average scores, a technique used on most occasions.

In the original exercise, 13 bids were funded, with average scores across the assessment ranging from 75 to 87. Had these been assessed using the calibrate with confidence method, only six projects originally given the highest marks would have received funding, while seven different projects would also have been allocated funds.

The method is already being used by Coventry University to assess internal funding applications, along with other organisations which have downloaded a free online tool to use the formula.

Robert MacKay, Professor of mathematics at the University of Warwick, said:

“Our ambition is for our method to be adopted as a decision support tool by every panel that has to evaluate a range of objects and whose members may have different standards and expertise.”

Ralph Kenna, Professor of theoretical physics from Coventry University said:

“The new approach is especially important for exercises such as the UK’s Research Excellence Framework (REF) where differences in standards within and between panels are currently a glaring weakness.”

Calibration with confidence: A principled method for panel Assessment can be accessed online. 

For more press information contact Kelly Baker-Adams, press officer, Coventry University on +44 (0)24 7765 9752 or email Kelly.baker-adams@coventry.ac.uk.