This session by Jonathan Adams (from Evidence Ltd)

Evidence Ltd have been working in this area for some years. Evidence Ltd believe that the UK’s international research status has increased over the last 20 years, and the REF needs to support this.

Jonathan is showing a graph of output from various European countries which shows that over the last 20 years, output from each country has been reasonably stable, except the Netherlands, which has experienced a large growth. He believes we need to avoid a system that leads to this disproportionate change (and is saying that the adoption of bibliometrics in the Netherlands caused their increased output)

Jonathan raising questions – will the introduction of metrics lead to manipulation, that invalidate the assumptions on which metrics have been judged and introduced.

Jonathan making the point that the recent DIUS announcement on peer review does not mean metrics are not to be used – in fact, any peer review in the REF must use the metrics – the panels will not be able sensibly to disregard what the metrics say.

Measurement of Impact (normalised) 1996-2000 shows high correlation to RAE2001 grade – however, he says there is a huge variation is residual data (I don’t understand what he means here).

One of the key questions (says Jonathan) is whether we assess total activity or selected papers – this makes a significant difference to the outcome of metrics you apply (although I would have thought the same was bound to be true of any measure of quality – allowing ‘nomination’ of papers is a filter for quality surely?)

Jonathan making the point (that Anthony did as well), that averages do not describe underlying profiles – the data is highly skewed (lots of low values, few very high values). So – we need to do profiling for different disciplines – we cannot rely on averages, as the data is so skewed. Comparing distributions is much better than averages, which is what these profiles are meant to allow us to do.

Another issue – there are many papers published that are never cited – we need to be clear how these will be dealt with.

It is clear that the Normalisation strategy adopted will have a significant impact on the outcome – Jonathan showing how three different normalisation strategies can change the outcome.

Any clustering model needs to fit a UK research model – need to be sure that a discipline is reviewed alongside similar disciplines from the bibliometric viewpoint. For example, Chemical engineering publications from the UK are published in physics journals – so need to think about it in these terms, not just lump it in with the rest of chemistry when deciding how to normalise etc.

Evidence Ltd have produced a RAE/REF Predictor (‘You are the REF’) which allows anyone (? or anyone from a UK HEI?) to decide on different weighting factors etc., and see what happens – this is apparently going to be available soon at http://RAE2008.com

Goodhart’s Law – once you use a metric for matters of public policy, it loses its effectiveness overtime (based on experience in the banking sector)