Today I’m attending the event “Beyond the RAE 2008: Bibliometrics, League Tables and the REF” at Kings. The event is sponsored by Thomson Scientific (who produce the Web of Knowledge and Citation indexes).

Irritatingly the Eduroam service which should allow me to join a wireless network here isn’t playing nice, and won’t let me on (I’ve had this problem at Kings before, but neglected to follow it up – now I wish I had).

The day is being opened with an introduction from Karen Stanton (Chief Information Officer and College Librarian), outlining the importance of bibliometric data, and how much is at stake for universities in terms of future funding depending on the outcome of the REF.

First up, Graeme Rosenberg from HEFCE is providing an update on the REF – following the recent announcements indicate a step back from the use of bibliometrics, this should be interesting:

He starts off by saying it may well be too early to answer the key questions the audience will have. However, they are expecting to publish some further details in May. However, the story so far is:

• In 2006, HEFCE were asked to develop a new framework, with a larger emphasis on bibliometrics
• HEFEC commissioned expert advice on the use of bibliometrics
• Consultation outcomes are now available on the HEFCE website at http://www.hefce.ac.uk/research/assessment/reform/

The main outcomes of the consultation are:

• A more unified framework, with assessment for all subjects, including some combination of metrics-based indicators, including bibliometrics where appropriate, as well as input from expert panels
• The timetable to design this more flexible framework is 12 months

What happens next?

• An early start on a pilot of the bibliometric indicator
• An early start to dialogue with the sector on developing light touch peer review and on choosing and using other metrics
• Further work on a number of other issues raised in the consultations
• Remain committed to full implementation of a new framework by 2014, with funding based on some aspects of the new framework to be gradually phased in from 2011-12

There are a number of issues that need to be resolved in the development of the bibliometric indicator:

• Coverage (which disciplines? all or selected staff? Do papers follow the author?)
• Technical and methodological issues (defining fields for normalisation, multi-authorship, self-citation, etc.)
• The use of bibliometric indicators within the REF
• Burden and other implications for the sector
• HEFCE will test issues through the pilot, then consult further and take decision during 2009

The plan is to have a bibliometrics exercise to happen in 2010.

The consultation essentially found that most responses felt bibliometrics were indicative of quality, but that there should be some input (often the word used was ‘moderation’) by expert panels.

The bibliometrics pilot should occur as follows:

• May-Jun 2008 – select HEIs/contractors
• Aug-Oct 2008 – Data collection
• Nov 2008 – early 2009 – Data analysis
• Spring 2009 – Pilot results

HEFCE are currently in contact with institutions who have shown interest in being part of the pilot. They are suggesting that the pilot is based on data submitted for the 2008 RAE as well as new material not covered in that submission.

The pilot will cover a broader range of disciplines – all where there is a reasonable amount of citation data available. However, they do not expect all disciplines to use bibliometrics in the actual REF – the pilot will establish some boundaries.