Graeme Rosenberg (REF Pilot Manager) from HEFCE presenting on this.

I’ve sat at the back with the only other blogger (afaik) in the room as we both run low on battery, and the only power sockets are next to the projector – which means I can’t hear so well 🙁

Some of this may be a  repetition of stuff I blogged at the earlier REF event at Kings College London. Following consultation the REF is going ahead with 2 key changes – assessment for all subjects will include some metrics and some peer-review process, and timescale lengthened.

The key features of the REF are:

• Unified framework for research assessment and funding
• Robust research quality profiles for all disciplines
• Emphasis on identifying and encouraging excellent research
• Greater use of metrics than at priesent – including bibliometrics “for all disciplines where these are meaningful”
• Reduced burden on HEIs

Timetable:

• Up to spring 2009 – bibliometrics pilot and other development work
• Spring/Summer 2009 – consult on all main features of the REF
• By Sept 2009 – decide on main operational features of the framework

Use of bibliometrics:

• Bibliometrics to be used in those disciplines in which they are meaningful – alongside other data and information
• Interpretation by expert panels
• To be based on citation rates per paper – not journal impact factors – and taking account of worldwide norms for the field, year of publication, and document type
• Results to be aggregated for substantial bodies of work; presented as a citation profile

About to run a pilot – starting imminently, with 22 institutions involved. This will look at a number of issues:

• Which disciplines?
• Which staff and papers should be included? Universal or selective coverage? Are papers credited to the research or the institution?
• How to collect data – and the implications for institutions (looking at Web of Science and Scopus for bibliometric data, but need institutions to at least identify the papers that ‘belong’ to them)
• Which citation database(s)? (as mentioned looking at WoS and Scopus – they have different coverage, and continue to develop – what is best for the pilot, may not end up being the best for the REF, or may change over time – need to pick the best one at the time)
• Refining the methods of analysis – including normalisation fields and handling self-citation
• Thresholds for the citation profile
• Interpretation by expert panels

Last point is key – allows flexibility in terms of what numbers are presented, as long as the expert panel know what is included and what is not (e.g. this could be a way of dealing with the self-citation issue)

It will be possible to compare the results to the 2008 RAE and investigate discrepancies and why they arise

The pilot institutions are:

• Bangor
• Bath
• Birmingham
• Bournemouth
• Cambridge
• Durham
• UEA
• Glasgow
• Imperial
• Institute of Cancer Research
• Leeds
• LSHTM
• Sussex
• …

The timetable is:

• May-Jun 08 – Select HEIs/contractors
• Aug-Oct 08 – Data collection
• Nov 08 – early 09 – Data analysis
• Spring 09 – Pilot results

Participating institutions will be asked to:

• Provide as much data as available on all researchers and publications eligible for the 2008 RAE (in relevant disciplines)
• To be matched to Web of Science initially, and supplemented by additional records
• We will evaluate issues of completeness and accuracy and seek feedback from the institutions
• JISC project to document the data systems requirements

Some issues for institutions:

• As REF developed need to assess the potential impact on the sector (accountability burden, equal opportunities and perceived behavioural incentives)
• Information management
  • Populating database for the initial bibliometrics exercise (Looking at Australia who are moving in a similar direction, and have had a requirement for collecting information on their published research outputs for some time)
  • Ongoing management of bibliographic data for multiple purposes
• Management information and internal resource allocation
• Relationships between citation data coverage and publication outlets – where are the gaps in WoS and Scopus?

For more information http://www.hefe.ac.uk/research/ref and/or join the REF-NEWS mailing list (details at the URL given here).