Case Study 1: King’s College London

Mary Davies (Deputy Direction of Information Services and Systems at KCL) is presenting this

Context

• In 2004, preparations for RAE 2008
• RAE 2001 had highlighted
  • Lack of linkage between systems
  • Data quality issues – discrepancies between central data and local (Dept) data
  • Time spent of data verification and manual data input
• Decided to build a ‘Research Gateway’ to collage information from our corporate systems for submission to RAE – regular data feeds from HR, student, grants and finance database

Mary showed a couple of screenshots from the Research Gateway.

The Research Gateway was meant to avoid duplication of effort and promote effective data management. Kings didn’t have a publications database, and found that searches by institutional name on Web of Knowledge revealed at least 25% of Kings research output not attributed to Kings – often because where institutions had merged with Kings, the researchers continued to use the name of the original institution, not KCL. Also confusion where staff work for both Kings and an NHS Trust, and where research was attributed.

Kings decided to have a College Citation Policy advising staff on the importance of citing KCL as the institution.

Methodology

Project setup for the RAE 2008. Eventually submitted 4897 outputs from 1589 staff, across 37 UoAs (units of assessment).

Survey of Schools, Depts etc. found publications stored in a wide variety of systems – they decided to capture electronically back to 2001 to provide initial database contents. Only two of the locally developed databases were mapped to the Research Gateway.

Methodology was to search databases with address/affiliation fields using variations of institutional name + postcodes – e.g.: Web of Science, ISI Proceedings, Medline, Cinahl, etc.

Much easier in Health schools that other departments – especially for book, book chapter and other non-journal article information. Lots of data extracted from Worldcat and other sources, imported into Reference Manager database, big de-duplication exercise.

Los of work to clean up data

Challenges/Problems

Interestingly Mary has identified as an problem that the UoAs were substituting selected publications up to the last minute – we had this problem as well, and in the library this meant we were checking the accuracy of records that weren’t submitted, and ever shifting target in terms of how many submissions that needed checking.

Plans for REF

• Develop the Research Gateway into a ‘Virtual Research Environment’, including integration with the Institutional Repository – attuned to requirements of REF (when these are clear)
• Continue to work with supplier concerning database indexing policies, where relevant
• Investigate research IDs and identifiers and institutional IDs developments
• Develop intern expertise in bibliometrics – sent staff on the Leiden course
• Relates to developing organisation culture of knowledge and comfort with bibliometrics
• Concerned about Source data issue and appropriate coverage
• Need to work on verification of citation information, especially if self citation excluded
• Need clarification of how open access forms of publication will be taken into account
• Capitalise on team work between academic and administrative staff
• Develop HE forum for sharing best practice

 

Case Study 2: University of Leicester

After RAE 2001, Leicester found many depts didn’t have good records, and were often trying to collect 6+ years of data in a few months. When they came to RAE 2008 decided to invest, in software etc. to collect data.

RAE 2008 submission was 2700+ outputs including 260 books, from c.800 staff

Leicester bought GENIUS from InfoEd to use as their database – which they called Research Expertise Database (RED) – similar to King’s Research Gateway. This has a web based interface, with users empowered to update records, and integration to the staff records from HR etc.

Journals selected from pick list to prevent mistypes and use of abbreviations. Allows you to indicate a publication should be used for RAE.

They then could export data from RED to that RAE software. The RAE software was not seen as usable by the administrators for the Units of Assessment – they just had to export from RED to Excel, do some ‘simple’ data manipulation, and import to RAE software.

I have to admit from the description of what the administrators had to do once the data was in Excel, it sounds quite complicated – lots of things to update. However, they say it worked, and that all the UoA administrators were able to follow the system OK.

Looking forward to REF, Leicester concerned about the amount of work involved in excluding self-citation. The Library is biding for a new post to cover both the management of the Institutional Repository, and to be a expert on bibliometrics.

Case Study 3: University of Southampton

Leslie Carr from Southampton presenting… (at last, some nice looking slides – nothing startling – but elegant and clean and a nice font!)

Leslie saying that ongoing research evaluation isn’t peculiar to HEFCE – essentially it is something that all those involved in research do – asking Why am I doing this research? Why are we funding you to do this research? etc.

The answer? Leslie suggests – because it’s important, it has impact, it is changing the world, it is saving lives, it is changing policy, it is saving money etc. – the point Leslie makes is that beyond the ‘measures’, there is a ‘story’ behind a piece of research.

At Southampton, they have used the research/institutional repository to generate data for the RAE.

• A repository helps an institution to collect an maintain its data, its impact measures and its story
• The library is a key agency in helping to collect and curate the research outputs, analyses and interpretation

(at this point Leslie’s mobile goes off!)

OK – back to the presentation

The RAE at Southampton made use of inter-service expertise – the management group drew from expertise across the University.

They used the repository to

• Store bibliographic detail of research outputs
• Store scanned PDFs (with appropriate levels of access to lock down RAE material)
• Store information to help write RA5 (don’t know what this is, probably should)
• Store RA2 (don’t know what this is either) descriptions for submission
• Output select and de-selection functionality
• Reporting functionality

The software they used was based on the JISC IRRA project, with additional development (1fte plus a 6 month secondment).

The repository collected the evidence, and handled as much Quality Assurance as possible, this evidence was then handed over to another system to add in finance data etc.

Again, had to bring together a number of existing databases in their ‘Schools’. Some were sophisticated – some were not… varying amounts of effort to import records and bring up to a reasonable standard of quality.

Core staffing was 1 repository editorial manager, 0.6 editorial assistant, 3 staff contributing a few hours a week, and at peak 7fte additional temp staff – this additional effort varied over 18 months.

They had issues with

• Recruiting and retaining staff with relevant experience for short term spells
• Staff training and supervision
• Dealing with high volume of legacy data (next time with ongoing use of the repository this should be less of an issue)
• Verification of publication status

There was a tension between the HEFCE preference for the DOI, and the remit of the Southampton respository storing PDFs of full-text. However, it was a good route to get researchers engaged with the repository, they got lots of feedback, and won a reputation for reliability of support. There were no RAE specifi
c posts – rather they were repository posts.

This involvement raised the profile for services like Library and Planning, giving them a voice in ongoing discussions about research management – broadly, not just REF focused.

Future issues:

• Liaise earlier to lobby HEFCE about any difficult requirements e.g. month for journal articles
• Some over-reliance on individual staff expertise to be addressed (certainly from the library perspective, we had this issue at Imperial as well)
• Further development of the research management and open access roles of the repository in the REF era
• Further development of the research management system to pull data together from core databases, provide analytical tools and re-presentation options

Wow – this morning has been a real race through the issues – now the Q and A session, if my battery survives…

Q: (Charles Oppenheim, Loughborough) A lot of misunderstanding of citation analysis – Charles has a particular question about ‘self-citation’ – why exclude self citation, as evidence indicates that this makes very little difference to outcome.

A: Leiden CTWS use self-citation as one of their indicators. Self citation tends to drop off quickly over time. They have found that in some cases 50-60% of citation is self-citation – some groups over self-citate. From HEFCE point of view, they don’t want to fund ‘self-citation’ – if there is any direct opportunity to ‘game play’ the system, then self-citation is the most obvious area that can be abused. However, HEFCE happy to look at again in the next set of work. Linda indicates that at a high level of aggregation, she doesn’t beleive it has much impact.

Jonathan indicating that it is perhaps dangerous to cut out self-citation, as cutting edge research groups are bound to be citing themselves – and that a message from HEFCE that says ‘self-citation is bad’ (I noted Graeme from HEFCE shaking his head at this phrase!), you start to really mess around with the building blocks which the concept of bibliometrics is built on (being quite scathing about HEFCE here…)

Q: Leiden have looked at links between age and any bias in bibliometrics (and found there was no link). However, shouldn’t they have been looking at ‘career age’ not physical age?

A: Yes. But there was a high correlation in the group they looked at. As part of pilot HEFCE want to link the researchers submitted to their HESA record so they can look at this as a factor.

Q: Is there any thought of how to reward sharing and re-analysis of data sets?

A: James, from Thomson, issue of data sets is at very early stage of development with publishers – should they be peer reviewed? How could they be cited? etc. Important, but no answers yet, and a long way to go. HEFCE – the question is what’s really important to the subject community – need to balance complexity vs

Q: (from THE) Why did HEFCE use Web of Science over Scopus, and how did they reach that decision

A: When HEFCE first looked at the issue, consultancy advised that Web of Science was the only real option. However, now further work has suggested that Scopus is viable as well. HEFCE hasn’t made a decision yet. They may possibly look at both for the pilot, and that would inform what was used in the REF itself (which could be one or the other, or both)

Jonathan noting that recent (significant) changes in the THE global rankings were linked to switch from using Web of Science to Scopus, and we might anticipate a similar effect on REF.

Q: What software was used in Australia for bringing together data mentioned in Linda’s presentation (staff data, student data, publications data)

A: No direct answer – but 37 universities in Australia have been looking at these issues – go talk to them!

Q: Question about what bibliometrics actual measure, and a comment to say that we could expect to see increased citations as a result of a bibliometrics based exercise (mentioning some specific things, e.g. negative citations – citing work to show it’s wrong)

A: Negative citations – not a big impact, only happens in very specific areas (e.g. cold fusion) – peer review and journal system tends to filter out bad papers, so issue doesn’t arise. However, outside science more of an issue, as you get different ‘schools of thoughts’ who simply disagree with each other – so more likelihood that you would cite something you didn’t see as ‘excellent’.

Re: Increased citations – there is a general trend towards increased citation. But you don’t have a lot of control over how this happens – so very difficult to coordinate an exercise to affect the outcome.

I personally think this misses the point a bit – the issue is not that people will actually coordinate to effect outcome, but they will act in a way that they think will affect the outcome – obviously I don’t know if this is going to be significant, but it seems to me it is likely to change the rules of the game to some extent – which comes back to the point that Jonathan made from the Banking Industry example – such measures are only relevant for a certain time frame before the action of those you are measuring renders them invalid.

Out of battery now…

Wow – this morning has been a real race through the issues – now the Q and A session, if my battery survives…

Q: (Charles Oppenheim, Loughborough) A lot of misunderstanding of citation analysis – Charles has a particular question about ‘self-citation’ – why exclude self citation, as evidence indicates that this makes very little difference to outcome.

A: Leiden CTWS use self-citation as one of their indicators. Self citation tends to drop off quickly over time. They have found that in some cases 50-60% of citation is self-citation – some groups over self-citate. From HEFCE point of view, they don’t want to fund ‘self-citation’ – if there is any direct opportunity to ‘game play’ the system, then self-citation is the most obvious area that can be abused. However, HEFCE happy to look at again in the next set of work. Linda indicates that at a high level of aggregation, she doesn’t beleive it has much impact.

Jonathan indicating that it is perhaps dangerous to cut out self-citation, as cutting edge research groups are bound to be citing themselves – and that a message from HEFCE that says ‘self-citation is bad’ (I noted Graeme from HEFCE shaking his head at this phrase!), you start to really mess around with the building blocks which the concept of bibliometrics is built on (being quite scathing about HEFCE here…)

Q: Leiden have looked at links between age and any bias in bibliometrics (and found there was no link). However, shouldn’t they have been looking at ‘career age’ not physical age?

A: Yes. But there was a high correlation in the group they looked at. As part of pilot HEFCE want to link the researchers submitted to their HESA record so they can look at this as a factor.

Q: Is there any thought of how to reward sharing and re-analysis of data sets?

A: James, from Thomson, issue of data sets is at very early stage of development with publishers – should they be peer reviewed? How could they be cited? etc. Important, but no answers yet, and a long way to go. HEFCE – the question is what’s really important to the subject community – need to balance complexity vs

Q: (from THE) Why did HEFCE use Web of Science over Scopus, and how did they reach that decision

A: When HEFCE first looked at the issue, consultancy advised that Web of Science was the only real option. However, now further work has suggested that Scopus is viable as well. HEFCE hasn’t made a decision yet. They may possibly look at both for the pilot, and that would inform what was used in the REF itself (which could be one or the other, or both)

Jonathan noting that recent (significant) changes in the THE global rankings were linked to switch from using Web of Science to Scopus, and we might anticipate a similar effect on REF.

Q: What software was used in Australia for bringing together data mentioned in Linda’s presentation (staff data, student data, publications data)

A: No direct answer – but 37 universities in Australia have been looking at these issues – go talk to them!

Q: Question about what bibliometrics actual measure, and a comment to say that we could expect to see increased citations as a result of a bibliometrics based exercise (mentioning some specific things, e.g. negative citations – citing work to show it’s wrong)

A: Negative citations – not a big impact, only happens in very specific areas (e.g. cold fusion) – peer review and journal system tends to filter out bad papers, so issue doesn’t arise. However, outside science more of an issue, as you get different ‘schools of thoughts’ who simply disagree with each other – so more likelihood that you would cite something you didn’t see as ‘excellent’.

Re: Increased citations – there is a general trend towards increased citation. But you don’t have a lot of control over how this happens – so very difficult to coordinate an exercise to affect the outcome.

I personally think this misses the point a bit – the issue is not that people will actually coordinate to effect outcome, but they will act in a way that they think will affect the outcome – obviously I don’t know if this is going to be significant, but it seems to me it is likely to change the rules of the game to some extent – which comes back to the point that Jonathan made from the Banking Industry example – such measures are only relevant for a certain time frame before the action of those you are measuring renders them invalid.

Out of battery now…

This by Linda Butler, from the Research Evaluation and Policy Project in Australia.

In Australia:

• 1990+ Research Quantum (RQ) formual funding
• 1993: Refinement of RQ formula – Universities establish publications collections (which were audited several times – although early audits found error rates of up to 60%)
• 2004: Research Quality Framework (RQF) announced
• 2006: Details of RQF process start to emerge – with metrics taking a leading role
  • Most Universities refine their MIS
  • Many import Thomson data directly into their publications databases
  • Link publication databases to those covering staff, student and grants data (either via proprietary software or through ‘home grown’ systems – that latter being rare, and seen as risky by Linda)
• 2007:  The RQF abandoned by new government, being replaced by new metrics based system – going in a very similar direction to the REF, and Linda believes they will be very similar

The data that is available already in the system that might be used for metrics in Australia is:

• Research Income – amount and source (but done at very broad level of aggregation)
• Publication counts – books, chapters, journal articles, conference papers
• Formulas also use separately collected Research student data – number of students; completions

Recommended RQF metrics were:

• Publication outlets ranked into tiers – journals, publishers, conferences, venues (for performing arts)
  • Journal ranking process has been ongoing for 6 months

I would have thought this really makes the likelihood of ‘game playing’ very high – already we can see the dominance of certain publications in some areas, and this will surely just reinforce this?

• External income – differentiate between types of income
  • However, this was dropped as universities could not supply the data at an appropriate level of detail
• Citation data – 1. Citations per publication; 2. Distribution across percentiles
  • standard methodology for disciplines with >50% coverage – include most science (inc. Maths) and Engineering asked to be included, although the coverage was closer to 40%

Linda is saying that the methodology chose directly affects costs (to the universities) and the transparency of the process – the type of thing you need to look at is:

• Level of aggregation – Institution, or discipline, or group (in order of ascending difficulty)
• Who a university can claim – on staff at census date (complex, time consuming), university appears on publication (straightforward)
• Who a university can submit – all staff (low cost), a subset of ‘research active staff’ (more complex and expensive
• Complexity of measures – treatment of collaborative papers, self-citations, etc.

What is the degree of risk that different methodologies will produce different outcomes?

Linda is expressing her own opinion here – only some of this currently backed up with empirical data. Talking about a high level of aggregation only – University or Discipline level:

• Level of aggregation – not much impact
• Who a university can claim – moderate impact – mostly not much impact, but perhaps an impact for a small number of universities. Generally you will gain some and lose some researchers
• Who a University can submit – low to moderate (depending on funding algorithm)
• Complexity of measures – low (this assessment has some empirical data behind it)

Essentially Linda seems to be saying – you may as well chose the cheapest approach, as she doesn’t believe it will impact significantly.

Linda draws the following lessons for the REF:

• The development of institutional information systems is complex and takes a long time
• Choice between simplicity and complexity has huge implication for the costs of the exercise
• Often those lobbying for a particular methodology have no real understanding of the cost implication of their preferred choice

Those ‘at the coalface’ need to communicate to others in the institution the impact of what they are asking for in terms of cost to the institution.

This session by Jonathan Adams (from Evidence Ltd)

Evidence Ltd have been working in this area for some years. Evidence Ltd believe that the UK’s international research status has increased over the last 20 years, and the REF needs to support this.

Jonathan is showing a graph of output from various European countries which shows that over the last 20 years, output from each country has been reasonably stable, except the Netherlands, which has experienced a large growth. He believes we need to avoid a system that leads to this disproportionate change (and is saying that the adoption of bibliometrics in the Netherlands caused their increased output)

Jonathan raising questions – will the introduction of metrics lead to manipulation, that invalidate the assumptions on which metrics have been judged and introduced.

Jonathan making the point that the recent DIUS announcement on peer review does not mean metrics are not to be used – in fact, any peer review in the REF must use the metrics – the panels will not be able sensibly to disregard what the metrics say.

Measurement of Impact (normalised) 1996-2000 shows high correlation to RAE2001 grade – however, he says there is a huge variation is residual data (I don’t understand what he means here).

One of the key questions (says Jonathan) is whether we assess total activity or selected papers – this makes a significant difference to the outcome of metrics you apply (although I would have thought the same was bound to be true of any measure of quality – allowing ‘nomination’ of papers is a filter for quality surely?)

Jonathan making the point (that Anthony did as well), that averages do not describe underlying profiles – the data is highly skewed (lots of low values, few very high values). So – we need to do profiling for different disciplines – we cannot rely on averages, as the data is so skewed. Comparing distributions is much better than averages, which is what these profiles are meant to allow us to do.

Another issue – there are many papers published that are never cited – we need to be clear how these will be dealt with.

It is clear that the Normalisation strategy adopted will have a significant impact on the outcome – Jonathan showing how three different normalisation strategies can change the outcome.

Any clustering model needs to fit a UK research model – need to be sure that a discipline is reviewed alongside similar disciplines from the bibliometric viewpoint. For example, Chemical engineering publications from the UK are published in physics journals – so need to think about it in these terms, not just lump it in with the rest of chemistry when deciding how to normalise etc.

Evidence Ltd have produced a RAE/REF Predictor (‘You are the REF’) which allows anyone (? or anyone from a UK HEI?) to decide on different weighting factors etc., and see what happens – this is apparently going to be available soon at http://RAE2008.com

Goodhart’s Law – once you use a metric for matters of public policy, it loses its effectiveness overtime (based on experience in the banking sector)

As This talk by Anthony van Raan from the University of Leiden (who was a consultant to HEFCE on the use of bibliometrics I think)

Anthony is at the Centre for Science and Technology Studies (CWTS), who have been working on bibliometrics for many years. CWTS have a licence agreement with Thomson for the use of Web of Science raw data, but has also created its own bibliometric data system using algorithms.

Apparently citation isn’t as complicated as quantum mechanics! That’s a relief…

What do citations measure:

• There is a positive correlation between citation based studies and qualitaty judgements by peers

Anthony is going through this at quite a pace, so these notes may be sketchy…

There is a set of total publications, and various systems list subsets of this ‘universe’ – Web of Science is a big subset, but also LNCS, Medline, arXiv. Now Scopus is another big subset, and CWTS are now comparing Web of Science to Scopus.

Anthony is now going of key indicators and what they help measure – however, he is going through it so fast I can’t keep up – it might not be quantum mechanics, but as this speed I can’t capture it all.

As Web of Science covers (as you might expect) mainly science – and it is the ‘science’ based disciplines that use journal publication as the major route for scholarly communication. In Humanities and Arts, books are used more, and CWTS is experimenting with a new kind of ranking – which measures if ‘your’ book is available in the top 250 university libraries (he doesn’t say how top 250 university libraries are measured) – if this gets adopted, will we start to see an increased demand for libraries to buy specific books to increase ranking?

Anthony demonstrating how Chemistry researchers publish across journals from many areas (e.g. Rheumatology, Cell Biology, Applied Physics) – so even an established discipline like Chemistry turns out to be very interdisciplinary – at least when it comes to publication.

Anthony indicating that bibliometric indicators can be manipulated and/or they discriminate certain persons/types of research – but establishing evidence for some of this needs work – for example CWTS have found that the assertion that bibliometrics discriminate against younger academics is not supported by the evidence. However, they have found that journal editors behave in such a way that maniuplates impact factor (CWTS published an article on this in the Journal of Documentation)

The session is by James Pringle from Thomson Scientific (now Thomson Reuters) (a.k.a. ISI)

I have to admit to some slight cynicism around Thomson’s involvement, as they clearly have one of the richest sets of citation based data, and so I feel that they have an innately biased approach – however, James has said he is speaking (as far as he can) not as a vendor in this context (but he’s hardly likely to say bibliometrics aren’t very helpful is he?)

James is indicating that Thomson believe that citation data need to be used wisely – to inform discussion rather than to replace discussions.

James is describing citations as “User-generated content” of formal scholarly communications (ok – but using a buzzword here – but all the published papers are ‘user-generated’ as well. I would think it would be more accurate to regard it as the ‘link’ of scholarly communications, which makes using it to generate bibliometrics akin to Google’s pagerank)

However, although citation-based evaluation is fundamental to effective evaluation and management, say James, it is frequently mis-used and misunderstood in an era of democratisation – the sector needs ‘best practices’ to avoid this (apparently  “Using bibliometrics in evaluating research” by David Pendlebury lists 10 rules for using citation-based evaluation). The availability of citation information leads researchers to make their own assessments – which may not be accurate – or at the least, not uniform. It also has lead to researchers submitting ‘corrections’ to Thomson, which challenges them to respond quickly and provide more accurate information.

James outlines how use is made of citation data for research evaluation at the level of Individuals, University Departments, University Management and External Entities (such as funding bodies). These different levels have different needs and requirements, so we need to producing citation data that can serve needs across this range.

Thomson have looked at Institutional Research Evaluation workflows:

• Identification of researchers and their work
• Data validation and metadata management
• Populating local systems with ongoing updates
• Enabling access to data via internal systems and respositories
• Data and metrics to create reports for external agencies

These are not unique to the UK – they are being tackled by institutions across the globe, although the detail is effected by national policies etc.

So, how do Thomson support research evaluation – they have a number of principals:

• First, do no harm – encourage and develop use of best practices
• Integrate and partner – work with institutions to define and develop integrated solutions
• Enhance content to meet emerging needs
  • Right Journal content
    • Careful increase journals in Web of Science
  • Recognize expanding scholarly communications
    • Integrate proceedings citation content into Web of Science
  • Engage researcher community
    • ResearcherID.com and Author disambiguation
  • Engage Libraries and Research Management
    • Institutional Identifiers and Institutional Disambiguation/Unification

All this sounds OK – but I’m worried about how it joins up with activity in the sector, and by other organisations. How does ‘ResearcherID.com’ link to OCLC Identities work? It would be great to see some joined up thinking across the library/information sector on this, as otherwise we will end up with multiple methods of identification.

Today I’m attending the event “Beyond the RAE 2008: Bibliometrics, League Tables and the REF” at Kings. The event is sponsored by Thomson Scientific (who produce the Web of Knowledge and Citation indexes).

Irritatingly the Eduroam service which should allow me to join a wireless network here isn’t playing nice, and won’t let me on (I’ve had this problem at Kings before, but neglected to follow it up – now I wish I had).

The day is being opened with an introduction from Karen Stanton (Chief Information Officer and College Librarian), outlining the importance of bibliometric data, and how much is at stake for universities in terms of future funding depending on the outcome of the REF.

First up, Graeme Rosenberg from HEFCE is providing an update on the REF – following the recent announcements indicate a step back from the use of bibliometrics, this should be interesting:

He starts off by saying it may well be too early to answer the key questions the audience will have. However, they are expecting to publish some further details in May. However, the story so far is:

• In 2006, HEFCE were asked to develop a new framework, with a larger emphasis on bibliometrics
• HEFEC commissioned expert advice on the use of bibliometrics
• Consultation outcomes are now available on the HEFCE website at http://www.hefce.ac.uk/research/assessment/reform/

The main outcomes of the consultation are:

• A more unified framework, with assessment for all subjects, including some combination of metrics-based indicators, including bibliometrics where appropriate, as well as input from expert panels
• The timetable to design this more flexible framework is 12 months

What happens next?

• An early start on a pilot of the bibliometric indicator
• An early start to dialogue with the sector on developing light touch peer review and on choosing and using other metrics
• Further work on a number of other issues raised in the consultations
• Remain committed to full implementation of a new framework by 2014, with funding based on some aspects of the new framework to be gradually phased in from 2011-12

There are a number of issues that need to be resolved in the development of the bibliometric indicator:

• Coverage (which disciplines? all or selected staff? Do papers follow the author?)
• Technical and methodological issues (defining fields for normalisation, multi-authorship, self-citation, etc.)
• The use of bibliometric indicators within the REF
• Burden and other implications for the sector
• HEFCE will test issues through the pilot, then consult further and take decision during 2009

The plan is to have a bibliometrics exercise to happen in 2010.

The consultation essentially found that most responses felt bibliometrics were indicative of quality, but that there should be some input (often the word used was ‘moderation’) by expert panels.

The bibliometrics pilot should occur as follows:

• May-Jun 2008 – select HEIs/contractors
• Aug-Oct 2008 – Data collection
• Nov 2008 – early 2009 – Data analysis
• Spring 2009 – Pilot results

HEFCE are currently in contact with institutions who have shown interest in being part of the pilot. They are suggesting that the pilot is based on data submitted for the 2008 RAE as well as new material not covered in that submission.

The pilot will cover a broader range of disciplines – all where there is a reasonable amount of citation data available. However, they do not expect all disciplines to use bibliometrics in the actual REF – the pilot will establish some boundaries.