This by Linda Butler, from the Research Evaluation and Policy Project in Australia.

In Australia:

• 1990+ Research Quantum (RQ) formual funding
• 1993: Refinement of RQ formula – Universities establish publications collections (which were audited several times – although early audits found error rates of up to 60%)
• 2004: Research Quality Framework (RQF) announced
• 2006: Details of RQF process start to emerge – with metrics taking a leading role
  • Most Universities refine their MIS
  • Many import Thomson data directly into their publications databases
  • Link publication databases to those covering staff, student and grants data (either via proprietary software or through ‘home grown’ systems – that latter being rare, and seen as risky by Linda)
• 2007:  The RQF abandoned by new government, being replaced by new metrics based system – going in a very similar direction to the REF, and Linda believes they will be very similar

The data that is available already in the system that might be used for metrics in Australia is:

• Research Income – amount and source (but done at very broad level of aggregation)
• Publication counts – books, chapters, journal articles, conference papers
• Formulas also use separately collected Research student data – number of students; completions

Recommended RQF metrics were:

• Publication outlets ranked into tiers – journals, publishers, conferences, venues (for performing arts)
  • Journal ranking process has been ongoing for 6 months

I would have thought this really makes the likelihood of ‘game playing’ very high – already we can see the dominance of certain publications in some areas, and this will surely just reinforce this?

• External income – differentiate between types of income
  • However, this was dropped as universities could not supply the data at an appropriate level of detail
• Citation data – 1. Citations per publication; 2. Distribution across percentiles
  • standard methodology for disciplines with >50% coverage – include most science (inc. Maths) and Engineering asked to be included, although the coverage was closer to 40%

Linda is saying that the methodology chose directly affects costs (to the universities) and the transparency of the process – the type of thing you need to look at is:

• Level of aggregation – Institution, or discipline, or group (in order of ascending difficulty)
• Who a university can claim – on staff at census date (complex, time consuming), university appears on publication (straightforward)
• Who a university can submit – all staff (low cost), a subset of ‘research active staff’ (more complex and expensive
• Complexity of measures – treatment of collaborative papers, self-citations, etc.

What is the degree of risk that different methodologies will produce different outcomes?

Linda is expressing her own opinion here – only some of this currently backed up with empirical data. Talking about a high level of aggregation only – University or Discipline level:

• Level of aggregation – not much impact
• Who a university can claim – moderate impact – mostly not much impact, but perhaps an impact for a small number of universities. Generally you will gain some and lose some researchers
• Who a University can submit – low to moderate (depending on funding algorithm)
• Complexity of measures – low (this assessment has some empirical data behind it)

Essentially Linda seems to be saying – you may as well chose the cheapest approach, as she doesn’t believe it will impact significantly.

Linda draws the following lessons for the REF:

• The development of institutional information systems is complex and takes a long time
• Choice between simplicity and complexity has huge implication for the costs of the exercise
• Often those lobbying for a particular methodology have no real understanding of the cost implication of their preferred choice

Those ‘at the coalface’ need to communicate to others in the institution the impact of what they are asking for in terms of cost to the institution.