This Consultation is now closed, it accepted submissions up until the 31st of October 2021.

Issue

The National Health and Medical Research Council (NHMRC) is reviewing the assessment of publications across its suite of track record reliant funding schemes, with the aim to:

  • support excellence by focussing on research quality and significance
  • increase emphasis on the quality and impact of research findings over publication numbers
  • ensure that track record assessment scores are underpinned by informed judgement on the qualityof publications
  • simplify the track record assessment of publications while maintaining rigour, transparency andequity
  • reduce burden on peer reviewers
  • align with international practice in research assessment.

Introduction

NHMRC funds outstanding health and medical research and researchers, identified in part by applicants’ previous research outputs and impact.

Assessing past research performance is important for recognising productivity, innovation and novelty, and plays a major role in determining academic success. There has been an increased movement in the academic community towards measuring quality, rather than quantity, of an individual’s research output. The most significant initiative towards this aim has been the Declaration on Research Assessment (DORA)1, which recognises the need to improve the ways in which researchers and the outputs of scholarly research are evaluated. NHMRC, along with many other international funding agencies, is a signatory of DORA.

The Leiden Manifesto for research metrics2, a best practice standard in research evaluation, also advocates that quantitative evaluation should only be used to support qualitative expert assessment. It argues that, while quantitative metrics can challenge bias tendencies in peer review and facilitate deliberation, assessors must not be tempted to cede all decision-making to numbers. Informed judgement on the local relevance of research, individual experience and expertise, and variation by field in publication and citation practice should all be taken into account by peer reviewers while making a holistic assessment of an applicant’s research and/or track record.

NHMRC’s 2019 survey of research culture in Australian NHMRC-funded institutions3 also provides some insight into the Australian health and medical research sector’s position on this issue, particularly in relation to the focus on quantity and the pressure to publish, and its consequential detrimental effect on research quality. Of those surveyed, 67% agreed that publication pressure leads some colleagues (whether intentionally or not) to cut corners; 51% indicated that they experience stress at the thought of their colleagues’ assessment of their publication output; and 49% felt that it is necessary to have a first-authored publication in a prestigious journal when seeking an academic position or promotion. Respondents also felt that this pressure negatively impacts reproducibility and transparent reporting of study design and methods.

In the Australian context, an increased emphasis on quality is particularly pertinent for Indigenous health research, as highly impactful research is often published in Australian journals with a limited and specialised audience. The application and impact of Indigenous health research also rely heavily on dissemination of knowledge to communities, which is often achieved via non-traditional publications, rather than the traditional ‘publications in highly ranked journals’ academic paradigm. As such, there is strong advocacy for collection, preservation and presentation of knowledge via an Indigenous-led open access forum4.

Comment

Current practice

The assessment of applications in many of NHMRC’s grant schemes includes consideration of the applicants’ track record. For some schemes, this includes the assessment of research outputs such as publications, commercial outcomes and patents. In other schemes, track record assessment is based on publications, leadership and research impact (as demonstrated by verifiable evidence of research outcomes). For the ‘Publications’ criterion, applicants are usually assessed based on a list of all their publications from the past 10 years (taking into account any Career Disruptions). Applicants are also
required to nominate their 5 best publications from those 10 years and provide explanations of why these publications have been selected, outlining the quality of the publications selected and their contribution to science.

Reviewing a list of all publications, in some cases more than 300 for senior researchers, is burdensome for peer reviewers and can lead to a ‘paper counting’ approach of assessment. This burden is compounded for peer reviewers assessing applications for those team-based schemes where the track record of each named Chief Investigator is assessed. A solely quantitative ‘paper counting’ approach disadvantages applicants whose productivity has been reduced due to the COVID-19 pandemic. It may also lead to bias against female researchers, who may publish fewer papers compared to their male counterparts over their career
spans5.

Although discouraged by NHMRC, many peer reviewers rely on bibliometrics to simplify the assessment process. Bibliometrics uses mathematical and statistical methods to measure scientific output and provides a quantitative, rather than a qualitative, assessment of an individual’s research performance. The most commonly used bibliometric indicators such as journal impact factor and number of citations, along with newer indicators such as h- and g-indices, all have their individual strengths and weaknesses6 but should not replace qualitative expert evaluation.

Publication assessment practices

The literature and associated expert opinion are mixed on the number of publications assessors should review to reach a rigorous and fair assessment of research quality that allows enough differentiation between applicants. Leading Australian academics and institution leaders have proposed that reviewing the top five, eight or ten publications would enable reviewers to judge the quality of an applicant’s published research7,8. Table 1 below contains a summary of how publications, or in most cases ‘research outputs’, are assessed by several other Australian and international agencies. The most significant track record-based funding schemes for each agency were explored for this analysis.

Table 1: Publication assessment practices across international and Australian funding agencies9
Agency Publication assessment practice
European Research Commission (ERC) 10,11 Early and mid-career applicants: Maximum two pages of important achievements, including most important publications (up to five for Starting Grant and up to ten for Consolidator Grant)

Senior applicants: Applicants are expected to have a proven track record of achievements in the past 10 years appropriate to their research field and at least matching one or more of the following benchmarks, for instance: up to 10 significant publications as main author
Medical Research Council (MRC) UK12,13 Applicants need to include their recent publications as a separate ‘List of Publications’ attachment, which may be a maximum of one A4 page.
Wellcome Trust 14,15,16 Applicants are asked to include up to 20 of their most significant research outputs, ensuring that at least five of these are from the last five years. For 10 of these outputs, applicants should also provide a statement describing their significance and contribution (up to 50 words per output). Applicants are also asked to enter the number of peer-reviewed publications they have authored/co-authored, including systematic reviews and meta analyses but excluding abstracts and literature reviews.
Canadian Institutes of Health Research (CIHR) 17 All publications in the past five years may be submitted by applicants.
National Institutes of Health (NIH) 18 Senior applicants: Up to five most significant contributions to science. The description of each contribution should be no longer than one half page, including citations. For each contribution, applicants may cite up to four publications or research products that are relevant to the contribution.

Graduate students and post doctorates: Same requirements as above with the following additional instruction: While all applicants may describe up to five contributions, graduate students and post doctorates may wish to consider highlighting two or three they consider most significant.
Australian Research Council (ARC)19 Applicants are asked to upload the following:
i. Publication context and contribution of no more than two A4 pages.
ii. A Publication list of no more than five A4 pages.
iii. ten career-best academic research outputs of no more than three A4 pages.
Medical Research Future Fund (MRFF)20 Applicants should include the five most significant publications over the past five years for assessment purposes.

In summary, most funding agencies:
• limit the number of top (or most significant) publications/research outputs to five or ten
• restrict the amount of material peer reviewers have to assess by providing either a page limit or a number limit for publications
• assess publications alongside other research outputs (patents, datasets, software, instruments, equipment, models, protocols, etc.).

Consultation

In formulating its proposal for track record assessment of publications, NHMRC considered its goals and rationale, advice from NHMRC’s Research Committee, international and national practices, and the following principles:

Excellence: support excellence in nationally relevant research across the health and medical research spectrum.
• Complementarity of quantitative and qualitative assessment: ensure that scores are underpinned by informed qualitative judgement
• Ensuring equity by accounting for variation:
o provide opportunities across fields with varying publication practices
o provide opportunities for talented researchers at all career stages, with different career trajectories and in different research settings
o seek to minimise both disadvantage and selective advantage
o avoid tailoring policies to exceptions
• Simplicity: ensure policies and processes are simple to understand and implement

Based on the information considered, NHMRC proposes that:

Applicants list their top 10 publications in the past 10 years (taking into account any Career Disruptions) and provide explanations of why these publications were selected, the author’s contribution to each, an outline of the quality of the publications selected and their contribution to science.
Key considerations:

  • This change is likely to result in significant reduction of peer reviewer burden and ensure increased focus on the quality of publications.
  • It will be equitable for applicants across all career stages and research fields.
  • It will align with publication assessment practices of many international and Australian funding agencies.
  • 10 publications may not provide peer reviewers with enough information to distinguish between the publication records (quality and contribution to science) of senior researchers. However, other elements of track record assessment, such as leadership and research impact, will assist with this issue.

The questions asked of organisations were:

  1. Details of the submitting organisation.
  2. Please indicate your view on NHMRC’s proposal for publication assessment using the drop-down menu below.(I support NHMRC’s proposal / I do not support NHMRC’s proposal / I am unsure about NHMRC’s proposal / I wish to suggest and alternative proposal) Provide comments below.
  3. Please provide any other comments from your organisation on the advantages, disadvantages or other consequences of NHMRC’s proposal for publication assessment?
  4. Please provide any other feedback from your organisation on NHMRC’s proposal for publication assessment? For example, will the proposal achieve the aims outlined at the start of this paper.


1 https://sfdora.org/read/
2 https://www.nature.com/articles/520429a
3 https://www.nhmrc.gov.au/research-policy/research-
4 Indigenous Voices Workshop - Open Access Week 2020 (https://aoasg.org.au/open-access-week-2020/)
5 https://doi.org/10.1371/journal.pone.0000127
6 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3338409/
7 https://doi.org/10.1038/d41586-019-00613-z
8 https://www.chiefscientist.gov.au/news-and-media/collective-actions-strengthen-research-integrity
9 Limits for pages, numbers of publications, time periods, etc. are emphasised in bold
10 https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/horizon/guidance/information-for-applicants_erc-stg_erc-cog_en.pdf
11 https://ec.europa.eu/research/participants/data/ref/h2020/other/guides_for_applicants/h2020-guide20-erc-adg_en.pdf
12https://mrc.ukri.org/documents/pdf/guidance-for-fellowship-applicants1/
13 https://mrc.ukri.org/skills-careers/fellowships/guidance-for-fellowship-applicants/3-the-application/#3.2.2
14 https://wellcome.org/grant-funding/schemes/sir-henry-wellcome-postdoctoral-fellowships
15 https://wellcome.org/grant-funding/schemes/research-career-development-fellowships
16 https://wellcome.org/grant-funding/schemes/senior-research-fellowships
17 https://cihr-irsc.gc.ca/e/38886.html
18 https://grants.nih.gov/grants/how-to-apply-application-guide/forms-e/fellowship-forms-e.pdf
19 https://www.arc.gov.au/file/6536/download?token=bou8I0cy
20 Announced and implemented in August 2021