Application-centric peer review is the matching of the most suitable peer reviewers to each application. In this way, each application can have a unique best-fit set of reviewers. It is used by NHMRC’s Investigator, Ideas and Synergy Grant schemes.
Why application-centric peer review?
Placing grant applications at the heart of the peer review process, the shift to application-centric peer review has contributed to NHMRC improving the independence and matching of expertise for peer review while retaining a balance between the rigour of review and burden on reviewers.
- Discover how application-centric peer review has benefited the Investigator Grants Scheme by viewing our infographic (available under Downloads below).
The application-centric peer review process
- Application matching:
- Research classification data from the peer reviewer’s Sapphire profile, such as fields of research, peer review areas and research keywords, are cross-referenced with the same data submitted by applicants for their research proposal, to generate a bespoke group of applications that best match a peer reviewer’s expertise, for the purpose of seeking conflicts of interest (CoI) and suitability declarations from a larger group of potential peer reviewers.
- CoI and suitability declarations:
- Potential reviewers self-assess their suitability to review an application and declare any CoIs based on comprehensive guidance. NHMRC examines declarations where peer reviewers seek policy clarification and where acceptable suitability is identified.
- Application allocation for peer review:
- Declarations inform the allocation of applications to the most suitable and non-conflicted peer reviewers. The allocation process aims to minimise workload while providing peer reviewers with an opportunity to benchmark their own approach to scoring to support consistency of assessments.
- Application assessment:
- NHMRC draws on the full pool of reviewers to allocate suitable reviewers for each application. Peer reviewers are instructed to undertake independent assessments and give constructive feedback that is provided verbatim to the applicant. Reviewers are supported by NHMRC secretariat staff, extensive documentation, a live briefing with Q&A and access to experienced peer reviewers who are available to provide guidance throughout the peer review period.
Development of application-centric peer review
In the past, our larger schemes relied on using ‘superpanels’ to group peer reviewers with groups of applications based on related research topics, then splitting them into multiple smaller grant review panels whose members met in person to assess applications.
In 2017–18, we undertook an extensive national consultation to inform the peer review processes for our then-new Investigator, Synergy, Ideas and Clinical Trials and Cohort Studies Grant schemes. Several themes emerged from this consultation including:
- preference for a larger number of independent assessments (minimum 4 or 5 per application)
- concerns that some panels did not have appropriate expertise and may have been overly reliant on an individual reviewer’s assessment
- concerns that applications were commonly reviewed by fewer than the target number of reviewers due to conflicts of interest
- the importance of feedback to applicants.
Following this public consultation, NHMRC has continued to consider ways to streamline and improve the quality of its peer review processes while striking the appropriate balance between rigour and burden. Feedback from the sector and the advice of NHMRC’s Research Committee and other advisory groups are instrumental in improving NHMRC peer review processes.
The COVID-19 pandemic necessitated innovative approaches to peer review, as lockdowns, illness, increased caring responsibilities and changed professional roles (such as front-line clinical and public health activities) limited the availability of peer reviewers to meet for panel discussions.
Analysis of scoring changes and outcomes of panel discussions from the 2019 Ideas Grant round, combined with data from published studies, showed that panel discussion does not significantly alter outcomes compared to independent evaluations. This prompted NHMRC to trial a hybrid application-centric peer review process for the 2020 Ideas Grant round, allowing the Ideas Grant scheme to save months in the process and deliver outcomes before the end of 2020.
Positive reviewer feedback from the trial and NHMRC analysis led to it being implemented for the 2021 rounds of Investigator, Synergy and Ideas Grants. In doing so, application-centric peer review has:
- made participating in peer review more flexible and inclusive by removing the need for peer reviewers to meet, allowing researchers to participate where they have caring/family commitments or other constraints on their ability to travel or spend large amounts of time in panel meetings
- reduced the average number of conflicts of interest and suitability declarations required of potential peer reviewers by pre-matching applications to a peer reviewer’s expertise
- reduced the average number of applications allocated to peer reviewers for full assessment, while ensuring a minimum number is allocated to improve the quality and consistency of assessments
- provided more flexibility in matching peer reviewers to applications to maximise the proportion of applications that receive the target of 5 assessments
- improved matching of applications to peer reviewer expertise, as evidenced by peer review survey results.
Transparency and accountability
Several steps have been introduced to improve transparency and accountability of application-centric peer review.
- Investigator, Ideas and Synergy Grant schemes now require that peer reviewers provide feedback to applicants about the strengths and weaknesses of the application which will be useful for future applications.
- Peer reviewer feedback comments are shared anonymously between reviewers of the same application (for Investigator and Ideas Grants) to increase transparency and accountability of the peer review process and allow peer reviewers to compare their assessments with those of other reviewers. Transcription errors may be identified through this process.
- NHMRC undertakes outlier screening to identify reviews where scores differ substantially from those of other reviewers, or where it is apparent comments do not align with scores. This may involve NHMRC contacting peer reviewers to clarify their assessments, noting that it is common and expected that peer reviewers will vary in their scoring and justifications for their conclusions. Further errors may be identified through this process.
Future developments
Substantial improvements in matching of peer reviewers to applications have been made over the last 2 years. With higher quality and complete data from applicants and peer reviewers and new digital tools, further improvements are expected.
Application-centric peer review is proving effective and efficient for NHMRC’s largest schemes. However, a panel-based approach will continue to be used for peer review of NHMRC’s smaller and targeted grant opportunities where good matching of groups of applications to panels can be achieved.
We recognise that many peer reviewers appreciated the opportunities that panel meetings in our larger schemes provided for in-person mentoring and professional development. We continue to refine and expand training and guidance opportunities for those involved in application-centric peer review, and to consider feedback about our schemes.
NHMRC acknowledges the substantial time and effort researchers commit to applying for and reviewing grants and the support their administering institutions provide. We will continue to explore ways to improve and strengthen NHMRC’s peer review processes, based on advice from peer reviewers, grant applicants and our advisory committees, and the experiences of other national and international funding agencies, to support the work of the health and medical research sector.
