The following data has been obtained from the 2019-2022 Postgraduate Scholarships (PGS) post-implementation review (PIR) surveys of peer reviewers. This document summarises key trends in these findings and details areas for improvement to better shape PGS peer review practices in the future.

Publication Data

Principles of peer review

Peer review is the impartial and independent assessment of research by others working in the same or a related field.

NHMRC’s Principles of peer review are high-level guiding statements that underpin NHMRC’s peer review processes and apply to all NHMRC funding schemes.

These principles include:

  • peer review processes are fair and seen to be fair by all
  • transparency applies to all stages of peer review
  • peer reviewers provide independent advice
  • peer review processes should achieve the highest-quality decision making.

Figure 1 shows, for each of the years from 2019 to 2022, the percentage of surveyed PGS peer reviewers who Agreed or Strongly Agreed that PGS peer review was consistent with these principles.

On average across all years, 83% of respondents Agreed or Strongly Agreed that PGS peer review processes were fair, transparent, independent and high quality.

Though, in 2022, some respondents felt that lack of visibility of other peer reviewer scores affected the transparency of the process.

Figure 1. Proportion of surveyed PGS peer reviewers who Agreed or Strongly Agreed that PGS peer review was consistent with NHMRC’s principles of peer review, by principle by year (2019–2022) (Figure description available at Table 1).
 

Column chart detailing proportion of surveyed PGS peer reviewers who Agreed or Strongly Agreed that PGS peer review was consistent with NHMRC’s principles of peer review, by principle by year (2019–2022). Refer to the figure description on this page for details.

Peer reviewer gender

NHMRC’s Guiding principles for peer reviewer nominations and appointments include balanced representation of gender as a goal. Figure 2 shows numbers of peer reviewers for the 2019 to 2022 PGS rounds by selected gender. On average across all rounds, 49.25% of peer reviewers selected female and 50.75% selected male.*

*Only these two gender selection options were available for these rounds. NHMRC expanded its gender selection options from 2023.

Figure 2. Number of peer reviewers by PGS round (2019–2022) by selected gender (Figure description available at Table 2).
 

Column graph shows numbers of peer reviewers for the 2019 to 2022 PGS rounds by selected gender. Refer to the figure description on this page for details.

Time spent reviewing applications

A PGS application includes a Report (which contains track record and application specific information), a Supervisor Report and an academic transcript.

Across the period 2019–2022, PGS peer reviewers spent an average of 1-2 hours assessing each application (refer to Figure 3).

Figure 3. Proportion of PGS peer reviewers by funding round (2019–2022) by time spent assessing each application (Figure description available at Table 3).

Column graph showing across the period 2019–2022, PGS peer reviewers spent an average of 1-2 hours assessing each application. Refer to the figure description on this page for details.

Assessment process

In 2022, an average of 85% of Peer reviewers Agreed or Strongly Agreed that they were comfortable with key elements of the PGS assessment process (refer to Figure 4).

Figure 4. Proportion of Peer reviewers that Agreed or Strongly Agreed that they were comfortable with key elements of the PGS assessment process, by year (2021–2022) by element (Figure description available at Table 4).
 

Column chart detailing proportion of Peer reviewers that Agreed or Strongly Agreed that they were comfortable with key elements of the PGS assessment process, by year (2021–2022). Refer to the figure description on this page for details.

Areas identified for improvement

The following areas for improvement were identified in the 2022 PGS round.

Stronger advice on Journal Impact Factors (JIFs)

Peer reviewers noted that some applicants included JIFs in the publication section of their application.

NHMRC is a signatory of the San Francisco Declaration on Research Assessment (DORA) which notes that it is not appropriate to use journal-based metrics such as JIFs as a surrogate measure of the quality of individual research articles when assessing an individual scientist’s contributions. The scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.

For 2023 and future PGS rounds, wording in the Grant Opportunity Guidelines will be updated to strengthen advice against inclusion of JIFs within applications.

Continued Sapphire improvements

Feedback from peer reviewers suggests that further improvements in accessing the Supervisor Reports in Sapphire will strengthen the PGS peer review experience.

Further information

Note

For the 4 years covered by this report, respondent numbers to the PGS PIR survey as compared with the total number of survey recipients were:

  • 22 of 42 (2019)
  • 19 of 42 (2020)
  • 19 of 35 (2021)
  • 16 of 35 (2022).

Figure descriptions

Figure 1

Table 1. Proportion of surveyed PGS peer reviewers who Agreed or Strongly Agreed that PGS peer review was consistent with NHMRC’s principles of peer review, by principle by year (2019–2022)
Principle 2019 2020 2021 2022
Fair 82% 90% 68% 81.3%
Transparent 91% 74% 74% 62.5%
Independent 95.5% 95% 100% 100%
High quality 81.8% 85% 79% 68.8%

Figure 2

Table 2. Number of peer reviewers by PGS round (2019–2022) by selected gender
Round Female Male
2019 50 50
2020 46 54
2021 46 54
2022 55 45

Figure 3

Table 3. Proportion of PGS peer reviewers by funding round (2019–2022) by time spent assessing each application
Round <1 hour 1–2 hours >2 hours
2019 36.4% 50% 13.6%
2020 36.9% 52.6% 10.4%
2021 36.9% 42.1% 20.9%
2022 25% 62.5% 12.6%

Figure 4

4. Proportion of Peer reviewers that Agreed or Strongly Agreed that they were comfortable with key elements of the PGS assessment process, by year (2021–2022) by element
Assessment process element 2021 2022
I had sufficient time to adequately complete my conflict of interest declarations 100.0% 87.5%
Overall, I had sufficient time to review the applications I was assigned 94.7% 75.0%
The information supplied in applications was sufficient to perform an assessment 89.5% 93.8%
The Category Descriptors were an effective tool in helping me complete my assessments 100.0% 93.8%
The workload involved in performing peer review for the PGS scheme was reasonable 68.4% 75.0%