NHMRC Evaluation Strategy Implementation Report 2024

Overview

Regular evaluation performance reporting is one of the activities set out within NHMRC's 2021 Evaluation Strategy. The present report is based on NHMRC staff responses to the 2024 Evaluation Strategy Implementation Survey along with other relevant sources.

During the reporting period, NHMRC continued to improve its evaluation practices, aligning them with government priorities and internal strategic objectives. Key accomplishments included enhanced collaboration with external partners, development of NHMRC Gender Equity Report Cards, participation in the Department of Finance’s Data Inventory Project, and progress in creating a new comprehensive grant program design and evaluation framework. Despite these achievements, some challenges remained. An assessment of NHMRC's data maturity, indicated that there was a need for a more structured, forward-thinking strategy. Limited resources and budget constraints also affected the scope and depth of evaluation activities.

Defining Evaluation-related Activities

The Australian Centre for Evaluation describes evaluation as the systematic and objective assessment of the design, implementation or results of a government program or activity for the purposes of continuous improvement, accountability and decision‑making.

Evaluation enables NHMRC to reflect on past actions to inform future progress (‘looking backward to move forward’), ensuring its policies and programs remain effective, responsive, and adaptable to Australia’s health and medical research sector.

This report provided details of NHMRC activities that support the continuous improvement and performance of NHMRC’s programs, policies, and strategic decisions. These activities encompass data collection, surveys, program reviews, performance monitoring, knowledge sharing, quality improvement initiatives, and analysis of internal processes and outcomes. NHMRC’s Evaluation Cycle (shown in Figure 1) categorises evaluation-related activities into five main themes: data collection, evaluation, communication of evaluation results, implementation of evaluation recommendations, and fostering an organisational culture of evaluation. Each activity in this report is aligned with the relevant theme(s) from the evaluation cycle for clarity and cohesiveness.

Figure 1 image description

An infographic explaining the culture of Evaluation at NHMRC and that it is made up of 4 key components:

1. Data collection
   Continual increase in the quality and comprehensiveness of the performance-related data that NHMRC collects (Flows to 2.)
2. Evaluation
   Continual increase in the quality, comprehensiveness and efficiency of NHMRC's evaluation efforts (Flows to 3.)
3. Implementation
   Continual improvement to NHMRC's activities, through implementation of evaluation recommendations Communication (Flows to 4.)
4. Communication
   Continual increase in NHMRC's reporting and evaluation promotion (Final segment.)

Grant Program

Centres of Research Excellence

Post-implementation Review – Centres of Research Excellence scheme 

(Evaluation)

In June 2024, NHMRC conducted a peer reviewer survey to gather feedback on the 2024 Centres of Research Excellence (CRE) grant opportunity. This feedback, along with input from Research Administration Officers (RAOs), panel members, panel Chairs and Community Observers throughout the peer review process informed the development of the 2024 CRE Post-implementation review (PIR). Based on recommendations from the PIR, the following changes were implemented for the 2025 CRE grant round:

• Updated CRE score descriptors to include assessment of the alignment between the objectives 
  of the proposed research and the objectives of the CRE scheme and application stream
• Grant proposal requirements were streamlined to avoid duplication of information and reduce 
  burden on applicants and peer reviewers
• Updated guidance for applicants and peer reviewers

Equipment Grants

Grant Opportunity Guidelines Evaluation

(Evaluation)

NHMRC has evaluated its Equipment Grants scheme for funding utilisation and appropriateness. A survey was distributed to RAOs in November 2023 as part of the evaluation process, requesting feedback on how their Administrating Institution (AI) allocated Equipment Grant funding from 2020 to 2022. 

Following this survey, NHMRC revised the Equipment Grant guidelines for the 2024 grant opportunity to further align with government policies, increase equity, and allow more flexibility on how the funding can be spent. To support sustainability in line with the Australian Government’s Net Zero Plan, Administering Institutions are now able to use funding for ongoing service, upgrades, maintenance and repair costs for equipment used on NHMRC funded Research Activities. Institutions can also attribute grant funds to charges associated with access to data, data processing or data analytics services. 

There was also a change to the grant amount offered under the 2024 Equipment Grant opportunity with grants now being awarded to a maximum of $300,000. This provided a more equitable approach to support institutions managing NHMRC grants, allowing more funding to smaller institutions to invest in larger pieces of equipment or improve sustainability through covering maintenance costs for a longer duration. Grant funds continue to be encouraged to support Australian made products.

Ideas Grants

Post-implementation Review – Ideas Grant Scheme

(Evaluation)

NHMRC conducted a peer reviewer survey in September 2023 to gather feedback on the peer review process for the 2023 Ideas Grant opportunity. Based on the survey results and additional insights from other peer reviewers, as well as the PIR of the scheme, the following recommendations were implemented to enhance the peer review processes:

• The Allocation of Research Experts to Applications (ARETA) and Allocation of Grant Applications To Assessors (AGATA) tools were used to improve suitable matching between peer reviewers and applicants for the Conflict of Interest and Suitability Survey
• Peer reviewer training was further refined by incorporating more examples of suitable feedback 
  for comments and conducting more interactive sessions with peer review mentors
• Enhanced communication was undertaken with Research Administration Officers to inform 
  them about the number of participants from their Administering Institution involved in the peer 
  review process.
• A more detailed timeline of tasks was provided for peer reviewers upfront, emphasising the importance of their availability throughout the entire peer review period. This facilitated better planning of their workloads

Investigator/Synergy Grants

Peer Review Survey

(Evaluation, Implementation)

NHMRC surveyed applicants and peer reviewers about the 2023 Investigator and Synergy Grant peer review processes. Based on survey feedback, the following changes were implemented to improve peer review processes:

• More detailed feedback was provided to unsuccessful Stage Two applicants in the 2024 Synergy Grant round
• Data quality was improved for the Research Impact criteria in the 2025 Investigator Grant application form
• Feasibility studies were conducted on sharing applicant feedback among Stage One peer reviewers and introducing career overviews in Sapphire for the 2025 Synergy Grant round.

Assessment and Eligibility Framework Review

(Evaluation)

Evaluation of the assessment and eligibility framework for Investigator and Synergy Grant schemes> is ongoing.

In February 2024, a working group was formed to advise on updates to research impact criteria 
and score descriptors. With advice from the working group, a public consultation was opened in 
November 2024 to gather input on three proposed updates:

• Changes to the Investigator and Synergy Grant score descriptors
• Revision of research impact assessment criteria

Changes were introduced to the 2026 Investigator Grant round which opened in June 2025.

Application-Centric Peer Review

(Evaluation, Communication)

The Improvements Gained Through Application-Centric Peer Review infographic was published in August 2024 on the NHMRC website.

NHMRC transitioned to an application-centric peer review process in 2021, using classification data to pre-match reviewers to applications. In early 2024, NHMRC evaluated this change and found it improved expertise matching, maintained review rigour, and reduced reviewer burden. A webpage featuring the infographic was created on the NHMRC website to share these findings with the sector.

Research Impact Track Record Assessment Framework Evaluation Report

(Evaluation, Communication, Implementation)

The Research Impact Track Record Assessment (RITRA) framework Evaluation Report was released in January 2024 on the NHMRC website. The report outlined the evaluation of the RITRA framework implementation and presented the key findings. The RITRA framework aims to enhance the translation of NHMRC-funded research into public health improvements by encouraging researchers to consider future impact. Its findings are now being used to support further policy development of Investigator and Synergy Grants.

Evaluation Outcomes

(Evaluation, Communication, Implementation)

Several changes have been made to the Investigator Grant Scheme following evaluations:

• Streamlined the cross-scheme eligibility framework to simplify the application process for CIAs and RAOs
• Consolidated the Research Impact section from three components into a single field, reducing duplication and applicant burden
• Added new field for applicants to provide evidence of their research impact
• Updated guidance on the knowledge gain criterion for better structure, clarity, and consistency in Research Proposals
• Shared outcomes of grant rounds publicly via the NHMRC website: 2024 Investigator Grants Outcomes Snapshot and 2024 Synergy Grants Outcome Snapshot.

Partnership Projects

Post-implementation Review – Partnership Projects Scheme

(Evaluation, Implementation, Communication)

A peer reviewer survey was conducted in June 2024 to gather feedback on the 2023 Partnership Projects grant opportunity. Findings from this survey, along with reports from panel Chairs and Community Observers, informed the development of the 2023 Partnership Projects PIR. Based on recommendations identified in the PIR, the following changes were implemented for the 2025 Partnership Projects grant round:

• The 2025 Partnership Project Score Descriptors for Assessment Criterion 3 have been updated to improve the alignment of the descriptors for each score
• Partner Contribution Guidelines have been updated to clarify the guidance on partner cash and in-kind contributions
• Grant Proposal requirements have been streamlined to reduce the burden on applicants and peer reviewers.

Postgraduate Scholarships

Post-implementation Review – Postgraduate Scholarships Scheme

(Evaluation)

The Postgraduate Scholarships (PGS) Post-implementation Review Survey Results infographic was released in October 2023 on the NHMRC website.

The article summarised key trends from the 2019–2022 PGS peer review surveys and recommended stronger guidance about Journal Impact Factors. As a result, clearer advice for applicants about not including Journal Impact Factors in applications was included in the PGS grant opportunity guidelines for future rounds.

Translation Programs

Large-scale Priority-driven Grant Opportunities Evaluation

(Evaluation)

The Evaluation of large scale priority-driven grant opportunities was released in June 2024 on the NHMRC website.

ORIMA Research was commissioned to evaluate the effectiveness of large-scale grant opportunities in fostering national collaboration, aiming to identify best practices for designing future priority-driven grant opportunities. Consultations, surveys, and interviews were used to collect qualitative and quantitative data on the grant application process, stakeholder satisfaction, and the impact on collaborative research. The evaluation concluded that large-scale priority-driven grants are effective in fostering national collaboration and addressing complex research problems.

Research and Grant Administration

Grant Administration and Policy

Direct Research Cost Guidelines Evaluation

(Evaluation, Implementation, Communication)

NHMRC evaluated the effectiveness of the NHMRC Direct Research Cost Guidelines in meeting its objective of providing guidance for researchers, RAOs, and Finance Officers on the appropriate allocation of NHMRC funds for salaries, stipends, and other research costs. Insights from this evaluation led to a revised policy, published in December 2023 on the NHMRC website.

Feedback was gathered from RAOs as part of the evaluation process, resulting in the following changes within the guidelines to address common concerns:

• Added additional details on allowable expenses
• Revised components of Salary Support Packages and Personnel Support Packages.

Grant opportunity Guidelines and Peer Review Guidelines Templates Evaluation

(Evaluation, Communication, Implementation)

In early 2024, NHMRC assessed the Grant opportunity Guidelines (GoGs) and Peer Review Guidelines (PRGs) templates. This assessment aimed to determine policy consistency across different NHMRC funding schemes and identify improvements to ensure compliance with whole-of-government requirements.

Consultations were conducted with NHMRC business areas, Senior Executive, Department of Finance (DoF), and the Program Coordination Committee as part of the evaluation process. NHMRC consolidated their feedback, resulting in 42 suggestions for the GoGs template and 14 suggestions for the PRGs template. Of these:

• 33 suggestions were implemented for the GoGs template, including additional instructions to ensure that CIAs who are also RAOs of their institutions cannot certify their own funding applications. This measure was implemented to manage Conflict of Interest and maintain a separation of roles in Administering Institutions.
• 14 suggestions were implemented for the PRGs template, including additional guidance for peer reviewers on declarations concerning Associate Investigators.

Grantee Variation Policy Evaluation

(Evaluation)

From 2022 to mid-2023, NHMRC evaluated its 2019 Grantee Variations Policy on how clearly it outlined the procedures for RAOs to request changes to a grant. Insights from this evaluation led to the revised 2023 Grantee Variations Policy, published in August 2023 on the NHMRC website.

Consultations were conducted with Chief Investigators, RAOs, and various committees and agencies as part of the evaluation process. NHMRC consolidated feedback from these consultations along with responses to the 2021 and 2022 RAO Surveys. The following changes were implemented to provide clearer guidelines within the policy:

• Updated terminology referencing the old grants management system to Sapphire
• Introduced new grant variation types for the Investigator Grant scheme
• Removed the requirement for Associate Investigators to notify NHMRC about changes
• Clarified the conditions under which a change to Chief Investigator will be approved
• Defined events that might qualify as exceptional circumstances.

MREA Final Report Template Review

(Data Collection, Implementation)

The revised Final Report template for Medical Research Endowment Account (MREA) grants was released in October 2024 and is accessible to all CIAs through Sapphire.

Updates to the Final Report template were informed by consultations with NHMRC principal committees, Research Administration Officers (RAO), and NHMRC’s data and research administration business areas. Changes were implemented to the Final Report template to improve the collection of data on personnel supported by NHMRC grants, and the research outputs, outcomes, and impacts generated from these grants.

Top 10 in 10 publications policy Evaluation Report

(Evaluation, Communication, Implementation)

The Top 10 in 10 Publications Policy Evaluation Report was released in May 2024 on the NHMRC website.

The evaluation examined whether a new policy for assessing publications of NHMRC’s track record-based schemes was effective. This policy focused on emphasising the quality and scientific contribution of an individual's publications rather than the quantity of publications produced.

As the evaluation did not reveal any negative unintentional consequences of the policy, the recommendation was to retain the policy as is. Implementation of this policy will continue to be monitored in the coming years to inform phase 2 of this evaluation work.

Research Administration Officer Engagement

RAO Seminar Evaluation Report

(Evaluation)

NHMRC periodically holds a RAO Seminar to provide research administrators with updates and developments from NHMRC and the Australian Research Council (ARC), along with networking opportunities with agency personnel. The 2023 RAO Seminar was held over 2 days from December 12-13, 2023, with Day 1 taking place at the Canberra Rex Hotel and Day 2 at NHMRC’s Canberra offices.

NHMRC evaluated the seminar’s effectiveness in accomplishing its objectives by conducting a post-seminar survey to gather feedback and observations from attendees.

The following changes were implemented for the 2024 RAO Seminar, held on December 2-4:

• Seminar was extended to three days to provide more time for detailed MRFF content
• Seminar was scheduled earlier in December to accommodate peak activity periods, reduced staffing, and higher travel costs
• All three seminar days were held at the Shine Dome, addressing attendee feedback for preference for a single venue.

RAO Survey Evaluation Report

(Evaluation)

The 2023 RAO survey was conducted in February 2023 to assess the effectiveness of post-award grant administration across NHMRC schemes and identified areas requiring improvement.

The results indicated that the most challenging aspects for RAOs included accessing outcomes, creating and submitting variations, and navigating the Sapphire system.

Research Ethics, Equity and Engagement

Ethics and Integrity

National Statement on Ethical Conduct in Human Research

(Evaluation, Implementation, Communication)

NHMRC evaluated the effectiveness of the 2018 National Statement on Ethical Conduct in Human Research (the National Statement) in meeting its objective as the primary ethical guideline for the design, review, and conduct of human research in Australia. Insights from this evaluation led to the revised 2023 National Statement, published in July 2023 on the NHMRC website.

Public consultations and feedback from the Australian Health Ethics Committee were conducted and consolidated as part of the evaluation process. The findings highlighted a need for new risk categories, expanded exemption criteria, and clarified research governance guidelines in the 2018 National Statement. As a result, substantial changes were made to Chapter 2.1: Risk and Benefit, and Section 5: Research Governance and Ethics Review of the revised 2023 National Statement.

Human Research Ethics Committees Activity Report

(Evaluation)

The 2023 Human Research Ethics Committees (HREC) Activity Report was published in 2024 on the NHMRC website. NHMRC requests annual reports from registered HRECs to monitor their adherence to the National Statement and Privacy Act guidelines.

ORIMA Research was commissioned to collect data from HRECs through the HREC Annual Survey. The findings from this survey were used to compile the 2023 HREC Activity Report.

Australian Research Integrity Committee Evaluation

(Evaluation, Communication)

The Australian Research Integrity Committee (ARIC) Evaluation Report was published in November 2023 on the NHMRC website. ARIC reviews institutional processes for managing and investigating potential breaches of the Australian Code for the Responsible Conduct of Research.

KPMG was commissioned in November 2022 to independently evaluate the effectiveness and performance of ARIC and make recommendations to enhance its efficacy and impact. The final evaluation report found that ARIC is effectively meeting its purpose but, made fifteen recommendations to improve its function and operation within the current ARIC Framework. NHMRC is currently leading the implementation of the recommendations, in collaboration with ARC.

Gender Equity

Gender Equity Strategy Evaluation

(Evaluation, Implementation, Communication)

The NHMRC Gender Equity Strategy 2022–2025 provides the framework for NHMRC’s actions to achieve gender equity. The Gender Equity Strategy Evaluation Plan was developed by NHMRC to monitor performance and evaluate the activities under the Gender Equity Strategy.

The Evaluation Plan includes the development of annual progress reports and Gender Equity Report Cards. Annual Gender Equity Report Cards for 2022 and 2023 were published on the NHMRC website, highlighting the achievements made by NHMRC in alignment with the Gender Equity Strategy 2022–2025.

Consumer Involvement in Research

Consumer Statement Position Paper Evaluation

(Evaluation, Communication)

The 2016 Statement on Consumer and Community Involvement in Health and Medical Research is being evaluated by NHMRC to assess its effectiveness and identify potential revisions.

Key evaluation activities and reports conducted to date include:

• November 2022: Online workshop conducted to gather stakeholder feedback on the 2016 Statement
• December 2022: Position Paper produced by the Consumer Statement Advisor Committee (CSAC) to guide stakeholder consultation
• December 2022: Environmental scan of global literature conducted by NHMRC
• March–May 2024: Australia-wide stakeholder workshops held to gather stakeholder suggestions for revisions to the Statement
• March–June 2024: Online survey conducted to gather stakeholder feedback on the Position Paper
• December 2024: Review of the 2016 Statement on Consumer and Community Involvement in Health and Medical Research - Online engagement report published on the NHMRC website, summarising all stakeholder feedback received thus far.

Insights from the evaluation are now being used to update the 2016 Statement, with the aim of enhancing its guidance on consumer and community involvement in health and medical research.

Indigenous Research

Indigenous Research Excellence Criteria (IREC) Review

(Evaluation, Implementation, Communication)

The Indigenous Research Excellence Criteria (IREC) Review Consultation Report was published in December 2023 on the NHMRC website. It summarised feedback received as part of the IREC Review national consultation, conducted by NHMRC from July to September 2023 through workshops, public webinars, submissions, and meetings.

The report made several recommendations for improving the IREC. One key recommendation was to require grant applications to address benefits to Aboriginal and/or Torres Strait Islander health. This question is being piloted in the 2025 CRE grant opportunity, and a fact sheet about the question was published on the NHMRC website.

NHMRC continues to update the IREC based on the report's recommendations.

NHMRC Data and Research Impact

Data Inventory and Strategy

Data Inventory Project

(Data Collection, Evaluation, Implementation)

NHMRC participated in the Data Inventory Project, led by the Office of the National Data Commissioner (ONDC), to enhance NHMRC's Data Asset Inventory (DAI) and align it with the Commonwealth Government Data Availability and Transparency Act 2022. As part of the project, NHMRC:

• Reviewed its data asset management and proposed solutions to meet ONDC and NHMRC metadata standards
• Created a DAI Management Manual with definitions, standards, controls, processes, and governance roles
• Enhanced its data asset management, showing significant improvement post-project
• Aligned the project with NHMRC priorities to boost data maturity and support decision-making.

NHMRC Data Strategy

(Data collection)

NHMRC is currently developing the NHMRC Data Strategy, which will serve as a long-term plan outlining the technology, processes, personnel, rules, and culture necessary to effectively manage our data.

To inform the development, NHMRC is evaluating data strategies from other agencies and facilitating discussions through the Information and Data Governance Working Group. NHMRC is also actively collaborating with the ONDC and other significant APS entities to ensure that NHMRC's Data Strategy aligns with national frameworks.

Data Maturity and Assessment Tool

(Data collection, Evaluation)

The NHMRC Data Maturity Assessment Report, released by DoF in August 2024, indicated NHMRC's data maturity and capability to be below the APS average.

The Data Maturity and Assessment Tool (DMAT) was developed by DoF to provide APS agencies with a consistent approach to measure and understand their data maturity. NHMRC’s DMAT report indicated that it currently sits at the Initial/Ad hoc stage. This stage reflects a history within NHMRC of reactive approaches to data management, focusing on identifying essential needs of the moment rather than adopting a structured, forward-thinking strategy.

Research Impact

Dementia and Diabetes

Evaluation of NHMRC Funded Dementia and Diabetes Research (Elsevier B.V.)

(Evaluation, Communication)

Traditionally, NHMRC has used publication bibliometrics to measure grant outputs, but this method does not fully capture the impact of NHMRC-funded medical research. To address this, NHMRC explored external partnerships and commissioned Elsevier B.V. to produce a detailed report evaluating NHMRC-funded dementia and diabetes research. The NHMRC funded dementia and diabetes research evaluation report was published on the NHMRC website in March 2025. The report provides a comprehensive analysis of health and medical research in dementia and diabetes, comparing NHMRC-funded research with other sources.

Impact Case Studies

(Evaluation, Communication)

NHMRC published eight Impact Case Studies during the reporting period. These case studies received 41 external promotions from NHMRC, partner organisations, and other institutes through various media channels.

Case studies were featured on NHMRC’s fortnightly newsletter, Tracker, which is accessible on the NHMRC website and distributed via email to subscribers. They were also shared on NHMRC's social media, including X (formerly Twitter), Instagram, and Facebook. Seven partner organisations and two peak bodies/professional networks promoted the case studies through their websites or by resharing NHMRC social media content.

Collaboration and Building Evaluation Culture

Coalition for Advancing Research Assessment

(Culture)

NHMRC joined the Coalition for Advancing Research Assessment (CoARA) and signed the Agreement on Reforming Research Assessment (the Agreement) in October 2024.

CoARA is a network of organisations committed to improving research evaluation methods. The Agreement advocates for holistic, transparent, and fair assessments that recognise diverse research contributions and impacts. As a CoARA member, NHMRC aims to refine grant application evaluations to ensure fair and evidence-based support for high-quality research.

Joining CoARA aligns with NHMRC’s Research Quality Strategy and NHMRC’s 2024-25 strategic priorities of optimising peer review and harmonising MREA and MRFF. The Action Plan for this initiative is currently being developed by NHRMC and is expected to be released in mid-2025.

Collaborations with the Australian Centre for Evaluation

The Australian Centre for Evaluation (ACE) commenced operations in July 2023. ACE was established to integrate evaluation evidence into policy design and decision-making across the Australian Public Service (APS).

Since the agency’s establishment, NHMRC collaborated with ACE on several initiatives. These included responding to its State of Evaluation in the Australian Government survey, developing an Evaluation Maturity Model for NHMRC, and participating in ACE’s Commonwealth Evaluation Community of Practice (CECoP) events.