Investigator Grants peer reviewer 2027 training webinar

Recorded on 21 September 2026 2:00–3:30 pm.

Video transcript

[Slide 0:00 - Investigator Grants Peer Review Webinar]

0:11 Lauren Spencer
Hello everyone, thank you for joining us today. We might just wait a few more moments so we can have more people join us and then we'll get started. Well, thank you all for joining everyone, and welcome to NHMRC's Investigator Grant Peer Review webinar. My name is Lauren and I'm an Assistant Director within the Investigator Grants team. On this call, we've also got my colleague Nick, who's an Investigator Grant section Director as well.

[Slide 1:13 - NHMRC acknowledges Aboriginal and Torres Strait Islander peoples as the Traditional Owners of the lands on which we all live, work, research and learn. We recognise the enduring strength of the knowledge systems, and the deep expertise Aboriginal and Torres Strait Islander peoples bring to health and medical research. We acknowledge and pay our respects to Elders past, present and emerging, and extend that respect to any Aboriginal and Torres Strait Islander people joining us today.]

I'd like to commence the meeting with an acknowledgement of country. So, NHMRC acknowledges Aboriginal and Torres Strait Islander peoples as the traditional owners of the land on which we live, work, research and learn. We recognise the enduring strength and knowledge systems and deep expertise Aboriginal and Torres Strait Islander peoples bring to health and medical research. We acknowledge and pay respects to elders past, present and emerging and extend respect to any Aboriginal or Torres Strait Islander people joining us on the call today.

[Slide 1:30 – What will be covered. 1. Scheme objectives, 2. The assessment process, - How to assess Statement of Expectations and Level, - How to use the Score Descriptors, - How to provide feedback, 3. Tricky parts – your most asked questions.]

So, this peer review webinar will cover the scheme objectives for Investigator Grants, the assessment process, including an interactive element on the statement of expectations so I would encourage you, if you haven't already, to familiarise yourselves with Appendix G of the Investigator Grants 2027 Peer Review Guidelines. We will then go into detail of the other scoring elements, how to use the score descriptors and providing applicant feedback, as well as a Q&A section as well. The Q&A element will be supported by our four peer review mentors for the round. They'll be able to be utilised in this session and also during the assessment process as well. So, our Peer Reviewer Mentors are Dr. Hao Song, Associate Professor Courtney Ryder, Professor Eric Chow, and Distinguished Professor Patsy Yates. We also have a Community Observer joining us in session today as well.

[Slide 2:30 - Support the highest-performing researchers at all career stages, with consolidated salary and research funding support. Provide the security of a 5-year government grant, while allowing the flexibility to pursue important new research directions and form new collaborations as the need arises.]

The Investigator Grants scheme is intended to support the highest performing researchers at all career stages, consolidating a salary and research support package. This grant opportunity provides five years of security while allowing for flexibility to pursue new and innovative research directions as the need arises and forming collaborations.

[Slide 3:00 - Outcomes snapshot 2026, 1894 Applications, 12.4% Total funded rate, $429,874,353 Total budget, 234 Grants awarded, 16.5% Funded rate for Aboriginal and Torres Straight Islander focused research, $1,837,070 Mean budget.]

This scheme is highly competitive and highly prestigious. On the slide now, you'll see an outcome snapshot from our 2026 round performed last year. You'll note that our funding rate is just a little over 10% and with the volume of applications received this round, we expect that it will be at a similar level.

[Slide 3:22 Date Stage 29 July 2026 Investigator Grant application submissions close.

Monday 10 August – Friday 21 August 2026 Conflicts of interest and suitability survey declarations. Mid-August - early September 2026 Assessments against the Indigenous Research Excellence Criteria (IREC) completed. Monday 21 September 2026 - Wednesday 23 September 2026 Investigator Grants Peer Review Training Webinars. Monday 28 September – Friday 23 October 2026 Assess allocated applications, submit scores and provide applicant feedback. Attend optional PRM Q&A sessions November 2026 Assessment Review Stage will be open for two weeks. December 2026 – January 2027 Peer review survey opens for feedback and continuous improvement. March 2027 Notification of outcomes under media embargo.]

This is a current round overview, so firstly, I just want to thank you all for completing your conflicts of interest and suitability declarations. This will allow us to be finalising our assessment allocations at the moment. Assessments against the Indigenous Research Excellence Criteria have been completed and so these reports will also be available with your assessments.

[Slide 3:45 IREC reviews complete Mid-August to Early September 2026; Peer Review Training Webinars 21–23 September 2026; Assessment period opens 28 September 2026; Assessment stage 28 September to 23 October 2026; Assessment review stage 2–15 November 2026; Assessments finalised; Funding recommendations; Funding recommendations sent to the Minister; Ministerial approval; Outcomes released under media embargo; Ministerial announcement; Embargo lifted]

We have now an assessment time frame, so just indicating where we're up to now through a little visual. So, this is one of our training webinars that we have introduced for this round in preparation for assessments opening next Monday. You will have four weeks to complete your reviews.

Throughout this time, there'll be Q&A sessions as well available to you. We recommend starting early on your assessments, as we acknowledge they take a lot of time as well. So that's something to bear in mind with your planning. After initial assessments are provided, the assessment review stage will open as well, which we'll talk about a bit later in the presentation. We thought it would be important to highlight as well once assessments are complete, the associated process for approval. So once your scores and comments are provided, these will get sent through to the Minister as well for approval with outcomes being released under a media embargo as pictured.

[Slide 4:45 Principles of Peer Review* Transparency; Independence; Fairness; Confidentiality; Impartiality; Quality and excellence; Appropriateness and balance; Research community participation; The NHMRC Obligations in Relation to Confidential Information establish confidentiality requirements that represent a lifelong commitment. Outlined in Appendix A of the Peer Review Guidelines.]

On that note, I'm going to hand over to my colleague, Nick, who is going to take you through the assessment stage of Investigator Grants and highlight some important elements that you'll need to know for your role. Thank you, Nick.

5:11 Nick Fairbairn
Thanks, Lauren. Hello everyone and thank you for joining us today. I'll be taking you through the core assessment elements to try and give you a better sense of your role by interpreting for you some of the key advice we've got in the guidelines. But first, some general reminders that as a peer reviewer, you must at all times uphold NHMRC principles of peer review, including your obligation in relation to confidentiality of applicant information. I'll remind you that this obligation extends beyond the peer reviewer round and essentially doesn't end. These responsibilities are outlined in Appendix A of the Peer Reviewer Guidelines. So, if you're not already familiar, please do read that before commencing your assessments. A reminder not to contact each other or the applicants about either the application or your assessments.

[Slide 5:56 Generative AI policy update; AI may be used to refine assessment comments. AI must not be used to assess, score, or review applications. Reviewers remain responsible for all comments and scoring decisions. Statements of Expectations; Updated guidance supports applicants to select the most appropriate Investigator Grant Level. Reviewers should consider the applicant's justification for their chosen Level when assessing applications.

And then a quick note on the use of AI, we have a policy on the use of AI. This is one of the two key changes that we want to reflect on quickly. Generally, what this means for you as a peer reviewer is that you're not to use AI really at all in the assessment of applications. It's up to you to understand the application yourself and to form your own views as to the merits or otherwise of each of the applications. What you can use AI for, and if you read our policy, it will tell you that as well, is that you can help AI to help finesse your applicant feedback to make that more helpful for applicants.

The other change we've made is another update to the statements of expectations, just to make it easier for applicants and for peer reviewers to better understand the most appropriate level for applicants to apply.

[Slide 6:38 The Statement on the Variables in research states that NHMRC-funded research: Must consider the Variables at all stages of every research project; Applicants must describe how each of the Variables is integrated in the proposed research and/or justify why any are not integrated; This information can be found in the Application Report and Grant Proposal, where relevant; Consider the appropriateness of the integration of the Variables and/or the strength of the justification for non-integration; Incorporate when scoring the Knowledge Gain criterion; For more information, refer to the Statement, FAQs, videos and resources on our gender equity webpage]

Another key change today, or key change for this round, is the introduction of the Sex, Gender, Variations of Sex Characteristics and Sexual Orientation, known as the Variables, into the assessment. Essentially, applicants are asked at the start of the application to indicate which of the variables is applicable to their research and to justify why. And then they're asked to embed that further in their peer review, sorry, in their research proposal. And your job as reviewers is, obviously, is to assess how well and how appropriately they have done so.

[Slide 7:10 Criteria and weighting Track record, relative to opportunity (70%), including selected Level Publications (35%) Research impact and pathway to impact (20%) ‘Reach and significance’ of the research impact (10%) ‘Applicant’s contribution’ to the research impact (10%) Leadership (15%) Knowledge Gain (30%)]

You will assess applications, we'll go on to the criteria now, you'll assess applications against four criteria. You can see them here on the slides with their weightings. We'll talk more about each of those in a minute, but for now, that's what you need to know. We know that many of you will commence your assessments offline and that's totally fine, but please do let your secretariats know so that we're not chasing you.
And when it comes time to, please make sure you're being really careful to put the right scores and comments against the right application when you put them back into Sapphire. Applicants will notice. If you have any questions throughout the process, your secretariat is your first port of call. If they can't answer your question, they can put you onto someone who can, like one of our Peer Reviewer Mentors, who Lauren has already spoken about.

[Slide 7:55 Peer Review Mentors Doctor Hao Song University of Queensland; Associate Professor Courtney Ryder Flinders health and medical research institute; Distinguished Professor Patsy Yates Queensland University of Technology; Professor Eric Chow Monash University]

We thought we'd give this opportunity to allow the Peer Review Mentors to introduce themselves to you. So, I'll hand over to Dr. Hao Song first to introduce himself and then we'll move through the other three peer reviewers and then I'll come back and talk to you at the end.

8:13 Hao Song
All right, OK. Thank you. Thank you. So, my name is Hao Song, I'm the group leader and NHMRC emerging leadership fellow at the Australian Institute for Bioengineering and Nanotechnology at the University of Queensland. I also have joint appointment with the School of Pharmacy and Pharmaceutical Sciences. So, my research basically focusing on using the nano materials, or like nano particles for drug delivery. It's finally for the currently using the nucleases for the vaccines and separate development. So, I've been joining the peer review for the NHMRC Investigator scheme for five years and I've been awarded twice of this, for the year one in 2020 and then year two in 2024. So, it's really a great pleasure to be a Peer Reviewer Mentor as well actually from last year and this year together to support the community.

9:09 Nick Fairbairn
Great, thank you. Professor Eric Chow, would you like to introduce yourself now, please?

9:15 Eric Chow
Thanks, Nick, and hi, everyone. My name is Eric Chow from Monash University, also Melbourne Sexual Health Centre at the Alfred Hospital. I'm a STI, HIV epidemiologist and biostatistician. So, my research focusing on HIV, STI, sexual health, epidemiology, and clinical trial. I am currently holding an L1 application and also have been a Peer Review Mentor in the last three years. Thank you.

9:45 Nick Fairbairn
Thank you, Senior Professor Courtney Rider, if you like to introduce yourself now.

9:49 Courtney Ryder
Namani, my name's Courtney Ryder. I'm an Aboriginal injury epidemiologist at Flinders University. So, I am the discipline lead for trauma and injury here and also co-director of the FIMRI Health Equity Impact Program. And this is my first time being a Peer Review Mentor. So please, everyone, be gentle on me. But in terms of that as well, so obviously a lot of the work I do is in the public health space, specifically trauma and injury, a lot of data work, but obviously a lot of work with Aboriginal and Torres Strait Islander communities. So, if you have got applications that you are assessing within that area, and certainly thinking about the Indigenous Research Excellence Criteria in all of those things, please feel free to reach out to me. I'm more than happy to have a yarn around all these things.

10:37 Nick Fairbairn
Great. Thank you, Courtney. Now, Patsy is running a little late. We might get an opportunity to allow Patsy to introduce herself to you all later on, but for now, we'll move along.

[Slide 10:48 Thoughtful Thorough Equitable Robust Fair Consistent]

But before I do, just to let you all know that the Peer Review Mentor role, they'll be available right throughout the peer review round to answer questions you might have, including at four Q&A sessions that we hold that you can come to and ask your questions directly of the PRMs. So, they're there for your support to feel in anything, like the role of a mentor for you to understand better what your role is. Now, we'll carry on and again talk about your role and really, essentially what it breaks down to is that you are as best you can to provide thoughtful, thorough and equitable reviews of your assigned applications so that NHMRC’s independent peer review can remain robust, fair and consistent. But what does this mean and how can we work towards achieving this?

[Slide 11:33 Key focus areas; Familiarise yourself with guidance materials; Leverage application centric peer review]

So, there are two key areas of focus that I think you should think about as a peer reviewer. The first is to ensure that you're across the guidance materials to understand your role really well. And the second is to ensure that, as best you can, you leverage that key advantage that we have through using the application-centric peer review model.

[Slide 11:53 Key assessment details; Consolidated into a single score descriptor appendix: what is expected of applicants; the elements you need to consider; provides a consistent framework to benchmark expectations. Outlined in Appendix C of the Peer Review Guidelines]

In relation to that first objective, we've tried to make this as easy as we can for you by consolidating all of the key assessment detail into a single score descriptor appendix. That's Appendix C of the Peer Review Guidelines. So, this is your one stop shop for understanding your role. It outlines for you what's expected of applicants, the elements of the assessment that you need to consider and provides you the information that you need to help construct a consistent framework upon which you can benchmark your expectations of applicants at each level and at each score.

[Slide 12:33 Score descriptors NHMRC recommends: Use the Score Descriptors as guardrails; Anchor expectations to scheme objectives; Exercise independent judgement; Use the full scoring range; Apply scores consistently; Align feedback with the score. NHMRC does not recommend: Treating the tables as prescribed scores; Worrying about how other reviewers might score; Introducing extraneous considerations; Searching for information not provided in the application.]

Indeed, most of what we'll be talking about or what I'll be talking about with you today, will be referring to and expanding on information that's contained in this appendix. So, the score descriptors attempt to walk you through your assessment, if you like, of each application. They lay out what the scheme is trying to achieve, how the assessment criteria that we use contribute to achieving those objectives, and then it explains what each assessment criteria is attempting to measure.

[Slide 12:58 The score descriptors and performance indicators IMAGE: Table 6…]

That last objective that the score descriptors do, that is achieved primarily by the score descriptor and performance indicator tables. So, the performance indicator table is new, and the score descriptor tables were updated recently to include more detail at each of the descriptions. So, how should we think about using them? These are guides. They don't tell you what you have to score. What they do do is provide some guardrails, some context, and that detail that you need to help you be more confident in baselining your own expectations from applicants. From our perspective, the reason why we've updated them to include more detail is because that allows NHMRC to be more confident than across many of you, hundreds of different peer reviewers from different backgrounds, both professional and personal backgrounds, and importantly from different research areas, to come to a more consistent understanding and a better calibrated understanding of what each score means to them. I think that's the really key thing that we want to drive home today in that your role is to determine what each score means to you and to apply that understanding consistently and fairly to each of your assigned applications, free from any unhealthy bias and from any considerations of extraneous information. And by that, we mean the assessment is of what is provided in the application. And while you're free to corroborate that externally if you wish to, or if you feel you need to, you shouldn't be looking externally for additional track record details or additional context about what the researcher is planning to do with their research in order to help inform your scores.
Peer review is naturally subjective. We encourage you to use the full range of scores and to align your applicant feedback with those scores. First of all, to justify them, but second of all, to make sure that the peer reviewers, sorry, the applicants have good feedback that's going to be as helpful as it can be.
It's also really important when you're reading and when you're considering for yourself what these scores mean to realise that each score descriptor appendix and each score descriptor is calibrated to the objectives of the scheme. So, for Investigator Grants, we intend to fund the highest performing researchers at all career stages in their research fields. And so, you need to anchor your expectations around that. And so that's why, for example, a score of four, which is, you know, this is a very competitive scheme. So, a score of four overall often does not see you being competitive. That description is ‘at expectations’. What that's supposed to mean is an applicant may present to you in their application, and that's what you're assessing the information in the application, as someone performing at what you would expect of a person in that field of research at that stage of their career with the opportunities that they've had available to them. And that's how you should think about your considerations, because as we've said, four
often would be outside of what the funding cutoff has been for this scheme, for pretty much since it's been implemented.

[Slide 16:09 Success in peer review Score against the Score Descriptor and Performance Indicator tables, and what they say is expected at each score. Success is not about guessing how other researchers read the descriptors or matching the scores of the other assessors on the application. Success is your expertise. Bring your own expert knowledge and experience so every assessment is as robust as it can be.]

Now, whilst I've just spent a bit of time talking about funding outcomes, it's really important to remind you that you shouldn't be considering the application's funding outcome when you're considering the most appropriate score. You don't know and we don't know what the funding cutoff will be for that application. And of course, you'll have no control over the final score because that's done by five peer reviewers averaging out their scores. So, your scores should be reflecting on the assessment criteria, so directly related to the assessment criteria, with reference to the score descriptors. A reminder that success in peer review isn't about guessing how other researchers or other peer reviewers are interpreting the score descriptors, the words and the phrases in them, or about guessing what they're scoring applications that you've been assigned. Success is you bringing your own expert knowledge and experience to the assessment. To ensure each application's assessment is as robust as it can be.

[Slide 17:00 Leverage application centric approach Application-centric peer review lets us match you, in almost all cases, with the applications you rated yourself most suited to review.]

Which segues really nicely into that second key focus area that we spoke about earlier, which is leveraging the advantage that we have using the application-centric peer review model. Which essentially allows us to allocate you, in almost all cases, applications which you've declared the highest level of suitability to review.
But how do we leverage that? How do we translate that into better outcomes for applicants, the scheme and ultimately Australian health?

[Slide 17:30 Relative to opportunity Every research career is distinct, and the track records of the very best researchers in the country can look quite different, depending on the opportunity they’ve had to: conduct independent research; form meaningful collaborations; win grant funding.]

So, the first key area where this is really important is your consideration of an applicant's opportunity to conduct research. And of course, that's done via the relative to opportunity [RTO] policy. A quick note on the policy, it's not a panacea. It can't fix everything that goes right or goes wrong in an applicant's career. It exists to acknowledge that every research career is different and to acknowledge that the research track records of the very best researchers around the country will look slightly different based on three key things. Their opportunity to conduct independent research, form meaningful collaborations and to win grant funding. These are three key things that you should be reflecting on when you are considering the applicant's opportunity and how that might have impacted their career trajectory.

So, how to adjust for RTO considerations? I've worked in the scheme for a very long time and I'm very aware of the difficulty that peer reviewers can have working in isolation and trying to benchmark or rank the opportunity that their applicants have had. And you may find it more helpful for you to assess everybody's or all of your applicants track records free from any consideration of their opportunity and then apply a scoring adjustment later. And you might do so by putting each of your applicants into a category of opportunity or their disruption to their opportunity. So, for example, limited or none, this applicant has had really no real impact to their career trajectory or opportunity to produce research outputs. Moderate, so there's been some, and significant. You would do that yourself and you would calibrate that yourself. And importantly, like I said before, the most important thing is to be consistent with how you're applying those adjustments. Remember, quality is quality. A publication's quality or impact doesn't change based on the applicant's opportunity to have contributed to it. However, your understanding of their contribution to that publication may well change based on what's in their relative to opportunity section of their application. It's really important as well to reflect on the fact that your scoring adjustments shouldn't be a one-size-fits-all approach. The nature and degree of the circumstances and the disruptions should determine your score, or your score adjustments, should you feel that's necessary. And by that I mean, you may well read the relative to opportunity sections of an applicant and decide that this has had more of an impact on their ability to have contributed to a meaningful impact, more than it did to their ability to contribute to a number of high quality publications or vice versa. So that's just a reminder to you to make sure that you're thinking flexibly about how you might want to go about changing your scores.

[Slide 20:11 Relative to opportunity cont. Career circumstances are personal or professional circumstances that might have affected research productivity for track record assessment, but do not meet the definition of a career disruption. Career disruptions are prolonged interruptions to the ability to work that are taken into account in track record assessment and to extend the 10 year assessment timeframe that determines an applicant’s eligibility to hold an Emerging Leadership Investigator Grant: pregnancy; major illness/injury; carer responsibilities including parental leave.]

So we'll talk more about RTO, which is important. So, I want to just quickly point out the difference between a circumstance and a career disruption. Basically, anything that's happened in an applicant's career can be a circumstance, so long as they can justify and explain how that might have impacted their ability. Again, thinking primarily of those three key things we talked about, conduct independent research,
form meaningful collaborations and win grant funding. Anything like that can be considered a circumstance. However, career disruptions are only these three categories, and therefore there's a minimum period of time as well that they have to happen. So, pregnancy, major illness or injury, and carer responsibilities, including parental leave.

[Slide 20:52 10 year assessment timeframe In an application with no career disruptions, the 10 year assessment timeframe commences 10 years prior to the funding round’s application close date (29 July 2016). Start of 10 year assessment timeframe 29 July 2016 Application open 3 June 2026 Application close 29 July 2026; Where a career disruptions occurs, the 10 year assessment timeframe extends commensurate with the period of disruption. A 12-month period of parental leave will extend the 10 year assessment timeframe back to 29 July 2015. Start of adjusted 10 year assessment timeframe 29 July 2015 Start of standard 10 year assessment timeframe 29 July 2016 Parental leave (12 months) 1 May 2018 – 1 May 2019 Application open 3 June 2026 Application close 29 July 2026]

Career disruptions extend the 10 year assessment time frame. So NHMRC have made a decision that the recent track record of applicants is the best indicator of future success. So, our track record assessment is based off the last 10 years. Now for this round, that 10 year assessment time frame begins on the 29th of July 2016, because that's 10 years prior to when the applications closed. What career disruptions do is they extend that 10 year assessment timeframe commensurate with a period of career disruption.

[Slide 21:23 Applicant Level exercise Please refer to the statement of expectations at Appendix G of the Peer Review Guidelines.  Research achievements, roles and responsibilities; Professional esteem and leadership; Academic level; Years post PhD]

What they also do is [they] can impact the most appropriate level that an applicant could be applying at. And that's something you need to make sure that you consider carefully. Now, for today and for our webinar, we've decided that the best use of an interactive exercise, we're going to ask, we're going to show you 3 example applicants and give you their demographics, if you like, their RTO section, so who they are, how long they've been in research, what they've been responsible for. And we're going to ask you all, once you've had an opportunity to read each applicant, each dummy applicant, to put them in the level that you think is most appropriate. Now, before we do this, I think it's really important just to cover a couple of key things and that, you know, it's not always a case of right and wrong when we're doing this. When it comes to the level of the selected, it's the degree of appropriateness. So, there might be some examples where there's more than one correct answer, and we'll talk about that when we come to it. The second thing to remember is that you shouldn't be relying on one metric too heavily, and we've actually put here, and this is taken straight from the statements of expectations, where we put an ordinance, we put a level of importance on the various characteristics that people have and relying upon them to put them in the most appropriate level. So, you can see that their research achievements, roles, responsibilities is the most important and the years post PhD is the least important. So, we'll go to the next slide where we've got the details for the first example applicant.

[Slide 22:49 Applicant 1 A cellular immunologist who was awarded their PhD 4 years ago.

The applicant has one strong publication as lead author, several high-quality publications as a co-author from postdoctoral work, and has recently secured institutional support to establish a small independent laboratory.

Their application emphasises technical expertise, a developing research vision and early evidence of independence. They are co-supervising one PhD student and serve as the EMCR representative on their department’s research committee.]

I'll let you read that for a few moments and then I'll ask, I think, Lauren to pop up the little interactive window will pop up in front of your screen. You'll select the level you think this applicant is most appropriate for. And then once everyone's had an opportunity to respond, we'll move on to the next. Okay, Lauren, if you want to pop up the window, I hope people have time to read it. I think you'll be able to move the screen out of the way when it does pop up in front of you. So, you can still read over the… I'll try it myself, make sure that's possible, that can happen. Doesn't seem to be for me. Okay, most people have had an opportunity to respond to that now. Okay, so most people, almost, yeah, a high percentage of people selected EL1, the next most probably being EL2, which is, that's good. Again, there's no right, it's not right or wrong, but that's showing that most people are pretty clear on where this applicant would lie. And of course, an acknowledgement there that if you've only ever excuse me, peer reviewed at the leadership category, might be less familiar to you to see some of these characteristics in applicants. So yeah, EL1 was the level that we had not notionally put into, which was the most by far the most popular response. Some of the detail that we that we pointed to that showed that was that there were developing, you know, developing in their research career. They're an EMRC rep and they're four years post PhD. And it's important to say in that order, the four years post PhD sort of helped inform it, but that's really those first two elements that were the bigger driver. So that's good. We'll move on to the second applicant now example.

[Slide 25:36 Applicant 2 A clinical science and medicine researcher, specialising in oncology and carcinogenesis. They are an Associate Professor at a highly regarded University, from which their PhD was awarded 11 years ago. The applicant has led several nationally-funded studies in radiology education, resulting in improvements in tools for national screening programs, which are being rolled into partnerships with prestigious national and international organisations. They have supervised 14 HDR students to completion, hold a Program Director role of a nationally-recognised undergraduate program, and serve on the boards of several national-level societies.]

I'll give a little bit more time because I don't think you can actually move that answer slide out of the way once it goes up on your screen, so I'll make sure that you have enough time to read this before we pop it up. Okay, we'll pop that up now. Thank you. Okay. Seems like most of you had opportunity to respond to that as well. And once again, the most popular response, well, actually, the most popular response was L1, which is not, that is fair and that is correct, but I guess it's an expected outcome. We'd actually put L2 as our most appropriate level. And I think we actually put this one here on purpose because it does present and look a lot like an L1. But there are some key things here that push it into the L2 category. And those were they lead nationally and are beginning to have international influence. So again, if you look at the statements of expectations, one of the key things that separates an L1 from an L2 is their ability or their influence outside of the national health and medical research sector. But L1 is still a correct answer and that is fine. What's more important, I suppose, because you know, the levels themselves are originally arbitrary, we could have made them any level of years between, like this post PhD years to PhD level. The most important thing is that when you are yourself considering the most appropriate level, that you are making sure that you're considering the opportunity this person has had to conduct research and assessing them fairly with the rest of the applications that you have. So, thank you, that's excellent. We'll move on to the final applicant now, the final example.

[Slide 28:48 Applicant 3 A health services researcher, focusing on nursing and midwifery in rural and remote health services. Their PhD was awarded 11 years ago, and there has been a career disruption of 1.4 years FTE within the assessment timeframe. The applicant has two EMCR awards from nationally recognised Societies, currently co-supervises 3 PhD students, and has supervised 2 Master students to completion. The applicant holds two governance roles in relevant peak bodies in Australia and has built a nationally recognised program in rural maternal health. The applicant reports 4.5 years FTE of research active time, with the rest of their FTE dedicated to teaching and departmental leadership roles.]

Okay, I'm gonna pop up the answers now, Lauren. Okay, looks like most people have responded to that as well now. And yeah, I think the first thing I'll say about this is that this is, we would deliberately put this example in here because I think it's a really common one where we have particularly emerging leader, emerging leadership peer reviewer come to us asking if this is okay, asking whether this person should have been applying at an L1 or whether it's okay for them to apply at an EL2. So, in that way, it's not necessarily right or wrong, but it is heartening to see that most people have chosen the same level that we have, which is EL2 for this individual. It's deliberately tricky though and they absolutely can pitch themselves as an L1. And if I worked at an institution, I might even encourage them to. If you look at the data around an applicant like this, if they were to apply at a higher level, we find that applicants that apply up relative to where you expect them to based on their years post PhD, they're actually more successful and vice versa when you apply down. So in this case, EL2 was the selection, primarily because of the EMCR awards, only three PhD students and the 4 1/2 years of active research time. But again, they could very well decide to pitch themselves as an L1 and really sell themselves to you. Because as I said before, the most important thing when selecting level is your achievements, your responsibilities, and your standing in the sector. So that's great. Hopefully some of you have understood a little bit more. If that raised questions for you, there's a Q&A session at the end as well. I'm happy to answer any of the questions you might have if there's still some lingering doubts.

[Slide 32:09 Indigenous Research Excellence Criteria Applicants will nominate if at least 20% of their research effort and/or capacity building focused on Aboriginal and/or Torres Strait Islander health. Indigenous health research peer reviewers assess these applications against the IREC (community engagement, benefit, sustainability and transferability, building capability) and determine if it meets or does not meet the 20%. Check the application's IREC status in Sapphire before reviewing each application. The IREC assessment report is available in Sapphire. If you have questions please ask our Indigenous health peer review mentor. Peer reviewers must consider the IREC assessment report. The IREC assessment report provides information on indigenous health considerations of an application.]

So we'll go on to our next slide and we'll talk about Indigenous Research Excellence Criteria [IREC], really important part of the assessment. And as Courtney said, Courtney Ryder is our Indigenous health expert PRM. So, every application that selects yes to the IREC question, which is to indicate that they believe that 20% of their research effort and or capacity building focuses on Aboriginal and Torres Strait Islander health, they will undergo a separate assessment by an IREC assessor who will produce an IREC assessment report, which you'll be notified exists and you'll get an e-mail and you can go retrieve that from Sapphire. Now, there isn't much written down at all about how to consider that IREC assessment report in your own assessment, and we do get asked this question. Essentially, the answer is tricky because there's no one answer. There's no hard and fast rule about how to consider that IREC assessment. As I said before, leveraging the advantage of app-centric peer review, you should be assigned applications which you are fairly familiar with to start with. So there's every chance that for some applications, you may not need to rely too heavily on what's in the IREC assessment in order to be confident to form your own views as to the quality of the proposed research. However, you need to consider things like how central is the IREC element to the overall research and how familiar are you with that area of health and medical research. So, I would just say, sum up by saying that the IREC report exists to fill in whatever gaps you may have in your understanding to help inform your understanding of the feasibility of the proposed work, how well it was designed, and or its potential for impact. So that's what it's there to do, it's to try to provide you more context on the appropriateness of the research plans. But again, we have the PRMs available to you if you do have any questions.

[Slide 34:04 Assessment criteria]

So we'll get now into the assessment criteria. This is where the rubber hits the road, if you like and we apply everything we've talked about so far to your scores.

[Slide 34:17 Publications (35%) Assess research quality and impact, not journal reputation; Use citation metrics as supporting evidence only; Consider the significance of the findings and the applicant’s contribution; Assess relative to career stage, opportunity and experience.]

The first criteria that we'll talk about is the publications criteria, it's the biggest, 35%. It's probably the oldest and I think easily the best understood. We don't get lots of questions about how to assess the publications. So, my comments here will focus primarily on what to avoid, some tricks and tips. First off, don't take shortcuts. So don't rely upon the reputation or the standing of the journal as a proxy or a shorthand for its quality, the quality of the paper. Don't rely too heavily on grandiose statements made in the accompanying explanations for each of the publications around its impact or the applicant's role contributing to it. You may find it helpful to read some or all of your nominated publications, but I really want to stress that this is not a requirement to do so. You may seek verification of some of these claims made in the application, and we encourage you to, where you have questions or you're uncertain about the claim,
But as I said, you are not required to and you shouldn't have to in order to, everything that's required for assessment should be in the application. That's something you should also be thinking about. Applicants are able to use field weighted citation metrics. That's fine. They may be really helpful for you, but again, don't rely on this solely as a proxy for to understand the publication's quality or impact. It may be helpful for you as well, when you're thinking about all of the track record assessment criteria, to think about them all as being really a measure of the applicant's impact. So, for publications, for example, you might want to be thinking about how many people have used these, in all of the papers, how many people have used these findings? How useful were they? What do they contribute to? How necessary were these findings? And that we mean, well, what would happen if they hadn't have been made, if the paper had never been written? So that's what you want to be thinking about. And that's how I would explain it about it's not just having done the publications. What did they do to actually impact, change the course or direction of something, the understanding of something, the knowledge of something?
So, and again, I want to make sure, and I'll say this at each of the three track record criteria, you obviously need to reflect upon the applicant's contribution, but you need to make sure that you're calibrating that, your expectations of that applicant's contribution responsibly, again, leveraging your expertise and knowledge, because mostly you'll be assessing people whose careers should be, should look fairly familiar to you when you're assessing the performance and their opportunity to have contributed to that publication. That's all I want to say about publications.

[Slide 36:52 Research impact and pathway to impact (20%) ‘Reach and significance’ of the research impact (10%) ‘Applicant’s contribution’ to the research impact (10%) Types of impact Knowledge Impact Health Impact Economic Impact Social Impact]

We'll move on to research impact and pathway to impact. Again, this was updated last year, so it's fairly new, but it has been well received. So hopefully it is something that is straightforward for you as well. So this is a newer, more challenging criteria. It still remains challenging, of course. Particularly, it's particularly hard the more junior you are, the less time you've had in research, obviously, to have contributed to something meaningful enough to want to put forward as your primary example of impact in research. At its heart, if you think about why we have this criteria, why we care, we want to be able to select for applicants who are most likely to have impact in the future. And we do that two ways, we do that by measuring the reach and significance of an impact that they have contributed to, because that could be a good proxy for the quality of the work that they do. But also really importantly, we measure that through their demonstrated attributes. And so by that, I mean, they're demonstrated, I think it's on the next slide.

[Slide 37:47 Proactive and deliberate; Planning; Did the applicant plan for their research to have impact?; How was this integrated throughout the research activities?; Targeted; Execution; Were the right stakeholders engaged at the most appropriate times (in the context of the type of research and/or planned impact)? Effective; outcome; How necessary / effective were the applicant’s contributions in realising, sustaining and/or maximising the reach and significance of the impact?]

Really want to be thinking about their proactive and deliberate leadership in research. Are they well targeted, do they well target the embedding of getting impact? So that might mean do they engage with the right people at the right time? Do they tell them the right things throughout the research plan and are they effective? So, did their contribution actually change the impact that the research had? So that's why I really think it's important to consider that we care about this criteria because we want to select the people to give Australian taxpayer money to, who are going to continue to have, who are most likely to have continued success and impact in their career.

[Slide 38:25 Leadership; Applicants are required to provide a single narrative that outlines their leadership achievements and their ability to identify and contribute to positive change, in either: Research mentoring; Research programs and team leadership institutional and research group; Institutional leadership; Research policy and professional leadership.]

OK, so talk about leadership next. Okay, just to make sure I've spoken about all those things. Last thing, just on that, on research impact, I think this is really important to reflect on quickly, is again, like I've said, we need to make sure that you're being appropriate with the expectations that you have of applicants, particularly if they're more junior, and to how influential, depending on the role that they have in the research team, that what they could have done, what they ought to have done given their role.

So, we will talk about leadership now. And again, it is about impact. It's not a box ticking exercise. The best applicants will be able to demonstrate how their leadership skills and performance across mentoring, training and institutional research group, positively impacts the people, places and research around them. So again, it's about impact, it's about change. Everybody at every level should be able to demonstrate the attributes of good leadership. But of course, as we keep on saying, you need to calibrate your expectations carefully as to the career stage and the role of these researchers when you're looking for their examples of leadership. And that's the last thing I think I want to say about leadership is that there are four, as you can see on your screen here, no, sorry, if we go back to the previous slide. So you can see the four elements, I think we call them, of leadership, research, mentoring, programmes, institutional research policy. It is not a requirement for applicants to talk about all four of them. Applicants can draw their example from across the four elements. However, it's the clarity of the applicant's narrative and the strength of the demonstrated leadership examples that determines the applicant's score, not whether they, you know, whether they reference one, two or more of those assessment, those leadership elements.

[Slide 40:20 Knowledge Gain NHMRC defines ‘knowledge gain’ for the Investigator Grant scheme as the quality of the proposed research and significance of the knowledge gained.]

So, we'll move on and talk about knowledge gain now, which is the final criterion. Again, this has changed over time. This used to be a much more open-ended assessment, which reflected the objective of the scheme, which was to fund the highest performing researchers at every career stage with the best ideas for their research features. But we heard peer reviewer feedback and we've acted and that it was very hard to assess these two, you know, multiple research proposals where there was nothing tying them to anything to be to be feasible at all, like practical within the resources that had available to them because it wasn't part of the assessment. So, we have included that to put some structure around the assessment for you. So you are now to assess, and this is really important, yes, most applicants will include details of previous work that led up to their program of work that they're applying for and other work that's happening concurrently with the existing research. And that's good and that is fine, and that might help put some context for you around what you're actually assessing. But a reminder that what the assessment is, is of the proposed new research. That's what you're scoring. So that's really important, I think, to remember.

[Slide 41:21 Transformative, Research design, Reproducibility, Achievable, Stakeholders, Global health, Expertise, Risk management, Impact, Knowledge, Gain, Research outputs, Proposed new research, Significance, Resources, Innovation, Feasibility]

We acknowledge that it's a bit, there's a duality in the assessment in that investigators are free to pursue new research directions and collaborations as the need arises.
That's fine, but you're to assess what they put in their research proposal at face value.
There are some other notes about it. Of course, it's fine, you're absolutely within your rights to reflect upon the applicant's track record when you're looking at the feasibility or the potential of that proposed research to be successful. However, it's really important that you are aware that they're not required to stay on the same career path. They can change research directions if they wish. They can go into a new field of research. That shouldn't be held against them. But again, the best applicants should be demonstrating to you how their immediate track record, their demonstrated skills and experience, would translate into what they are attempting to do. Okay, also, it's really important to remember, we really care about impact. So, we obviously just talked about the research, the track record element of impact, but we've also deliberately embedded more requirements on applicants to talk about the impact, how they're going to plan for having impact in their research proposal. So the best applicants will be talking about how they're going to engage with stakeholders at the appropriate times, both in the formation of their research questions, but also their plans, the research plans and design, and also plans to disseminate information at the end of the research, so all throughout the research lifecycle.

Final point I want to make on the knowledge gain is around global health and research. We do get questions, including before this webinar on that issue, so NHMRC’s grant program exists to support the agency to build a healthier Australia. However, we welcome international collaboration and global health research. Applicants are free to nominate research that is conducted in or on areas and populations outside of Australia. Ultimately, I think it's fairly easy for us all to see how any research into human health, or by Australian health medical researchers, has the potential to be relevant in the Australian context. However, applicants are reminded in the score descriptors that where the five-year research plan or vision proposes research that is predominantly in or on international populations, applicants should articulate how it advances global health and medical knowledge, builds Australia's research capability and or achieves better health outcomes for the Australian communities. So, there is a clear requirement there are applicants where they're proposing research which is essentially wholly done overseas and potentially by researchers based overseas that they at least draw that link for you back to the relevance to the Australian context. Feel free to include that in your applicant feedback if the applicant did a really good or a poor job of doing that, if indeed that impacted your scores. So that's enough from me and I'm going to pass it back to Lauren to talk to you more about applicant feedback and I'll be available back at the end when we do our Q&A session. Thanks.

[Slide 44:30 Applicant feedback]

44:37 Lauren Spencer
Thank you, Nick.

[Slide 44:38 Applicant feedback Character limit 100 – 1500 characters including spaces and punctuation; Highlight the strengths and weaknesses within an application; Provide constructive feedback that will assist applicants with future applications; Use language from the score descriptors where appropriate and frame comments around the scores provided; If an applicant has applied at an inappropriate level this can be noted in feedback; Avoid inclusion of duplicate funding, eligibility, integrity concerns or possible plagiarism in applicant feedback]

So, in addition to the five scores that you provide across the assessment criteria, considering the score descriptors, you're required to provide applicant feedback. Our applicant feedback is the single qualitative response for the applicant ranging between 100 to 1,500 characters, including spaces and punctuation. We know a lot of peer reviewers work offline. If you're doing this, please ensure that if you are doing a lengthy applicant feedback comment, that it does adhere to the character limit, so applicants aren't missing key pieces of information of your review. When writing applicant feedback, we encourage peer reviewers to highlight the strengths and weaknesses of the application. As mentioned earlier, a large volume of applicants will be unsuccessful, and this applicant feedback will be accompanying their application assessment summary. So, we ask that people be professional, kind and courteous. You can be critical and do this in a way that's not disrespectful.

We encourage linkages from the score descriptors and framing your comments around the scores you have provided. By this we mean if you provide exclusively positive feedback that your score should be similarly reflecting that sort of quality as well.

Another element of applicant feedback we encourage people to consider, similar to the exercise we performed earlier on category and level, if you believe that someone has not applied at the appropriate level or that their justification has been weak, that you can note this in the applicant feedback response for the applicant to consider.

We also encourage you to not provide any comments around duplicate funding, eligibility, integrity concerns, plagiarism concerns. These should be provided to NHMRC directly rather than in your applicant feedback comments.

[Slide 46:35 Common traps to avoid; Applicant feedback comments should avoid: Information outside of the application; Derogatory, inappropriate or unprofessional language; Your suitability or expertise to review the application; Rhetorical questions; Emotive language; Scores; Comparative language; Artificial intelligence; Acronyms or terminology; Jargon]

So, what are some common traps to avoid with applicant feedback? So, as part of your role, you're able to verify the claims provided by the applicant within the application, but this doesn't mean going to consider additional information that the applicant has not provided in their application.

We advise to avoid derogatory, inappropriate or unprofessional language. Similarly, to avoid commenting on your suitability to review, so comments such as, ‘though not in my primary area of expertise’, should not be included. Rhetorical questions, we would also encourage you to avoid using those as there's not a rebuttal process and the applicant could misinterpret your response. Overly emotive language should also not be used, such as this application deserves to be funded or similar comments. You should not provide your individual criterion scores in your applicant feedback
comment as this could confuse the applicant and provide them with insights that otherwise they wouldn't get through their average overall scores per criteria.

We also advise you to avoid using a comparative language. Each application, as mentioned earlier, should be assessed relative to its own opportunity and not compared amongst the cohort.

Similarly, with the AI guidance that has been provided, peer reviewers can use AI to adjust their comments, but they should not be using AI to perform any aspect of their review. There should be no indication in any comments provided that AI has been used, as this could confuse the applicant to think that there wasn't a human, these expert peer reviewers, performing the review. Similarly, please avoid acronyms, terminology, and jargon that the applicant wouldn't understand.

[Slide 48:31 Assessment review stage (ARS) In response to sector feedback calling for greater transparency and accountability. Investigator Grants piloted an ARS in 2026. Following positive feedback, the ARS will continue in Investigator and Ideas Grants schemes. ARS shares individual criterion scores, applicant feedback comments and identities of peer reviewers for all assigned applications. 79% of respondents agreed the Assessment Review Stage improved transparency; 81% of respondents agreed the Assessment Review Stage improved peer reviewer confidence; 63% of respondents used the Assessment Review Stage to check and improve feedback; 65% of respondents agreed that the Assessment Review Stage improved overall quality]

After all initial assessments are completed, the Assessment Review Stage occurs. This stage builds on the previous comment sharing stage and was piloted in the 2026 round of Investigator Grants. This was in response to application centric schemes needing greater transparency and accountability. You'll see on the right of this slide, this was warmly received and improved the transparency and other metrics as well. So the ARS will continue within the Investigator Grants scheme and is now being rolled out to the Ideas Grant scheme as well as this scheme, thank you.

[Slide 49:11 Assessment Review Stage; Peer review stage; assigned application; independent assessment (avg 13.6 applications/assessor; Assessment Review Stage; review scores, review feedback, identity sharing, raise concerns to NHMRC, update scores and feedback (if required), resubmit assessments]

The Assessment Review Stage happens after the initial independent assessments are received. The Assessment Review Stage allows participants to see the scores, identities and feedback from their peers who assess the same applications. It is important to remind you all that this is still an independent exercise and the strict confidentiality requirements that you have. By this, it requires you to not contact applicants or your fellow peer reviewers about the information that you see during this stage as well. If you identify any concerns during the assessment review stage of another peer reviewer assessment, please raise this with us directly.

This stage is an opportunity to supplement some of the knowledge learnings that you would have had through panel model schemes as well. So this is an opportunity for you to look at your assessments and see if there's anything that perhaps you've overweighted or underweighted, or missed in your initial reviews, giving you the opportunity to revise your assessments if required. Noting that you are not obligated to revise your reviews, and we still encourage the full range of the one to seven scoring scale.

[Slide 50:29 Finalising assessments; Peer reviewer: Peer reviewers submit their applicant feedback in their initial assessments; Peer reviewers review their feedback during the assessment review stage and resubmit their assessments; Peer reviewers must not contact applicants or other peer reviewers about their assessments; NHMRC: NHMRC review a portion of applicant feedback comments to ensure that feedback is appropriate according to NHMRC guidance; If an error is identified NHMRC contacts the relevant peer reviewer via email; NHMRC completes an outlier score check to identify and clarify any outlier scores; Peer reviewer: Peer reviewer receives email from NHMRC with advice to reconsider or remove an inappropriate comment; Peer reviewer considers NHMRC advice and revises comments]

After the assessment review stage, there are additional internal measures that NHMRC conducts, such as outlier score checks, a targeted applicant feedback QA, as well as duplicate funding checks as well. During this time, NHMRC may be in contact with you as well, so we’d appreciate timely responses when we're getting back to you as well on these.

[Slide 51:08 Next steps in peer review: Assessments open on Monday 28 September 2026; Assess Applications; PRM Q&A sessions; Assessment Review Stage; PRM Q&A sessions: Date, Time: Thursday 8 October 2026 11am; Tuesday 13 October 2026 2pm; Thursday 15 October 2026 11am; Monday 19 October 2026 2pm]

So the next steps, as mentioned earlier, we plan to open assessments next Monday and there'll be the four week window for you to be able to assess applications. During this time, there'll be 4 peer reviewer mentor Q&A sessions as shown below and this would have been provided to you in previous communications. After this, the assessment review stage will open as well. This stage isn't mandatory, so if you are unable to participate due to availability, your initial assessments will be brought across and those will be the reviews considered in our funding recommendations.

[Slide 51:40 Q&A; Doctor Hao Song; Distinguished Professor Patsy Yates; Associate Professor Courtney Ryder; Professor Eric Chow]

On that, we will now move to the Q&A element. So we'll bring our peer reviewer mentors onto the screen. Some of you have already started popping some questions in, but we do have a Q&A feature enabled as well. Thank you. As Patsy arrived a little late, we'll just give Patsy the opportunity to introduce herself as well.

52:09 Patsy Yates   
Good afternoon everyone. My name is Patsy Yates. I've been an NHMRC peer reviewer for a few years and really enjoy the process and working with peer reviewers to ensure that we do a great job. I'm Executive Dean, Faculty of Health at QUT and an NHMRC Investigator Fellow. Thank you.

52:29 Lauren Spencer   
Thank you, Patsy. We will kick off with our first question. What advice do you have to assessors who have always assessed Leadership level applications and are now assessing Emerging Leadership? How do we ensure that we don't impose Leadership level standards on Emerging Leadership applicants? I might hand over Patsy as you've just spoken to have a first response and then we'll open up for the rest of the peer reviewer mentors.

52:55 Patsy Yates
Yes. And sure. Look, I think you're right. You can sometimes get in that mindset, can't you, that you've sort of been seeing people at a certain sort of level. But the main thing I always keep encouraging people to do is to go back to your category descriptors, go back to the criteria and make sure that you are using the criteria at the level that you're judging. And that's the main piece of advice. I think if you go back and you have a look at that, you really can't go too far wrong. But it is a good, it's good to actually be aware of that. You know, sometimes there's those unconscious biases. So, it's actually good that you're asking that question because it actually already highlights that, yeah, I need to make sure that I haven't got that bias when I'm reviewing a different type of application.

53:46 Lauren Spencer
Thank you, Patsy. I'm opening up to the other peer reviewer mentors if they have anything further to add.

53:53 Courtney Ryder
I think as well, just being really quite reflective that it is, it's emerging. So, while you have been doing sort of those leaderships, we're talking about people who are sort of more mid-career sort of research level to senior career research level. So, I don't know, cast your days back to when you were an early career researcher and kind of
thinking about some of those things and putting that in that context, but at the same time, rightfully so, as Patsy said, making sure that you're really drawing on the NHMRC guidelines and frameworks in that area to help sort of shape that as well.

54:27 Lauren Spencer
Thank you, Courtney. We might move on to the next question, which is, researchers must supply verifiable information for their application, but this does not apply to their career circumstance or career disruption. How can we verify this claim is legitimate? We might, we might go to Eric.

54:57 Eric Chow   
Sorry, the question is probably more about like to verify the statement, right? I think, yeah, so I think what Nick was trying to say, also mentioned, is like, I think you should probably rely on mostly from the application, but if there's something really suspicious and very a bit off, like you probably can also like just to look for external documentation. Like for example, for publication claim, you can probably go back to the publication they nominate and try to see what they claim is actually reflect and also consistent with what the publication listed as the author contribution. Leadership thing, you can probably try to have a look at the university profile or […] profile, like to have a look and see whether the claim is actually justified and consistent.

55:54 Lauren Spencer
Thanks, Eric. Nick, do you have anything further to add?

55:59 Nick Fairbairn
Yeah, thanks Lauren. Just to, so it's not your role as a peer reviewer to go and verify these claims. Both the CIA and the administering institution need to hold evidence of career disruptions and other career circumstances that they're claiming in applications. So that's actually up to NHMRC to verify those claims that are made in application. So that's not something you need to worry about. You need to take the application at face value.

56:24 Lauren Spencer
Thank you, Nick. We have another question on how many years of experiences do researchers need to lead the research as a CI if they've only recently completed their PhD? We might go to Hao for this question.

56:41 Hao Song
I think actually this is basically for assessing EL1 [Emerging Leadership 1], if it's just a completing PhD. I think it's okay, actually, basically, as long as actually they have been demonstrating their achievements, that's good enough. So it's not just limiting to the years post PhD. As we can see from the guideline, actually, it's probably went to like the appendix, so it's not just previously probably we say, okay, we are expecting like in each level or each of the category actually we're expecting after PhD how many years or which kind of like the levels you are at, but now actually basically that's in the relatively lower position, actually, we are now focusing on the achievements of the people. So that's why actually we see, okay, again, actually with the previous two questions, but basically that also highlights the importance of how the statement of expectation in this application. We need to assess the relative to opportunity and then how their career has been grown and then how the achievements accumulated over this process.

57:50 Lauren Spencer
Great, thank you, Hao. Do we have any other comments from the PRMs on the line?

57:58 Patsy Yates
Not really. I'll just say that it, you know, you do get that whole picture of the other examples of things that the applicant has done, which gives you that confidence that, you know, they've completed, you know, whether it was their PhD or other research work that gives you confidence that they have got the skills that are needed to deliver on that. And of course, that also goes to the sort of feasibility question in the sense of, you know, that what they're actually, who they're working with and what else is wrapped around them to sort of be able to understand that they, that that feasibility is there as well, but it is much more than just years of PhD. I can't agree with that more, yeah.

58:45 Lauren Spencer
Great, thanks. We've got a next question for the peer reviewer mentors as well. When there is one exceptional publication, but multiple publications that are below expectations, how would you advise to score?

59:00 Courtney Ryder
I think you very much need to, and as Nick was talking about some of this before as well, in terms of the publications, you need to sit with the criteria. So there might be, I don't know, a Lancet, a Nature Medicine article where you're like, wow, that's really cool. But what kind of like what was their role in that and what was kind of like the impact or the evidence of the impact? Has it been cited in policies? Has it been cited in models of care and all of those things? And how have they articulated that? So yeah, you could have someone who's got an amazing Nature Medicine article, but then maybe they've got something in the Australian New Zealand Journal of Public Health – not saying that's a bad journal, just saying it's not the same level as Nature Medicine, but that might have been cited in four models of care, you know, which is going to have more impact in terms of that. So, and also I think being very careful about field weighted citation index and old metrics sort of outputs, particularly in that first year as well, which can be very hard for those EL1 and EL2 applications. So just keep that in mind, in terms of sort of the frameworks and guidelines in that area.

1:00:11 Lauren Spencer
Fantastic. Does anyone else have anything to add for that question?

1:00:18 Eric Chow
Yeah, I think it's more about like the body of the evidence. I also agree what that's been said that like you shouldn't actually rely on one publication and also probably the name of the journal and you also probably focusing on like the research impact. And it's good that I'm totally - mentioned about the metric score and also kind of always very tricky because sometimes it depends on the topic and also not really depends on the impact. So you should probably focusing on like what the impact and how it essentially translate into better health.

1:00:56 Lauren Spencer
Great. We have a question for NHMRC. Beyond averaging the five assessor scores, can you clarify how NHMRC normalises, adjusts scores to account for assessor variation, for example, for the average score is provided to the minister?

1:01:19 Nick Fairbairn
It's a very short answer, we don't. We've talked about normalisation and other processes, but we've had some, we've liaised with some external experts on what that would mean for the assessment, but the decision has been taken that we don't do that. We rely upon, as I said, as I guess, I spoke a bit about the advantage we have in application-centric peer reviewer, be able to assign 5 expert peer reviewers who should have a really good indication of the applicant's - understanding of the applicant's career and proposed research and making sure that we provide this sort of training, this sort of advice and help with the peer reviewer mentors as well, to make sure that each assessment can be as robust as it can be.

1:02:01 Lauren Spencer
Right. We also perform NHMRC does outlier score checks on each application assessment provided as well. So that's something that we also do as an additional measure. Further information on our outlier score checks is available through our published peer review analysis committee report as well. So that's something that we can circulate out of session, for those who are interested.

Next question we have, how much weight are we supposed to put on the sex and gender responses in Sapphire and what is in the knowledge gain? So this is an introduction for this round, a key change. I might open up and see if we have a peer reviewer mentor to volunteer to address this one.

1:03:00 Courtney Ryder
I was waiting to see if anyone else was going to jump in before me, but that's okay. Look, it is new and I obviously haven't examined in this, but again, I think that NHMRC provides some very specific guidelines with examples. I was brushing myself up over them over the weekend in kind of thinking about how gender and sex is actually considered in that. So really making sure that you are actually thinking how an applicant has done that, whether they're playing with big data, whether they're doing focus groups or other sort of quant or qualitative measures or so forth. So thinking about that. And then in terms, so I'd be thinking about it more in terms of that context in how it's been played a significant role in sort of their methodology and their methods and so forth. And then hopefully it will translate into knowledge gain, but probably sort of less emphasis on that part there. That's my interpretation of it anyway.

1:04:01 Lauren Spencer   
Thank you, Courtney.

1:04:01 Hao Song
I think exactly, actually, apart from that, actually, when you have a look actually for the knowledge gain for the assessment criteria part, actually, there's a part of that actually is integrated in the assessment as a one of the part of the scoring. It's not as individual score, but part of the first score with the knowledge gain. So I think it's linked to the methodology development details. It's indicated whether the applicant has already has this kind of very detailed methodology in place in this application. And also, that's also linked to whether that will have like a broad, like the impact with this research. Probably that's part of that integrated in your assessment.

1:04:51 Lauren Spencer   
Thank you, Hao. We have a question here for NHMRC about identifying integrity concerns for an application, so inaccurate evidence provided in impact or contributions to publications, citation order, etc. and how to, I guess, address these when you see them. Please provide these to NHMRC for review. We'll then be able to investigate these matters as appropriate. If something is also a serious integrity concern, we have information as well, an integrity fact sheet on what sort of process NHMRC follows as well.

Another question we have is, a particular challenge can be assessing levels for clinician researchers. Quite often they will claim career circumstances that limit their available research time and thus negatively affecting their output, but then some then cite the same career circumstances as evidence of their leadership and impact. this can be quite difficult to assess. Advice? I might throw to Patsy to kick this one off.

1:06:02 Patsy Yates
Yeah, I think you've probably answered your own question there almost, isn't it? You know, when there's a clinician that can be take you away from your research time, being a clinician can, but it also can be, um, enable your research in other ways and you can be participating. So, I think it's really up to the applicant to make the case.
And, you know, and you need to judge the information that you have at hand there. And if it's contradictory, well, that's, you know, you've really got to weigh up, you know, what do you think is the best position that this person is in. So. it is a holistic assessment and that's where, you know, again, that sort of post PhD issue, it comes into play as well. You know, someone may have had a lot of research experience as a clinician and done a PhD late. So again, it's that holistic piece. If someone has contradictory information, you as an assessor really need to weigh up what you think is the most accurate position based upon the information that they've given you.
Because it can, being a clinician can, it can actually enable and it can actually get in the way. So it depends upon the evidence that they provide. Yeah.

1:07:14 Lauren Spencer
Thanks, Patsy. Does anyone have anything further to add for that question? Otherwise, we might move on. We've got a comment here. Thanks for clarifying the three category relative to opportunity score adjustment approach. Can you clarify how scores should be adjusted? Lower scores of applicants without any relative to opportunity considerations or increased scores for applicants with ROPE considerations. We might get Nick to start us off on this one.

1:07:50 Nick Fairbairn
Thanks, Lauren. It's a good question. I'm glad it was asked. You don't ever, you should never be reducing someone's score for RTO. I mean, it's brought in to acknowledge that not everybody gets a clean shot at their research career. Excellence is excellence, like I said. An output that is the world-leading output doesn't become less good because the person who contributed to it had a clean shot, had no interruptions before coming to that. So I think it's really important for everyone to understand that RTO is just an acknowledgement that some career, because some careers can look different and the very best track records can look a bit different because of the opportunity that's been afforded to them. So it should always be up. In terms of how much up - I realise that in a seven point scoring system where there probably aren't that many scores that are relevant for funding, it can seem like a blunt instrument. But yes, 1 score or 2 score, depending on the degree to which you are impressed, I suppose, by the achievement of that individual in the absence of opportunity to produce that track record. That's what I say.

1:08:53 Lauren Spencer
Thanks, Nick. Is there anything peer reviewer mentors would like to add on to that question? No?

We have another question here about the assessment review stage. For the assessment review stage and the final assessment stage, is there any level of engagement between reviewers or is it through NHMRC? Could you please share the time frame again? Thanks.

During the assessment review stage and post the assessment review stage, there should not be any contact between peer reviewers or applicants. The confidentiality agreement is essentially a lifelong agreement that you won't be talking about specific details or contacting applicants or fellow reviewers about this. Once peer review is finalised, that doesn't mean that you can't say share learnings as a peer reviewer, obviously not noting specific application information, scores, etc. to your peers, but as a way to mentor others, that's something that you can do, but there must be no contact between applicants and peer reviewers about the assessments they are reviewing.

And the time frame as well for the assessment review stage, is the first week in November, so November 2nd, and then up until I believe it's the 15th of November as well.

We have another question here. What is the incentive for an applicant to apply for higher leadership categories? And why would an experienced researcher not apply for a lower level where they look better compared to less experienced peers? Should we factor this into our assessment? We might open the peer reviewer mentor to start us off on this one.

1:10:46 Eric Chow
I can start first. I think, well, we probably just previously worked through a few examples. It really depends on the applicants, like what level or what category they don't choose that best fit their career. But obviously, I think sometimes when peer review, peer reviewer also thought like probably may not speak what they actually apply for. So, so there's NHMRC document that actually showing like if you think they actually apply for a wrong category or wrong level, like you can actually apply for a different adjustment based on what you think and then you can put that adjustment in your scoring matrix.

1:11:38 Lauren Spencer
Are there any other further comments from the peer reviewer mentors or Nick?

1:11:43 Courtney Ryder
I guess it's all up to the individual, but read the guidelines and apply for the level that fits you.

1:11:56 Lauren Spencer
OK, thank you. We have a question here about where to access the variable information. So that's something that's available through NHMRC website. So that's something I'll make sure everyone has access to out of session. And I'll get one of my colleagues as well to reply to that as well in the Q&A thread as well.

Another question we have is, my understanding from the presentation was that you do not have to address all four leadership categories. However, in the assessment criteria, all four are listed. So if an applicant doesn't address all of these, how do we score those that weren't addressed?

1:12:45 Patsy Yates
I’ll start. I mean, it's like all of these areas, isn't it? We've got such diversity in our research community, and so it's got to be a holistic assessment. And so, you know, if someone's really done a whole great deal of, got a whole great deal of achievements in one aspect of leadership, you know, that may be really impressive. But you may have actually done something across 4 aspects of leadership, but actually it hasn't been that significant or impactful. So it's really that overall assessment, again, going back to the selection criteria. So yeah, so that I think that really goes for all of the areas that we assess. Yep.

1:13:29 Lauren Spencer
Nick, is there anything that you want to comment on to do with the leadership response? No? Perfect. We have another question on should we utilize the full range of scores? I often see people stick to fives and sixes. Courtney, I might go to you on this one.

1:13:52 Courtney Ryder
Yeah, I think you might find that people are sticking to fives or sixes because everyone who's applying is amazing. And you know, it's so hard because you've got people who are at the top of their game applying for these. So it's not just that people are kind of just sticking to those ones. It's just that every applicant is so good that they're fitting into those realms. Yes, you can use the whole rating system, but you're going to probably find that it is going to be kind of around there because these people are amazing who are applying.

1:14:29 Lauren Spencer
Thanks, Courtney. We have a question for NHMRC, so I'll go to you, Nick. Are you able to explain how NHMRC identifies and selects its peer reviewers and how assessors are chosen for these schemes?

1:14:45 Nick Fairbairn
Okay, I can speak a little bit about this. It's not my role in the team. We've got our own dedicated membership team who do an excellent job. There's a self-nominations process where people put themselves forward, but we also, of course, have people's profile records in system where we use a tool to match, but that's why the minimum data period exists so that we can get all the information we need to out of applications to go out and source those peer reviewers that are going to be most likely to be appropriate to assess. So that's basically the process. I don't know, more than that, we can probably get back to you with some more detail from the membership team, but that's why they're not, there aren't enough people if we just went for Investigator Grant, previous grant holders, we need to spread it out a bit further than that.

1:15:28 Lauren Spencer
Thank you, Nick. Another question. Are there common biases or scoring pitfalls that new peer reviewer should be particularly aware of, and what strategies can mitigate them? Might open up to a peer reviewer mentor to volunteer for this one.

1:15:55 Courtney Ryder
I was going to say, I think one of the hard things when you're first coming into it is that, okay, for people doing it the first time, do your first assessment and then revisit it again. I think it's very hard. You've got to make sure that you are wholeheartedly looking at each application individually and not doing sort of like a cross comparison across all of the applications. So treat that application as it is, in its own right against the NHMRC guidelines and frameworks. So I think that that's one of the most common pitfalls that people can fall into in terms of implicit bias is that cross comparison type thing across the applications.

1:16:40 Lauren Spencer
That's great, Courtney. We might go round to the rest of the peer reviewer mentors for this question to give their own kind of tips as well on certain strategies to mitigate bias in peer review. So we might go to Eric.

1:16:58 Eric Chow
I think one of the major potential bias probably will be like whether you actually seeing like someone from like a particular institution or from a particular team. So you probably will let you some bias and also probably we also mentioned about the publication. Like, whether you have just one Lanset paper, Nature paper, and then you actually score the whole application like really, really well. So I think you should probably like have a look at different section and also make sure you're familiar with the assessment criteria and you actually score the application against the criteria rather than just based on one or two evidence from the applications.

1:17:46 Lauren Spencer
Great, thanks, Eric, and I'll go to Hao as well.

1:17:50 Hao Song   
Right, I think I think exactly Eric mentioned actually, so so the papers only part of that actually. A lot of the like the assessment, I see, .O.K. paper actually means a lot, but definitely actually that's part of the achievement, but actually we are more looking at like the the applicant’s role and the achievement, the leadership are actually, is a more well-rounded actually capability for the applicants. So it's not just about the research excellence itself. I think another actually, actually basically people might have is focusing a lot on the proposal. So, so for people actually assessing actually like Ideas, actually they're focusing more a lot on the the proposal itself, probably that's good. But now actually for NHMRC, they they the Investigator scheme actually, we are more assessing or most assessing actually try to give some of the feedbacks for the applicants to see the applicants themselves.

1:18:52 Lauren Spencer
And Patsy.

1:18:54 Patsy Yates
Yeah, I think that those, that's great advice. I think that, you know, the first thing I would sort of say is just being aware that you may have biases, that you may not even be conscious of yourself is probably the starting point. And so, and those biases come from all of those sorts of areas. You're quite right, you know, or gee, they're from X institution, so it must be good. Or gee, they've got one publication, and so it must be good, or they they've worked with X, they, you know, it's got to be really, really careful about those things. And so just being aware, I think the what I again, it's always helpful to go back and test yourself after you've done, you know, have I really been, have I?
been, have I judged something in a particular way that may imply that I've been biased? So going back to your criteria always again is so, so critical as peer reviewers. But yeah, just the first step is actually just remembering that we all do come with biases. It's not, it's not a wrong thing. It's wrong if we don't actually, though, acknowledge them and make sure that we've got that filter on, that lens on all the time.

1:20:08 Lauren Spencer
Great, thank you, Patsy. We have a question on, could you please clarify how we adjust scores based on experience? Isn't the major criterion for level selected based on years post PhD? Nick, if you want to kick us off on this one.

1:20:26 Nick Fairbairn
Thank you. There is actually a rubric in the statements of expectations which tells you how you might want to go about adjusting your scores. And the only thing I'll say is just a reminder that actually we've tried to make it really clear that PhD is not the major, in fact it's the least relevant of the four criteria. And we've done that by
I think in the past, at the SoEs, we used to introduce each level by saying, we expect an applicant to be at this many years and then went on to explain their roles and responsibilities. We've removed that reference now so that we just talk about a person, we talk about the various elements and the PhD is last. So really trying to hammer that home to all of you that that's not what you should be relying on overly, to think about what level the person's most appropriate at.

1:21:13 Lauren Spencer
Thank you, Nick. We have a question here about how many applications are peer reviewer is expecting to review this round? Is normally a cap? In past rounds, individuals have said it's been quite challenging within the timeframe available.

We're aiming this round to have everyone under 20 assessments to review, so that's something that we always try to achieve is having those lower workloads whilst having a minimum amount of applications for benchmarking purposes. If you do have concerns with the time frame available, please reach out to your secretariat as well and we may be able to accommodate an extension in those circumstances as well.

We probably have time for maybe a final question, which is, what is 1 suggestion you have for reviewers to keep in mind when undertaking their assessments? So this one, we might go across the peer reviewer mentor panel. We'll start with Courtney.

1:22:18 Courtney Ryder
Okay, mine is, it's all going to make sense to you when you start assessing and you go to the NHMRC guidelines and the rubrics and the frameworks, you go, oh, this all makes obvious sense. One of my favorite ones, or one of the ones that I really recommend having a look on, is the research life cycle for the impact, and really thinking about those three parts of how it was strategic, directed and so forth. So, you know, that's one of my top tips for diving into it.

1:22:53 Lauren Spencer
Thank you, Courtney. Hao?

1:22:58 Hao Song
Right, so my suggestion probably is just assessing the application as a whole. Actually, it's not just about the publication, not just about like the proposal or the impact. Actually, that's also considering the relative to opportunity, the statement of the expectations, a lot of things actually building the people that's making this application to you. See, so probably you need to read a lot of this and then make that in mind and then start to assess the application.

1:23:28 Lauren Spencer
Great. Thank you, Hao. Eric.

1:23:34 Eric Chow
I think my advice is we are all very busy with our work and also it takes time to review all this application and they are very important to every person that who apply because it's a five years research grant for the salary and also research programme. So I would actually say do definitely like need to delegate time within a time frame and as Lauren mentioned, like ask for extension if you need to. My tips usually I would, I think depends on like how you want to score or review application. I usually find it very helpful like if you're doing this first time, I would do publication across all the application 1st and try to get yourself like very familiar with each section and then you can actually move towards like the other section.

1:24:30 Lauren Spencer
Thanks, Eric. Over to you, Patsy.

1:24:35 Patsy Yates
They're all great tips, and I was also going to, can't emphasise enough, as I've been saying, that the resources that are there, like, you know, NHMRC has been doing this now for many, many years and has taken on board the feedback and has got really, really good guidance. And if you've got questions, often, the answers to those questions are in those guides. And if it's not, I think this process that's available now to reach out to, you know, your contacts at NHMRC to participate in the peer review sessions is another great way to sort of address, because it's not a precise science. All these questions you've raised today are very valid questions.

But so, you know, what's my tip then? You know, probably I think my tip would be in addition to what's already been said, though, is that a practical thing that I do is I often really start with ones with, you know, to get to get warmed up. I start with those applications, which I probably might say, like, I feel that's more in my comfort zone. I'm going to work with, you know, to work on that first and then you get warmed up. And once you get on a roll, I think it's just getting that start is often the hardest bit, you know, and once you sort of get on that roll, as I said, it gets easier and easier. So, you know, it's an important role that you play and you wouldn't have put your hands up to do it, as some peer reviewer, as you all know, I'm sure. Those of you who have applied, it's such a critical role.

And I just would add the bit of feedback. I really like that slide that that was presented around the feedback. You know, you all, if you've been on the end of getting that feedback, you know how that's taken by you know, you personally. So don't forget to spend just that bit of time being careful in what you craft, because it means a lot to researchers and it can be really helpful to someone's career. So there's probably a few tips I've given there, I think, yeah.

1:26:37 Lauren Spencer
Thanks, Patsy. And at time now, so I'd like to just thank Eric, Hao and Courtney and Patsy for their time today and for lending us their expertise as well across the approaching four peer reviewer mentor Q&A sessions as well that if you have further questions, we would encourage you to attend those and ask your questions, or get in contact with an NHMRC Secretariat.

I just want to thank you on behalf of our section as well, everyone for your time today and for the I guess commitment and dedication to the Investigator Grants scheme. Without you participating in peer review, our scheme delivery and awarding the highest performing researchers wouldn't be possible. So thank you all for your time.
Have a lovely day.

1:27:27 Nick Fairbairn
Thanks everyone.

End of transcript.