The A*STAR and NHMRC joint peer review is designed to provide a rigorous, fair, transparent and consistent assessment of the merits of each application according to the Australian Code for the Responsible Conduct of Research to ensure that only the highest quality, value for money research is recommended for funding (see section 11.2 of the NHMRC Funding Rules 2017).
All applications are assessed against the Assessment Criteria as set out in the section 3 of the scheme-specific Funding Rules 2017, using the Category Descriptors at Attachment A. Applications that are accepted by NHMRC as relating to health research involving Aboriginal and Torres Strait Islander peoples are also assessed against the Indigenous Research Excellence Criteria as set out in section 6.3 of the NHMRC Funding Rules 2017.
Applications are assessed relative to opportunity, taking into consideration any career disruptions (see section 6.2 of the NHMRC Funding Rules 2017).
An overview of the NHMRC/A*STAR peer review process can be found at section 1 of this document with further detail about each step provided below.
3.1 Before the GRP Meeting
3.1.1 Receipt and Initial Processing of Applications
NHMRC and A*STAR staff will verify that applications meet eligibility criteria.
3.1.2 Appointment of joint GRP and identification of Conflicts of Interest
NHMRC and A*STAR will establish a joint GRP and appoint members to provide assessments against submitted applications.
Panel members will be provided access, via RGMS to the Snapshot Summary Report of each application assigned to the GRP, and will declare their CoI in accordance with the guidance on the management of CoI (refer to section 4.3 of the Guide to NHMRC Peer Review 2017)
Panel members will be given access to the full application only if they have no or a low CoI. Where panel members declare they have a high CoI, they will not be granted access to the full details of the application.
Some GRP members may have a CoI for which they require a ruling. In this instance, NHMRC staff will assess the information in the declaration and specify a particular level of participation. Members are requested to ensure they include sufficient detail in their declaration to ensure an accurate CoI assessment can be made by NHMRC staff. Important details include:
- In the case of collaborations and relationships (e.g. publications, grants, etc.), did these activities occur five or more years ago, or are they more recent?
- Is the collaboration (e.g. publications, grants, etc.), with a Chief Investigator, or an Associate Investigator?
The answers to these questions will help NHMRC to assess CoI. The peer review process is more rigorous if experts are not unnecessarily excluded from the assessment process due to ambiguity arising from excessively brief CoI declarations.
CoIs must be declared at the beginning of the peer review process. However CoIs may be declared at any stage of the peer review process if new conflicts become apparent.
GRP members a must not approach applicants, and should ensure that they cannot be identified by applicants at any point during the review process.
3.1.3 Assessment of applications with an Aboriginal and Torres Strait Islander health focus
Applications relating specifically to Aboriginal and Torres Strait Islander Peoples’ health will be identified by information provided by the applicant in their application. NHMRC Assigners Academy members with Aboriginal and Torres Strait Islander health expertise will confirm that these applications have at least 20% of their research effort and capacity building focused on Aboriginal and Torres Strait Islander health.
For applications confirmed as relating specifically to Aboriginal and Torres Strait Islander health, NHMRC will endeavor to obtain at least one Indigenous researcher or relevant expert to take on the role of an external assessor to complete the peer review of applications with a focus on Aboriginal and Torres Strait Islander health.
The external assessor’s review will have a particular focus on the Indigenous Research Excellence Criteria (section 6.3 of the NHMRC Funding Rules 2017).This assessment is to be considered by GRP members when scoring.
3.1.4 Allocation of Spokespersons
Panel members will indicate their ability to act as a Primary Spokesperson (1SP) or Secondary Spokesperson (2SP) on particular applications based on the closest match with their expertise. NHMRC staff will aim to allocate an Australian and Singapore Spokesperson to each application based on the indicated suitability and declared conflicts of interest of each panel member. Panel members will be notified of their allocations accordingly.
3.1.5 GRP Members Access Applications
All panel members will be provided with access to the full application where there is a no or low CoI. When accessing the full application, panel members should again check whether they have a CoI not previously evident.
GRP members who become aware of any previously undeclared CoI should contact the NHMRC Project Officer immediately. The panel member will be required to delete or destroy any files in their possession pertaining to applications with which they have declared a late high CoI.
3.1.6 Preparation of Spokespersons reports and initial scores
Spokespersons will submit their written assessments into RGMS for the applications they have been assigned the role of Spokesperson. Assessments will address all four assessment criteria, and scores will be provided for each criterion using the Category Descriptors at Attachment A as a guide).
NHMRC and A*STAR will not review GRP members’ comments. Therefore, GRP members must ensure that their assessments do not contain inappropriate or defamatory remarks.
The 1SP and 2SP prepare a Spokesperson report that will be provided to the applicant. The Spokesperson report should discuss the strengths and weaknesses of the application (see Attachment B for further guidance) and include questions on those aspects of the application that require clarification, including the appropriateness of the requested budget. To ensure impartiality and independence of assessments the SPs must not enter into dialogue, nor will they have access to other SP reports, thereby ensuring an independent assessment.
When determining questions for the applicant, the SPs should:
- be clear and concise so as to not mislead the applicant;
- not provide an opportunity for the applicant to modify the research plan in any major way;
- not provide a report which has so many questions/ critiques that the applicant cannot respond within the three page limit;
- not provide arbitrary or irrelevant commentary on the application; and
- not identify themselves as the 1SP or 2SP
Questions may seek to:
- ask additional scientific questions that may assist the GRP to understand the application;
- gain further justification of any perceived weaknesses in the project;
- clarify budget issues, including appropriateness of PSPs, DRCs and equipment (where applicable) requested. If it is felt that the budget is overstated, it is essential that the applicant is given a chance to defend their proposal;
- clarify relationships with other applications, funding sources and existing grants held by the applicants; and
- give the applicant the opportunity to explain any identified issues or problems with their track record, or where relevant, the composition of the proposed research team.
At this point, the 1SP and 2SP will also provide initial scores in RGMS against each assessment criterion, using the Category Descriptors at Attachment A.
3.1.7 Release of Spokesperson Reports
The Spokesperson Reports will be made available to applicants through RGMS. This report will be released to applicants within a set timeframe (see overview in section 1) and can only be released when both spokespersons have submitted their written comments.
Spokesperson reports may not be released within the anticipated published timeframes if the reports have not been submitted.
Note: The comments provided in the Spokesperson Report should not be considered an indication of the final outcome of the application.
3.1.8 Applicant Response
The Spokesperson Report will be provided to applicants in the release period: 18-28 May 2017.
Applicants will be given a clear indication as to the required format of their response and the date that it is due at the release of their Spokesperson Report.
The response should address the questions raised and is not an opportunity to modify the proposed research plan.
Applicants will be allowed up to 10 days, inclusive of weekends and public holidays, in which to submit their Applicant Response.
The provision of Spokesperson Report to applicants is dependent upon the timely provision and availability of reports from the Spokespersons and therefore may occur outside of the indicative time period. Where significant delays occur, applicants will be notified.
3.1.9 Spokesperson reassessment of applications
Once the Applicant Responses have been received, the 1SP and 2SP for each application will consider the research proposal in conjunction with the Applicant Response. The 1SP and 2SP will then be asked to review their initial scores and may rescore the application in RGMS against each of the four assessment criteria.
3.1.10 Removing Less Competitive Applications - Not For Further Consideration (NFFC)
The Not for Further Consideration (NFFC) process aims to identify and remove applications that are assessed as being less competitive than others on the panel.
The Spokespersons’ scores following review of the Spokesperson Report and Applicant response will determine the identification of applications considered to be the least competitive of those assessed the GRP.
An application may only be included on the NFFC list if NHMRC has received a score from both the 1SP and 2SP. The bottom 50% of applications will be included on the NFFC list. The NHMRC will review all applications appearing on the NFFC list to confirm that no applications have scores from the Spokespersons which are two or more points away from each other.
A NFFC list, tailored for conflicts of interest, will be provided to panel members before the GRP meeting. If a panel member feels strongly that an application warrants rescuing from the NFFC list (and should proceed to full review), they have an opportunity to nominate one application only for consideration by the panel. If a member would like to rescue an application, they should notify the NHMRC Project Officer via email within the given timeframe. Those applications remaining on the NFFC list will be removed from the list for detailed discussion at the GRP meeting.
If a late CoI is declared by the 1SP or 2SP for applications that appear on their NFFC list, a new 1SP or 2SP will be assigned to the application, and the application will be reviewed in detail by the panel. The scores from the conflicted Spokesperson/s will be discarded.
It is important to note that applications on the NFFC list are subject to CoI considerations, as are all applications, and therefore should not be discussed between members.
Once the NFFC list has been finalised, the GRP secretariat will release a running order for the GRP meeting.
Applications not appearing on the NFFC list will automatically proceed to full review.
3.2 GRP Meeting (Videoconference)
The GRP will meet via videoconference to review each application that is not on the NFFC list.
The GRP will commence with an induction session for all members. The induction session will provide an opportunity for members to ask questions and clarify any matters relating to the peer review process. Attendance is compulsory.
3.2.1 Declaration of inter-relationships - (suggested time limit – 10 minutes)
When all members connect, each panel member will be invited to briefly describe their expertise and previous experience sitting on any review panels. During their introductions, members will be asked to declare any relationships with other panel members including:
- current collaborations and previous collaborations;
- former student/teacher/mentoring relationships;
- common employment/institutional relationships; and
- other relationships that may, or be seen to, impair fair and impartial judgement.
This information is sought for the benefit of panel members, who may raise any concerns arising from declarations with NHMRC and/or A*STAR staff.
Review of applications will take place via videoconference and will be conducted as follows:
3.2.2 Chair to announce the application - (suggested time limit – 2 minutes)
The Chair will announce the application to be discussed including the title, institutions, and the names of all applicants in the research team.
The Chair will then identify any members that have previously identified a CoI with an application. Those members with a high CoI will be asked to mute/disconnect from the meeting.
The Chair will also invite members to disclose if they have since identified an interest with the application. Where a member declares a new interest at the joint GRP meeting, the following process will take place:
- The declaring member must disclose in detail, the nature and extent of the interest and how it relates to the work of the committee.
- If the panel determines that the interest is not a personal material interest, and no panel member objects to that determination, the member may remain in on the line.
- If the panel determines that the interest is a personal material interest, the member must mute/disconnect from the meeting.
- If the disclosing member is required for the assessment of the application for quorum or particular expertise, the panel can make a determination by majority vote that the disclosing member can participate and determine how that participation will occur. For example, the panel may decide that the disclosing member can answer questions, or contribute to the discussion of the application but not participate in the scoring, or that they should not remain in the meeting or on the line.
- If the interest is disclosed by the Chair, NHMRC Senior Research Scientists will make the final determination as to whether the interest is a personal material interest that precludes the Chair from involvement. The panel cannot be involved in this determination.
- No application can proceed until all CoIs have been considered and dealt with appropriately. The Chair will highlight if any CoIs have been noted.
If a CoI is declared at the GRP meeting by a Spokesperson, which prevents them from participating in the assessment of the application, a new Spokesperson will be assigned to the application. Discussion of the application will be moved to a later time to give the new Spokesperson/s time to prepare.
Once highly conflicted members have left the meeting, those with a low CoI are allowed to remain), the Chair will then identify the Spokespersons and ask them to begin the assessment discussion.
3.2.3 1SP to comment on the application (Suggested time limit - 7 minutes)
The 1SP will:
- Outline the nature of the career disruption(s) (where applicable).
- Provide a concise summary of the application and discuss the strengths and weaknesses against the assessment criteria taking into consideration the Aboriginal and Torres Strait Islander health research assessment (if applicable). The 1SP will assume that the GRP members are familiar with documentation relating to the application.
- Ensure that relevant considerations (e.g. track record relative to opportunity, career disruptions) are taken into account.
- Not make reference to the budget at this stage.
3.2.4 2SP to comment on the application (Suggested time limit- 3 minutes)
The 2SP will:
- Briefly highlight their agreement/disagreement with the 1SP.
- Comment on the applicant’s response to the Spokespersons Report and whether the questions were suitably addressed.
- Not make reference to the budget at this stage.
3.2.25 Full panel discussion (Suggested time limit- 10 minutes)
The application will then be opened to the panel for general discussion. GRP members have an opportunity to ask questions of both SPs discuss the strengths and weaknesses of the application and ensure that relevant considerations are taken into account. The Chair must ensure adequate review of the application occurs, that all members get a fair opportunity to comment and no member exerts undue influence over others.
A quorum must be present for discussion and scoring to occur. For the purposes of GRP meetings, a quorum is one member more than half the total number of scoring members on the GRP. NHMRC and A*STAR will endeavour to identify, prior to the GRP meeting, those applications that do not have a quorum and obtain further panel members.
3.2.6 Scoring by members (Suggested time limit- 5 minutes)
Following the panel’s discussion, the Chair will ask the SPs to provide their scores against the four criteria.
The Chairs will then ask if any GRP member intends to score two or more away from any of the SPs’ four criterion scores. The GRP member must declare this to the GRP and provide a brief justification, which will be recorded by the Secretariat.
All non-conflicted GRP members, excluding the Chairs and Co-Chair, will then confidentially score the application via RGMS Electronic Scoring (E-Scoring). All scoring GRP members will submit their score against each of the four Assessment Criteria using the Category Descriptors at Attachment A.
Collation of the members’ scores will be managed by the Secretariat. At the completion of scoring, the Secretariat will announce the following results to the GRP:
- Rating - The rating will be determined by including each scoring member’s score for each of the assessment criteria. The rating, as calculated arithmetically to three decimal places, will take account of the weighting of each criterion; and
- Category – this will be deemed, based on the calculated rating, as follows:
6.501 - 7.000
|deemed as Category 7|
5.501 - 6.500
|deemed as Category 6|
4.501 - 5.500
deemed as Category 5
3.501 - 4.500
deemed as Category 4
2.501 - 3.500
deemed as Category 3
1.501 - 2.500
deemed as Category 2
1.001 - 1.500
deemed as Category 1
The Co-Chair and Secretariat will record these scores. Where members are uncertain or have concerns regarding the final score, the Chair should invite further discussion. If any member still disagrees with the outcome, members will be invited to re-score for that application.
3.2.7 Discussion of proposed budget (Suggested time limit- 5 minutes)
All applications that receive a deemed category of 5 or above will trigger a discussion of the proposed Australian and Singapore budgets.
The Chair will lead the budget discussions to ensure equity is achieved between applications. Each SP will be prepared to discuss the proposed budget for the aspect of the proposal being conducted in their country, and to comment on the appropriateness of the outlined costs. Other panel members may also provide relevant assessment. Where the GRP deems that the proposed budget is in excess of that required to accomplish the research objectives, appropriate reductions may be recommended.
When reviewing the Australian component of the budget, the Australian GRP members will consider the elements of the budget, including the justification, and provide advice on an appropriate budget for the application (refer to section 8 of the NHMRC Funding Rules 2017). The Co-Chair and Secretariat will then record budget recommendations as agreed to by the panel. The rationale for differences between the recommended and requested budget will be annotated. The Chair will verify that the budget recommendations have been recorded correctly.
The Singapore component of the budget will be reviewed by the Singapore GRP members, according to the Guidelines for Managing A*STAR Grants.
NHMRC and A*STAR reserve the right to amend the budgets recommended by the GRP for any grant.
NOTE: Australian budgets greater than $350,000 (AUD) will be reduced to $350,000 (AUD) (the maximum allowed) or less. A*STAR budgets greater than $350,000 (SGD) will be reduced to $350,000 (SGD) (the maximum allowed) or less.
3.2.8 Reconciliation of applications
At the end of each day’s deliberations, a reconciliation of applications reviewed will take place. This process gives GRP members an opportunity to raise any concerns regarding applications that have been reviewed throughout the day or earlier in the week.
Where a GRP member believes an application may have been reviewed in an inconsistent manner, they should raise the matter with the GRP Chair. The NHMRC secretariat will ensure that panel member with high CoIs leave the discussion prior to details of the application and the circumstances of concern being outlined to the panel.
In the event that an application needs to be reassessed, the application will be reopened for discussion and rescored by the panel at the next opportunity (i.e. the following day).
The Chair may also revisit budget discussions at the end of each day to ensure consistency was achieved.
3.3 Post GRP Meeting
The following actions will occur after the GRP meeting concludes:
- Confirmation of final ranking – NHMRC will discuss and confirm with A*STAR the ranked list provided at the end of the panel meeting, prior to developing the funding recommendations.
- Provisions of funding recommendations –NHMRC will review the list of applications and determine which applications will be recommended for funding. Research Committee recommends those applications to be funded through NHMRC Council to the CEO, who submits them for approval to the Minister for Health.
- Announcement of outcomes – subsequent to Ministerial approval, applicants and Research Administration Officers will be advised of the outcome of their application.
- Preparation of Application Assessment Summary – Applicants will receive feedback on the application following the approval process where the application was discussed in full by the GRP. Applicants whose applications have been deemed NFFC will receive a letter indicating their application was assessed to be among the least competitive applications.
3.4 Retention of GRP Documentation
GRP members are to retain their notes made during the peer review process for six months after the GRP meeting. After this date, both hard copy and electronic notes should be destroyed to ensure the maintenance of confidentiality. In exceptional circumstances, NHMRC may request a GRP member to comment on issues raised in a complaint to the NHMRC.