Canada's First Experience With a Patient-Adjudicated Cancer Research Grant Competition: A Case Study

Abstract

Although patient partners have been part of grant review panels, they are rarely given decision-making authority and never given sole responsibility for determining what research to fund. In patient-oriented research, however, patient partners may be in the best position to determine what is needed and whether the proposed research will engage patients in meaningful ways. The objective of this case study was to demonstrate that patient partners can adjudicate patient-oriented research proposals without the inclusion of researcher/expert reviewers.

Key Points

• Adjudication of patient-oriented research applications can be undertaken entirely by patient partners.

• There was consensus among patient partners in their ratings of research proposals, particularly for the highest and lowest rated proposals.

• Grant adjudication provides a useful experiential learning opportunity for patient partners—helping them to better understand the research process and build their skills as research partners.

Introduction to the Issue

The Canadian Cancer Research Alliance (CCRA) has hosted a pan-Canadian cancer research conference on a biennial basis since 2011. A parallel program, PIP (Patient Involvement in Cancer Research Program), was initiated in 2017. PIP provides patient partners with an opportunity to learn about leading-edge science from Canada's cancer researchers and trainees and to, likewise, offer the scientific community an opportunity to learn from and interact with patient partners keen on building their knowledge of cancer research and demonstrating the value that engaged patients can bring to the cancer research process.

To facilitate an enriched learning experience, organizations supporting PIP requested the development of a small research grant competition to be adjudicated entirely by participants of the 2021 PIP. This research grant was designed to provide a “hands-on” (virtual as necessitated by the pandemic) learning opportunity for patient partners and fund a research project with a patient-oriented focus. To our knowledge, this is the first such research grant competition determined solely by patient partners.

Description of the Intervention

A small grant competition was developed that offered a single one-year grant in the amount of CAD$15,000. The competition targeted trainees and early career investigators. Grant guidelines and an application form were created, the latter being brief by design and focused on the essential elements of the proposed project and the anticipated outcomes/benefits for patients. This simplified application was designed to reduce applicant burden (particularly relevant given the grant budget and tight timelines) as well as reviewer burden. Curriculum vitae and letters of reference were not required as part of the application and the form was designed so that the identifiers could be easily stripped to facilitate a blinded review process. A blinded review process was identified as an important means to level the playing field for all applicants (ie, remove potential bias that may arise for student applications) and remove the potential for gender and other biases that have been reported in the literature (1-3). In addition, it was felt that a blinded process would help mitigate perceptions of power imbalances well described in the patient engagement literature (4-6).

The grant guidelines detailed the rating criteria and general process for grant evaluation. There was a 4-week turn-around between the call for applications and the application deadline. Nineteen applications were received. Applicants had a mix of qualifications: 3 masters candidates, 5 doctoral candidates, 7 postdoctoral fellows/residents, and 4 early career investigators.

In the first phase of the adjudication, 22 PIP participants were paired (11 dyads) and randomly assigned applications to review. Ten dyads reviewed 5 applications and 1 dyad reviewed 6, which means that each application was reviewed by 3 dyads (ie, 6 participants in total reviewed each application). Applications were reviewed over a 16-day period and evaluated on 4 criteria using 5-point rating scales: proposed impact of the research; patient integration/involvement approach; readability of the proposal; and feasibility of the work plan. The maximum possible score was 60 points per application (20 points per dyad). Scores ranged from 30 to 57 (M = 43.8, SD = 7.6).

The 3 top-rated applicants were selected to proceed to the second phase of adjudication for which each applicant participated in a 30-min online videoconference session during which the first 10 min was spent on a presentation of their proposed research and the remaining 20 min on questions and feedback from PIP participants. This allowed PIP participants to identify areas for improvement in terms of patient engagement for each proposal.

The 3 online sessions were held over 2 consecutive days. The sessions were recorded so that PIP participants unable to attend the live session could view at their convenience. Each PIP participant was asked to submit their individual ratings within the 2 days that followed on 4 criteria using 5-point rating scales: patient integration/involvement approach; clarity of the presentation; receptiveness to feedback; proposed impact of the research. One of the participants who participated in the first phase of the review did not provide ratings for the second phase.

The PIP participants met as a group 3 days after the online presentations concluded to discuss the ratings for the 3 applicants and to determine the successful applicant. After discussion, consensus was that the highest rated applicant should be conferred the award, although participants indicated that all 3 projects were deserving of funding. As such, the winner and other 2 top-rated applicants were all highlighted at the 2021 conference.

Evidence of Impact

A summary of the dyad ratings is provided in Figure 1. There was a high degree of concordance among the averaged ratings, apart from 4 applications (marked with an asterisk). In these instances, 1 of the 3 dyads had a divergent and lower rating.

Figure 1.

Dyad ratings per application (19 applications).

In the second phase of online presentations, ratings were very similar for the 3 top-rated applicants (A01 M = 17.7, SD = 2.2; A02 M = 16.6, SD = 2.8; A03 M = 16.5, SD = 3.2) and mirrored the initial ranking from the dyad rating process.

In a post-program survey, all PIP participants felt that PIP grant adjudication was valuable (89%) or somewhat valuable (11%). The experience yielded a sense of empowerment, and many patient partners have since gone on to participate in further grant reviews in researcher-led panels. In their own words:

The grant review helped build bench strength for patient partners to be involved in the grant review process going forward.

I loved the PIP grant review. I had a great partner and we learned from one another.

In my view, the review process was a success, and it clarified several points/doubts for me.

Because of our experience with the PIP grant, there are now 21 PIPs across the country who are much more knowledgeable about grant processes. Having knowledgeable patient partners will go a long way to lessening the dreaded “Tokenism” that exists.

Working in dyads for the first round of ratings was suggested by the patient partners who were part of the PIP working group (paper's authors). This was an essential component of the success of this approach.

PIP participants, the grant awardee, and some of the other grant applicants (voluntary basis) provided feedback on how best to adjust and improve this opportunity going forward. Suggestions are provided in a report on the 2021 PIP available on the CCRA website and will be major considerations for program planning in 2023.

Conclusions

The PIP participants were a diverse group in terms of where they lived, their demographic characteristics, and cancer experience. They were highly educated, and most were actively engaged in advisory, advocacy, and support activities in health services and oncology. While background reading was suggested, no formal orientation to grant review was provided to reduce the overall program time commitment. It is recognized that formalized preparation in a group setting in advance of the adjudication process would be helpful, particularly for those participants with less exposure to patient engagement and research.

In the future, comparing patient partner adjudication to a hybrid approach involving researchers and patient partners would be useful to ascertain if adjudication outcomes would differ. Furthermore, although the process was designed to minimize bias, a controlled experiment to test for the influence of applicant and reviewer factors on ratings would help advance the field. A recent review suggests a realist lens for renovating peer review, with incremental change and innovation as strategies for process improvement 7.

Notwithstanding these limitations, this initial pilot effort suggests that grant adjudication can be successfully undertaken by patient partners, fulfilling the dual aims of providing an experiential learning opportunity on research grant adjudication and determining important patient-oriented research to support.