Testing a modified electronic version of the Edmonton symptom assessment system-revised for remote online completion with ambulatory cancer patients in Alberta,Canada

Abstract

Objective

The cancer program in Alberta, Canada routinely collects patient-reported outcomes using the Edmonton symptom assessment system-revised (ESAS-r). The program recently launched the province's new clinical information system which has expanded functionality, allowing patients to complete symptom questionnaires remotely online, instead of completing a paper form at the clinic. This study aimed to test a modified electronic version of the ESAS-r [(e)ESAS-r] with patients, to assess the feasibility of completion and questionnaire clarity.

Methods

Staff, patients, and other stakeholders worked to create modified definitions for ESAS-r symptoms, to aid in patient understanding. Patient and family advisors were recruited to test the questionnaire. Participants completed an online mock-up of the (e)ESAS-r and answered questions about technical issues. One-to-one cognitive interviews were held to discuss each symptom definition in detail. Modifications were made based on the feedback and a second round of interviews was held to finalize the wording.

Results

In total, 19 patients and 7 family advisors participated. All but one (96.2%) completed the questionnaire without assistance and had no technical issues. Participants requested certain wording modifications and that definitions be added for all symptoms for consistency. Very few participants reported any confusion with the final definitions.

Conclusions

The (e)ESAS-r was tested for clarity and ease of completion and was determined to be suitable for remote online use with ambulatory cancer patients. The enhanced definitions on the new questionnaire were clear to patients and helped ensure they understood the meaning of each symptom they were asked to rate.

Keywords

Patient-reported outcomes PROs symptom reporting feasibility study electronic completion cognitive interviews person-centered care remote patient monitoring

Background

Symptom management is an important aspect of caring for patients living with a cancer diagnosis. Identifying and monitoring symptoms and concerns is key to providing personalized and timely support. Routine collection of patient-reported outcomes (PROs) can enhance patient care when properly integrated into and used to guide clinical practice, by providing clinical teams with a more comprehensive picture of patients’ symptoms.¹ When used to tailor care based on a patient's specific concerns, PROs can lead to improvements in health-related quality of life and extended survival in certain cancer patient populations.^2,3 Using PROs also helps facilitate patient-centered communication and shared decision-making about symptom management, allowing patients to be active participants in their cancer care.^4–6

Since 2012, PROs have been routinely collected throughout the provincial cancer program in Alberta, Canada, as a standard of patient care.⁷ The PRO measure includes a symptom-rating tool called the Edmonton symptom assessment system-revised (ESAS-r).⁸ This self-report tool has patients rate the severity of nine symptoms from 0 to 10 (with 10 being the most severe): pain, tiredness, drowsiness, nausea, lack of appetite, shortness of breath, depression, anxiety, and well-being.⁹ Five symptoms have brief definitions to aid in understanding: tiredness, drowsiness, depression, anxiety, and well-being. These definitions were added based on patient feedback on the original ESAS tool, as patients found these symptoms difficult to fully understand.¹⁰ Appetite was also unclear to patients, however, instead of adding a definition, this symptom was renamed to indicate “lack of appetite” specifically. The ESAS-r is preferred by patients over the original ESAS¹¹ and has been widely used with cancer patients worldwide.⁹

The entire healthcare system in Alberta is transitioning to a new province-wide clinical information system (CIS) called Connect Care, which uses functionality by Epic Systems.¹² Cancer Care Alberta (CCA) launched Connect Care in November 2022. Connect Care has features that the previous system lacked, including a patient portal that patients can use to complete questionnaires online prior to appointments. CCA has traditionally collected this information using a paper form; however, direct online entry by patients has certain advantages. Previously, staff would manually input information from the paper form into the online system, to translate the information into electronic PROs (ePROs) so that trended data could be viewed online.¹³ Direct patient entry means that staff members will save time. The use of the paper form was also disrupted considerably when virtual care became increasingly used in CCA at the start of the COVID-19 pandemic, with the majority of patients not completing a PROs questionnaire in their virtual appointment.¹⁴ This issue could be effectively managed using online questionnaire entry.

In the health research literature, meta-analytic reviews in 2008 and 2015 confirmed that the same PRO measure is equivalent when completed on paper or on a computer.^15,16 However, this is dependent on each item or symptom being clear enough that patients can understand and respond accurately using both formats. The ESAS-r was designed for self-reporting by patients, however in Alberta it has always been completed in a healthcare setting, where patients can ask for help if needed.¹⁷ To ensure patients, as well as staff, felt comfortable transitioning to remote online entry, we wanted to be certain that patients could understand well enough to complete the questionnaire without assistance from a healthcare professional. To enhance clarity, definitions were modified for many of the nine original symptoms on the ESAS-r, with permission from the developers of this tool.¹¹

Prior to launching the electronic tool in Connect Care, we set out to test the modified questionnaire in an electronic environment. The objectives of this study were to assess if remote online completion was feasible for patients and to finalize the wording of the revised symptom definitions and validate their clarity.

Methods

Modifying the questionnaire

Over a period of several months, we engaged with key stakeholders in CCA to determine which definitions on the ESAS-r were unclear and if other symptoms would benefit from a definition. We worked with the Alberta Health Services (AHS) Human Factors team, clinical staff, and a dedicated working group of patient and family advisors (PFAs) to draft modified definitions for the five symptoms with existing definitions, as well as add a new definition for lack of appetite. PFAs found the existing ESAS-r definitions to be too short and lacking detail, so we worked to lengthen the definitions to clarify the meaning of the symptom. For example, the original definition for drowsiness is “feeling sleepy”; our modified definition was “feeling sleepy or having trouble staying awake,” providing additional clarification to help patients understand what they are being asked to rate. CCA's Patient Education team reviewed all drafted definitions to ensure patient-friendly language. Original and proposed definitions are presented in Table 1. The redesigned questionnaire was named the electronic version of the ESAS-r [(e)ESAS-r].

Table 1.

Original, proposed, and final symptom definitions for the electronic version of the Edmonton symptom assessment system-revised [(e)ESAS-r].

Symptom	ESAS-r definition	Proposed definition
Pain	None	None
Tiredness	Lack of energy	Lack of energy that does not go away with rest
Drowsiness	Feeling sleepy	Feeling sleepy or having trouble staying awake
Nausea	None	None
Lack of appetite	None	Loss of interest in food or eating
Shortness of breath	None	None
Depression	Feeling sad	Sadness that does not go away, loss of interest or feeling hopeless
Anxiety	Feeling nervous	Feeling nervous, agitated or worried
Well-being	How you feel overall	How I feel overall

Study participation

Participants were recruited from the PFA networks within CCA and the broader AHS program. Current and former patients and family members voluntarily join these networks to participate in projects and research opportunities requesting patient input. All PFAs who responded to the call for participation were included, meaning that some participants were family advisors rather than patients themselves. While PROs are, as the name specifies, intended to be completed by patients, family members may be asked by patients to provide assistance completing PROs remotely online, either by providing technical assistance or by clarifying symptom meanings in the absence of a healthcare professional. Because of this, we felt it was important to test the questionnaire with family advisors as well.

A sample size of 20–30 is often recommended for interview-based studies.^18,19 Similar studies related to testing health questionnaires have included between 16 and 30 participants^20,21; based on all of this, we set a similar goal for our study. The study received full ethics approval from the Health Research Ethics Board of Alberta Cancer Committee prior to initiating any contact with PFAs (HREBA.CC-21–0214). When individuals sign up to be PFAs, they consent to receive participation opportunities like this via email. All participants then provided implied consent by reaching out to the researchers to express interest in the study as well as verbal consent prior to the start of each cognitive interview (this method will be discussed later in this section).

Feasibility of remote online completion

A mock-up version of the (e)ESAS-r was built into an online survey platform. Participants were emailed a link to this online tool and were asked to use a computer to complete it. As this study was intentionally completed prior to Connect Care being launched, it was not possible for participants to complete the questionnaire through the actual patient portal. The mock-up included the same instructions, rating scales, and symptom titles and definitions, in the same order, as would be built into the ePRO questionnaire in the portal. In the future, it will be important to understand patients’ experiences completing the questionnaire online through the patient portal, to determine if additional instructions or other modifications may be beneficial. At the end of the questionnaire, participants were presented with a short series of questions asking about technical issues, completion problems, and their personal comfort level with computers. The questions in this section were designed based on previous studies of a similar type.²² Free-text comment boxes were included for each question to allow participants to leave more detailed feedback if they desired. Descriptive analyses were carried out on these data.

Assessing clarity of symptom definitions: cognitive interviews

After participants had completed the online questionnaire, one-to-one virtual cognitive interviews were scheduled. Cognitive interviews are a useful method for collecting detailed feedback on tools intended for use with patients, as they allow for more in-depth responses than a survey and help ensure that the wording of specific items is clear.^20,23,24 Each symptom was discussed individually in the interviews, with the current definition (if present on the ESAS-r) and proposed definition (if applicable) displayed on the screen. For symptoms with a proposed definition, participants were asked: “Is there any confusion with the definition?” and, if applicable, “Is the new definition stronger than the old one?” For symptoms without a definition, participants were asked, “Would a definition be helpful?” Participants were encouraged to provide detailed comments, positive or negative, and suggestions. The data were analyzed descriptively to determine the number and percentage of participants who responded “Yes” and “No” to each question, and qualitatively to identify common themes in participants’ comments.

After reviewing the results from the first round of interviews, some definitions were added or modified based on participants’ feedback. A second round of cognitive interviews took place to finalize these modifications and the data were descriptively analyzed. CCA's Patient Education team reviewed for plain language and approved the final definitions, with very minor wording alterations.

In addition to the nine individual symptoms, the ESAS-r also includes an “Other Problem” space, providing patients with the opportunity to write in their own symptom. The description accompanying this option is “For example constipation.” We did not modify this “Other Problem” option, and as such did not assess this piece during our study.

Results

Sample demographics

In total, 26 PFAs participated in the study. Descriptive information on the characteristics of the sample is presented in Table 2. The age of participants ranged from 24 to 79 years old, with the majority (61.5%) falling into the 51–70 age group. The sample was 76.9% female and 23.1% male. Most participants lived in a metro area (80.8%) and were patient advisors (69.2%) rather than family members (26.9%). There were more participants from CCA's PFA network (57.7%) than from the broader AHS network (42.3%). Most participated in both rounds of cognitive interviews (88.5%).

Table 2.

Participant demographics (N = 26).

Characteristics	Frequency (%)
Age
≤30	1 (3.8)
31–50	5 (19.2)
51–70	16 (61.5)
71+	4 (15.4)
Sex
Male	6 (23.1)
Female	20 (76.9)
Location
Metro	21 (80.8)
Urban	3 (11.5)
Rural	2 (7.7)
Affiliation
CCA	15 (57.7)
Other AHS	11 (42.3)
Type of advisor
Patient	18 (69.2)
Family	7 (26.9)
Both	1 (3.8)
Participated in both interviews
Yes	23 (88.5)
No	3 (11.5)

AHS: Alberta Health Services; CCA: Cancer Care Alberta.

Ease of electronic completion

Responses to the electronic completion questions revealed that only one participant (3.8%) had technical problems (Table 3). This was an issue related to the text-to-speech feature not working properly. We did not explicitly ask patients to test this feature, as this was only a mock-up; however, we took note to ensure that this feature was tested in the final patient portal. No other technical issues were reported, although one participant (3.8%) did ask for assistance from a family member as they did not feel confident completing the electronic questionnaire alone. Most participants indicated a high comfort level with computers (88.5%). Comments from participants such as “Loved filling it out electronically” and “Very simple and clear” reinforced that electronic completion was straightforward overall.

Table 3.

Summary of responses to technical questions (N = 26).

	Frequency (%)	Select participant comments^a
Technical problems
Yes	1 (3.8)	Issue with text-to-speech feature
No	25 (96.2)
Required assistance
Yes	1 (3.8)	Was not confident to electronically complete it alone
No	25 (96.2)	Loved filling it out Very simple and clear I doubt most people would need help
Comfort level with computers
Low	1 (3.8)
Moderate	2 (7.7)
High	23 (88.5)

Comments were selected to exemplify key themes found in the analyses. For additional comments, please contact the corresponding author.

Cognitive interviews

In the first round of interviews, most participants requested that definitions be added for two symptoms (Table 4), nausea (65.4%) and shortness of breath (61.5%). This left pain as the only symptom without a definition, and although only 23.1% of participants thought a definition was needed, many commented that, for consistency, all symptoms should have a definition. Very low levels of confusion were reported for the proposed definitions of tiredness (3.8% indicated some confusion), drowsiness (0%), lack of appetite (7.7%), depression (7.7%), anxiety (0%), and well-being (3.8%). Additionally, a clear majority of participants (ranging from 84.6% to 100%) felt that the modified definitions for these symptoms were stronger than the original definitions provided on the ESAS-r, and further modifications to these definitions were not needed.

Table 4.

Summary of cognitive interviews, round 1 (N = 26).

	Any confusion with the definition? n (%)		Stronger than the original definition? n (%)
	Yes	No	Yes	No
Tiredness	1 (3.8)	25 (96.2)	24 (92.3)	2 (7.7)
Drowsiness	0 (0.0)	26 (100.0)	25 (96.2)	1 (3.8)
Lack of appetite	2 (7.7)	24 (92.3)	^a	^a
Depression	2 (7.7)	24 (92.3)	26 (100.0)	0 (0.0)
Anxiety	0 (0.0)	26 (100.0)	24 (92.3)	2 (7.7)
Well-being	1 (3.8)	25 (96.2)	22 (84.6)	4 (15.4)
		Would a definition be helpful? n (%)		Participant suggestions
		Yes	No	Participant suggestions
Pain		6 (23.1)	20 (76.9)	All symptoms should have a definition for consistency
Nausea		17 (65.4)	9 (34.6)	Feeling queasy or sick Having the urge to vomit Feeling sick to your stomach
Shortness of breath		16 (61.5)	10 (38.5)	Having trouble breathing Indicate when, is it during activities or at rest? Difficulty taking deep breaths

The Edmonton symptom assessment system-revised (ESAS-r) does not have a definition for lack of appetite, so this question was not applicable to this symptom.

The second interviews were shorter and focused on the newly added definitions for pain (“physical discomfort or suffering”), nausea (“feeling sick to my stomach or like I need to vomit”), and shortness of breath (“difficulty breathing or taking a full breath”). As seen in Table 5, most participants found the new definitions very clear. Results ranged from 95.7% of participants indicating no confusion, for pain, to 100% of participants, for nausea and shortness of breath.

Table 5.

Summary of cognitive interviews, round 2 (n = 23).

Symptom	Any confusion with the definition? n (%)
Symptom	Yes	No
Pain	1 (4.3%)	22 (95.7%)
Nausea	0 (0.0%)	23 (100.0%)
Shortness of breath	0 (0.0%)	23 (100.0%)

The full questionnaire, which includes the final versions of all definitions, is included in Figure 1 in the Appendix and can be freely downloaded, to be built into an electronic platform for use with ambulatory cancer patients.

Discussion

The (e)ESAS-r was easily completed remotely through an online platform by patients with cancer and family members. Cognitive interviewing allowed for rich conversations with all participants and enabled the collection of detailed feedback on each symptom definition. The wording of each definition was thoroughly discussed to ensure clarity and understanding among the majority of patients. Assessing and modifying the definitions and testing the ease of online completion prior to the launch of Connect Care helped ensure a smooth transition to the patient portal, as small issues could be identified and addressed in advance.

Importance of testing an electronic ESAS-r

As digital health strategies are continuously being integrated into healthcare delivery, the electronic completion of PROs will become increasingly important. The ESAS-r is already used as a digital tool in multiple healthcare settings, including Canadian cancer programs in Ontario and Quebec.^25–27 However, clinicians in CCA were concerned that digital completion, when not taking place in-person, could be a problem for some patients who may not clearly understand all symptoms and would typically ask for assistance from a healthcare provider in the clinic. Careful assessment of whether the symptom definitions were adequate for remote patient reporting was needed to ensure the ePROs information generated from the (e)ESAS-r was reliable. If a patient is unable to clearly understand the specific symptoms they are being asked to rate, their results may be skewed, providing an inaccurate depiction of the patient's symptom burden. This study helped ease clinician concerns by demonstrating that the symptoms were clear and the questionnaire was easy to complete independently.

Ever since the ESAS-r was implemented for routine use in CCA, clinic staff have commented that, in their experience, patients often seem confused regarding certain symptoms, with clinicians needing to clarify with patients if they truly understood the symptom they responded to. This was commonly an issue with lack of appetite, which does not have a definition on the ESAS-r, and well-being, which is somewhat counterintuitive to respond to as a score of 0 indicates that the patient is feeling well. Both symptoms have been identified as problematic items on the ESAS-r in previous studies.^10,11 Assessing, modifying, and finalizing the wording of these and all other definitions with patients and family members confirmed that we can be confident that patients will be able to better respond to each symptom and complete the questionnaire, with few exceptions.

ePROs as facilitators of effective communication and real-time response

Regardless of format, routine PRO tools can help encourage effective communication between patients and providers. A previous study demonstrated that patients reported more symptoms using a systematic assessment tool than they reported verbally to their provider.¹ Cancer Care Ontario routinely collects ePROs using digital entry kiosks in clinic waiting rooms, with positive results: communication and shared decision-making have improved, and patients feel better able to clearly share their experience with their care team.^4,6,28 ePROs can further improve communication by enabling real-time clinical response, which can help prevent worsening of symptoms. A study conducted in the United States compared patients using ePRO tools to patients receiving standard care and found improved health-related quality of life and increased survival among the ePRO cohort.² One aspect of the ePRO intervention in the study was that nurses received email alerts triggered by patients reporting severe or worsening symptoms, enabling the notification and ability to phone the patients and intervene.² This type of real-time response may have significant benefits for patients and will be considered in the future in CCA, using the enhanced functionality of the new CIS. The remote completion of ePRO tools also creates the possibility of patients reporting their symptoms between clinic appointments.^2,29 This can further enhance communication between patients and providers by enabling care teams to see a full picture of a patient's symptom trends over time.

Limitations

Although the results of this study were positive, it is important to recognize the limitations. The sample of PFAs is not representative of all cancer patients in Alberta, with the majority being female and living in a metro area. The vast majority of participants indicated a high comfort level with computers which may not accurately reflect all patients in Alberta, although a recent survey of cancer survivors in Alberta suggests that regular computer and smartphone use is widespread.³⁰ Comfort with and use of technology does not necessarily translate into digital health literacy, and this should be considered in future studies involving Connect Care's patient portal.³¹ Participants from CCA likely had previous experience working with the paper ESAS-r, which may have aided in their completion and understanding of the electronic version. This limitation is minimized by the fact that the AHS advisors were also able to complete the online questionnaire without difficulty. The (e)ESAS-r was only built and tested in English; should the tool be translated into other languages, testing should be repeated to ensure clarity in each language.^32,33 Finally, the (e)ESAS-r tested in this study was a mock-up only and has minor differences in format compared to the final questionnaire built into the patient portal. Now that Connect Care has launched, we can plan additional studies to understand patients’ experience with the portal, including experiences with the mobile-friendly version, to determine what impact, if any, the final format has on the ease of completion.

Conclusions

The (e)ESAS-r was determined to be feasible for remote electronic completion in CCA, and the final definitions were clear and helpful to participating advisors. This study was part of a larger endeavor to further redesign and validate CCA's PROs questionnaire for use as a comprehensive ePRO tool in the new patient portal. Cancer programs or individual providers who are interested in remotely administering an electronic version of the ESAS-r to patients can use the version tested in this study and can be confident that completion should be feasible for most patients. The collection of PROs through remote electronic completion has been underway in CCA since November 2022, although a paper copy of the questionnaire is still available at in-person clinic appointments, as not all patients have signed up for the patient portal at this time. Evaluation of the final version of the questionnaire, completed through the actual patient portal, is an important next step and will help determine what strategies are needed to further assist patients and staff members in effectively utilizing ePROs.

Footnotes

Acknowledgements

The authors thank the Canadian Partnership Against Cancer for funding this work. They also acknowledge Alberta Health Services Operations for their support and allocation of resources to this work.

Contributorship

All authors contributed to the study's conception and design. CL and SQ researched the literature and conceived the methodology. CL developed the protocol and managed the ethics approval. CL, AD, LC, and AH were involved in patient recruitment and data collection. Data analysis was performed by CL. LW, AD, and LB provided supervision. CL wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This study was approved by the Health Research Ethics Board of Alberta—Cancer Committee (HREBA.CC-21-0214, approved 9 June 2021).

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the The Canadian Partnership Against Cancer (CPAC) (grant number AHS-2018-05-9176).

Guarantor

ORCID iD

Claire Link

Appendix

References

Homsi

Walsh

Rivera

, et al. Symptom evaluation in palliative medicine: Patient report vs. systematic assessment. Support Care Cancer 2006; 14: 444–453.

Basch

Deal

Kris

, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized control trial. J Clin Oncol 2016; 34: 557–567.

Denis

Basch

Septans

A-L

, et al. Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA 2019; 321: 306–307.

Basch

Barbera

Kerrigan

, et al. Implementation of patient-reported outcomes in routine medical care. Am Soc Clin Oncol Educ Book 2018; 38: 122–134.

Zhang

Burgess

Reddy

, et al. Provider perspectives on the integration of patient-reported outcomes in an electronic health record. JAMIA Open 2019; 2: 73–80.

Yang

Manhas

Howard

, et al. Patient-reported outcome use in oncology: a systematic review of the impact on patient-clinician communication. Support Care Cancer 2018; 26: 41–60.

Cuthbert

Watson

, et al. Patient-reported outcomes in Alberta: Rationale, scope, and design of a database initiative. Curr. Oncol 2019; 26: 503–509.

Canadian Partnership Against Cancer. 2012 cancer system performance report. Updated 2012, Ontario, Canada. https://www.systemperformance.ca/report/performance-report-2.

Nekolaichuk

Watanabe

Beaumont

. The Edmonton symptom assessment system: a 15-year retrospective review of validation studies. Palliat Med 2008; 22: 111–122.

10.

Watanabe

Nekolaichuk

Beaumont

, et al. The Edmonton symptom assessment system – what do patients think? Support Care Cancer 2009; 17: 675–683.

11.

Watanabe

Nekolaichuk

Beaumont

, et al. A multicenter study comparing two numerical versions of the Edmonton symptom assessment system in palliative care patients. J Pain Symptom Manage 2011; 41: 456–468.

12.

Alberta Health Services. Connect Care Manual: Access to Epic UserWeb. Updated 2023, Alberta, Canada. https://manual.connect-care.ca/access/UserWeb.

13.

Watson

DeIure

, et al. Utilizing patient reported outcome measures (PROMs) in ambulatory oncology in Alberta: Digital reporting at the micro, meso and macro level. J. Patient Rep. Outcomes 2021; 5: 97.

14.

Watson

DeIure

, et al. Virtual cancer care during the COVID-19 pandemic in Alberta: Evidence from a mixed methods evaluation and key learnings. JCO Oncol Pract 2021; 17: e1354–e1361.

15.

Gwaltney

Shields

Shiffman

. Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review. Val Health 2008; 11: 322–333.

16.

Muehlhausen

Doll

Quadri

, et al. Equivalence of electronic and paper administration of patient-reported outcome measures: a systematic review and meta-analysis of studies conducted between 2007 and 2013. Health Qual Life Outcomes 2015; 13: 167.

17.

Nekolaichuk

Beaumont

Watanabe

, et al. Edmonton symptom assessment system – revised (ESAS-r): Administration manual. Updated 2019, Edmonton, Canada. Alberta Health Services (AHS) Edmonton Zone Palliative Care Program; Covenant Health (CH) Palliative Institute; Division of Palliative Care Medicine, Department of Oncology, University of Alberta. https://www.albertahealthservices.ca/assets/info/peolc/if-peolc-ed-esasr-admin-manual.pdf.

18.

Creswell

. Qualitative inquiry and research design: Choosing among five approaches. 2nd ed. London: Sage, 2007.

19.

Vasileiou

Barnett

Thorpe

, et al. Characterising and justifying sample size sufficiency in interview-based studies: Systematic analysis of qualitative health research over a 15-year period. BMC Med Res Methodol 2018; 18: 148–166.

20.

Garcia

. Cognitive interviews to test and refine questionnaires. Public Health Nurs 2011; 28: 444–450.

21.

Chan

CWH

Tam

Cheng

KKF

, et al. Piloting electronic self report symptom assessment – cancer (ESRA-C) in Hong Kong: a mixed method approach. Eur J Oncol Nurs 2011; 15: 325–334.

22.

W-W

Johnson

Schepp

, et al. Electronic self-report symptom and quality of life for adolescent patients with cancer: a feasibility study. Cancer Nurs 2011; 34: 479–486.

23.

Taylor

Fern

Solanki

, et al. Development and validation of the BRIGHTLIGHT survey, a patient-reported experience measure for young people with cancer. Heath Qual Life Outcomes 2015; 13: 107.

24.

Reeve

McFatrich

Pinheiro

, et al. Cognitive interview-based validation of the patient-reported outcomes version of the common terminology criteria for adverse events in adolescents with cancer. J Pain Symptom Manage 2017; 53: 759–766.

25.

Hui

Bruera

. The Edmonton symptom assessment system 25 years later: Past, present and future developments. J Pain Symptom Manage 2017; 53: 630–643.

26.

Barbera

Moody

. A decade in review: Cancer care Ontario’s approach to symptom assessment and management. Med Care 2019; 57: S80–S84.

27.

Tran

Zomer

Chadder

, et al. Measuring patient-reported outcomes to improve cancer care in Canada: an analysis of provincial survey data. Curr Oncol 2018; 25: 176–179.

28.

Howell

Molloy

Wilkinson

, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol 2015; 26: 1846–1858.

29.

Mooney

Berry

Whisenant

, et al. Improving cancer care through the patient experience: How to use patient-reported outcomes in clinical practice. Am Soc Clin Oncol Educ Book 2017; 37: 695–704.

30.

Ester

McNeely

McDonough

, et al. A survey of technology literacy and use in cancer survivors from the Alberta cancer experience program. Digital Health 2021; 7: 1–10.

31.

Canada Health Infoway. Consulting Canadians on future of their health system: a health dialogue (executive summary). November 2020, Ontario, Canada. https://www.infoway-inforoute.ca/en/component/edocman/resources/reports/3850-a-healthy-dialogue-executive-summary.

32.

Yokomichi

Morita

Nitto

, et al. Validation of the Japanese version of the Edmonton symptom assessment system – revised. J Pain Symptom Manage 2015; 50: 718–723.

33.

Carvajal

Hribernik

Duarte

, et al. The Spanish version of the Edmonton symptom assessment-revised (ESAS-r): First psychometric analysis involving patients with advanced cancer. J Pain Symptom Manage 2013; 45: 129–136.