Development of a Patient-Reported Experience Measure Tool for Ambulatory Patients With Acute Unexpected Needs: The APEX Questionnaire

Abstract

Background: The aim of this study was to develop a patient-reported experience measure (PREM) for comparing the experience of care received by ambulatory patients with acute unexpected needs presenting in emergency departments (EDs), walk-in clinics, and primary care practices. Methods: The Ambulatory Patient EXperience (APEX) questionnaire was developed using a 5-phase mixed-methods approach. The questionnaire was pretested by asking potential users to rate its clarity, usefulness, redundancy, content and face validities, and discrimination on a 9-point scale (1 = strongly disagree to 9 = strongly agree). The pre-final version was then tested in a pilot study. Results: The final questionnaire is composed of 61 questions divided into 7 sections. In the pretest (n = 25), median responses were 8 and above for all dimensions assessed. In the pilot study, 63 participants were enrolled. Adjusted results show that access, cleanliness, and feeling treated with respect and dignity by nurses and physicians were significantly better in the clinics than in the ED. Conclusion: We developed a questionnaire to assess and compare experience of ambulatory care in different clinical settings.

Keywords

emergency department primary care practices ambulatory care patient experience quality of care patient-reported experience measure (PREM)

Introduction

Overcrowding in emergency departments (ED) has been a problem for decades. Many observers point out that a significant number of ED visits could be directed to primary care to prevent or treat low-acuity conditions.¹ It is generally believed that non-ED alternatives such as primary care practices and walk-in clinics offer good quality of care at a lower cost to low-acuity ambulatory ED patients. However, data supporting these assumptions are scarce and often contradictory.^2,3 Moreover, to our knowledge, very few studies have assessed this issue from the patients’ perspective.⁴

In a patient-centered system, assessing the patient's perspective is crucial to providing optimal care pathways and quality of care. Some evidence suggests that this practice can break down the barriers between health professionals and patients, ensure a more transparent system, shift emphasis onto patients’ priorities, and reduce healthcare costs.^5,6 Moreover, the quintuple aim adopted by many healthcare organizations (improving population health, patient experience and care providers’ work life while reducing the cost of care and improving health equity),^7,8 underscores the importance of the patients’ experience to successful organizations. It is therefore pivotal to measure patient experience using validated tools to effectively improve health services.

The aim of this study was to develop a patient-reported experience measure (PREM) in English and French suitable for assessing and comparing the experience of ambulatory care patients with acute unexpected needs presenting in EDs, walk-in clinics, and primary care practices.

Methodology

Study Design

The APEX (Ambulatory Patient EXperience) questionnaire was developed in five phases using a mixed-methods research design, in accordance with the Guidelines for Cross-Cultural Adaptation of Self-Report Measures.⁹

Phase 1: Creation of the prototype;

Phase 2: Forward–reverse translation;

Phase 3: Pretesting;

Phase 4: Pilot study;

Phase 5: Synthesis of the above four phases.

In each phase, problems with the intelligibility, reliability, and structure of the questionnaire were identified and resolved before beginning the next phase. Modifications after each phase were made by a steering committee composed of the principal investigators, the patient partners, a linguist, and the research coordinator. Prior approval for the study was received from the CHU de Québec-Université Laval research ethics board (authorization number MP-20-2021-5224).

Phase 1: Creation of the Prototype

To create the first APEX prototype, two validated English PREMs were merged, one used in EDs (Ontario emergency department patient experience of care survey [EDPEC] - 48 questions¹⁰), the other in primary care settings (Ontario Primary Care Patient Experience Survey [PCPES] - 39 questions¹¹). Since both surveys include questions specific to their particular care setting, neither is wholly suited to compare multiple care settings. A panel of 12 experts was therefore assembled to identify the questions from the two source PREMs that would be most relevant to assessing the experience of care in all targeted settings (ED, walk-in clinic, primary care practice). Panelists were asked first to rate each candidate question independently on a scale of 1 (not relevant) to 9 (very relevant) using an online questionnaire (REDCap). Questions with a median score between 1 and 3 were excluded, while those with a median score between 7 and 9 were included in the APEX prototype. Questions with a median score between 4 and 6 were discussed by videoconference. Panelists were asked to reach a consensus on the usefulness of these questions. They were also asked to propose minor modifications to the questions thus retained (eg, replace “emergency department” with “care facility”) to adapt them for use in any setting. If similar questions from the two source PREMs were retained, panelists were asked to merge the questions to avoid redundancy in the new questionnaire.

Phase 2: Forward–Reverse Translation

The APEX prototype was translated into French then back to English to identify potential inconsistencies or conceptual errors by comparing the original version with the translations. Two French native-speaker professional translators were first tasked to translate the prototype into French. After comparing the two translated versions, discrepancies and ambiguities were resolved in discussion with the translators, the linguist, and the patient partners to ensure the intelligibility of all questions. The APEX prototype was then translated back into English by two other different native English-speaking translators. The same process of reconciling discrepancies and eliminating ambiguities was carried out on this new English version.

Phase 3: Pretesting

In the third phase, a convenience sample of ambulatory adults was recruited in the CHUL (Centre hospitalier de l'Université Laval) ED, a teaching hospital in Québec City (Canada) receiving about 80,000 visits per year. Participants were asked to answer the APEX prototype and then rate the different items on a scale of 1 to 9 (1 = totally disagree to 9 = totally agree) to evaluate the clarity, utility, discriminability, face validity, content validity, and lack of redundancy¹² of the questionnaire. Respondents were also allowed to provide free-text comments on these domains.

Phase 4: Pilot Study

In the fourth phase, a prospective cohort study was conducted between June 8 and October 3 of 2021 to test the APEX prototype in a real-world clinical setting and compare the experience of care in the CHUL ED to that of two nearby walk-in clinics. Because recruitment took place during the COVID-19 pandemic and the availability of walk-in health services was limited, we included walk-in clinics designated for the management of patients with symptoms of respiratory infections. A convenience sample was recruited by research staff, between 10:00 and 20:00, Monday to Sunday, in the ED, and between 9:00 and 17:00, weekdays, in walk-in clinics. Patients were included if they met the following 4 criteria: (1) were at least 18 years of age; (2) were ambulatory throughout their visit at the participating care facility; (3) had a low-acuity acute respiratory condition, namely common cold, otitis media, acute tonsillitis or pharyngitis, acute laryngitis, acute sinusitis, COVID-19, influenza, pneumonia, bronchitis, acute exacerbation of chronic obstructive pulmonary disease or asthma; and (4) were able to provide written informed consent. Patients were excluded if they: (1) were transported by ambulance; (2) were not covered by provincial health insurance; (3) had consulted for a similar symptom in the previous 30 days; (4) were living in a long-term care facility or were incarcerated; and (5) were receiving palliative care. After giving on-site consent, participants were called back between 24 and 72 h later to answer the APEX questionnaire. We planned to recruit at least 30 participants, 15 from the ED and 15 from the clinics since previous studies have shown that 90% of the problems associated with using a questionnaire can be identified by testing it on a sample of 15 respondents.¹³

Phase 5: Synthesis of the 4 Previous Phases

Following the pilot study, the expert panel re-evaluated the questionnaire and the notes taken during each phase (screening and selection of questions, translation, pretesting, and pilot study) to ensure that no issues relating to content, translation, or usability were overlooked.

Statistical Analysis

Changes to the APEX prototype in each phase were recorded and are reported as counts. The APEX pretest in phase 3 is reported as medians of participants’ responses for each of the six domains studied (clarity, utility, discriminability, face validity, content validity, lack of redundancy). A median score between 7 and 9 was determined a priori to be satisfactory for each domain assessed. In phase 4, descriptive statistics and frequencies were generated from the data to provide trends in patients’ experience of care. Wilcoxon–Mann–Whitney tests were used to compare continuous characteristics across care settings (ED and clinics), and Fisher's exact tests were performed to explore the presence of associations between categorical characteristics and settings. Multivariate generalized linear regression models were used to compare care settings adjusted by propensity score. The propensity score was estimated using a logistic regression model with care settings as outcome and the APEX responses for age, gender, sex, ethnicity, comorbidities (Charlson index score and number of regular medications), patient perception of illness severity, and length of stay as independent variables. All statistical tests have a significance threshold of 0.05. Analyses were performed using SAS (v9.4).

Results

The final APEX is composed of 61 questions and 7 sections: (1) “Upon your arrival,” (2) “During your visit,” (3) “At your departure from the healthcare setting,” (4) “Your overall experience during your most recent visit,” (5) “Your usual healthcare,” (6) “About you,” and (7) “Comments.” Of these questions, 2 use a 4-step Likert scale, 25 use a 5-step Likert scale, 29 offer multiple choice answer, and 5 are open-ended. The final APEX in French and English is available in the Web appendices. Figure 1 shows the workflow during the 5 phases of the APEX questionnaire development.

Figure 1.

Workflow of the APEX questionnaire development and testing process.

Phase 1: Creation of the Prototype

We assembled a panel consisting of two patient partners, two clinicians, an expert linguist in information design, an expert in methodology, an emergency room nurse manager, an expert in patient experience measurement and improvement, the principal investigator and three co-investigators. From the 87 questions that made up the two PREM source questionnaires, the panel's selection of questions and the process of merging similar questions on the same dimension of experience resulted in a prototype with 63 questions. Questions about ambulance use, medication administered during the visit, ease of scheduling appointments, and the quality of the long-term relationship with the family physician were excluded since they were not applicable in all of the targeted care settings. Following a recommendation from the expert panel, two questions from the EDPEC questionnaire, asking separately if the doctor and the nurses had spent enough time with the patient, were merged into one question to evaluate the patient satisfaction with the time spent with all health professionals during the visit. This recommendation addressed the fact that not all clinics have the same staffing model, some having no nurses and others offering appointments with nurse practitioners rather than physicians. However, the panel decided also to evaluate physicians and nurses separately on several aspects of care quality, to detect possible differences in patient experiences of physicians and nurses. Seven additional questions were thus generated by separating seven PCPES questions for this purpose. Based on discussion of the demographic questions during the panel meeting, two anthropologists were consulted, which led to modifying section 6 (“about you”) to obtain a more inclusive questionnaire, one that could measure whether patient experience is related to demographic characteristics such as ethnicity, gender, and sexual orientation.

Phase 2: Forward–Reverse Translation

After forward and reverse translations, one question was split in two to assess separately patient experience of the advice received regarding medication and investigations. Two questions were excluded and two were merged to reduce redundancy (Figure 1). Some questions were rephrased without changing their meaning to make them more relevant and applicable to the three clinical settings. The modifications are reported in the Web appendices.

Phase 3: Pretesting

The questionnaire was pretested on 25 participants, of which 48% were over 40 years old and 52% were women. With median scores between 8 and 9 for all dimensions (Table 1), the APEX prototype was rated as clear, low in redundancy and useful, with good discriminatory properties and good face and content validity. Based on respondents’ comments and suggestions, the following three additions were made: (1) a question to evaluate the convenience of the waiting area (eg, access to restrooms, food, outlet for charging phone batteries); (2) a clarification (examples of the type of information the healthcare professional can give on a specific medication; question 2.5 in the Web appendices) to make one question more understandable; and (3) an introduction to the section “about you” to explain the purpose of potentially sensitive questions (Figure 1). After this phase, the APEX prototype had 61 questions and 7 sections.

Table 1.

APEX Pretest Results (n = 25 Respondents/Potential Users) (Phase 3).

	Median (Q1; Q3) ^a
Questions are simple and easy to understand.	9 (8;9)
Questions elicit frank and honest answers.	9 (8;9)
Questions are able to identify the differences between responders’ experience of care.	8 (7;9)
Questions are important and relevant.	8 (8;9)
Questions are able to measure all elements to assess patients’ experience of care.	8 (7;9)
Questions are not repetitive or redundant.	8 (7;9)

Median score on a scale of 1 to 9 (1 = totally disagree; 5 = neutral; 9 = totally agree).

Phase 4: Pilot Study

During phase 4, 91 eligible participants were approached: 23 declined participation, 2 ED patients were excluded because they were not ambulatory throughout their visit and 3 others because they did not fill out the APEX completely. Overall, 63 participants were included in the analysis, 34 from the walk-in clinics and 29 from the ED. The demographic characteristics of the participants are shown in Table 2. The mean length of stay was 55 min (SD: 96) in the clinics and 170 min (SD: 191) in the ED. More men were recruited from the clinic than from the ED (47% vs 21%). No participant reported a gender identity other than male or female. Comparing patient experience between settings, adjusted results show that access, cleanliness, and feeling treated with respect and dignity by nurses and physicians were significantly better in the clinics than in the ED (Table 3). However, the perception that the professional spent enough time with them was similar. No statistically significant difference in the overall care experience and the tendency to recommend the facility to family and friends was found between the two settings. Table A1 in the Appendix shows the crude results for each APEX question.

Table 2.

Demographic Characteristics of the Pilot Study Participants (Phase 4)^a.

	Clinics (n = 34)	ED (n = 29)
Mean (SD) age, years	42 (17)	37 (13)
Mean (SD) length of stay, minutes	54.7 (95.8)	170.0 (190.7)
Male sex	16 (47.1)	6 (20.7)
Caucasian	30 (88.2)	22 (75.9)
Perception of the severity of the disease
Not severe	2 (5.9)	0 (0)
Not very severe	8 (23.5)	5 (17.2)
Moderately severe	13 (38.2)	15 (51.7)
Severe	11 (32.4)	8 (27.6)
Very severe	0 (0)	1 (3.4)
Perception of his/her physical health
Poor	1 (3.0)	0 (0)
Acceptable	6 (18.2)	6 (20.7)
Good	6 (18.2)	7 (24.1)
Very good	12 (36.4)	12 (41.4)
Excellent	8 (24.2)	4 (13.8)
Missing	1 (2.9)	0 (0)
Perception of his/her mental health
Poor	0 (0)	0 (0)
Acceptable	4 (12.1)	1 (3.4)
Good	9 (27.3)	11 (37.9)
Very good	7 (21.2)	14 (48.3)
Excellent	13 (39.4)	3 (10.3)
Missing	1 (2.9)	0 (0)
Highest level of education
High school or equivalent	7 (21.2)	6 (20.7)
College, CEGEP or other equivalent level or pre-university diploma	8 (24.2)	8 (27.6)
Undergraduate degree	13 (39.4)	9 (31.0)
Graduate degree (Masters or PhD)	5 (15.2)	6 (20.7)
Missing	1 (2.9)	0 (0)
Gender
Male	15 (45.5)	5 (17.2)
Female	17 (33)	24 (82.8)
Transgender, non-binary, fluid, other	0 (0)	0 (0)
Prefer not to answer	1 (3.0)	0 (0)
Missing	1 (2.9)	0 (0)
Identification/ethnicity (can answer more than one)
White	30 (88.2)	22 (75.9)
Black or Afro-Caribbean origin	1 (2.9)	2 (6.9)
First Nations, Metis, Inuit, Mixed, others	0 (0)	0 (0)
Arab	1 (2.9)	3 (10.3)
Latino-American	1 (2.9)	0 (0)
Chinese	0 (0)	1 (3.4)
Korean	0 (0)	1 (3.4)
Missing	1 (2.9)	0 (0)

SD, standard deviation.

All results are reported as n(%), unless otherwise indicated.

Table 3.

Between-Settings Differences in Adjusted Means (95% CI) for Selected APEX Questions (n = 63).

	Clinic	ED	P-value
Overall experience^a	4.3 (4.0;4.6)	3.8 (3.4;4.3)	.19
Ease of access to care^a	4.1 (3.6;4.5)	3.1 (2.7;3.5)	.007
Overall experience with reception staff^a	4.3 (4.0;4.8)	3.8 (3.2;4.3)	.13
Would you recommend the healthcare setting to your family and friends^b	1.2 (1.0;1.4)	1.5 (1.2;1.8)	.15
Perception that the professional spent enough time with you^c	1.4 (1.2;1.6)	1.4 (1.1;1.7)	.89
Perception of waiting too long before obtaining care^c	2.8 (2.6;3.0)	1.9 (1.6;2.8)	<.0001
General cleanliness of the healthcare setting^a	4.7 (4.4;4.9)	3.7 (3.2;4.2)	.003
Meeting with doctor or nurse practitioner, how do you rate this person for:
Listened to your needs and your preferences^a	4.5 (4.2;4.8)	4.1 (3.7;4.5)	.14
Treated you with dignity and respect^a	4.8 (4.6;5.0)	4.4 (4.1;4.7)	.05
Overall experience with this person^a	4.4 (4.1;4.7)	4.20 (3.9;4.6)	.39
Meeting with nurses, how do you rate these people for:
Listened to your needs and your preferences^a	4.8 (3.8;5.0)	3.9 (3.3;4.5)	.23
Treated you with dignity and respect^a	5.0 (4.6;5.0)	4.1 (3.6;4.5)	.02
Overall experience with these people^a	4.2 (3.5;4.9)	4.1 (3.6;4.5)	.83

Results are estimated by generalized linear regression models adjusted by propensity score. Propensity score was calculated using age, sex, gender, ethnicity, comorbidities, patients’ perception of illness severity, and length of stay.

Results with the 5-point Likert scale; 1 = poor, 2 = acceptable, 3 = good, 4 = very good, 5 = excellent.

Results with the following scale; 1 = definitely yes, 2 = probably yes, 3 = probably not, 4 = definitely not.

Results with the following scale; 1 = yes, definitely, 2 = yes, somewhat, 3 = no.

Phase 5: Synthesis of the 4 Previous Phases

The panel re-evaluated the four preceding phases. Only one point was raised regarding the accuracy of the vocabulary. The panel decided to replace the term “consultation” with “visit” because the latter encompasses the entire episode of care and not just the consultation with a specific care provider.

Discussion

Interpretation

In this study, we developed, adapted, and produced evidence supporting the validity of a PREM designed to assess the care experienced by ambulatory patients presenting with unexpected needs. Called APEX, this questionnaire can be used in EDs, primary care practices, and walk-in clinics. Consisting of 61 questions divided into 7 sections, it can be used to identify and evaluate differences in care experience between these care settings and to assess the quality of care provided to ambulatory patients with acute conditions.

Previous Studies

The development of a new PREM was made necessary because we found no robust, validated tool in the scientific literature to achieve our objective of comprehensively comparing the experience of care in EDs, walk-in, and primary care clinics^14–20 Indeed, to our knowledge, in only one study was such a tool designed to compare the three settings from the patient's perspective. Hutchison et al⁴ assessed patient satisfaction using three domains of questions: (1) Perception of patient-centered communication; (2) Doctor's attitude; and (3) Delay in the waiting room.⁴ Conducted in Ontario (Canada), this study found that patients were more satisfied with the doctor's attitude and with the waiting time in walk-in clinics than in EDs. Our study showed similar results: significantly more participants felt that they were treated with respect and dignity by doctors in the clinics than in the ED, and more participants reported unacceptable waiting times in the ED than in clinics. However, Hutchison et al. only surveyed patient satisfaction with doctors. The tool they used was not designed to evaluate other important dimensions of the patient experience such as the relationship with non-medical staff (eg, nurses), continuity of care, medication administration, and support for self-care.²¹ Our PREM tool addresses each of these dimensions to some extent.

Clinical Implications

The evaluation of quality of care provided in walk-in clinics represents a blind spot in many health jurisdictions since these establishments operate without formal monitoring by health authorities. Our PREM tool can be used to evaluate the patient experience in walk-in clinics and provide benchmarks against other facilities treating the same population.

Research Implications

Our goal in developing a new PREM was to allow a multicenter study that would compare the value of care provided in EDs, walk-in clinics, and primary care practices from the patient's perspective. This study is currently ongoing in 14 institutions in Canada.²² The APEX will also enable other research initiatives intended to evaluate different care pathways offered to ambulatory patients with unexpected needs, such as redirection protocols whereby patients are redirected from the ED to outpatient clinics.

Limitations

This study has a number of limitations. First, although developed in English and French, our PREM tool was piloted in a primarily French-speaking community with less socio-cultural diversity than is found in other Canadian cities of similar size. However, we expect the English version of the APEX to perform equally for two reasons: (1) the tool was created by merging two existing PREMs validated in English; and (2) it has been reviewed by two professional translators whose native language is English. Another limitation is that the pilot study was conducted with a convenience sample that included only respiratory conditions, which may not be wholly representative of the population of ambulatory patients with acute conditions. However, the primary objective of the pilot study was to identify and correct usability problems. We are confident that our sample size was large enough to achieve this goal and secondarily to explore potential differences in the experience of care between the sites. A third limitation is that although the APEX assesses most of the dimensions of patient-centered care endorsed by the Institute of Medicine and other internationally recognized societies, it provides no evaluation of the involvement of patients, their families, and caregivers in the care received.^23,24 Despite originating from two validated PREMs and benefitting from numerous opportunities for amendment by patients, the final tool contains no direct questions on this aspect of healthcare. Finally, in adapting the PREM for use in different facilities where emergency ambulatory patients may be treated, some setting-specific questions were excluded. While the final tool facilitates comparison across settings, it may not be the most appropriate tool if only one type of setting (eg, ED) is assessed.

Conclusion

We have developed the APEX questionnaire, a PREM to compare the care experience of ambulatory patients in EDs, walk-in clinics, and primary care practices. Through a 5-phase process, we have addressed several usability problems and accumulated evidence of validity such that the tool appears to be very usable in the ambulatory care context regardless of setting. The pilot study on a sample of patients has shown some differences in the experience of the care provided in one ED and two walk-in clinics. The APEX tool is currently being used in a multicenter study to provide robust evidence to determine which alternative offers the best value of care to ambulatory patients.

Key points

It is generally believed that non-ED (emergency department) alternatives such as primary care practices and walk-in clinics offer good quality of care at a lower cost to low-acuity ambulatory ED patients, but data supporting these assumptions are scarce.

We developed the Ambulatory Patient EXperience (APEX) questionnaire, a patient-reported experience measure designed to assess and compare the experience of care in EDs, walk-in, and primary care clinics.

The comparative data obtained from this questionnaire will help health authorities identify optimal care pathways for ambulatory patients with acute unexpected needs.

Supplemental Material

sj-xlsx-1-jpx-10.1177_23743735241229373 - Supplemental material for Development of a Patient-Reported Experience Measure Tool for Ambulatory Patients With Acute Unexpected Needs: The APEX Questionnaire

Supplemental material, sj-xlsx-1-jpx-10.1177_23743735241229373 for Development of a Patient-Reported Experience Measure Tool for Ambulatory Patients With Acute Unexpected Needs: The APEX Questionnaire by Myriam Nadeau, Dominique Chabot, Mylaine Breton, Jason R. Guertin, Laurie Harvey Labbé, Danièle Roberge, Gabrielle Lefebvre, Myriam Mallet, Sandrine Beaulieu, Éric Kavanagh, Nathalie Cloutier, Philippe Garant, Lynda Bélanger, Samuel Vaillancourt, Tarek Boumenna, Kathryn Bareil, Joanie Savard, David Simonyan, Mahukpe Narcisse Ulrich Singbo and Simon Berthelot in Journal of Patient Experience

Footnotes

Author Contributions

Each author substantially contributed to (1) the study’s conception and design, or acquisition of data, or analysis and interpretation of data; (2) drafting the article or revising it critically for important intellectual content; (3) final approval of the version to be submitted.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Canadian Institutes of Health Research [PJM 175404].

Ethical Approval

Prior approval for the study was received from the CHU de Québec-Université Laval research ethics board (authorisation number MP-20-2021-5224).

ORCID iDs

Myriam Nadeau

Éric Kavanagh

Simon Berthelot

Supplemental Material

Supplemental material for this article is available online.

Appendix

Table A1.

Crude results of phase 4 for each APEX question.

	Clinic (n = 34)	ED (n = 29)	P-value
Ease of access to care (%)
Poor	1 (2.9)	4 (13.8)	.007
Acceptable	1 (2.9)	2 (6.9)
Good	7 (20.6)	12 (41.4)
Very good	10 (29.4)	9 (31.0)
Excellent	15 (44.1)	2 (6.9)
Waiting time at the reception desk/in the waiting room (%)
Poor	0 (0)	8 (27.6)	< .0001
Acceptable	1 (2.9)	11 (37.9)
Good	3 (8.8)	3 (10.3)
Very good	7 (20.6)	4 (13.8)
Excellent	23 (67.6)	3 (10.3)
Overall experience with reception staff (%)
Poor	0 (0)	2 (6.9)	.11
Acceptable	3 (8.8)	2 (6.9)
Good	3 (8.8)	8 (27.6)
Very good	7 (20.6)	6 (20.7)
Excellent	21 (61.8)	11 (37.9)
Length of time waiting in the examination room before to speak to a healthcare provider (%)
Poor	1 (2.9)	3 (10.3)	.01
Acceptable	4 (11.8)	8 (27.6)
Good	4 (11.8)	7 (24.1)
Very good	8 (23.5)	8 (27.6)
Excellent	17 (50.0)	3 (10.3)
Missing	1 (2.9)	0 (0)
Meeting with doctor or nurse practitioner, how do you rate this person for :
Attentive to your concerns (%)
Poor	0 (0)	1 (3.4)	.41
Acceptable	0 (0)	1 (3.4)
Good	5 (15.2)	5 (17.2)
Very good	7 (21.2)	9 (31.0)
Excellent	21 (63.6)	13 (44.8)
Missing	1 (2.9)	0 (0)
Used words that you could understand (%)
Poor	0 (0)	0 (0)	.48
Acceptable	0 (0)	0 (0)
Good	2 (6.1)	4 (13.8)
Very good	6 (18.2)	7 (24.1)
Excellent	25 (75.8)	18 (62.1)
Missing	1 (2.9)	0 (0)
Explained things in an easily comprehensible manner (%)
Poor	0 (0)	1 (3.4)	.35
Acceptable	0 (0)	1 (3.4)
Good	3 (9.4)	6 (20.7)
Very good	5 (15.6)	5 (17.2)
Excellent	24 (75.0)	16 (55.2)
Missing	2 (5.9)	0 (0)
Listened to your needs and your preferences (%)
Poor	0 (0)	1 (3.4)	.27
Acceptable	0 (0)	3 (10.3)
Good	5 (15.2)	4 (13.8)
Very good	6 (18.2)	6 (20.7)
Excellent	22 (66.7)	15 (51.7)
Missing	1 (2.9)	0 (0)
Treated you with dignity and respect (%)
Poor	0 (0)	0 (0)	.24
Acceptable	0 (0)	2 (6.9)
Good	2 (6.3)	2 (6.9)
Very good	4 (12.5)	7 (24.1)
Excellent	26 (81.3)	18 (62.1)
Missing	2 (5.9)	0 (0)
Gave you clear instructions regarding what you were supposed to do after your consultation (%)
Poor	0 (0)	0 (0)	.70
Acceptable	3 (9.1)	1 (3.4)
Good	5 (15.2)	3 (10.3)
Very good	6 (18.2)	8 (27.6)
Excellent	19 (57.6)	17 (58.6)
Missing	1 (2.9)	0 (0)
Overall experience with this person (%)
Poor	0 (0)	0 (0)	.39
Acceptable	0 (0)	2 (6.9)
Good	5 (15.2)	6 (20.7)
Very good	7 (21.2)	7 (24.1)
Excellent	21 (63.6)	14 (48.3)
Missing	1 (2.9)	0 (0)
Meeting with nurses, how do you rate these people for :
Attentive to your concerns (%)
Poor	0 (0)	0 (0)	.80
Acceptable	1 (7.1)	3 (10.3)
Good	1 (7.1)	5 (17.2)
Very good	3 (21.4)	7 (24.1)
Excellent	9 (64.3)	14 (48.3)
Missing	20 (58.8)	0 (0)
Used words that you could understand (%)
Poor	0 (0)	0 (0)	.51
Acceptable	1 (6.7)	1 (3.4)
Good	0 (0)	2 (6.9)
Very good	4 (26.7)	12 (41.4)
Excellent	10 (66.7)	14 (48.3)
Missing	19 (55.8)	0 (0)
Explained things in an easily comprehensible manner (%)
Poor	1 (7.1)	0 (0)	.34
Acceptable	0 (0)	2 (6.9)
Good	1 (7.1)	3 (10.3)
Very good	2 (14.3)	9 (31.0)
Excellent	10 (71.4)	15 (51.7)
Missing	20 (58.8)	0 (0)
Listened to your needs and your preferences (%)
Poor	1 (7.1)	2 (6.9)	.46
Acceptable	0 (0)	2 (6.9)
Good	0 (0)	2 (6.9)
Very good	3 (21.4)	10 (34.5)
Excellent	10 (71.4)	13 (44.8)
Missing	20 (58.8)	0 (0)
Treated you with dignity and respect (%)
Poor	0 (0)	1 (3.4)	.94
Acceptable	0 (0)	1 (3.4)
Good	1 (7.7)	3 (10.3)
Very good	3 (23.1)	8 (27.6)
Excellent	9 (69.2)	16 (55.2)
Missing	21 (61.8)	0 (0)
Gave you clear instructions regarding what you were supposed to do after your consultation (%)
Poor	0 (0)	2 (8.7)	.72
Acceptable	0 (0)	1 (4.3)
Good	2 (20.0)	2 (8.7)
Very good	2 (20.0)	7 (30.4)
Excellent	6 (60.0)	11 (47.8)
Missing	24 (70.6)	6 (20.7)
Overall experience with these people (%)
Poor	1 (5.9)	1 (3.4)	.91
Acceptable	0 (0)	1 (3.4)
Good	3 (17.6)	5 (17.2)
Very good	4 (23.5)	10 (34.5)
Excellent	9 (52.9)	12 (41.4)
Missing	17 (50.0)	0 (0)
Perception that the professional spent enough time with you (%)
Yes, definitely	24 (70.6)	19 (65.5)	.91
Yes, somewhat	8 (23.5)	8 (27.6)
No	2 (5.9)	2 (6.9)
Have received medication (%)
Yes	1 (2.9)	14 (48.3)	<.0001
Have blood sampling, an x-ray or any other examination performed or prescribed (%)
Yes	16 (47.1)	22 (75.9)	.02
Understanding of your main health problem (%)
Yes	29 (85.3)	25 (86.2)	1.00
Understanding which symptoms or additional health concerns to watch at home (%)
Yes	27 (79.4)	24 (82.8)	.76
New medication prescribed for home (%)
Yes	15 (44.1)	20 (69.0)	.07
Understanding of the new medication prescribed (%)
Yes, definitely	12 (80.0)	14 (70.0)	.70
Yes, somewhat	3 (20.0)	6 (30.0)
No	0 (0)	0 (0)
Missing	19 (55.9)	9 (31.0)
Need a follow-up visit (%)
	13 (38.2)	12 (41.4)	1.00
Asked if capable of obtaining follow-up (%)
Yes	7 (53.8)	6 (50.0)	1.00
Missing	21 (61.8)	17 (58.6)
General cleanliness of the healthcare setting (%)
Poor	0 (0)	3 (10.3)	.004
Acceptable	0 (0)	1 (3.4)
Good	3 (8.8)	7 (24.1)
Very good	5 (14.7)	8 (27.6)
Excellent	26 (76.5)	10 (34.5)
Confidence toward the doctor/healthcare professionals (%)
Poor	0 (0)	1 (3.4)	.76
Acceptable	0 (0)	0 (0)
Good	6 (17.6)	4 (13.8)
Very good	6 (17.6)	7 (24.1)
Excellent	22 (64.7)	17 (58.6)
Respect of the confidential nature of the information regarding your health status (%)
Poor	0 (0)	1 (3.4)	.10
Acceptable	0 (0)	2 (6.9)
Good	1 (2.9)	2 (6.9)
Very good	5 (14.7)	8 (27.6)
Excellent	28 (82.4)	16 (55.2)
Overall experience (%)
Poor	0 (0)	2 (6.9)	.43
Acceptable	1 (2.9)	2 (6.9)
Good	6 (17.6)	6 (20.7)
Very good	8 (23.5)	8 (27.6)
Excellent	19 (55.9)	11 (37.9)
Perception of waiting too long before obtaining care (%)
Yes, definitely	3 (8.8)	12 (41.4)	< .0001
Yes, somewhat	2 (5.9)	8 (27.6)
No	29 (85.3)	9 (31.0)
Would you recommend the healthcare setting to your family and friends (%)
Definitely yes	28 (82.4)	18 (62.1)	.29
Probably yes	5 (14.7)	8 (27.6)
Probably not	1 (2.9)	2 (6.9)
Definitely not	0 (0)	1 (3.4)

References

New England Health Institute. A Matter of Urgency: reducing emergency department overuse. Published March 2010. https://www.nehi-us.org/writable/publication_files/file/nehi_ed_overuse_issue_brief_032610finaledits.pdf. Accessed January 27, 2022

Campbell

Silver

Hoch

, et al. Re-utilization outcomes and costs of minor acute illness treated at family physician offices, walk-in clinics, and emergency departments. Can Fam Physician. 2005;51(1):82-3.

Chen

Mehrotra

. Walk-in clinics versus physician offices and emergency rooms for urgent care and chronic disease management. Cochrane Database Syst Rev. 2017;2(2):CD011774. doi:https://doi.org/10.1002/14651858.CD011774.pub2

Hutchison

Østbye

Barnsley

, et al. Patient satisfaction and quality of care in walk-in clinics, family practices and emergency departments: the Ontario walk-in clinic study. CMAJ. 2003;168(8):977-83.

Aubin

Hebert

Eurich

. The importance of measuring the impact of patient-oriented research. CMAJ. 2019;191(31):E860-4. doi:https://doi.org/10.1503/cmaj.190237

Dean

Al Sayah

Johnson

A . Measuring value in healthcare from a patients’ perspective. J Patient-Reported Outcomes. 2021;5(2):88. doi:https://doi.org/10.1186/s41687-021-00364-4

Sikka

Morath

Leape

. The quadruple aim: care, health, cost and meaning in work. BMJ Qual Saf. 2015;24(10):608-10. doi:https://doi.org/10.1136/bmjqs-2015-004160

Nundy

Cooper

Mate

. The quintuple aim for health care improvement: a new imperative to advance health equity. JAMA. 2022;327(6):521-2. doi:https://doi.org/10.1001/jama.2021.25181

Beaton

Bombardier

Guillemin

Ferraz

. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25(24):3186-91. doi:https://doi.org/10.1097/00007632-200012150-00014

10.

Ontario Hospital Association. Ontario Emergency Department Patient Experience of Care survey (Ontario EDPEC). Published 2020. https://www.oha.com/Documents/Survey%20-%20EDPEC.pdf. Accessed January 30, 2022

11.

Health Quality Ontario. Primary Care Patient Experience Survey. Published 2020. https://www.hqontario.ca/Portals/0/documents/qi/primary-care/primary-care-patient-experience-survey-en.pdf. Accessed January 30, 2022

12.

Cook

Guyatt

Jaeschke

, et al. Determinants in Canadian health care workers of the decision to withdraw life support from the critically ill. JAMA. 1995;273(9):703-8.

13.

Virzi

RA.

Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? Hum Factors. 1992;34(4):457-68. doi:https://doi.org/10.1177/001872089203400407

14.

Parast

Hays

, et al. Development and validation of a patient experience of care survey for emergency departments. Health Serv Res. 2022;57(1):102-12. doi:https://doi.org/10.1111/1475-6773.13853

15.

Sabapathi

Roberts

Fuller

, et al. Validation of a 5-item tool to measure patient assessment of clinician compassion in the emergency department. BMC Emerg Med. 2019;19(1):63. doi:https://doi.org/10.1186/s12873-019-0279-5

16.

Bull

Crilly

Latimer

Gillespie

. Establishing the content validity of a new emergency department patient-reported experience measure (ED PREM): a Delphi study. BMC Emerg Med. 2022;22(1):65. doi:https://doi.org/10.1186/s12873-022-00617-5

17.

Wang

Chen

Yang

Burström

. Validation of the patient-reported experience measure for care in Chinese hospitals (PREM-CCH). Int J Equity Health. 2021;20(1):25. doi:https://doi.org/10.1186/s12939-020-01370-6

18.

Bos

Sizmur

Graham

van Stel

. The accident and emergency department questionnaire: a measure for patients’ experiences in the accident and emergency department. BMJ Qual Saf. 2013;22(2):139-46. doi:https://doi.org/10.1136/bmjqs-2012-001072

19.

Ogaji

Giles

Daker-White

Bower

. Findings and predictors of patient-reported experience of primary health care in Nigeria. J Patient Exp. 2016;3(3):69-80. doi:https://doi.org/10.1177/2374373516667005

20.

Burt

Campbell

Abel

, et al. Improving Patient Experience in Primary Care: a Multimethod Programme of Research on the Measurement and Improvement of Patient Experience . NIHR J Library. 2017. http://www.ncbi.nlm.nih.gov/books/NBK436541/. Accessed July 6, 2023

21.

Haggerty

Burge

Lévesque

, et al. Operational definitions of attributes of primary health care: consensus among Canadian experts. Ann Fam Med. 2007;5(4):336-44. doi:https://doi.org/10.1370/afm.682

22.

Berthelot

Breton

Guertin

, et al. A value-based comparison of the management of ambulatory respiratory diseases in walk-in clinics, primary care practices, and emergency departments: protocol for a multicenter prospective cohort study. JMIR Res Protoc. 2021;10(2):e25619. doi:https://doi.org/10.2196/25619

23.

Tzelepis

Sanson-Fisher

Zucca

Fradgley

. Measuring the quality of patient-centered care: why patient-reported measures are critical to reliable assessment. Patient Prefer Adherence. 2015;9:831-5. doi:https://doi.org/10.2147/PPA.S81975

24.

The Picker Principles of Person Centred care. Picker. https://picker.org/who-we-are/the-picker-principles-of-person-centred-care/. Accessed July 6, 2023

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB