Nonurgent Patients’ Preferences for Emergency Department Versus General Practitioner and Effects of Incentives: A Discrete Choice Experiment

Abstract

Objective. This study investigates potential of a new financial incentive policy, the GP-referral discount scheme introduced in Singapore, in reducing nonurgent emergency department (ED) visits, and compares it with alternative interventions. Methods. A discrete choice experiment (DCE) was designed to elicit patients’ preferences for ED and general practitioner (GP) under hypothetical nonurgent medical conditions. Through latent class multinomial logistic regression, choice models were estimated to quantify how patients’ choices are influenced by GP-referral discount, other ED/GP attributes (waiting time, test facilities, and payment), patient demographics, and their perception of severity. The choice models were used to predict uptake of the GP-referral discount scheme and other countermeasures suggested by these models. Results. Survey responses from 849 respondents recruited from a public hospital in Singapore were included in the study. The choice model identified two prominent classes of patients, one of which was highly sensitive to GP-referral discount and the other to test-facility-availability. Patients’ perceptions of severity (“critical” v. “not critical” enough to go to ED directly) were highly significant in influencing preference heterogeneity. Predictive analysis based on the choice model showed that GP-referral discount is more effective when patients visit ED expecting “shorter” waits, as opposed to test-facility provision at GPs and perception-correction measures that showed stronger effects under “longer” expected waits. Conclusions. The new GP-referral financial incentive introduced in Singapore can be effective in reducing nonurgent ED visits, if it reasonably covers the (extra) cost of visiting a GP. It may serve as a complement to test-facility provision at GPs or perception-correction measures, as the financial incentive and the latter two measures appear to influence distinct classes (discount-sensitive and facility-sensitive) of patients.

Keywords

discrete choice experiment emergency department crowding GP-referral discount nonurgent emergency department visits patient perception patient choice model

Introduction

Nonurgent patients’ visits to emergency departments (EDs) has been a global problem for many years. In several countries these patients constitute a large proportion of ED demand; for example, 9% to 60% in the United States and Canada, 19.6% to 40.9% in Europe, and 30% to 57% in Singapore and Hong Kong.^1–3 The nonurgent use of ED causes crowding and longer waiting times at EDs, unnecessary testing, excessive health care spending, and weaker patient-primary care provider relationships.⁴ Despite such severity, nonurgent ED visits largely remain an unresolved problem.^1,5

A big challenge in addressing nonurgent ED visits is that hospitals cannot bar patients from directly accessing ED, and that nonurgent patients often “prefer” ED to general practitioners (GPs) due to factors such as wrong perception of severity, belief that ED provides better care, or lack of incentives to visit GPs even when they are accessible.² In Singapore, where this study was conducted, there are about 1700 private GP clinics that provide within walking distance primary care service in most neighborhoods, and accept walk-in patients without requiring advance appointment. Yet, nonurgent ED visits remain an issue.³ Therefore, in general, interventions/incentives must be designed to encourage nonurgent patients to choose primary care.

In the past, some countries have tried interventions such as higher ED co-payment, tele-advice services, education on proper ED usage, strengthening primary care linkage, and improving access to GP care. Among studies that examined tele-advice services in the United Kingdom and the United States, most found no significant reduction in ED visits,⁶ but explanations are lacking whether this is due to limitations of tele-advice services or other potential factors. Public education campaigns in the past showed mixed results in the United States.⁷ In Singapore, such campaigns were initially effective, but a few months after each campaign, ED visits reverted to precampaign levels.⁸ Some recent studies in the United Kingdom and Australia have shown positive impact of extended GP hours and in-hospital GP walk-in centers, but in the past, opening of such centers in Singapore showed no effect.⁹ As discussed in reviews,^7,10 studies on interventions have often failed to reach a consensus about their potential/effectiveness due to lack of understanding of the interplay of various factors.

Compared to other interventions, ED co-payment/fee increase has shown more consistent positive results.^7,10 It has been observed though that such disincentives need to be significant to have an impact. For example, ED co-payments for Medicaid beneficiaries in the United States showed reduction in ED visits in Oregon (with $50 co-payment per ED visit), but no influence on ED visits in other states (with $2–$8 ED co-payment).¹⁰ In Singapore, ED fee increases between 1987 and 1997 showed reduction in ED visits only after ED fees significantly exceeded the charges levied by primary care services.⁸ Though high financial penalties seem to be effective, there have been concerns on imposing high ED co-payments/fees, as penalties for ED use could delay the needed care, especially for low-income populations.¹⁰

Recently, a new financial incentive policy—the GP-referral discount scheme—was introduced as a pilot by a public hospital in Singapore (here onward referred to as the Public Hospital) so as to encourage nonurgent patients to first visit GPs instead of directly visiting the ED. Under this scheme, patients who are referred to the Public Hospital ED by participating GPs receive a discount on their ED fee, whereas those who directly go to the ED receive no discount.³ (In 2017, ED fee at the Public Hospital was flat S$120, and the pilot scheme offered S$50 discount for GP-referred patients.) As opposed to increase in ED co-payment/fee (which undesirably imposes penalty for ED use even on urgent patients), the above incentive has the advantage that it does not impose penalty for ED use. Such an incentive also has the potential to be adopted by health care systems where ED fee increase is not even feasible (e.g., universal publicly funded systems*). As it is the first-of-its-kind incentive scheme, it is of interest to explore its influence; which is one of the main motivations of our study.

In this study, we evaluate the potential of the above GP-referral discount scheme in reducing nonurgent ED visits, and also compare it with other possible interventions in Singapore context, for example, test-facility provision at GPs and publishing of ED waiting times. To understand the interplay of various factors and appropriately evaluate the interventions, we follow a discrete choice experiment (DCE)¹¹ approach that enables us to quantify patients’ sensitivity to different factors (including GP-referral discount, other service attributes of ED/GP such as waiting time, test-facilities and payment, as well as patients’ personal characteristics) in choosing ED versus GP. By virtue of DCE, we are able to investigate effectiveness of GP-referral discounts other than the specific discount offered in the pilot, which provides useful insights for further implementation/adoption of the scheme. Furthermore, we are able to assess and compare it with other possible interventions before their implementation. Through our DCE design, we also quantify effects of presenting nonurgent medical conditions and patients’perceptions (self-perceived urgency of the medical condition) on their choices. Though some previous studies^12–14 have observed importance of perceptions in patients’ choices between ED and GP, we are the first to quantify its influence and thereby assess the benefits of perception-improvement measures (e.g., tele-advice services).

We are among the first to use a DCE approach to quantify nonurgent patients’ preferences, and thereby assess interventions for nonurgent ED visit reduction. Previous studies on nonurgent patients’ preferences have mostly pursued qualitative investigation,¹⁵ which have identified their primary driving factors as cost, perceived severity, expecting to require tests, greater trust on ED services, and lack of access to timely care outside ED. A few studies have explored quantitative choice models using “revealed preference” data of patients’ visits to EDs/GPs.^16–18 However, they have estimated only the influence of patients’ sociodemographic characteristics and cost of care. Recently, DCE with mixed logit regression has been used^12,19,20 to quantify influence of ED/GP service attributes on patients’ preferences for ED/GP, and to compare effects of different medical scenarios. Some common findings from these studies are that patients prefer lower costs, shorter waiting times, and quality care.

Our study differs from the above-mentioned DCE studies in multiple aspects. First, we investigate influence of the “GP-referral discount” along with other factors. Second, we study patients’ preferences under four different nonurgent scenarios (previous studies considered only one or two nonurgent scenarios) that capture the most common health conditions for which nonurgent patients visit EDs in Singapore. A comprehensive analysis of these helps not only study heterogeneity due to medical scenarios but also to better understand the potential of GP-referral discount scheme and other investigated measures in reducing nonurgent ED visits. Third, we also quantify influence of patients’ perceptions (self-perceived urgency for the presented scenario) on their choices, and thereby also investigate implications of perception improvement, which has not been done before. Thus, our study and experiment design differ significantly from previous DCE studies, and aim to provide a good understanding of patients’ tradeoffs that can provide useful guidelines for designing effective interventions/reforms.

Methods

Study Setting

This study was conducted in Singapore in collaboration with a public hospital, Changi General Hospital (CGH). The study was ethically reviewed and approved by SingHealth Centralized Institutional Review Board (Reference Nos. 2016/2006, 2016/2619, 2016/2620, 2017/2927) and Singapore University of Technology and Design Institutional Review Board (Reference No. 15-088).

In Singapore, patients can seek primary care treatment at private GP clinics or subsidized polyclinics, the former of which meet about 80% of the total primary care demand.²¹ Emergency care is provided by EDs at both private and public acute hospitals. EDs, polyclinics, as well as GPs accept “walk-in” patients. Compared to an ED, a GP has limited test facilities and fewer senior doctors, but has lower wait (compared to nonurgent patients’ wait at EDs) and lower consultation costs in general. For this study, we focused on patients’ choices between ED and GP, in particular for “nonurgent” health conditions that constitute nonurgent ED demand.

Study Design

A DCE is a commonly used technique in health economics (and many other fields) to elicit patient preferences for health care products and services.¹¹ A DCE presents respondents with a sequence of choice tasks, where they are asked to make a choice between alternatives (e.g., health care providers) that are described by a set of attributes (e.g., waiting time, payment, etc.). These attributes can be designed to capture various characteristics of health care services and policies of interest. By varying attribute-levels among the choice tasks, respondents’ tradeoffs for attributes can be observed, which enables estimation of choice models using statistical techniques.²²

For our study, DCE was the most suitable approach for the following reasons. First, since GPs in Singapore have private management and do not record patients’ visit statistics, we did not have patients’ data from the GPs to be able to analyze effectiveness of the new GP-referral discount scheme. Second, we were also interested in examining influence of discount amounts other than the specific amount offered in the pilot, which was not possible with pilot data. Third, our factors of interest also included patients’ perceptions of medical conditions and ED/GP service attributes. For these a revealed preference study is not possible in general because ED/GP visit data lacks information on patients’ perceptions, and though real data of service attributes (waiting time, payment, etc.) can be observed for the medical provider a patient visits, corresponding data for the other provider does not exist. DCE thus provided a feasible approach for our objectives, whereby we could elicit patients’ perceptions and could explicitly specify ED/GP attributes one of which also captured the GP-referral discount.

Unlike previous DCE studies on preferences between ED and GP, which used unlabeled* DCE designs, we used a labeled DCE^23,24 where choice alternatives were labeled (Option-ED, Option-GP) and were constructed from ED/GP-specific attribute-levels. Labeled design enabled us to capture influence of the GP-referral discount scheme (as the discount applies only if patients choose the GP option), and estimate patients’ inherent preference for ED/GP. Furthermore, it offered the advantage that we could specify ED/GP-specific waiting times and test facilities.

Study Protocol

Attribute and Level Identification for DCE

To identify relevant attributes for DCE design, we first conducted focus group discussions (FGDs) with 20 patients recruited from CGH, who had visited in the past 3 months 1) an ED, without any referral (8 patients); 2) an ED after referral from GP (7 patients); or 3) a GP and were not referred to ED (5 patients). In the appendix, we provide participants’ demographics (Table 6) and details of FGD execution and analyses. Based on FGD results and our follow-up discussions with ED physicians and administrators, we included in our DCE the ED/GP attributes shown in Table 1. Since we were particularly interested in investigating influence of the aforementioned GP-referral discount scheme, we also included “GP-referral discount” as an attribute in the DCE.

Table 1

List of Attributes and Levels Used for Discrete Choice Experiment Design

Attributes	ED Levels	GP Levels
Waiting time to see a doctor	30 minutes	15 minutes
	1 hour	30 minutes
	2 hours	1 hour
	4 hours	2 hours
Test facilities (blood test, X-ray, ECG, etc.)	All available	None available All available
Out-of-pocket payment	S$50	S$50
	S$100	S$100
	S$200	S$200
GP-referral discount (when GP refers to ED)	No discount	No discount
		S$50 discount on ED fee
		Entire GP fee waived

ECG, electrocardiogram; ED, emergency department; GP, general practitioner.

In our labeled DCE, each choice task asked respondents to choose between an ED and a GP alternative comprising of the four attributes shown in Table 1. For each attribute, the ED/GP-specific levels (also shown in Table 1) were chosen based on Singapore health care setting, after discussions with ED physicians and administrators. Details of these levels’ selection is provided in the appendix. For the attribute “GP-referral discount,” the GP-level “No discount” represented the status quo, “S$50 discount on ED fee” represented the pilot scheme offer, and ‘Entire GP fee waived” represented a possible alternative for future. If in a choice task, GP’s out-of-pocket payment was S$50 (respectively, S$100, S$200) and GP-referral discount was “Entire GP fee waived,” then it implied that the patient would get back her S$50 (respectively, S$100, S$200) GP payment if the GP referred her to ED.

DCE Design

Having specified Table 1, we adopted a blocked fractional factorial design²⁵ (using Lighthouse Studio, version 9.5.3, Sawtooth Software, Inc.) to generate 40 choice tasks. Each task contained one ED level and one GP level from Table 1 corresponding to each attribute. Figure 1 shows an example of one such random choice task. We randomly divided the above tasks into five blocks with eight tasks each. The conventional approach to investigating rationality of DCE subjects is to include non-satiation tests where one alternative is made worse than the other in all attributes.²⁶ Since our DCE used a labeled design, such validity tests were not suitable because subjects might make inferences based on the identity tag (ED v. GP) on alternatives.²⁷ Therefore, for validity test in our labeled DCE, we constructed two holdout choice tasks,²⁸ as shown in Figure 1. The first of these represented the status quo, whereas in the second all ED attributes were dominated by those of GP. We added these two holdout tasks to all five choice task blocks. Thus, in each block, there were 10 choice tasks with eight being different across the blocks (random tasks) and two being common (holdout tasks). As we will explain later, we estimated patient preference models based on their responses to random choice tasks, and used their responses to holdout tasks to validate the estimated preference models.

Figure 1

Example of a random choice task and two holdout tasks.

Survey Design

The survey was designed to have two parts. The first part contained questions about patients’ demographics (age, gender, race, employment status, income range, and travel times to patients’ preferred ED and GP). The second part contained the DCE. At the beginning of the DCE, respondents were presented with one of the four hypothetical medical scenarios depicted in Figure 2. These scenarios represent the most common cases of nonurgent patient visits to ED, and were developed through discussions with ED physicians of CGH. The survey asked the respondents about their perception of the presented scenario (Do you think this condition is critical enough to go to ED directly), and their belief about GP referral under that scenario (Do you think your GP will (a) provide adequate medical treatment for this scenario; or (b) will refer you to a specialist; or (c) will refer you to ED for urgent treatment?). Respondents were then presented with one of the five choice task blocks described under “DCE design.” In each task, they were asked to choose between their preferred ED and GP under the presented conditions, imagining that they faced the presented medical scenario.

Figure 2

Hypothetical nonurgent medical scenarios.

To help the respondents properly interpret all attributes before starting with the choice tasks, the DCE also contained several warm-up questions, which were presented before the hypothetical medical scenario and the choice task block. In addition, at the beginning of the choice task block, an example choice task was presented to clearly explain how to interpret a choice task.

Study Population

The survey was conducted in CGH from December 2017 to February 2018. Respondents were recruited from the ED, specialist outpatient clinics, discharge lounge, and public areas in CGH such as food court and cafes. Because of the reasons described later in “Limitations” section, we used convenience sampling for recruitment. The eligibility criteria for the respondents were 1) Singapore citizen or Permanent Resident, 2) English speaking, and 3) 21 years old or above. Trained research assistants (RAs) screened the eligibility of participants at the beginning of the survey and obtained eligible participants’ written consent. The survey was conducted using electronic tablets, and RAs accompanied each respondent throughout the survey to assist in case of any queries.

According to Orme’s rule of thumb formula,²⁹ the minimum sample size required for our survey design is 157 respondents (500 × maximum number of levels for any attribute (5 levels corresponding to “waiting time”) ÷ number of random choice tasks in each survey (8) ÷ number of alternatives (2), i.e. $(500 \times 5 \div 8) \div 2$ ). In the survey, a total of 849 respondents were recruited, among which 204 were given scenario 1 (fever and runny nose), 216 were given scenario 2 (ankle injury), 218 were given scenario 3 (back pain), and 211 were given scenario 4 (stomach upset and vomiting). Among these respondents, 7% were recruited from specialist outpatient clinics, 8% from ED, 27% from public areas in the hospital and food court, and 58% from the waiting area at the entrance of inpatients’ discharge lounge.

Analysis

We first estimated patient choice models using latent class multinomial logit regression.³⁰ Multinomial logit is widely used for analyzing DCE data.³¹ With latent class it incorporates preference heterogeneity by identifying (based on choice data) groups of respondents sharing similar preferences.²³ Thus, it essentially generates different choice models for different groups.

To generate the latent class model we used data from only random choice tasks from the survey. We used effects coding³² to construct attribute-levels such that in the generated model, the coefficients of all levels within an attribute sum up to zero. For the attribute-level “Entire GP fee waived,” the amount of discount depended on the value of GP’s out-of-pocket payment (explained under “Study Protocol”). Therefore for our analysis we divided “Entire GP fee waived” into three sublevels: “Entire GP fee waived when GP payment is S$50,”“Entire GP fee waived when GP payment is S$100,” and “Entire GP fee waived when GP payment is S$200.” We conducted all statistical analyses using Sawtooth Software (Latent Class, version 9.5.3, Sawtooth Software, Inc.).

To investigate whether specific respondent characteristics made them more likely to belong to the different classes identified by the latent class model, we performed logistic regression (using R, version 3.4.1), which mapped respondents’ class memberships generated by the latent class model to their presented medical scenario, their perception and belief about GP referral for presented scenario, and their demographics collected in the survey. In the appendix we provide mathematical description of our latent class and logistic class membership models.

Next, we tested the validity of the choice model and class membership model generated above. First, we predicted class membership of each respondent using logistic regression, and using the corresponding class’ choice model we predicted each respondent’s choice under the two holdout tasks. For each task we then computed the proportion of respondents predicted to choose ED. Because holdout tasks were completed by all survey respondents, we also had the actual proportion of respondents from the survey who selected ED under each holdout task. We compared these actual proportions with predicted proportions above, and showed that they are similar, suggesting that patients’ preference are reasonably captured by our models.

Last, we predicted uptake of the GP-referral discount scheme and other countermeasures suggested by the choice models. For this we considered four “status quo” tasks from Table 2 that capture differences in GP fees and waiting times between day and night hours, and account for possibilities that patients’ decisions to visit ED/GP may depend on “short” or “long” expected ED wait (attribute-level-selection for Table 2 is explained in the appendix). For each status quo task, we predicted the proportion of survey respondents who would choose ED (following similar prediction method as for holdout tasks). We then predicted this proportion under different countermeasures, each of which changed one specific attribute-level in the status quo tasks (e.g., “GP-referral discount” for GP alternative changed to “Entire GP fee waived”). With the above predicted proportions $P_{ED}^{s}$ (under status quo) and $P_{ED}^{c}$ (under countermeasure), we computed “percentage reduction in nonurgent ED visits” due to the countermeasure as

\frac{P_{ED}^{s} - P_{ED}^{c}}{P_{ED}^{s}} \times 100 .

To predict such reduction for a countermeasure that corrects patients’ misperceptions of severity, we generated a “perception-corrected” sample from the survey sample by changing all respondents’“perception” to “not critical” (cf. Table 3). For this sample, we computed the proportion choosing ED under the status quo tasks ( $P_{ED}^{pc}$ ), and compared it against that of survey sample ( $P_{ED}^{s}$ ) to obtain the corresponding reduction in nonurgent ED visits as

\frac{P_{ED}^{s} - P_{ED}^{pc}}{P_{ED}^{s}} \times 100 .

Table 2

Attribute-Levels for Status Quo Tasks

	Task 1		Task 2		Task 3		Task 4
	Short ED Wait				Long ED Wait
	Day Hour		Night Hour		Day Hour		Night Hour
Medical provider	ED	GP	ED	GP	ED	GP	ED	GP
Waiting time to see a doctor	30 minutes	30 minutes	30 minutes	15 minutes	4 hours	30 minutes	4 hours	15 minutes
Test facilities	All	None	All	None	All	None	All	None
Out-of-pocket payment	S$100	S$50	S$100	S$100	S$100	S$50	S$100	S$100
GP-referral discount	No discount	No discount	No discount	No discount	No discount	No discount	No discount	No discount

Table 3

Characteristics of Survey Respondents

Patient Characteristics	Full Sample (n = 849)	Scenario 1 (n = 204)	Scenario 2 (n = 216)	Scenario 3 (n = 218)	Scenario 4 (n = 211)	Facility-sens’ class (n = 305)	Discount-sens’ class (n = 544)	Class Difference and 95% CI
Age in years, mean (SD)	35.9 (13.3)	38.2 (14.3)	36.6 (13.9)	34.5 (12.4)	34.5 (12.5)	37.6 (14.4)	35.0 (12.6)	2.6 (0.7, 4.5)
Gender, n (%)
Men	394 (46.4)	90 (44.1)	116 (53.7)	102 (46.8)	86 (40.8)	140 (45.9)	254 (46.7)	−0.8 (−7.8, 6.2)
Race, n (%)
Chinese	565 (66.5)	131 (64.2)	143 (66.2)	137 (62.8)	154 (73.0)	189 (62.0)	376 (69.1)	−7.1 (−13.8, −0.4)
Malay	202 (23.8)	52 (25.5)	53 (24.5)	60 (27.5)	37 (17.5)	85 (27.9)	117 (21.5)	6.4 (0.3, 12.5)
Indian	60 (7.1)	17 (8.3)	15 (6.9)	14 (6.4)	14 (6.6)	24 (7.9)	36 (6.6)	1.3 (−2.4, 5.0)
Other	22 (2.6)	4 (2.0)	5 (2.3)	7 (3.2)	6 (2.8)	7 (2.3)	15 (2.8)	−0.5 (−2.7, 1.7)
Yearly household income, n (%)
Less than S$25,000	205 (24.1)	52 (25.5)	55 (25.5)	50 (22.9)	48 (22.7)	88 (28.9)	117 (21.5)	7.4 (1.3, 13.5)
S$25,000 to S$49,999	246 (29.0)	45 (22.1)	66 (30.6)	71 (32.6)	64 (30.3)	76 (24.9)	170 (31.3)	−6.6 (−12.8, −0.4)
S$50,000 to S$99,999	241 (28.4)	67 (32.8)	66 (30.6)	54 (24.8)	54 (25.6)	78 (25.6)	163 (30.0)	−4.4 (−10.6, 1.8)
S$100,000 and above	157 (18.5)	40 (19.6)	29 (13.4)	43 (19.7)	45 (21.3)	63 (20.7)	94 (17.3)	3.4 (−2.1, 8.9)
Employment status, n (%)
Full-time	577 (68.0)	141 (69.1)	145 (67.1)	148 (67.9)	143 (67.8)	195 (63.9)	382 (70.2)	−6.3 (−12.9, 0.3)
Part-time	102 (12.0)	28 (13.7)	29 (13.4)	24 (11.0)	21 (10.0)	45 (14.8)	57 (10.5)	4.3 (−0.4, 9.0)
Other: retired, homemaker, etc.	170 (20.0)	35 (17.2)	42 (19.4)	46 (21.1)	47 (22.3)	65 (21.3)	105 (19.3)	2.0 (−3.7, 7.7)
Perception,^an (%)
Critical	153 (18.0)	70 (34.3)	23 (10.6)	30 (13.8)	30 (14.2)	76 (24.9)	77 (14.2)	10.7 (5.0, 16.4)
Not critical	696 (82.0)	134 (65.7)	193 (89.4)	188 (86.2)	181 (85.8)	229 (75.1)	467 (85.8)	−10.7 (−16.4, −5.0)
Belief about GP referral,^bn (%)
No	733 (86.3)	144 (70.6)	198 (91.7)	204 (93.6)	187 (88.6)	253 (83.0)	480 (88.2)	−5.2 (−10.2, 0.2)
Yes	116 (13.7)	60 (29.4)	18 (8.3)	14 (6.4)	24 (11.4)	52 (17.0)	64 (11.8)	5.2 (−0.2, 10.2)
Travel time to ED − Travel time to GP in minutes, mean (SD)	8.5 (10.9)	9.94 (11.3)	8.5 (12.1)	7.9 (10.6)	7.9 (9.2)	7.5 (10.5)	9.1 (11.0)	−1.6 (−3.1, −0.1)

CI, confidence interval; ED, emergency department; GP, general practitioner.

Perception: For the perception question, “Do you think this condition is critical enough to go to ED directly?” the response Yes was coded as “critical”; and No was coded as “not critical.”

Belief about GP referral: For the question “Do you think your GP will (a) provide adequate medical treatment for this scenario; or (b) will refer you to a specialist; or (c) will refer you to ED for urgent treatment?” responses (a) and (b) were coded as “No” and response (c) was coded as “Yes.”

Results

Sociodemographic Profiles and Perception

Table 3 describes socioeconomic characteristics of the survey respondents, and their perceptions under the presented scenarios. In summary, the average age of respondents was 35.9 years with 46.4% being males and 66.5% Chinese. As we explain later under “Limitations,” the survey sample reasonably represents the nonurgent patient population of interest that considers choice between ED and GPs. The survey response showed that 96% respondents knew that ED has priorities in serving patients; and 97% respondents knew that they would have to wait longer at the ED if they were nonurgent.

An interesting observation from Table 3 is respondents’ perception of severity for the four scenarios. As per ED physicians, all four scenarios described conditions that could be managed adequately by GPs and would not require urgent treatment at the ED. However, patients’ perceptions varied. For example, the percentage of respondents who perceived their presented scenario as “critical enough to go to ED directly” was 34.3% among those who got the fever-related scenario, but only 10.6% among those who got the ankle injury scenario.

Choice Models and Class Membership

Our latent class model identified two classes of respondents based on their choices and provided distinct choice models for each class. The model validity test described under “Analysis” generated the following choice percentages, which suggests that patients’ preferences are reasonably captured by these models. The actual proportions of respondents from the survey who chose ED under holdout task-1 and holdout task-2 were 16% and 2.8%, respectively. By predicting respondents’ choices based on the generated models, we obtained the corresponding proportions as 21.3% and 5.5%, respectively.

Table 4 presents parameter estimates for the two-class model. The two classes clearly differ in their sensitivities to availability of test-facilities and GP-referral discount; hence, we will refer to them as facility-sensitive and discount-sensitive class, respectively. These classes contain 38.5% and 61.5% respondents, respectively. The model indicates that discount-sensitive class has an inherent preference for GP, as their attribute-level coefficient for GP (0.32) is higher than that for ED (−0.32).

Table 4

Results From the Discrete Choice Experiment: Latent Class Model

	Facility-Sensitive Class (38.5%)			Discount-Sensitive Class (61.5%)
	Coefficient (95% CI)	SE	P Value	Coefficient (95% CI)	SE	P Value
Attribute-level coefficients
ED	0.02 (−0.09 to 0.13)	0.06	0.722	−0.32* (−0.43 to −0.22)	0.05	<0.001
GP	−0.02 (−0.13 to 0.09)	0.06	0.722	0.32* (0.22 to 0.43)	0.05	<0.001
Out-of-pocket payment
Payment	−1.21* (−1.33 to −1.09)	0.06	<0.001	−1.67* (−1.81 to 1.53)	0.07	<0.001
GP-referral discount
No discount	−0.02 (−0.20 to 0.16)	0.09	0.819	−0.60* (−0.83 to −0.38)	0.11	<0.001
S$50 discount on ED fee	−0.02 (−0.21 to 0.16)	0.09	0.824	−0.29^† (−0.53 to −0.06)	0.12	0.033
Entire for S$50^a	0.06 (−0.25 to 0.36)	0.16	0.710	0.67^‡ (−0.04 to 1.39)	0.37	0.092
Entire for S$100^a	0.00 (−0.24 to 0.25)	0.13	0.971	0.47* (−0.12 to 0.82)	0.18	0.022
Entire for S$200^a	−0.02 (−0.33 to 0.29)	0.16	0.897	−0.25 (−0.54 to 0.05)	0.15	0.130
Test facility
No facility	−0.87* (−0.98 to −0.76)	0.06	<0.001	−0.08^‡ (−0.18 to 0.01)	0.05	0.104
All facilities	0.87* (0.76 to 0.98)	0.06	<0.001	0.08^‡ (−0.01 to 0.18)	0.05	0.104
Patient waiting time
15 minutes	1.11* (0.88 to 1.34)	0.12	<0.001	1.50* (1.28 to 1.73)	0.11	<0.001
30 minutes	0.55* (0.41 to 0.69)	0.07	<0.001	1.12* (0.97 to 1.26)	0.07	<0.001
1 hour	0.40* (0.28 to 0.53)	0.06	<0.001	0.56* (0.44 to 0.68)	0.06	<0.001
2 hours	−0.49* (−0.61 to −0.36)	0.06	<0.001	−0.62* (−0.75 to −0.49)	0.07	<0.001
4 hours	−1.58* (−1.81 to −1.34)	0.12	<0.001	−2.56* (−2.82 to −2.29)	0.14	<0.001

CI, confidence interval; ED, emergency department; GP, general practitioner.

“Entire for S$50,”“Entire for S$100,” and “Entire for S$200” represent “Entire GP fee waived” with GP’s out-of-pocket payment being S$50, S$100, and S$200, respectively, in the choice task.

, ^†, and ^‡ indicate 1%, 5%, and 10% levels of significance respectively.

To compare the impacts of different ED/GP attributes on the two classes, we plot in Figure 3 the attribute-level coefficients of each class against attribute-levels. For a meaningful comparison of the two classes, we generated these plots using normalized coefficients corresponding to Table 4 (obtained by dividing the coefficients in each class by the absolute value of the out-of-pocket payment coefficient of that class).

Figure 3

Normalized attribute-level coefficients versus attribute-levels.

Figure 3A and B show combined attribute-level coefficient of GP-referral discount and out-of-pocket payment for the two classes. To see how it is computed, consider this example. For facility-sensitive class, the combined coefficient of “Entire GP fee waived” discount and S$100 out-of-pocket payment (for GP) is calculated as: Normalized “Entire for S$100” coefficient for facility-sensitive class+Coded value for S$100 payment × (−1). Thus, in each Figure 3A and B, the three points corresponding to “no discount” represent combined coefficients of “no discount” and S$50/S$100/S$200 out-of-pocket payment (for either ED or GP). They imply that higher the ED/GP payment, lower is the utility for ED/GP. The red (solid), green (dotted), and blue (dashed) plots further show how the utility for GP changes with different levels of GP-referral discount, when out-of-pocket payment for GP is S$50, S$100, and S$200, respectively. The figures imply that GP-referral discount has very mild influence on facility-sensitive class, but has significant impact on discount-sensitive class, especially when GP payment is S$50 or S$100.

Figure 3C shows test-facility-level coefficients for the two classes. It implies that with test-facility provision, utility for GP significantly increases for the facility-sensitive class, but remains almost unchanged for discount-sensitive class.

Figure 3D shows that effects of waiting time to see a doctor (at either ED or GP) are similar on both classes—both classes prefer shorter waiting times, with their attribute-level coefficients decreasing almost linearly in waiting time.

The logistic class membership model in Table 5 shows that among all patient characteristics collected in the survey, only age, employment status, difference in travel times to ED and GP, and patients’ perception of severity significantly influence their class membership. It shows that patients younger than 40 years or with longer travel time to ED are more likely to belong to the discount-sensitive class. This implies, based on the choice model from Table 4, that such patients have higher inherent preference for GP and are more sensitive to GP-referral discount. Patients older than 40 years or with part-time employment exhibit higher likelihood of belonging to the facility-sensitive class. This implies their higher sensitivity to test facilities than GP-referral discount, which may be because older patients may care more about health diagnosis than cost. Also, since part-time workers have more time, they may be willing to wait longer at the ED, which has all test-facilities.

Table 5

Results of Logistic Class Membership Analysis^a.

	Coefficient (95% CI)	SE	P Value
Facility-sensitive class
Constant	−0.15 (−0.52 to 0.22)	0.19	0.422
Age¹: >40 years	0.48^† (−0.19 to 0.78)	0.15	0.001
Employment²: PartTime	0.41^‡ (−0.03 to 0.84)	0.22	0.068
Employment²: EmployOther	0.15 (−0.21 to 0.52)	0.18	0.402
Travel time to ED − Travel time to GP	−0.01^‡ (−0.03 to 0.00)	0.01	0.067
Perception³: NotCritical	−0.71* (−1.07 to −0.35)	0.18	<0.001
Log-likelihood	−538.1

CI, confidence interval; ED, emergency department; GP, general practitioner.

Level of reference: 1, Age: 21–40; 2, Employment: full-time; 3, Perception: critical.

, ^†, and ^‡ indicate 1%, 5%, and 10% levels of significance, respectively.

An important observation from Table 5 is that patients’ perceptions of their severity significantly influence their class membership and thus heterogeneity of preferences. Patients who perceived their presented scenario as “critical enough to go to ED directly” exhibit much higher likelihood of belonging to the facility-sensitive class, and hence of preferring ED. Another important observation from this model is that the “Scenario” variable is not significant, indicating that the effects of the different nonurgent scenarios on patients’ preference are sufficiently captured by patients’ perceptions (“critical” v. “not critical”) of those scenarios.

Predicted Uptake of Countermeasures

Figures 4 to 7 show percentage reduction in nonurgent ED visits due to the different countermeasures, as per our predictive analysis described in “Analysis” section. In all figures, 0% reduction corresponds to the status quo attribute-level (e.g., “no discount” in Figure 4), and other attribute-levels on x-axis represent countermeasures. A common observation for all countermeasures is that their uptake is similar for day versus night hours if, under both situations, patients visit ED expecting the same wait. However, if patients visit ED expecting “short” versus “long” waits, then uptakes are quite different.

Figure 4

Percent reduction in nonurgent ED visits versus “GP-referral discount.”

Figure 5

Percent reduction in nonurgent ED visits versus % of patients whose GPs have “all test facilities.”

Figure 6

Percent reduction in nonurgent ED visits due to perception correction.

Figure 7

Percent reduction in nonurgent ED visits versus patients’ belief on “ED wait” (based on published ED waits): Circled point represents assumed patients’ belief under status quo.

Figure 4 shows that the effect of GP-referral discount is stronger when patients visit ED expecting “short” wait, with 23.5% reduction in nonurgent ED visits in the best case. Figures 5 and 6 show that test-facility provision at GPs and correcting patients’ misperceptions of severity have stronger effect when patients visit ED expecting “long” waits, with best case reductions of 74.8% and 7.8%, respectively. Finally, Figure 7 shows that publishing ED wait information could reduce up to 76.2% nonurgent ED visits if actual waits are much higher than current beliefs (Figure 7A); however, it could also drastically increase nonurgent ED visits if actual waits happen to be lower than common beliefs (Figure 7B).

Discussion

We now discuss policy implications of our results. The latent class multinomial logit model presented in the previous section identified two patient classes. Among these, for the discount-sensitive class, introduction of “GP-referral discount” significantly increases preference for GPs. In particular, providing entire GP fee waiver upon referral to ED appears to be quite effective with standard day/night GP charges in Singapore (represented by S$50/S$100 in the survey). Though the discount shows very mild influence on facility-sensitive class, the large size of discount-sensitive class suggests that hospitals can benefit overall from such incentive schemes for diverting nonurgent patients from ED to GPs.

Our patient choice model also suggests other potential countermeasures that may be useful in reducing nonurgent ED visits, which we discuss below.

The model shows that patients are heterogeneous in their preferences for test-facility-availability, with facility-sensitive class being highly sensitive to it and discount-sensitive class not. Given significant size of the former class, increasing test facilities at GPs can also help in reducing nonurgent ED visits. This observation is similar to those of concurrent DCE studies^19,20 conducted in China, which also found that improvements to medical equipment/skill at primary care level could be effective in directing patients to primary care.

Previous DCE studies^12,19,20 that explored patients’ preferences between ED and GP have identified waiting time as an important attribute. Our model corroborates this as it shows that both patient classes are highly time-sensitive and have clear preferences for shorter waiting time. Currently, hospitals in Singapore as well as in many other countries do not publish ED wait information online. Since patients are highly time-sensitive, information made available about long ED waits should discourage nonurgent patients from visiting ED. According to the follow-up questions to our DCE, 73% respondents would like to have a mobile application for ED estimated waiting times.

The logistic class membership model shows that patients who perceived their presented nonurgent scenarios as “critical enough to go to ED directly” have significantly higher preference for ED. Furthermore, patients’“perceptions” sufficiently capture preference heterogeneity due to the presented nonurgent scenarios. These results suggest that measures to correct patients’ misperception of their ailments may also be useful. In our survey, 77% respondents said that they would like to have a call facility or application which can help them assess their symptoms prior to arrival at ED.

The predictive analysis further shows that uptake of GP-referral discount would be higher when patients visit ED expecting “short” wait, whereas uptakes of test-facility provision at GPs and perception-correction would be higher when patients visit ED expecting “long” waits, because patients’ choice of ED despite expecting long wait implies their strong preference for test facilities or misperception of severity. Publishing of ED wait information, on the other hand, can have good uptake if actual ED waits are much higher than patient’ beliefs, but it could also drastically increase nonurgent ED visits if actual waits happen to be lower than common beliefs. Therefore, before considering this measure, it is important to understand how patients’ beliefs compare to actual ED waits.

Overall, the predictive analysis shows that compared to other countermeasures, test-facility provision at GPs has much higher uptake, whereas perception-correction has much smaller uptake as it only influences preferences of those with “critical” perception, which is a relatively small proportion of patients. Interestingly, for GP-referral discount the analysis shows that, for the same GP fee of S$50, describing the discount as “entire GP fee waived” as opposed to “S$50 discount on ED fee” is much more effective, which indicates that describing the incentive as “entire GP fee waiver” (possibly with minimum discount of S$50) may be a more effective strategy for further implementation of the scheme.

Limitations

There are some potential limitations of our study that are worth mentioning. The first limitation comes from the use of DCE which uses hypothetical scenarios. It could be argued that respondents may behave differently under real versus hypothetical situations.^33,34 Nevertheless, to minimize such differences, we designed the attributes and levels in our survey as close as possible to those in a real-world situation. Also, though we emphasized to make choices based on the hypothetical situation, it is possible that some respondents made choices based on their past medical experience. To avoid misinterpretation of attributes, we constructed several warm-up questions to help the respondents properly understand all attributes before starting with choice tasks.

The other limitation relates to sampling of participants as we do not know the profile of real patient population that considers choice between ED and GPs, in particular for the “nonurgent” conditions of interest (those constituting nonurgent ED demand). Although nonurgent patient statistics were available for CGH-ED, it was not available for GPs as GPs in Singapore do not record such statistics. Because of this nontriviality we chose to use convenience sampling to recruit participants. However, after building the choice model, we predicted the distribution of patients from the survey sample that would choose ED under status quo ED/GP attribute-levels (from Table 2). We found that this is comparable to the real profile distribution of nonurgent patients at CGH-ED, which implies that the survey sample is a reasonable representation of the patient population of interest.

Conclusions

This study showed that the new GP-referral financial incentive introduced in Singapore can be effective in reducing nonurgent ED visits, if it reasonably covers the (extra) cost of visiting a GP. The ease of implementation of such financial incentive makes it a more viable solution compared to other infrastructure-heavy solutions such as test-facility-provision at GPs or perception-correction measures. In Singapore context, however, our results implied that increase in GP test facilities can have much bigger overall impact; hence, it may be worth considering for a long-term solution. In fact, the financial incentive may serve as a complement to the other two measures, as the former and the latter appear to influence distinct classes (discount-sensitive and facility-sensitive) of patients.

Results of this study should be of interest to health care professionals in other places too, because GP-referral financial incentive can be easily adapted to other systems. Compared to financial disincentives such as ED co-payment/fee increase that have been tried in some countries in the past, this novel incentive has the advantage that it does not impose penalty for ED use. Hence, it can provide a better/feasible alternative where ED fee increase may not be preferable/feasible.

Supplemental Material

sj-bib-1-mpp-10.1177_23814683211027552 – Supplemental material for Nonurgent Patients’ Preferences for Emergency Department Versus General Practitioner and Effects of Incentives: A Discrete Choice Experiment

Supplemental material, sj-bib-1-mpp-10.1177_23814683211027552 for Nonurgent Patients’ Preferences for Emergency Department Versus General Practitioner and Effects of Incentives: A Discrete Choice Experiment by Yuliu Su, Shrutivandana Sharma, Semra Ozdemir, Wai Leng Chow, Hong-Choon Oh and Ling Tiah in MDM Policy & Practice

Supplemental Material

sj-docx-2-mpp-10.1177_23814683211027552 – Supplemental material for Nonurgent Patients’ Preferences for Emergency Department Versus General Practitioner and Effects of Incentives: A Discrete Choice Experiment

Supplemental material, sj-docx-2-mpp-10.1177_23814683211027552 for Nonurgent Patients’ Preferences for Emergency Department Versus General Practitioner and Effects of Incentives: A Discrete Choice Experiment by Yuliu Su, Shrutivandana Sharma, Semra Ozdemir, Wai Leng Chow, Hong-Choon Oh and Ling Tiah in MDM Policy & Practice

Footnotes

Appendix

Acknowledgements

We would like to thank the administration of Changi General Hospital, Singapore, for their support in the recruitment of study participants.

We would also like to thank Dr. Adam Oei, Mr. Tony Lim, Ms. Suxin Liaw, and Mr. Andrew Tan for survey administration, and Professors Yow Wei Quin, Bikramjit Das, and Costas Courcoubetis from Singapore University of Technology and Design for their valuable feedback during study design.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this study was provided by grants from SUTD-MIT International Design Center (IDC) (Grant Award No. IDG21500103) and Ministry of Education, Singapore (MOE AcRF Tier-2 Grant No. MOE2019-T2-2-168). Any findings, conclusions, or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the IDC or the Ministry of Education, Singapore. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

Authors’ Note

Institutions where work was done: The work was done at the Engineering Systems and Design Pillar, Singapore University of Technology and Design, Singapore, and Health Services Research Department, Changi General Hospital, Singapore.

Presentations: The work presented in this manuscript has been orally presented at the following conferences. These conferences do not have any published proceedings.

POMS-HK International Conference, Hong Kong, 2020 (POMS-HK Chapter; January 4–5, 2020; HKU Business School).

INFORMS Healthcare Conference 2019 (INFORMS; July 27–29, 2019; MIT, Cambridge, MA).

INFORMS MSOM Conference 2019 (INFORMS; July 1–2, 2019; National University of Singapore, Singapore).

ORCID iDs

Shrutivandana Sharma

Semra Ozdemir

Supplemental Material

Supplementary material for this article is available on the Medical Decision Making Policy & Practice website at .

*

Such systems could possibly implement a referral-based incentive by compensating the patients referred from GP to ED for the extra costs (due to travel, wait, etc.) incurred in visiting a GP.

*

In the unlabeled DCEs, respondents’ choice tasks consisted of generic health care provider alternatives (Option A v. Option B), both of which were constructed from the same set of attribute-levels.

*

Averaged between January 2013 and June 2016.

*

Across different EDs in Singapore, the daily (averaged between January 2013 and June 2016) median waiting time (to see the doctor) of the lower-urgency P3 patients ranges from 47 to 68 minutes, and their 95th percentile waiting time ranges from 97 to 221 minutes.³⁶

References

Lega

Mengoni

. Why non-urgent patients choose emergency over primary care services? Empirical evidence and managerial implications. Health Policy. 2008;88(2–3):326–38.

Pines

Hilton

Weber

, et al. International perspectives on emergency department crowding. Acad Emerg Med. 2011;18(12):1358–70.

Hamarian

. To cut AE load, hospital offers discount to patients with GP referrals [cited December 1, 2018]. Available from: https://www.todayonline.com/singapore/cut-ae-load-hospital-offers-discount-patients-gp-referrals

Uscher-Pines

Pines

Kellermann

Gillen

Mehrotra

. Deciding to visit the emergency department for non-urgent conditions: a systematic review of the literature. Am J Manag Care. 2013;19(1):47–59.

Ahmed

Khan

Mehmood

Ahad

Hussain

Ullah

. A study on the factors associated with non-urgent utilization of emergency department. J Saidu Med Coll. 2018;8(2).

Turner

Coster

Chambers

, et al. What Evidence Is There on the Effectiveness of Different Models of Delivering Urgent Care? A Rapid Review. NIHR Journals Library; 2015.

Morgan

Chang

Alqatari

Pines

. Non–emergency department interventions to reduce ED utilization: a systematic review. Acad Emerg Med. 2013;20(10):969–85.

Anantharaman

. Impact of health care system interventions on emergency department utilization and overcrowding in Singapore. Int J Emerg Med. 2008;1(1):11–20.

Morley

Unwin

Peterson

Stankovich

Kinsman

. Emergency department crowding: a systematic review of causes, consequences and solutions. PLoS One. 2018;13(8):e0203316.

10.

Raven

Kushel

Penko

Bindman

. The effectiveness of emergency department visit reduction programs: a systematic review. Ann Emerg Med. 2016;68(4):467–83.e15.

11.

Ryan

Gerard

Amaya-Amaya

. Using Discrete Choice Experiments to Value Health and Health Care. Vol 11. Springer Science & Business Media; 2007.

12.

Harris

Whitty

Kendall

, et al. The Australian public’s preferences for emergency care alternatives and the influence of the presenting context: a discrete choice experiment. BMJ Open. 2015;5(4):e006820.

13.

Leung

Chan

SSC

Chau

PKY

Chua

. Using conjoint analysis to assess patients’ preferences when visiting emergency departments in Hong Kong. Acad Emerg Med. 2001;8(9):894–8.

14.

Durand

Palazzolo

Tanti-Hardouin

Gerbeaux

Sambuc

Gentile

. Nonurgent patients in emergency departments: rational or irresponsible consumers? Perceptions of professionals and patients. BMC Res Notes. 2012;5(1):525.

15.

Kraaijvanger

van Leeuwen

Rijpsma

Edwards

. Motives for self-referral to the emergency department: a systematic review of the literature. BMC Health Serv Res. 2016;16(1):685.

16.

Yip

Wang

Liu

. Determinants of patient choice of medical provider: a case study in rural China. Health Policy Plan. 1998;13(3):311–22.

17.

Habtom

GMK

Ruys

. The choice of a health care provider in Eritrea. Health Policy. 2007;80(1):202–17.

18.

Qian

Pong

Yin

Nagarajan

Meng

. Determinants of health care demand in poor, rural China: the case of Gansu province. Health Policy Plan. 2009;24(5):324–34.

19.

Liu

Kong

de Bekker-Grob

. Public preferences for health care facilities in rural China: a discrete choice experiment. Soc Sci Med. 2019;237:112396.

20.

Liu

Kong

Wang

Zhong

van de Klundert

. The impact of hospital attributes on patient choice for first visit: evidence from a discrete choice experiment in Shanghai, China. Health Policy Plan. 2020;35(3):267–78.

21.

Singapore Ministry of Health. Primary care survey 2014 report [cited February 1, 2020]. Available from: https://www.moh.gov.sg/resources-statistics/reports/primary-care-survey-2014-report

22.

Kjaer

. A Review of the Discrete Choice Experiment—With Emphasis on Its Application in Health Care. Syddansk Universitet Denmark; 2005.

23.

Louviere

Hensher

Swait

. Stated Choice Methods: Analysis and Applications. Cambridge University Press; 2000.

24.

David

John

William

. Applied Choice Analysis. 2nd ed. Cambridge University Press; 2015.

25.

Jaynes

Wong

. Using blocked fractional factorial designs to construct discrete choice experiments for healthcare studies. Stat Med. 2016;35(15):2543–60.

26.

Miguel

Ryan

Amaya-Amaya

. “Irrational” stated preferences: a quantitative and qualitative investigation. Health Econ. 2005;14(3):307–22.

27.

Lancsar

Louviere

. Deleting “irrational” responses from discrete choice experiments: a case of investigating or imposing preferences? Health Econ. 2006;15(8):797–811.

28.

Orme

Johnson

. Including Holdout Choice Tasks in Conjoint Studies. Sawtooth Software Inc; 2015.

29.

Orme

. Sample size issues for conjoint analysis. In: Getting Started With Conjoint Analysis: Strategies for Product Design and Pricing Research. 2nd ed. Research Publishers LLC; 2010.

30.

Greene

Hensher

. A latent class model for discrete choice analysis: contrasts with mixed logit. Transp Res Part B Methodol. 2003;37(8):681–98.

31.

Train

. Discrete Choice Methods With Simulation. Cambridge University Press; 2009.

32.

Bech

Gyrd-Hansen

. Effects coding in discrete choice experiments. Health Econ. 2005;14(10):1079–83.

33.

Johnson

Özdemir

Mansfield

, et al. Crohn’s disease patients’ risk-benefit preferences: serious adverse event risks versus treatment efficacy. Gastroenterology. 2007;133(3):769–79.

34.

Brown

Johnson

Poulos

Messonnier

. Mothers’ preferences and willingness to pay for vaccinating daughters against human papilloma virus. Vaccine. 2010;28(7):1702–8.

35.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

36.

Yap

JCH

Qian

. Understanding Hospital Waiting Times. Saw Swee Hock School of Public Health, National University of Singapore; 2019.

37.

MoneySmart. 24 hour clinics in Singapore a guide to rates & surcharges [cited January 31, 2020]. Available from: https://blog.moneysmart.sg/healthcare/24-hour-clinics-singapore

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB

0.45 MB