Sage Journals: Discover world-class research

Abstract

Background

The decision to admit patients to the intensive care unit (ICU) is complex. Structuring the decision-making process may be beneficial to patients and decision-makers alike. The aim of this study was to investigate the feasibility and impact of a brief training intervention on ICU treatment escalation decisions using the Warwick model- a structured decision-making framework for treatment escalation decisions.

Methods

Treatment escalation decisions were assessed using Objective Structured Clinical Examination-style scenarios. Participants were ICU and anaesthetic registrars with experience of making ICU admission decisions. Participants completed one scenario, followed by training with the decision-making framework and subsequently a second scenario. Decision-making data was collected using checklists, note entries and post-scenario questionnaires.

Results

Twelve participants were enrolled. Brief decision-making training was successfully delivered during the normal ICU working day. Following training participants demonstrated greater evidence of balancing the burdens and benefits of treatment escalation. On visual analogue scales of 0–10, participants felt better trained to make treatment escalation decisions (4.9 vs 6.8, p = 0.017) and felt their decision-making was more structured (4.7 vs 8.1, p = 0.017).

Overall, participants provided positive feedback and reported feeling more prepared for the task of making treatment escalation decisions.

Conclusion

Our findings suggest that a brief training intervention is a feasible way to improve the decision-making process by improving decision-making structure, reasoning and documentation. Training was implemented successfully, acceptable to participants and participants were able to apply their learning. Further studies of regional and national cohorts are needed to determine if training benefit is sustained and generalisable.

Keywords

intensive care critical care decision making clinical decision-making triage

Introduction

The decision to escalate treatment to the intensive care unit (ICU) is complex and influenced by multiple factors. These factors include organisational and clinician-based factors in addition to clinical and patient-focussed information.^1–5 Decision-making is also influenced by cognitive bias; a systematic pattern of deviation from rational decision-making. An example of cognitive bias includes seeking information that only supports an initial hypothesis.⁶ This bias can result in predictive patterns of behaviour which may influence ethical decision-making.^7,8 This adds further complexity to treatment decisions which are frequently made in time-pressured environments using uncertain information.⁹

There is variability in decision-making between clinicians, leading to concerns regarding equity and ethical justification of decision-making.^10,11 These decisions have profound consequences for patients, their families, and healthcare systems.^12,13 Structured frameworks to support clinician decision-making have been developed, such as the Warwick Model.^1,8,14

The decision-making process involved in ICU admission remains relatively understudied.^9,15 Studying the decision-making process is problematic because reliance on clinical documentation, whilst providing some insight, may not truly reflect underlying thought processes.¹⁶ Real-time observation of clinical decision-making may improve investigation but is intensive, costly, intrusive to clinicians and may be influenced by the presence of an observer.¹⁵ Alternative methods to study the process of decision-making in treatment escalation are therefore needed.

Simulated clinical scenarios in objective structured clinical examinations (OSCEs) are well-established in the assessment of clinical competencies.¹⁷ They enable the assessment of analytical and evaluative skills alongside ethical reasoning skills.^18,19 Thus, OSCEs are a strong potential candidate for the assessment of decision-making.

The purpose of this study was to investigate the feasibility of a brief training intervention in ICU treatment escalation decisions using a structured model of decision-making.

Feasibility would be evaluated in terms of the acceptability (whether the training was suitable and acceptable to participants), practicality of the intervention (whether participants are able to apply intervention learning), and implementation (whether the intervention could be successfully delivered).²⁰

Methods

Setting and participants

This feasibility study was carried out in a single acute NHS hospital in the United Kingdom. Participants were anaesthetic and intensive care registrars at specialist training level (ST-3-7), equivalent to postgraduate year 5–9. Participants were recruited via email adverts which outlined the study. Written consent was obtained prior to the study. Participants were excluded if they had previously received training in the Warwick model.¹⁴

Scenarios

Development and delivery of scenarios

Two scenarios concerning treatment escalation were developed by the authors who included two ICU junior doctors, a consultant intensivist and an ethicist. Scenarios were piloted with an ICU junior doctor and refined following feedback. Participants were initially presented with a brief clinical history and examination findings. Further information including investigation results and clinic letters were available upon request. An actor played the patient’s relative to allow participants to explore patient values. Scenarios were designed so that the escalation decision was unclear and the ICU admission decision could be justifiable either way. Participants were advised of the study aims prior to the clinical scenarios. Two investigators (HMR, AJB) facilitated the scenarios and regularly prompted participants to verbalise their thoughts. Participants undertook one scenario followed by brief training and then the second scenario on the same day (Figure 1). Half of the participants completed scenario A first and the remainder completed scenario B first. Details of the clinical scenarios are in Supplementary file 1.

Figure 1.

Study flowchart.

Decision-making training

The Warwick model of structured decision-making provides a framework for treatment escalation decisions.^1,14 At the time of this study no comparable frameworks for clinical decision-making were available for comparison. This model supports a structured approach to decision-making surrounding escalation of therapy and subsequently, the decision whether or not to admit to the ICU. It was developed from a multi-centre observational study of decision-making and subsequent expert opinion, refined through an iterative approach and a consensus conference. The model is intended to guide a full process of decision-making in a pressured environment and codifies best practice in decision-making. The model is applied by clinicians working through three domains during the assessment of a patient. Firstly, gathering evidence, secondly using this evidence in a reasoning exercise to recommend a treatment and thirdly, implementing the treatment by considering communication, review arrangements and mitigating for the inability to deliver treatment promptly. A schematic representation of the model is contained in Figure 2. This was given to participants as an aide memoire after the study.

Figure 2.

Schematic of the Warwick model.

While all components of the model were addressed in the training, the OSCE-style clinical scenarios are best suited to assess evidence gathering and reasoning domains. Evidence gathering focusses on information that will enable the clinician to determine the following: (1). Whether the patient requires ICU for organ support or other reasons and includes their acute diagnosis and relevant chronic conditions. (2). Their capacity to recover from the acute illness, evidenced by the trajectory of their illness, frailty score, nutritional status and other prognostic markers. (3). The patient’s values and wishes, gathered from the most appropriate available source.

The reasoning domain of the model guides clinicians to identify relevant outcomes, balance the burdens and benefits of treatment, and identify a recommended treatment option. All participants received 90 minutes of training on the Warwick model via a small group tutorial using an identical slide set delivered by one ICU consultant (CB). It included education on the theory surrounding decision-making, information on cognitive bias, the impact of clinician factors such as fatigue and hunger, and a worked example of decision-making in practice.

Data collection

Data was collected in a standardised format during the pre- and post-training scenarios. A pre-specified checklist was used to collect data as participants verbalised their thought process (Supplementary file 2). The checklist consisted of 10 items directly mapped to the domains of the Warwick model. Two investigators (HMR, AJB) independently completed the checklist during the scenario and compared findings to reach consensus. Participants also made a clinical note entry of their assessment and escalation decision. A post-scenario questionnaire explored the participants’ decision-making process in more depth and is summarised in Table 1. This questionnaire was based on key elements of the Warwick model and explored experiences of decision-making. Specific prompts were given with free-text responses to explore participant reasoning. Participants indicated levels of confidence, ease and comfort in their response to questions using 10 cm visual analogue scales (VAS) (Table 1). A further questionnaire evaluated the Warwick model using Likert scales and free-text questions.

Table 1.

Post-scenario questionnaire components.

Did you recommend admission and how did you come to your decision?

How confident are you with the decision?

How easy did you find the decision?

How comfortable are you with the decision?

Was ICU admission needed to stop this patient dying now, what information did you use?

How confident are you with your answer?

How easy did you find it to answer the question?

How comfortable are you with your answer?

If the patient was admitted to ICU, how likely are they to recover to leave hospital?

How confident do you feel about this answer?

How easy did you find it to answer the question?

How comfortable are you with your answer?

To what extent do you think this patient would want ICU admission for themselves?

How confident do you feel about this answer?

How easy did you find it to answer the question?

How comfortable are you with your answer?

At what stage did you make your decision?

Did you change your mind at any point, if so, what information did you use?

Overall

How comfortable are you with your decision-making process?

Do you feel adequately trained?

How uncertain do you feel about your decision?

How structured was your decision-making process?

Do you feel you considered all the factors?

How realistic did you find the scenario?

Do you feel you have learnt something?

VAS questions in italics.

Data analysis

Descriptive statistics were used to summarise quantitative data. Wilcoxon signed-rank tests were used to compare checklist and questionnaire items before and after decision-making training. Clinical note entries and questionnaire free-text responses were managed with NVivo software (QSR, International Pty Ltd) and analysed using a thematic approach. Statistical analyses were performed using SPSS 22 (IBM Corp. Armonk, NY) with p < 0.05 taken as statistically significant.

Ethics

The study was approved locally by the Research and Development department at University Hospitals Coventry and Warwickshire NHS Trust. A participant information sheet was distributed and written informed consent obtained.

Results

Participant characteristics are summarised in Table 2. One participant completed scenario A but was unable to attend the decision-making training and therefore did not complete scenario B.

Table 2.

Participant characteristics.

Characteristic	N=12
Median age (IQR) – years	37 (33-38)
Gender – no. (%)
Male	7 (58%)
Female	5 (42%)
Grade – no. (%)
ST7	5 (42%)
ST6	2 (17%)
ST5	3 (25%)
ST3	2 (17%)
Median medical experience (IQR) – years	10 (9–12)
Median ICU experience (IQR) – months	18 (11–27)
Previous decision-making training – no. (%)
Yes	3 (25%)
No	9 (75%)

Practicality of the decision-making training intervention

The practicality of the intervention was evaluated as the ability of the participants to successfully apply learning to a simulated clinical scenario. This assessment was made through analysis of scenario checklist assessments, clinical note entries and post-scenario questionnaires.

Scenario checklist assessments

Scenario checklist data showed that both before and after decision-making training, all participants demonstrated evidence of assessing the acute clinical information and the patient’s capacity to recover (Table 3). Before decision-making training, most participants assessed the chronic clinical information (8, 73%), reversibility (9, 82%), patient wishes (9, 82%), demonstrated evidence of balancing (8, 73%) and communicated their decision (9, 82%). However, irrespective of training grade, fewer participants assessed family wishes (6, 55%), considered the availability of resources (5, 45%) and made review arrangements (5, 45%).

Table 3.

Practicality of decision-making intervention: Table showing the effect of the intervention on participants’ inclusion of the components of the Warwick model in their decision-making process.

	Before decision-making training	After decision-making training	P-value
Assessment of acute clinical information	11 (100%)	11 (100%)	1.000
Assessment of chronic clinical information	8 (73%)	11 (100%)	0.083
Assessment of reversibility	9 (82%)	10 (91%)	0.317
Assessment of patient capacity to recover	11 (100%)	11 (100%)	1.000
Assessment of family wishes	6 (55%)	8 (73%)	0.317
Assessment of patient wishes	9 (82%)	8 (73%)	0.564
Balancing burdens versus benefits	8 (73%)	9 (82%)	0.655
Consideration of resources	5 (45%)	4 (36%)	0.564
Arrangements for review	5 (45%)	6 (55%)	0.564
Communicates decision	9 (82%)	9 (82%)	1.000

Data reported as number of participants (% of total), with p-value from Wilcoxon Signed-Rank test.

Analysis of during-scenario checklists investigated the domains of the Warwick model. Participant experience and training background did not appear to influence the evaluation of domains. After decision-making training, elements of the evidence domain were considered by more participants, including the patient’s medical background (8, 73% vs 11, 100%) and family wishes (6, 55% vs 8, 73%). However, elements of the implementation domain remained low among both junior and senior trainees, with only four (36%) participants considering the availability of resources and six (55%) considering review arrangements. Prior to decision-making training, almost all participants demonstrated evidence of assessment of reversibility (9, 82%), assessment of patient’s wishes (9, 82%), and balancing the burdens and benefits of treatment options (8, 73%). These proportions remained similar after training (Table 3).

Clinical note entries

Qualitative analysis of clinical note entries identified several themes which were combined and broadly categorised into “assessment of clinical information” and “decision-making reasoning”.

Assessment of clinical information

All participants showed evidence of reviewing clinical information both before and after decision-making training; however clinical note entries after training followed a more systematic approach for both junior and senior registrars. Assessments included recognition of comorbidities, assessment of the acute clinical information and response to treatments.

Decision-making reasoning

There were disparities between the data collected from the scenario checklists and that collected from clinical note entries. Prior to decision-making training, only two (18%) clinical note entries showed evidence of balancing the burdens and benefits of treatment, irrespective of ICU admission recommendation. This contrasted with the data captured by checklists during the scenarios where most participants (8, 73%) demonstrated evidence of balancing. As such, whilst balancing may have been part of the cognitive process, it was evident in only a few clinical note entries. However, following decision-making training, more participants (6, 55%) clearly demonstrated evidence of balancing the burdens and benefits of treatment within clinical note entries. This finding differed slightly from data captured by during scenario checklists where even more participants (9, 82%) showed evidence of balancing. As part of this balancing process, participants appeared more confident in suggesting potential limits to care.

Following decision-making training, there were no observed differences in how participants formulated an impression of the patient’s functional status. Participants considered the patient’s level of mobility and social support through discussion with a family member. This appeared to heavily influence the ICU admission decision. Review arrangements were also cited more consistently following decision-making training. This was particularly of note when the decision was not to admit to ICU, where participants frequently mentioned use of critical care outreach review. Only one participant considered available resources in their clinical note entry either before or after training.

Acceptability of the decision-making training intervention

Acceptability of the intervention was determined by post-scenario VAS questionnaires. These evaluated how comfortable and confident participants were in making assessments related to components of the Warwick model, alongside their overall confidence in decision-making practice before and after the training intervention.

After decision-making training participants felt more adequately trained to make treatment escalation decisions (median 4.9 vs 6.8, p = 0.017) (Figure 3). Furthermore, responses indicated that participants felt their decision-making process was more structured (4.7 vs 8.1, p = 0.017) and that they had considered all relevant factors (5.2 vs 7.9, p = 0.032). There were no differences in participant comfort (6.4 vs 6.3, p = 0.575) or uncertainty (5.4 vs 5.3, p = 0.859) in their decision-making process (Table 4). Participant confidence, ease, and comfort in their assessment of whether they recommended ICU admission, whether ICU admission was required to prevent imminent death, whether the patient had capacity to recover from an ICU admission to leave hospital, and whether the patient would want ICU admission, all remained unchanged after decision-making training.

Figure 3.

Participant self-rated factors regarding ICU treatment escalation decision-making process before and after training with the Warwick model. *=p < 0.05, ns=p > 0.05.

Table 4.

Acceptability of decision-making intervention: Table showing effect of the intervention on participants experience of decision-making process.

	Decision-making training (10 cm VAS, 0 = not at all, 10 = completely)
Factor	Before	After	P-value
Recommendation for admission	-		-
Confidence	7.0 (3.4)	6.6 (2.0)	0.879
Ease	6.6 (5.8)	6.2 (1.8)	0.262
Comfort	8.4 (2.3)	6.6 (3.0)	0.109
Requirement for admission to prevent death	-		-
Confidence	7.4 (1.8)	6.0 (3.3)	0.625
Ease	7.2 (1.1)	6.4 (1.8)	0.656
Comfort	6.9 (2.1)	6.2 (3.4)	1.000
Likely to recover and leave hospital	-		-
Confidence	5.4 (2.7)	6.0 (3.6)	0.722
Ease	5.4 (4.8)	5.6 (3.3)	0.484
Comfort	5.1 (3.9)	6.1 (2.7)	0.450
Would patient want ICU admission	-		-
Confidence	6.4 (3.9)	6.4 (4.0)	0.833
Ease	6.2 (4.2)	6.1 (4.6)	0.624
Comfort	7.4 (3.6)	6.6 (3.5)	0.624
Decision-making process
Comfort	6.4 (2.3)	6.3 (2.4)	0.575
Adequately trained	4.9 (4.0)	6.8 (2.0)	0.017*
Uncertainty	5.4 (4.6)	5.3 (4.0)	0.859
Structured	5.7 (1.6)	8.1 (1.5)	0.022*
Considered all relevant factors	5.2 (2.6)	7.9 (3.9)	0.032*

Data reported as median (interquartile range), with p-value from Wilcoxon Signed-Rank test. *Significance at p < 0.05.

Further information on acceptability of the intervention was gained from free-text comments. These were positive with many citing the usefulness of a structure to organise the decision-making process. One participant stated that the training should be delivered regionally to ICU trainees and others felt the training would be useful at consultant level too. Participants rated the Warwick model highly, reporting that training had provided them with a structure for decision-making and would improve their treatment escalation decision-making.

Implementation of the decision-making training

Decision-making training was successfully delivered to small groups (1–5 participants) during the normal clinical working day in an opportunistic fashion. Participant feedback showed that training delivery was convenient and was managed within the workload and setting of a busy tertiary teaching hospital.

Discussion

This study piloted a brief training intervention based on the Warwick model of decision-making and sought to improve decision-making surrounding escalation of treatment to ICU. It provided data on whether such an intervention was feasible and worthy of further development. Our findings show that the intervention could be implemented as part of the normal ICU working day, was acceptable to participants and that participants were able to apply their learning.

Participants provided positive feedback and reported an improvement in how prepared they were for the task of making treatment escalation decisions, alongside the structuring of these decisions. While this study did not show any change in participant feelings of confidence and comfort in making more granular assessments, within the overall process they did feel more prepared for this task. This discordance may be appropriate given the complexities of decision-making in clinical practice. Indeed, case realism was reported as high and cases were considered representative of typical patients encountered by participants, in contrast to previous studies of ICU clinician decision-making.²¹

There are difficulties in teaching clinical decision-making ²² and at the time of development of the Warwick model, no other guidance was available to support a clinician at the patient’s bedside. Where ethical frameworks can provide theoretical guidance, this does not account for the real-world challenges faced by clinicians. Training and teaching should account for the impact of fatigue, stress and levels of confidence, as well as clinical and ethical reasoning. The decision-making framework and training tested in this study considered both clinical and ethical reasoning, providing a structure in which clinicians can formulate their thoughts and presented the training in the context of the clinician’s experience of a busy environment. This may account for the clear acceptability of the intervention above the fact that no previous training or model was available. It is also notable that this intervention was well received at the same time as improving documentation.

Investigating the decision-making process is troublesome. Previous research has predominantly focussed on how individual factors influence binary ICU admission decisions,^23,24 rather than the process of formulating decisions. Charlesworth and colleagues observed and interviewed ICU clinicians responding to referrals.¹⁵ Whilst they were able to explore the decision-making process for individual clinicians, the study was not able to investigate how decision-making processes vary between individuals when exposed to the same case. In our study, using OSCE-style clinical scenarios, we were able to directly compare specific elements of the decision-making process between clinicians and to assess the impact of training.

The decision-making training intervention was found to have a broadly positive impact in our study. Following decision-making training, registrars felt better trained, felt they were given more structure and that they better considered relevant factors in ICU treatment escalation decisions. Clinical note entries after decision-making training also showed more evidence of balancing the burdens and benefits of treatment escalation. This is a major strength of the decision-making training which may help ICU clinicians make more reasoned decisions. Our analysis demonstrated that treatment decisions and case interpretation varied considerably for the same scenarios, even after decision-making training. These findings are consistent with those previously reported.²⁵ This may reflect the complexity of scenarios used in our study. In addition, whilst training may not alter consensus for ICU admission decisions, it improves transparency, accountability and justification for decisions.

Our study found a discrepancy between real-time observation and clinical note entries with respect to the balancing of burdens and benefits of treatment escalation. This illustrates some of the difficulties in using clinical note entries in isolation to study decision-making and highlights the need for more transparency in clinical note entries.

The experiences and seniority of decision-makers can influence their practice.^26,27 Our study investigated registrars with a range of experience levels. Prior to decision-making training, it was predicted that senior registrars would better structure their decision-making compared to junior registrars. However, this was not evident in our study. Whilst senior registrars have more experience of treatment escalation decisions, this may not translate to improvements in decision-making structure due to cognitive bias and lack of formal training.⁶ Though senior clinicians may be resistant to changes in practice,²⁸ potentially limiting the response to decision-making training, our study showed that all participants were supportive of decision-making training and the Warwick model.

Limitations

As a feasibility study only, our cohort consisted of a small sample size from a single centre. This may reduce the generalisability of our findings. However, by comparing participants working in the same clinical and social environment the variability of decision-making was less likely to be influenced by external factors.

The study used simulated clinical scenarios rather than direct clinical exposure to explore the decision-making process. It is possible that participants would not respond the same way in a real clinical situation. Direct observation of participants also risks altering their practice.²⁹ Participants completed the second scenario immediately after decision-making training which may limit the testing of decision-making training effectiveness. However, our methodology enabled us to investigate the formulation of treatment decisions to greater depth than would be possible in the clinical setting.

Conclusion

Our findings show that a brief training intervention is a feasible way to improve the process of decision-making. This intervention improves decision-making structure, increases the prevalence of clear reasoning surrounding a decision, improves documentation and makes clinicians feel better prepared for this task. Given the findings we believe that this intervention is feasible and further implementation should be considered. Studies of regional and national cohorts with extended follow up will help to determine if training benefit is sustained.

Supplemental Material

Supplemental Material - Investigating the impact of brief training in decision-making on treatment escalation to intensive care using objective structured clinical examination-style scenarios

Supplemental Material for Investigating the impact of brief training in decision-making on treatment escalation to intensive care using objective structured clinical examination-style scenarios by Hisham M Riad, Adam J Boulton, Anne-Marie Slowther and Christopher Bassford in Journal of the Intensive Care Society

Footnotes

Author contributions

HMR and AJB contributed equally to this paper and are recognised as joint first authors. Study design: HMR, AJB, CB. Study conduct: HMR, AJB, CB. Data analysis and figure production: HMR, AJB. Drafting of manuscript: HMR, AJB. Review of manuscript: HMR, AJB, AMS, CB.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Hisham M Riad

Adam J Boulton

Supplemental Material

Supplemental material for this article is available online.

References

Bassford

Griffiths

Svantesson

, et al. Developing an intervention around referral and admissions to intensive care: a mixed-methods study. Health Serv Deliv Res 2019; 7(39).

Gopalan

Pershad

. Decision-making in ICU - a systematic review of factors considered important by ICU clinician decision makers with regard to ICU triage decisions. J Crit Care 2019; 50: 99–110.

Sprung

Geber

Eidelman

, et al. Evaluation of triage decisions for intensive care admission. Crit Care Med 1999; 27(6): 1073–1079.

Garrouste-Orgeas

Montuclard

Timsit

, et al. Predictors of intensive care unit refusal in French intensive care units: a multiple-center study. Crit Care Med 2005; 33(4): 750–755.

Dallison

Jones

Matthews

. Analysis of critical care referrals. Crit Care 2010; 14(Suppl 1): P469.

Stiegler

Tung

. Cognitive processes in anesthesiology decision making. Anesthesiology 2014; 120(1): 204–217.

Rogerson

Gottlieb

Handelsman

, et al. Nonrational processes in ethical decision making. Am Psychol 2011; 66(7): 614–623.

Harvey

DJR

Gardiner

. 'MORAL balance' decision-making in critical care. BJA Educ 2019; 19(3): 68–73.

Bassford

. Decisions regarding admission to the ICU and international initiatives to improve the decision-making process. Crit Care (London, England) 2017; 21(1): 174.

10.

Hanson

Danis

Garrett

, et al. Who decides? Physicians' willingness to use life-sustaining treatment. Arch Intern Med 1996; 156(7): 785–789.

11.

Garland

Shaman

Baron

, et al. Physician-attributable differences in intensive care unit costs: a single-center study. Am J Respir Crit Care Med 2006; 174(11): 1206–1210.

12.

Adhikari

Fowler

Bhagwanjee

, et al. Critical care and the global burden of critical illness in adults. Lancet (London, England) 2010; 376(9749): 1339–1346.

13.

Chin-Yee

D'Egidio

Thavorn

, et al. Cost analysis of the very elderly admitted to intensive care units. Crit Care (London, England) 2017; 21(1): 109.

14.

Rees

Bassford

Dale

, et al. Implementing an intervention to improve decision making around referral and admission to intensive care: results of feasibility testing in three NHS hospitals. J Eval Clin Pract 2020; 26(1): 56–65.

15.

Charlesworth

Mort

Smith

. An observational study of critical care physicians' assessment and decision-making practices in response to patient referrals. Anaesthesia 2017; 72(1): 80–92.

16.

Campling

Cummings

Myall

, et al. Escalation-related decision making in acute deterioration: a retrospective case note review. BMJ Open 2018; 8(8): e022021.

17.

Harden

Stevenson

Downie

, et al. Assessment of clinical competence using objective structured examination. Br Med J 1975; 1(5955): 447–451.

18.

Patrício

Julião

Fareleira

, et al.

Is the OSCE a feasible tool to assess competencies in undergraduate medical education?

Med Teach 2013; 35(6): 503–514.

19.

Waltz

Davis

Cadigan

, et al. Professionalism and ethics: a standardized patient observed standardized clinical examination to assess ACGME pediatric professionalism milestones. MedEdPORTAL 2020; 16: 10873.

20.

Bowen

Kreuter

Spring

, et al. How we design feasibility studies. Am J Prev Med 2009; 36(5): 452–457.

21.

Barnato

Mohan

Lane

, et al. Advance care planning norms may contribute to hospital variation in end-of-life ICU use: a simulation study. Med Decis Making 2014; 34(4): 473–484.

22.

Lighthall

Vazquez-Guillamet

. Understanding decision making in critical care. Clin Med Res 2015; 13(3–4): 156–168.

23.

Escher

Perneger

Chevrolet

. National questionnaire survey on what influences doctors' decisions about admission to intensive care. BMJ 2004; 329(7463): 425.

24.

Dahine

Mardini

Jayaraman

. The perceived likelihood of outcome of critical care patients and its impact on triage decisions: a case-based survey of intensivists and internists in a Canadian, quaternary care hospital network. PloS One 2016; 11(2): e0149196.

25.

Barnato

Hsu

Bryce

, et al. Using simulation to isolate physician variation in intensive care unit admission decision making for critically ill elders with end-stage cancer: a pilot feasibility study. Crit Care Med 2008; 36(12): 3156–3163.

26.

Cohen

Eichorn

Silver

. Admission decisions to a medical intensive care unit are based on functional status rather than severity of illness. a single center experience. Minerva Anestesiol 2012; 78(11): 1226–1233.

27.

Stelfox

Hemmelgarn

Bagshaw

, et al. Intensive care unit bed availability and outcomes for hospitalized patients with sudden clinical deterioration. Arch Intern Med 2012; 172(6): 467–474.

28.

Cabana

Rand

Powe

, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999; 282(15): 1458–1465.

29.

Parsons

. What happened at Hawthorne?: New evidence suggests the Hawthorne effect resulted from operant reinforcement contingencies. Science (New York, NY) 1974; 183(4128): 922–932.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB