Sage Journals: Discover world-class research

Abstract

Background and objectives:

To help house officers adjust quickly to their new responsibilities, a departmental handbook is provided. As this had not been updated recently and was underutilised, a quality improvement project was undertaken to improve this resource.

Methods:

The handbook was updated and new material added based on feedback from house officers and senior medical staff. Subsequently, house officers were surveyed on their utilization rate of the new handbook and its usefulness in various clinical scenarios. An open-book clinical quiz was administered to gauge their clinical competency.

Results:

There was a 26.8% increase in the proportion of house officers who used the handbook more than once every 2–3 days compared to pre-intervention. There was a significant improvement in quiz scores from 26.7/40 to 31.1/40 between the pre-intervention (n=19) and post-intervention (n=37) groups (p<0.001), as well as perceived usefulness of the handbook in various clinical scenarios as measured on a 10-point Likert scale.

Conclusions:

We find that improvement in the quality of a handbook for paediatric house officers is associated with increase in its utilization and benefits to their clinical competency and confidence level, especially on call. To increase the utilization rate of the handbook, both the user interface and content need to be improved.

Keywords

Junior doctors house officers paediatric handbook medical education

Introduction

The transition from a medical student to a post-graduate year 1 (PGY1) house officer (HO) is probably the most challenging phase in one’s medical career.^1,2 Similar to many centres worldwide, HO training in Singapore lasts a year and comprises of three or more clinical postings.³ It is imperative that HOs acquire knowledge and learn how to manage the new responsibilities and clinical problems encountered in each posting.

Traditionally, junior doctors have been recommended to read clinical textbooks or guidebooks which vary in the level of details included.^4,5 However, these publications do not provide rapid access to high-yield information, especially on call, and are unable to cover institution-specific protocols and practical information such as contact numbers or prescription guidelines which are crucial to the needs of HOs. To bridge this gap, a departmental handbook for junior doctors is a common solution.⁶ Other resources studied include the provision of mobile electronic devices for residents, an online editable wiki, or a mobile application (app).^7–9

KK Women’s and Children’s Hospital (KKH) houses the largest paediatric unit in Singapore and the Department of Paediatrics receives 80–100 new HOs annually. The supervision of the paediatric HOs’ training and welfare is headed by the PGY1 committee which is comprised of paediatric specialists. Since 2015, a soft-copy departmental HO handbook was introduced to help new paediatric HOs integrate more seamlessly into their new roles and on-call duties. However, common feedback was that the current version was too cumbersome to use and difficult to apply to clinical patient management, thus limiting the original intent of the handbook. Therefore, the Department of Paediatrics decided to improve the effectiveness of the paediatric HO handbook through a quality improvement (QI) project.

Methods

QI methodology including plan-do-study-act (PDSA) cycles was utilised. This study was exempted from consent-taking by the Singhealth Centralised Institutional Review Board (CIRB ref. no: 2019/2942).

Aims

The study aims were to improve the effectiveness of the handbook by increasing the proportion of HOs utilising the handbook at least 2–3 times per week by 25% within a year, and improving the clinical application of the content and overall satisfaction with the new handbook.

Participants

All HOs posted to the Department of Paediatrics, KKH between May 2019–April 2020 participated in the study. There were six batches of HOs during this period: May, July, September, November 2019, and January and March 2020. For all participants, it was their first time working in a paediatric department and their posting lasted for 4 months. The first three batches (May–September 2019) used the existing handbook and formed the pre-intervention group for stakeholder discussion and collection of baseline data. The last three batches (November 2019 and January–March 2020) used the new handbook and formed the post-intervention groups to test the effectiveness of the improved paediatric handbook.

Background assessment

A workgroup consisting of senior residents, chief resident and PGY1 core-faculty was tasked to lead this project. We interviewed all HOs from the May and July 2019 batches to obtain feedback on the effectiveness of the existing HO handbook at the end of their posting. They highlighted that the handbook was most effective during the first month of their posting and this was also reflected in higher usage of the handbook. This behaviour was similar to other educational resources designed to help junior doctors during on call duty.¹⁰ Therefore, the study team used the frequency of usage to assess the effectiveness of the new handbook and the target of 2–3 times a week was to match the maximum number of on call duties per week. In KKH, four HOs per day are rostered for on call duties, and therefore we aimed for an increment of 25% in the usage of the handbook – the equivalent of one more HO using the handbook while on call. The possible reasons for low usage of the HO handbook are represented by the Ishikawa diagram as shown in Figure 1.

Figure 1.

Ishikawa diagram showing possible reasons for low usage of the house officer (HO) handbook.

Survey form

The design of the paper survey was based on existing literature^11,12 and in consultation with the first two batches of HOs in May and July 2019. The survey form consisted of four sections: frequency of usage, effectiveness in clinical work, improvement matrix and open-ended comments (see Appendix 1: Survey form). The survey was administered within the first month of the HO posting and to assess the frequency of using the handbook during this period, the HOs were required to select a single response ranging from no usage to using it multiple times a day. The HOs were also asked to rate the effectiveness of the handbook in various clinical settings, such as on-call duty, and clinical tasks, such as prescription of medications. A 10-point Likert scale was used to rate its effectiveness, where 1 denotes ‘not useful at all’ and 10 denotes ‘extremely useful’. The workgroup members also developed a 3×6 matrix consisting of suggested areas for improvement, based on their interaction with the first two batches of HOs and their personal experience in the department. The HOs were allowed to circle multiple choices on the matrix that they thought would be useful to include in the handbook. The last section consisted of open-ended questions for the HOs to highlight what was useful and what could be improved. The survey was conducted at the end of their first month of posting to better reflect the effectiveness of the handbook in their transition to new paediatric HOs. We surveyed HOs from the September 2019 batch for baseline data and HOs from the last three batches for post-intervention data.

Clinical quiz

The workgroup developed a 12-question clinical quiz to assess the handbook content, which was administered at the end of their first week of posting to emphasise the application of the handbook. The questions were designed as common clinical scenarios managed by paediatric HOs and included both multiple-choice and short open-ended questions.¹³ The HOs were allowed to refer to the handbook for the quiz, and it was marked by two workgroup members using a standardised answer scheme.

PDSA cycle 1 (PDSA1)

The content of the handbook was restructured to provide updated clinical information and guidelines, inclusion of essential information required for HO daily duties, and inclusion of frequently used resources. The clinical information was structured by organ system for easy reference, except for general clinical information such as normal vital signs and calculation of maintenance fluids which were important enough items to merit their own section at the start of the handbook (Figure 2). Each section focused on approaches to symptom complexes and conditions commonly encountered in paediatrics. New sections were added and all guidelines were updated. Essential practical information was also included to aid HOs in their daily tasks such as titration of inhaled salbutamol doses and review of intravenous drip rates. This was followed by an ‘Appendix’ containing institution-specific resources such as dosages for commonly used medications, contact numbers and resuscitation algorithms. Each section was written or updated by one of the workgroup members, and proofread by a second member to ensure accuracy. The completed handbook was sent to senior medical staff in the department for a final round of proofreading and comments.

Figure 2.

Contents page of the handbook showing the information included in each section. (NGT: Naso-gastric tube, ACLS: Advanced Cardiac Life Support)

We improved the user-interface by converting the handbook into digital copy, incorporating diagrammatic representation of information, and inclusion of quick access functions. The handbook was in PDF format that could be downloaded by the HOs onto their mobile devices. Wherever possible, diagrams or workflows were included for easy reference to guide clinical management. Summary tables were also used to compile dense information such as differential diagnosis. Hyperlinks were added to the content page for quick access to the respective sections and a search function was included to facilitate quick retrieval of relevant information. This cycle was completed in time for the updated handbook to be disseminated to the HOs that joined the department in November 2019.

PDSA cycle 2 (PDSA2)

The user-interface was enhanced by improving the compatibility with various platforms, such as tablet and mobile phones, and with various operating systems, such as iOS and Android systems. The workgroup downloaded and simulated the use of the handbook on the different platforms and operating systems to troubleshoot any issues with access or function. The sharing of handbook was also improved by optimising its transfer via mobile messaging applications, unlike the previous handbook which could only be downloaded from the hospital Intranet.

To improve people and system factors, we promoted the awareness of the new HO handbook among the new HOs and existing medical staff. During the orientation programme for new HOs, they were introduced to the new handbook and encouraged to download it to their mobile phones during the session. The QI project was shared with existing medical staff to inform them of the revisions to the HO handbook and also to recommend that new HOs be allowed to refer to the handbook during clinical duties. This cycle was completed in time for the updated handbook to be disseminated to the HOs that joined the department from January 2020.

Statistical analysis

Statistical analysis was performed using SPSS (version 19.0, IBM, USA) software. Comparison of the means was performed using the two-sample t-test. A p-value of less than 0.05 was taken to be statistically significant.

Results

A total of 87 HOs participated in the QI project and their participation rate is represented in Table 1. In the pre-intervention group, there were three batches of HOs (n=50) who utilised the previous handbook. Of these, 19 were from the September 2019 batch which took part in the survey and clinical quiz for baseline information on the existing handbook. The baseline response rate in this group was 94.7% for the feedback survey and 100% for the clinical quiz. The remaining 37 HOs were from the November 2019 (n=11), January 2020 (n=20) and March 2020 (n=6) batches which used the new handbook, and together form the post-intervention group. The overall response rate in the post-intervention batch was 73% (for the feedback survey and 100% for the clinical quiz. All HOs in each cycle participated in the clinical quiz as this was included in their orientation programme. However, a few HOs from each cycle did not do the survey as they were off duty and the survey for the March 2020 HOs was disrupted by the COVID-19 pandemic.

Table 1.

Number of house officers (HOs) in each batch, and utilization of the handbook by HOs in the first month of their rotation.

HO batch	Pre-intervention			Post-intervention
HO batch	May 2019^a	July 2019^a	September 2019	November 2019(PDSA1)	January 2020(PDSA2)	March 2020^b (PDSA2)
Frequency of usage of handbook, number (%)
Multiple times per day	-	-	2 (11.1)	1 (10.0)	1 (5.9)	-
Once per day	-	-	3 (16.7)	1 (10.0)	2 (11.8)	-
Once every 2–3 days	-	-	5 (27.8)	4 (40.0)	11 (64.7)	-
Once per week	-	-	3 (16.7)	2 (20.0)	1 (5.9)	-
Once every few weeks	-	-	1 (5.6)	2 (20.0)	2 (11.8)	-
I did not refer to the handbook	-	-	4 (22.2)	0 (0)	0 (0)	-
Subtotal
At least once every 2–3 days	-	-	10 (55.6)	6 (60)	14 (82.4)	-
Total
Number participating in survey (%)	-	-	18 (94.7)	10 (90.9)	17 (85)	-
Number participating in clinical quiz (%)	-	-	19 (100)	11 (100)	17 (100)	6 (100)
Total number of HOs	23	8	19	11	20	6

PDSA: plan-do-study-act; PDSA1: PDSA cycle 1; PDSA2: PDSA cycle 2.

HOs in these batches participated in stakeholder discussions and no survey or clinical quiz was instituted.

HOs in this batch did not participate in the survey due to COVID-19 disruption.

The survey on utilization of the HO handbook revealed that 55.6% of the pre-intervention group used the existing handbook at least 2–3 times a week during the first month of their posting. This proportion of users improved modestly to 60.0% in PDSA1 and increased significantly to 82.4% in PDSA2. The increment of 26.8% in PDSA2 from the pre-intervention group also exceeded the study team’s target of 25% increment. Another improvement was noted in the proportion of HOs who did not use the handbook. Of the HOs in the pre-intervention group, 22% (4 out of 19) did not refer to the previous handbook at all, but all HOs in both PDSA1 and PDSA2 referred to the new handbook at least once. The results are summarised in Table 1.

Table 2 shows the scores obtained on the clinical quiz. There was a significant improvement in scores between the pre-intervention group and all post-intervention groups. Table 2 also summarises the effectiveness of the handbook. In the pre-intervention group, the existing handbook was perceived to be generally useful in various clinical settings and tasks, with the mean score ranging from 5.9 –7.5. It was most useful during on call duty (mean score 7.5) and when clerking new cases (mean score 7.1). The new handbook was rated higher in all areas with the mean score ranging from 7.3–8.4 in PDSA1 across different scenarios and further improved in PDSA2 to a range of 7.6–8.9. Similar to the pre-intervention group, the new handbook was most highly rated during on call duty with a mean score of 8.4 in PDSA1 and 8.9 in PDSA2. However, the new handbook was almost equally effective in both clerking of new cases and reviewing existing patients in PDSA2 with a mean score of 8.4 and 8.5 respectively. The new handbook was consistently rated higher globally when the results of both PDSA cycles were combined, and the p-values for the post-intervention group were statistically significant for use in all scenarios and in improving the overall confidence of the HOs.

Table 2.

Clinical quiz results (total score of 40) and survey on effectiveness of handbook in various clinical settings (based on a 10-point Likert scale), expressed as mean score with standard deviation in brackets.

HO batch	Pre-intervention	PDSA1 and PDSA2	PDSA1	PDSA2	p-Value^a
Clinical quiz score	26.7 (3.4)	31.1 (2.8)	29.7 (2.8)	31.7 (2.6)	<0.001
Scenario	Perceived effectiveness of the handbook on a 10-point Likert scale
Daily ward tasks	6.8 (1.6)	8.3 (1.4)	8.1 (1.0)	8.5 (1.6)	0.001
On-call tasks	7.5 (1.4)	8.7 (1.0)	8.4 (0.8)	8.9 (1.1)	0.002
Clerking of new cases	7.1 (1.6)	8.3 (1.3)	8.2 (0.9)	8.4 (1.5)	0.006
Reviews on existing patients	6.7 (1.4)	8.1 (1.6)	7.5 (2.0)	8.5 (1.2)	0.004
Prescribing medications	5.9 (2.1)	7.5 (1.6)	7.3 (1.8)	7.6 (1.6)	0.008
Overall confidence as a paediatric house officer	7.0 (1.4)	8.2 (1.5)	7.3 (1.1)	8.7 (1.5)	0.011

PDSA: plan-do-study-act.

Value of p is compared between combined PDSA1 and PDSA2 with pre-intervention group.

Discussion

Our study showed more frequent usage, better clinical applicability and improved overall satisfaction with the updated handbook. Departmental handbooks for junior doctors are common resources in many institutions, and play a vital role in helping them transition from medical students to house officers. However, there is a dearth of literature measuring their educational impact and how to improve their quality. A range of measures of improvement have been used, from user ratings on a Likert scale,^11,12 pre- and post-intervention quizzes to performance on existing knowledge-based assessments.^12,14 One study even tracked attendance at departmental conferences as a surrogate marker for efficiency,¹² on the assumption that residents would be able to attend conferences on time if the handbook helped them to complete their tasks faster.

There are four inter-related areas to consider when revising an educational resource – content, interface, systems and the user. The content needs to be updated and contain department and role-specific information. It is important that those guiding the development of the handbook have had experience working in the department and understand what the trainees need and their knowledge gaps. Recently, there is an increasing trend towards crowdsourcing to find out what the trainees themselves require and where they themselves think their information gaps lie. Some approaches that have been tried include editable wikis⁸ or free digital resources that can be made open access such that trainees, senior medical staff and nurses can request the creation of new pages or add information.¹¹ Our approach was to compile a matrix of possible additions to the handbook which the HOs could ‘vote’ on to indicate which additions would be more useful. This can be edited with new suggestions (from senior medical staff, or from the HOs’ own open ended responses). Furthermore, we updated the handbook with patient care-centric and HO role-specific information that could be easily applied to bedside management. We also adopted an electronic format to improve the user interface, with inclusion of graphic representations and a search function for easy retrieval of information.

Improvements made to the content and user interface of the handbook, as well as increasing awareness of the handbook among HOs, were successful in improving its utilization rate. However, it was evident that improvements to the content contained in the handbook alone are not enough. The increase in the proportion of HOs who used the handbook more than 2–3 times per week only surpassed 25% in PDSA2, which focused on improving the compatibility with end-users’ electronic devices, as well as promoting awareness of the handbook among HOs by emphasising it in their orientation program and at doctors’ meetings. With the increasing usage of smartphones and other personal electronic devices, even seemingly minor considerations such as optimising compatibility of the handbook with different platforms and mobile operating systems, or including hyperlinks in the text to improve searchability, can greatly improve utilization of educational resources.

Of the various clinical settings, the new handbook was most highly rated when used during on-call duty, and this agrees with the authors’ experience that being on-call in an unfamiliar department, when senior medical staff are busier and less available for consultation, would be the main reason for use of such a handbook. Junior doctors are also more susceptible to psychological ill-health and on-call duties have been highlighted as highly stressful events.¹⁵ In 2006, the Royal College of Physicians found it necessary to set up a working group to come up with a practical guide for junior doctors to prepare, survive and recover from working night shifts.¹⁶ We believe that a junior doctors’ handbook that is crafted with the aim of helping them provide safe and effective patient care, especially during on-call duties, will go a long way towards alleviating the stress associated with being on call. This will be vital to combat burnout among the junior doctors.

There were some limitations to our study. Firstly, there was possible recall bias given the nature of the study as the participants were surveyed on their usage of the handbook during the first month of their posting. However, we found it necessary to average the use of the handbook over a month to account for factors such as decreased patient numbers, being on leave or on call which might affect the usage of the handbook. Secondly, the survey for March 2020 HOs was disrupted by the sudden impact of the COVID-19 pandemic. The study team planned to use online surveys in future due to this disruption.

Conclusions

Updating and improving a handbook for paediatric HOs is associated with increased utilization rate of this handbook, as well as concomitant improvement in their clinical knowledge and their confidence level when dealing with clinical scenarios, especially on call. Despite being common resources in many institutions, the effectiveness of departmental handbooks is not well studied and our study shows that they are still useful and relevant in the present day, when a wide range of clinical information is freely available online. Further research can focus on whether different groups of junior doctors (for example older trainees, graduate versus undergraduate students, or those at different levels of training) would benefit from different modalities of instruction or different types of educational resources.

Footnotes

Appendix 1: Survey form

Dear HO,

Please take a moment to fill up this survey! It will help us decide what areas of the HO handbook to improve so that future batches can benefit.

Please circle your answer (one answer only):

________________________________________________________________________________________________________

Thank you for your time!

(Abbreviations: HO: house officer; VF: ventricular fibrillation; VT: ventricular tachycardia; PEA: pulseless electrical activity; CTSP: “called to see patient”, i.e. clinical review of admitted patients; GE: gastroenteritis; NNP: neonatal pyrexia; IT: information technology.)

Acknowledgements

The authors would like to thank Drs Ng Yong Hong, Oh Jean Yin, Cristelle Chow, Chia Shiyun, Koh Ai Ling, Samantha Lee, Joyce Tan and Celeste Yap for developing the 2015 department handbook for HOs, as well as A/Prof Chong Chia Yin, Mr William Yap and Mr John Wong for their insights on the QI process.

Authors’ contributions

WLYJ designed and edited the departmental handbook, designed and administered the data collection instrument, collected study data, performed the statistical analyses, and drafted the initial manuscript. TCY designed, compiled and edited the departmental handbook, designed and administered the data collection instrument, and critically reviewed the manuscript. CDT conceptualised the study, designed the departmental handbook, revised and administered the data collection instrument, and critically reviewed the manuscript. KSY and KZL designed the departmental handbook, revised and administered the data collection instrument, and critically reviewed the manuscript. LBKG conceptualised the study, edited and vetted the departmental handbook, revised the data collection instrument, suggested additional analyses, and critically reviewed the manuscript. All authors approved the final manuscript as submitted.

Availability of data and materials

The datasets generated and analysed during the current study are available from the corresponding author.

Conflict of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This study was exempted from review by the Singhealth Centralised Institutional Review Board (CIRB Ref. No: 2019/2942) as it is a quality improvement project.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

Informed consent

This study was exempted from consent-taking by the Singhealth Centralised Institutional Review Board (CIRB Ref. No: 2019/2942) as it is a quality improvement project.

ORCID iD

Justin Liang Yi Wee

References

Wall

Bolshaw

Carolan

From undergraduate medical education to pre-registration house officer year: How prepared are students?

Med Teach 2006; 28: 435–439.

Bligh

The first year of doctoring: Still a survival exercise. Med Educ 2002; 36: 2–3.

MOH Holdings, Singapore. Medical service career path, https://www.physician.mohh.com.sg/medicine/medical-service-career-path (2018, accessed 6 June 2020).

Kliegman

Stanton

Schor

, et al. Nelson textbook of pediatrics. 19th ed. Philadelphia: Saunders, 2011.

Soh

Jamuar

Puthucheary

The baby bear book: A practical guide on paediatrics. 3rd ed. Singapore: SingHealth Academy, 2015.

McDonagh

Clarke

Veale

Write a departmental handbook for junior staff. BMJ 1993; 307: 553–555.

Lobo

Crandley

Rumph

, et al. Pilot study of iPad incorporation into graduate medical education. J Grad Med Educ 2013; 5: 142–144.

Kohli

Bradshaw

JK.

What is a wiki, and how can it be used in resident education?

J Digit Imaging 2011; 24: 170–175.

Holtkamp

Eye

Theeler

BJ.

Implementation of the neurology resident handbook mobile application improves patient care. Mil Med 2017; 182: 1581.

10.

Hafiz

Mcmanus

‘Asked to see patient’ (ATSP)–improving patient safety and confidence in junior doctors. US China Educ Rev A 2015; 5: 277–285.

11.

Sutzko

Goldman

Boniakowski

, et al. Development, launch, and evaluation of an open-access vascular surgery handbook through house officer curriculum crowdsourcing. Ann Vasc Surg 2019; 58: 309–316.

12.

Ortiz

Ottolini

Agrawal

Written and online residency guidebook to improve resident efficiency and knowledge of best patient care practices. MedEdPORTAL 2016; 12: 10424.

13.

Epstein

RM.

Assessment in medical education. N Engl J Med 2007; 356: 387–396.

14.

Gibson

Nguyen

Davidson

, et al. Effectiveness of an educational handbook in improving psychiatry resident knowledge of second-generation antipsychotics. Acad Psychiatry 2015; 39: 154–159.

15.

Lam

Wong

, et al. Psychological well-being of interns in Hong Kong: What causes them stress and what helps them. Med Teach 2010; 32: e120–e126.

16.

Horrocks

Pounder

Group

RW.

Working the night shift: Preparation, survival and recovery–a guide for junior doctors. Clin Med 2006; 6:61–67.

Improving the effectiveness of a paediatric handbook for new house officers