Assessing the Effectiveness of Observation Training for Clinical Instructors

Abstract

INTRODUCTION: Many respiratory care programs rely heavily on observation instruments to assess student performance in the clinical setting. In addition, the practice of using volunteer clinical instructors (CI) to assess student performance is commonplace. The combination of these variables raises questions about the re-liability of this method of multiobserver assessment. The purpose of this study was to determine whether a training program using videotapes prepares CI to ac-curately record clinical performance on checklist instruments as measured by an interobserver agreement ≥ 0.85. METHODS & SUBJECTS: In a randomized con-trol group posttest-only design, CI were assigned to a control group (n = 17) or an experimental group (n = 17). The CI in the experimental group attended a session during which they viewed segments of videotapes showing tracheal aspiration (TA) and chest physiotherapy (CPT) performed according to commonly accepted stan-dards of care. The videotapes had been developed by the American Association for Respiratory Care (AARC) for use by supervisory personnel responsible for evalu-ating clinical performance. Experimental group CI reviewed checklist instruments and procedural specifications for TA and CPT that corresponded to the video-tapes. The dichotomously scored checklist instruments identified tasks essential to the correct performance of TA and CPT. The instruments were developed by the investigators and pilot tested by an expert panel to establish interrater reliability and face and content validity. One week later, both groups attended a posttest ses-sion during which segments of the TA and CPT videotapes with incorrectly per-formed or omitted tasks were shown. The CI completed the checklist instruments as they viewed the videotapes. RESULTS: Data were statistically analyzed using one-tailed t tests for independent samples (a = 0.05) and chi-square tests (α = 0.05). Mean interobserver agreement was greater in the experimental group than in the control group for both TA (0.885 vs 0.672) and CPT (0.926 vs 0.680), and the dif-ference was significant for both procedures (p < 0.001 for t tests and p < 0.00001 for chi-square). Calculation of effect size demonstrated practical significance. CONCLUSION: The experimental group was more accurate than the control group in using the checklist instruments to record clinical performance shown in the videotapes. Educators may consider developing similarly designed training programs for preparing CI to use checklist instruments to evaluate clinical per-formance.