Sage Journals: Discover world-class research

Abstract

We previously developed a workplace-based tool for assessing point of care ultrasound (POCUS) skills and used a modified Delphi technique to identify critical items (those that learners must successfully complete to be considered competent). We performed a standard setting procedure to determine cut scores for the full tool and a focused critical item tool. This study compared ratings by 24 experts on the two checklists versus a global entrustability rating. All experts assessed three videos showing an actor performing a POCUS exam on a patient. The performances were designed to show a range of competences and one included potentially critical errors. Interrater reliability for the critical item tool was higher than for the full tool (intraclass correlation coefficient = 0.84 [95% confidence interval [CI] 0.42–0.99] vs. 0.78 [95% CI 0.25–0.99]). Agreement with global ratings of competence was higher for the critical item tool (κ = 0.71 [95% CI 0.55–0.88] vs 0.48 [95% CI 0.30–0.67]). Although sensitivity was higher for the full tool (85.4% [95% CI 72.2–93.9%] vs. 81.3% [95% CI 67.5–91.1%]), specificity was higher for the critical item tool (70.8% [95% CI 48.9–87.4%] vs. 29.2% [95% CI 12.6–51.1%]). We recommend the use of critical item checklists for the assessment of POCUS competence.

Keywords

assessment standard setting internal medicine work-based point-of-care ultrasound

Get full access to this article

View all access options for this article.

References

AIUM Practice Parameter for the Performance of Point-of-Care Ultrasound Examinations. (2019). Journal of Ultrasound in Medicine, 38(4), 833–849. https://doi.org/10.1002/jum.14972

American College of Physicians Statement in Support of Point-of-Care Ultrasound in Internal Medicine. (2018). American College of Physicians Online. https://www.acponline.org/meetings-courses/focused-topics/point-of-care-ultrasound/acp-statement-in-support-of-point-of-care-ultrasound-in-internal-medicine

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Angoff

. (1971). Scales, norms, and equivalent scores (2nd ed.). American Council on Education.

Blanco

Volpicelli

(2016). Common pitfalls in point-of-care ultrasound: A practical guide for emergency and critical care physicians. Critical Ultrasound Journal, 8(1), 15. https://doi.org/10.1186/s13089-016-0052-x

Cook

D. A.

Brydges

Ginsburg

Hatala

(2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49(6), 560–575. https://doi.org/10.1111/medu.12678

Cunnington

J. P. W.

Neville

A. J.

Norman

G. R.

(1996). The risks of thoroughness: Reliability and validity of global ratings and checklists in an OSCE. Advances in Health Sciences Education, 1(3), 227–233. https://doi.org/10.1007/BF00162920

Daniels

V. J.

Bordage

Gierl

M. J.

Yudkowsky

(2014). Effect of clinically discriminating, evidence-based checklist itemson the reliability of scores from an internal medicine residency OSCE. Advances in Health Sciences Education: Theory and Practice, 19(4), 497–506. https://doi.org/10.1007/s10459-013-9482-4

Downing

S. M.

Tekian

Yudkowsky

(2006). Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teaching and Learning in Medicine, 18(1), 50–57. https://doi.org/10.1207/s15328015tlm1801_11

10.

Dreyfus

S. E.

(2004). The five-stage model of adult skill acquisition. Bulletin of Science, Technology & Society, 24(3), 177–181. https://doi.org/10.1177/0270467604264992

11.

Ericsson

K. A.

(2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79(10 Suppl), S70–81. https://doi.org/10.1097/00001888-200410001-00022

12.

Expert Round Table on Ultrasound in ICU. (2011). International expert statement on training standards for critical care ultrasonography. Intensive Care Medicine, 37(7), 1077–1083. https://doi.org/10.1007/s00134-011-2246-9

13.

Faraggi

Reiser

(2002). Estimation of the area under the ROC curve. Statistics in Medicine, 21(20), 3093–3106. https://doi.org/10.1002/sim.1228

14.

Gofton

W. T.

Dudek

N. L.

Wood

T. J.

Balaa

Hamstra

S. J.

(2012). The Ottawa surgical competency operating room evaluation (O-SCORE): A tool to assess surgical competence. Academic Medicine, 87(10), 1401–1407. https://doi.org/10.1097/ACM.0b013e3182677805

15.

Hatala

Ginsburg

Hauer

K. E.

Gingerich

(2019). Entrustment ratings in internal medicine training: Capturing meaningful supervision decisions or just another rating? Journal of General Internal Medicine, 34(5), 740–743. https://doi.org/10.1007/s11606-019-04878-y

16.

Hodges

McIlroy

J. H.

(2003). Analytic global OSCE ratings are sensitive to level of training. Medical Education, 37(11), 1012–1016. https://doi.org/10.1046/j.1365-2923.2003.01674.x

17.

Hodges

Regehr

McNaughton

Tiberius

Hanson

(1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine, 74(10), 1129–1134. https://doi.org/10.1097/00001888-199910000-00017

18.

Ilgen

J. S.

I. W. Y.

Hatala

Cook

D. A.

(2015). A systematic review of validity evidence for checklists vs global rating scales in simulation-based assessment. Medical Education, 49(2), 161–173. https://doi.org/10.1111/medu.12621

19.

Kane

M. T.

(1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527.

20.

Lammers

R. L.

Davenport

Korley

Griswold-Theodorson

Fitch

M. T.

Narang

A. T.

Evans

L. V.

Gross

Rodriguez

Dodge

Hamann

C. J.

Robey Iii

W. C.

(2008). Teaching and assessing procedural skills using simulation: Metrics and methodology. Academic Emergency Medicine, 15(11), 1079–1087. https://doi.org/10.1111/j.1553-2712.2008.00233.x

21.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

22.

Lewiss

R. E.

(2018). “The Ultrasound Looked Fine”: Point-of-care ultrasound and patient safety. Patient Safety Network. https://psnet.ahrq.gov/web-mm/ultrasound-looked-fine-point-care-ultrasound-and-patient-safety

23.

Liu

R. B.

Donroe

J. H.

McNamara

R. L.

Forman

H. P.

Moore

C. L.

(2017). The practice and implications of finding fluid during point-of-care ultrasonography: A review. JAMA Internal Medicine, 177(12), 1818–1825. https://doi.org/10.1001/jamainternmed.2017.5048

24.

LoPresti

C. M.

Jensen

T. P.

Dversdal

R. K.

Astiz

D. J.

(2019). Point of care ultrasound for internal medicine residency training: A position statement from the alliance of academic internal medicine. The American Journal of Medicine, 132(11), 1356–1360. https://doi.org/10.1016/j.amjmed.2019.07.019

25.

I. W. Y.

Desy

Woo

M. Y.

Kirkpatrick

A. W.

Noble

V. E.

(2020). Consensus-based expert development of critical items for direct observation of point-of-care ultrasound skills. Journal of Graduate Medical Education, 12(2), 176–184. https://doi.org/10.4300/JGME-D-19-00531.1

26.

I. W. Y.

Pugh

Mema

Brindle

M. E.

Cooke

Stromer

J. N.

(2015). Use of an error-focused checklist to identify incompetence in lumbar puncture performances. Medical Education, 49(10), 1004–1015. https://doi.org/10.1111/medu.12809

27.

McGaghie

W. C.

Issenberg

S. B.

Barsuk

J. H.

Wayne

D. B.

(2014). A critical review of simulation-based mastery learning with translational outcomes. Medical Education, 48(4), 375–385. https://doi.org/10.1111/medu.12391

28.

McGaghie

W. C.

Issenberg

S. B.

Petrusa

E. R.

Scalese

R. J.

(2010). A critical review of simulation-based medical education research: 2003–2009. Medical Education, 44(1), 50–63. https://doi.org/10.1111/j.1365-2923.2009.03547.x

29.

Meineri

Bryson

G. L.

Arellano

Skubas

(2018). Core point-of-care ultrasound curriculum: What does every anesthesiologist need to know? Canadian Journal of Anesthesia/Journal canadien d’anesthésie, 65(4), 417–426. https://doi.org/10.1007/s12630-018-1063-9

30.

Moore

C. L.

Copel

J. A.

(2011). Point-of-care ultrasonography. New England Journal of Medicine, 364(8), 749–757. https://doi.org/10.1056/NEJMra0909487

31.

Payne

N. J.

Bradley

E. B.

Heald

E. B.

Maughan

K. L.

Michaelsen

V. E.

Wang

X. Q.

Corbett

E. C.

Jr . (2008). Sharpening the eye of the OSCE with critical action analysis. Academic Medicine, 83(10), 900–905. https://doi.org/10.1097/ACM.0b013e3181850990

32.

Regehr

MacRae

Reznick

R. K.

Szalay

(1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine, 73(9), 993–997. https://doi.org/10.1097/00001888-199809000-00020

33.

Soni

N. J.

Schnobrich

Matthews

B. K.

Tierny

D. M.

Jensen

T. P.

Dancel

Cho

Dversdal

R. K.

Mints

Bhagra

Reierson

Kurian

L. M.

Liu

G. Y.

Candotti

Boesch

LoPresti

C. M.

Lenchus

Wong

Johnson

… Lucas

B. P

. (2019). Point-of-care ultrasound for hospitalists: A position statement of the society of hospital medicine. Journal of Hospital Medicine, 14, E1–E6. https://doi.org/10.12788/jhm.3079

34.

Ultrasound Guidelines: Emergency, Point-of-Care and Clinical Ultrasound Guidelines in Medicine. (2017). Annals of Emergency Medicine, 69(5), e27–e54. https://doi.org/10.1016/j.annemergmed.2016.08.457

35.

Van Der Vleuten

C. P. M.

(1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1(1), 41–67. https://doi.org/10.1007/bf00596229

36.

Van Der Vleuten

C. P. M.

Schuwirth

L. W. T.

(2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317. https://doi.org/10.1111/j.1365-2929.2005.02094.x

37.

Walters

S. J.

(2009). Consultants’ forum: Should post hoc sample size calculations be done? Pharmaceutical Statistics, 8(2), 163–169. https://doi.org/10.1002/pst.334

38.

Walzak

Bacchus

Schaefer

Zarnke

Glow

Brass

McLaughlin

I. W. Y.

(2015). Diagnosing technical competence in six bedside procedures: Comparing checklists and a global rating scale in the assessment of resident performance. Academic Medicine, 90(8), 1100–1108. https://doi.org/10.1097/ACM.0000000000000704

39.

Wood

T. J.

Pugh

(2020). Are rating scales really better than checklists for measuring increasing levels of expertise? Medical Teacher, 42(1), 46–51. https://doi.org/10.1080/0142159X.2019.1652260

40.

Yudkowsky

Otaki

Lowenstein

Riddle

Nishigori

Bordage

(2009). A Hypothesis-driven physical examination learning and assessment procedure for medical students: Initial validity evidence. Medical Education, 43(8), 729–740. https://doi.org/10.1111/j.1365-2923.2009.03379.x

41.

Yudkowsky

Park

Y. S

Downing

(2020). Assessment in health professions education (2nd ed.). Routledge.

42.

Yudkowsky

Park

Y. S.

Riddle

Palladino

Bordage

(2014). Clinically discriminating checklists versus thoroughness checklists: Improving the validity of performance test scores. Academic Medicine, 89(7), 1057–1062. https://doi.org/10.1097/acm.0000000000000235

43.

Yudkowsky

Tumuluru

Casey

Herlich

Ledonne

(2014). A Patient safety approach to setting pass/fail standards for basic procedural skills checklists. Simulation in Healthcare, 9(5), 277–282. https://doi.org/10.1097/SIH.0000000000000044

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

Use of Critical Items in Determining Point-of-Care Ultrasound Competence

Abstract

Keywords

Get full access to this article

References

Supplementary Material