Diagnostic Radiology Residency Assessment Tools: A Scoping Review

Abstract

French

Purpose:

The multifaceted nature of learning in diagnostic radiology residency requires a variety of assessment methods. However, the scope and quality of assessment tools has not been formally examined. A scoping review was performed to identify assessment tools available for radiology resident training and to evaluate the validity of these tools.

Methods:

A literature search was conducted through multiple databases and on-line resources. Inclusion criteria were defined as any tool used in assessment of radiology resident competence. Data regarding residents, evaluators and specifics of each tool was extracted. Each tool was subjected through a validation process with a customized rating scale using the 5 categories of validity: content, response process, internal structure, relations to other variables, and consequences.

Results:

The initial search returned 447 articles; 35 were included. The most evaluated competency being overall knowledge (31%), most common published journal was Academic Radiology (24%); evaluations were most commonly set in the United States (57%). In terms of validation, we found low adherence to modern integrated validity, with 34% of studies including a definition of validity. When specifically examining the 5 domains of validation evidence presented, most were either absent or of low rigor (70%). Only one study presented a modern definition of validation (3%, 1/35).

Conclusion:

We identified 35 evaluation tools covering a variety of competency areas. However, few of these tools have been validated. Development of new validated assessment tools or validation of existing tools is essential for the ongoing transition to a competency-based curriculum.

Keywords

medical education radiology resident competency-based assessment radiology resident education validity

Get full access to this article

View all access options for this article.

References

Probyn

Lang

Tomlinson

Bandiera

. Multisource feedback and self-assessment of the communicator, collaborator, and professional CanMEDS roles for diagnostic radiology residents. Can Assoc Radiol J. 2014;65(4):379–384.

Jippes

van Engelen

Brand

Oudkerk

. Competency-based (CanMEDS) residency training programme in radiology: systematic design procedure, curriculum and success factors. Eur Radiol. 2010;20(4):967–977.

Harris

Bhanji

Topps

, et al. Evolving concepts of assessment in a competency-based world. Med Teach. 2017;39(6):603–608.

Frank

Snell

Cate

, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–645.

Kogan

Holmboe

. Realizing the promise and importance of performance-based assessment. Teach Learn Med. 2013;25(Suppl 1):S68–S74.

Canada RCoPaSo. Competence by design. Canada’s model for competency-based medical education. 2020. Accessed October 5, 2020. https://www.royalcollege.ca/rcsite/cbd/competence-by-design-cbd-e

Canada RCoPaSo. EPAs and CanMEDS Milestones. 2020. Accessed October 5, 2020. https://www.royalcollege.ca/rcsite/cbd/implementation/cbd-milestones-epas-e

Kwan

BYM

Mbanwi

Cofie

, et al. Creating a competency-based medical education curriculum for Canadian diagnostic radiology residency (Queen’s Fundamental Innovations in Residency Education)-Part 1: Transition to Discipline and Foundation of Discipline Stages. Can Assoc Radiol J. 2020:846537119894723.

McGaghie

Miller

Sajid

Telder

. Competency-based curriculum development on medical education: an introduction. Public Health Pap. 1978;(68):11–91.

10.

Powell

Carraccio

. Toward competency-based medical education. N Engl J Med. 2018;378(1):3–5.

11.

Schmitt

Scanlon

Servaes

Levin

Cook

. Milestones on a shoestring: a cost-effective, semi-automated implementation of the new ACGME requirements for radiology. Acad Radiol. 2015;22(10):1287–1293.

12.

Vydareny

Amis

Becker

, et al. Diagnostic radiology milestones. J Grad Med Educ. 2013;5(1 Suppl 1):74–78.

13.

Nasca

Philibert

Brigham

Flynn

. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051–1056.

14.

Frank

Danoff

. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29(7):642–647.

15.

ten Cate

Scheele

. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 2007;82(6):542–547.

16.

Collins

Hyde

Gray

, et al. Radiology resident evaluation: a form that addresses the six competencies of the accreditation council for graduate medical education. Acad Radiol. 2002;9(7):815–816.

17.

Collins

de Christenson

Gray

, et al. General competencies in radiology residency training: definitions, skills, education and assessment. Acad Radiol. 2002;9(6):721–726.

18.

Isupov

McInnes

Hamstra

, et al. Development of RAD-Score: a tool to assess the procedural competence of diagnostic radiology residents. AJR Am J Roentgenol. 2017;208(4):820–826.

19.

Wood

Collins

Burnside

, et al. Patient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills. Acad Radiol. 2004;11(8):931–939.

20.

Collins

Herring

Kwakwa

, et al. Current practices in evaluating radiology residents, faculty, and programs: results of a survey of radiology residency program directors. Acad Radiol. 2004;11(7):787–794.

21.

Epstein

. Assessment in medical education. N Engl J Med. 2007;356(4):387–396.

22.

Bari

. Direct observation of procedural skills in radiology. AJR Am J Roentgenol. 2010;195(1):W14–W18.

23.

Holmboe

Sherbino

Long

Swing

Frank

. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676–682.

24.

Harris

Nousiainen

Reznick

. Competency-based resident education—The Canadian perspective. Surgery. 2020;167(4):681–684.

25.

Norcini

Anderson

Bollela

, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33(3):206–214.

26.

Beckman

Cook

Mandrekar

. What is the validity evidence for assessments of clinical teaching? J Gen Intern Med. 2005;20(12):1159–1164.

27.

LeClaire

Nihira

Hardré

. Validity: applying current concepts and standards to gynecologic surgery performance assessments. Adv Health Sci Educ Theory Pract. 2015;20(3):817–828.

28.

Chuan

Wan

Royse

Forrest

. Competency-based assessment tools for regional anaesthesia: a narrative review. Br J Anaesth. 2018;120(2):264–273.

29.

Kogan

Holmboe

Hauer

. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–1326.

30.

American Psychological Association, American Educational Research Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing, Vol. viii. American Psychological Association; 1985:100.

31.

American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational and Psychological Testing (U.S.). Standards for Educational and Psychological Testing, Vol. ix. American Educational Research Association; 2014:194.

32.

Samuel

. Validity. In: Linn

, ed. Educational Measurement. The American Council on Education/Macmillan Series on Higher Education. 3rd ed. American Council on Education; 1989:13–103.

33.

Kane

. Validation. In: Brennan

, ed. Educational Measurement. 4th ed. Praeger Publishers; 2006:17–64.

34.

Kane

. Current concerns in validity theory. J Educ Meas. 2001;38(4):319–342.

35.

Tricco

Lillie

Zarin

, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–473.

36.

37.

Cook

Beckman

. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2):166.e7–16.

38.

Andreatta

Marzano

Curran

. Validity: what does it mean for competency-based assessment in obstetrics and gynecology? Am J Obstet Gynecol. 2011;204(5):384.e1–6.

39.

Wallis

Edey

Prothero

McCoubrie

. The Bristol Radiology Report Assessment Tool (BRRAT): developing a workplace-based assessment tool for radiology reporting skills. Clin Radiol. 2013;68(11):1146–1154.

40.

Curtis

Amis

Cruess

Riordan

. Evaluation of radiology resident cognitive performance. Invest Radiol. 1985;20(6):640–642.

41.

Holmboe

. The Milestones Guidebook. Accreditation Council for Graduate Medical Education. 2020. Accessed December 10, 2020. http://www.acgme.org/Portals/0/MilestonesGuidebook.pdf

42.

CanMEDS 2015 Physician Competency Framework. Royal College of Physicians and Surgeons of Canada; 2015.

43.

Downing

. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–837.

44.

Beckman

Ghosh

Cook

Erwin

Mandrekar

. How reliable are assessments of clinical teaching? A review of the published instruments. J Gen Intern Med. 2004;19(9):971–977.

45.

Messick

. Standards of validity and the validity of standards in performance assessment. Edu Meas: Issues Prac. 1995;14(4):5–8.

46.

Messick

. Meaning and values in test validation: the science and ethics of assessment. Edu Res. 1989;18(2):5–11.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB