Assessing the Quality of Vascular Ultrasound Examinations Using the IAC QI Self-Assessment Tool: Inter-Rater and Intra-Rater Agreement Between Vascular Technologists and Interpreting Physicians

Abstract

Introduction:

Vascular laboratories need to improve efficiency while maintaining examination quality. Vascular ultrasound examinations were retrospectively evaluated using the Intersocietal Accreditation Commission (IAC) Quality Improvement (QI) Self-Assessment Tool to evaluate the quality of the examinations and to determine inter-rater and intra-rater agreement between groups of vascular technologists and interpreting physicians.

Methods:

Vascular technologists and attending interpreting physicians evaluated 100 carotid Doppler and 100 lower extremity venous Doppler examinations utilizing the IAC QI Self-Assessment Tool. Gwet’s First-Order Agreement Coefficient (AC₁) statistic and 95% confidence intervals were calculated to determine both inter-rater and intra-rater agreement.

Results:

The inter-rater agreement for 91% of all questions ranged from substantial (AC₁ = 0.6) to almost perfect (AC₁ = 1) for both examination types across all reviewers. Agreement on 2 questions was suboptimal: specifically, the technical quality section for the vascular technologists showed slight agreement for carotid Doppler and venous Doppler examinations (AC₁ = 0.108 and 0.087, respectively), and the case review section for the interpreting physicians showed poor agreement for carotid Doppler examinations (AC₁ = –0.087) and fair agreement for venous Doppler examinations (AC₁ = 0.329). The intra-rater agreement among the interpreting physicians for both examinations was in the almost perfect range (Carotid AC₁ = 0.875-1; Venous AC₁ = 0.871-1). The intra-rater agreement among the vascular technologists for both examinations ranged from slight to almost perfect (Carotid AC₁ = 0.191-1; Venous AC₁ = 0.155-1).

Conclusion:

Substantial to almost perfect agreement among all reviewers was observed for most of the quality review questions. The IAC QI Self-Assessment Tool aided in identifying areas for improvement in vascular studies and the comments captured in the technical quality section of the tool were instrumental in establishing areas for quality improvement.