Multicenter Interrater Reliability in the Endoscopic Assessment of Velopharyngeal Function Using a Video Instruction Tool

Abstract

Objective

Assess interrater agreement of endoscopic assessment of velopharyngeal (VP) function before and after viewing the video instruction tool (VIT). We hypothesized improvement in interrater agreement using the Golding-Kushner scale (GKS) after viewing the VIT.

Study Design

Prospective study.

Setting

Multi-institutional.

Methods

Sixteen fellowship-trained pediatric otolaryngologists who treat velopharyngeal insufficiency (VPI) rated 50 video segments using the GKS before and after watching the VIT. Raters assessed gap size percentage and lateral pharyngeal wall (LPW), soft palate (SP), and posterior pharyngeal wall (PPW) movement. Intraclass correlation coefficient was estimated for these continuous measures. Raters also indicated the presence of a palatal notch, Passavant’s ridge, and aberrant pulsations (categorical variables). Fleiss κ coefficient was used for categorical variables. Wilcoxon signed-rank test was performed on the difference between the pre/post individual video ratings.

Results

Reliability improved for all continuous variables after watching the instructional video. The improvement was significant for PPW (0.22-0.30, P < .001), SP (left: 0.63-0.68, P < .001 and right: 0.64-0.68, P = .001), and LPW (left: 0.49-0.54, P = .01 and right: 0.49-0.54, P = .09) but not significant for gap size (0.65-0.69, P = .36). Among categorical variables, agreement on Passavant’s ridge significantly improved (0.30-0.36, P = .03).

Conclusion

Exposure to a video instruction tool improves interrater agreement of endoscopic assessment of VP function. Significant improvement was observed in our primary end points, specifically posterior pharyngeal wall movement, soft palate movement, and lateral pharyngeal wall movement. There was less impact of the VIT on the interrater agreement of the categorical variables, palatal notch, Passavant’s ridge, and aberrant pulsations.

Keywords

velopharyngeal insufficiency VPI interrater reliability

Get full access to this article

View all access options for this article.

References

Eliasziw

Young

Woodbury

Fryday-Field

. Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther. 1994;74:777-788.

Sie

Starr

Bloom

et al . Multicenter interrater and intrarater reliability in the endoscopic evaluation of velopharyngeal insufficiency. Arch Otolaryngol Head Neck Surg. 2008;134:757-763.

Golding-Kushner

Argamaso

Cotton

et al . Standardization for the reporting of nasopharyngoscopy and multiview videofluoroscopy: a report from an International Working Group. Cleft Palate J. 1990;27:337-347.

Yoon

Starr

Perkins

Bloom

Sie

. Interrater and intrarater reliability in the evaluation of velopharyngeal insufficiency within a single institution. Arch Otolaryngol Head Neck Surg. 2006;132:947-951.

Tieu

Gerber

Milczuk

et al . Generation of consensus in the application of a rating scale to nasendoscopic assessment of velopharyngeal function. Arch Otolaryngol Head Neck Surg. 2012;138:923-928.

Donner

Eliasziw

. Sample size requirements for reliability studies. Stat Med. 1987;6:441-448.

Walter

Eliasziw

Donner

. Sample size and optimal designs for reliability studies. Stat Med. 1998;17:101-110.

Harris

Taylor

Thielke

Payne

Gonzalez

Conde

. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377-381.