To investigate the validity and reliability of multiple listener judgments of hypernasality and audible nasal emission, in children with repaired cleft palate, using visual analog scaling (VAS) and equal-appearing interval (EAI) scaling.
Design
Prospective comparative study of multiple listener ratings of hypernasality and audible nasal emission.
Setting
Multisite institutional.
Participants
Five trained and experienced speech-language pathologist listeners from the Americleft Speech Project.
Measures
Average VAS and EAI ratings of hypernasality and audible nasal emission/turbulence for 12 video-recorded speech samples from the Americleft Speech Project. Intrarater and interrater reliability was computed, as well as linear and polynomial models of best fit.
Results
Intrarater and interrater reliability was acceptable for both rating methods; however, reliability was higher for VAS as compared to EAI ratings. When VAS ratings were plotted against EAI ratings, results revealed a stronger curvilinear relationship.
Conclusions
The results of this study provide additional evidence that alternate rating methods such as VAS may offer improved validity and reliability over EAI ratings of speech. VAS should be considered a viable method for rating hypernasality and nasal emission in speech in children with repaired cleft palate.
BaylisA.L.MunsonB.MollerK.T.Audible nasal emission in speakers with cleft palate: a comparative study of listener judgments. Cleft Palate Craniofac J.2011; 48: 399–411.
2.
BrancampT.U.LewisK.E.WattersonT.The relationship between nasalance scores and nasality ratings obtained with equal appearing interval and direct magnitude estimation scaling methods. Cleft Palate Craniofac J.2010; 47: 631–637.
3.
ChapmanK.L.BaylisA.Trost-CardamoneJ.Nett CorderoK.DixonA.DobbelsteynC.ThurmesA.WilsonK.Harding-BellA.SweeneyT.. The Americleft Speech Project: A training and reliability study. Cleft Palate Craniofac J. In press.
4.
ChengD.T.H.Direct Magnitude Estimation Versus Visual Analogue Scaling in the Perceptual Rating of Hypernasality.Hong Kong: Division of Speech and Hearing Sciences, University of Hong Kong;2006. Undergraduate dissertation.
5.
CicchettiD.V.Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess.1994; 6: 284–290.
6.
ConleyS.F.GosainA.K.MarksS.M.LarsonD.L.Identification and assessment of velopharyngeal inadequacy. Am J Otolaryngol.1997; 18: 38–46.
7.
EadieT.L.DoyleP.C.Direct magnitude estimation and interval scaling of naturalness and severity in tracheoesophageal (TE) speakers. J Speech Lang Hear Res.2002; 45: 1088–1096.
GescheiderG.A.Psychophysics, Method, and Theory.Hillsdale, NJ: Lawrence Erlbaum;1976.
10.
GrantS.AitchisonT.HendersonE.ChristieJ.ZareS.McMurrayJ.DargieH.A comparison of the reproducibility and the sensitivity to change of visual analogue scales, Borg scales, and Likert scales in normal subjects during submaximal exercise. Chest.1999; 116: 1208–1217.
11.
HallgrenK.A.Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol.2012; 8: 23–34.
12.
JamiesonS.Likert scales: how to (ab)use them. Med Educ.2004; 38: 1217–1218.
13.
JohnA.SellD.SweeneyT.Harding-BellA.WilliamsA.The cleft audit protocol for speech-augmented: a validated and reliable measure for auditing cleft speech. Cleft Palate Craniofac J.2006; 43: 272–288.
14.
KelchnerL.N.BrehmS.B.WeinrichB.MiddendorfJ.deAlarconA.LevinL.ElluruR.Perceptual evaluation of severe pediatric voice disorders: rater reliability using the consensus auditory perceptual evaluation of voice. J Voice.2010; 24: 441–449.
15.
KuehnD.P.MollerK.T.Speech and language issues in the cleft palate population: the state of the art. Cleft Palate Craniofac J.2000; 37: 348–383.
16.
KummerA.W.BriggsM.LeeL.The relationship between the characteristics of speech and velopharyngeal gap size. Cleft Palate Craniofac J.2003; 40: 590–596.
17.
KummerA.W.CurtisC.WiggsM.LeeL.StrifeJ.L.Comparison of velopharyngeal gap size in patients with hypernasality, hypernasality and nasal emission, or nasal turbulence (rustle) as the primary speech characteristic. Cleft Palate Craniofac J.1992; 29: 152–156.
18.
LandisJ.R.KochD.D.The measurement of observer agreement for categorical data. Biometrics.1977; 33: 159–174.
19.
LohmanderA.OlssonM.Methodology for perceptual assessment of speech in patients with cleft palate: a critical review of the literature. Cleft Palate Craniofac J.2004; 41: 64–70.
20.
LohmanderA.WilladsenE.PerssonC.HenningssonG.BowdenM.HuttersB.Methodology for speech assessment in the Scandcleft project—an international randomized clinical trial on palatal surgery: experiences from a pilot study. Cleft Palate Craniofac J.2009; 46: 347–362.
21.
MetzD.E.SchiavettiN.SaccoP.R.Acoustic and psychophysical dimensions of the perceived speech naturalness of nonstutterers and posttreatment stutterers. J Speech Hear Disord.1990; 55: 516–525.
22.
MollerK.T.An approach to evaluation of velopharyngeal adequacy for speech. Clin Commun Disord.1991; 1: 61–75.
23.
NormanG.Likert scales, levels of measurement and the “laws” of statistics. Adv Health Sci Educ Theory Pract.2010; 15: 625–632.
24.
NormanG.R.StreinerD.L.Biostatistics: The Bare Essentials.Hamilton, Ontario: BC Decker: 2008.
25.
OhdeR.N.SharfD.J.JacobsonP.F.Phonetic analysis of normal and abnormal speech. J Acoust Soc Am.1992; 92: 3452.
26.
PereiraV.SellD.TuomainenJ.The relationship between equal appearing interval scales (EAI) and visual analog scales (VAS) for rating hypernasality: findings from an osteotomy study. Abstract presented at the 9th European Craniofacial Congress; September 2011; Salzburg, Austria.
PriceD.D.McGrathP.A.RafiiA.BuckinghamB.The validation of visual analogue scales as ratio scale measures for chronic and experimental pain. Pain.1983; 17: 45–56.
29.
ReipsU.D.FunkeF.Interval-level measurement with visual analogue scales in Internet-based research: VAS Generator. Behav Res Methods.2008; 40: 699–704.
30.
SchiavettiN.MartinR.R.HaroldsonS.K.MetzD.E.Psychophysical analysis of audiovisual judgments of speech naturalness of nonstutterers and stutterers. J Speech Hear Res.1994; 37: 46–52.
31.
SchiavettiN.MetzD.E.OrlikoffR.F.Evaluating Research in Communicative Disorders.Boston: Pearson;2011.
32.
SellD.JohnA.Harding-BellA.SweeneyT.HegartyF.FreemanJ.Cleft audit protocol for speech (CAPS-A): a comprehensive training package for speech analysis. Int J Lang Commun Disord.2009; 44: 529–548.
SouthwoodH.P.WeismerG.Listener judgments of the bizarreness, acceptability, naturalness, and normalcy of the dysarthria associated with amyotrophic lateral sclerosis. J Med Speech Lang Pathol.1993; 1: 151–161.
36.
StevensS.S.Perceptual magnitude and its measurement. In: CateretteE.C.FriedmanM.P., eds. Handbook of Perception. Vol. 2. New York: Academic Press; 1974: 22–40.
37.
StevensS.S.Psychophysics: Introduction to its Perceptual, Neural, and Social Prospects.New York: Wiley;1975.
38.
TonerM.A.EmanuelE.H.Direct magnitude estimation and equal appearing interval scaling of vowel roughness. J Speech Hear Res.1989; 32: 78–82.
39.
TonerM.A.EmanuelF.W.ParkerD.Relationship of spectral noise levels to psychophysical scaling of vowel roughness. J Speech Hear Res.1990; 33: 238–244.
40.
Trost-CardamoneJ.E.A controlled sentence sample for assessing speech outcome. Poster presented at the annual meeting of the American Cleft Palate Craniofacial Association; April 18, 2012; San Jose, CA.
41.
van AlphenA.HalfensR.HasmanA.ImbosT.Likert or Rasch? Nothing is more applicable than a good theory. J Adv Nurs.1994; 20: 196–201.
42.
WhitehillT.L.ChengJ.JonesD.L.Rating hypernasality: direct magnitude estimation (DME) versus visual analogue scaling (VAS). Abstract presented at the annual meeting of the American Cleft Palate Craniofacial Association; April 2007; Broomfield, CO.
43.
WhitehillT.L.LeeA.S.ChunJ.C.Direct magnitude estimation and interval scaling of hypernasality. J Speech Lang Hear Res.2002; 45: 80–88.
44.
WuytsF.L.De BodtM.S.Van de HeyningP.H.Is the reliability of a visual analog scale higher than an ordinal scale? An experiment with the GRBAS scale for the perceptual evaluation of dysphonia. J Voice.1999; 13: 508–517.
45.
ZraickR.I.LissJ.M.A comparison of equal-appearing interval scaling and direct magnitude estimation of nasal voice quality. J Speech Lang Hear Res.2000; 43: 979–988.
46.
ZraickR.I.LissJ.M.DormanM.F.CaseJ.L.LaPointeL.L.BealsS.P.Multidimensional scaling of nasal voice quality. J Speech Lang Hear Res.2000; 43: 989–996.
