To compare reliability in auditory-perceptual assessment of hypernasality for 3 different methods and to explore the influence of language background.
Design:
Comparative methodological study.
Participants and Materials:
Audio recordings of 5-year-old Swedish-speaking children with repaired cleft lip and palate consisting of 73 stimuli of 9 nonnasal single-word strings in 3 different randomized orders. Four experienced speech-language pathologists (2 native speakers of Brazilian–Portuguese and 2 native speakers of Swedish) participated as listeners. After individual training, each listener performed the hypernasality rating task. Each order of stimuli was analyzed individually using the 2-step, VISOR and Borg centiMax scale methods.
Main Outcome Measures:
Comparison of intra- and inter-rater reliability, and consistency for each method within language of the listener and between listener languages (Swedish and Brazilian–Portuguese).
Results:
Good to excellent intra-rater reliability was found within each listener for all methods, 2-step: κ = 0.59-0.93; VISOR: intraclass correlation coefficient (ICC) = 0.80-0.99; Borg centiMax (cM) scale: ICC = 0.80-1.00. The highest inter-rater reliability was demonstrated for VISOR (ICC = 0.60-0.90) and Borg cM-scale (ICC = 0.40-0.80). High consistency within each method was found with the highest for the Borg cM scale (ICC = 0.89-0.91). There was a significant difference in the ratings between the Swedish and the Brazilian listeners for all methods.
Conclusions:
The category-ratio scale Borg cM was considered most reliable in the assessment of hypernasality. Language background of Brazilian–Portuguese listeners influenced the perceptual ratings of hypernasality in Swedish speech samples, despite their experience in perceptual assessment of cleft palate speech disorders.
AltmanDG. Practical Statistics for Medical Research. London, England: Chapman & Hall/CRC Press; 1991:404.
2.
AndersonRT. Nasometric values for normal Spanish-speaking females: a preliminary report. Cleft Palate Craniofac J. 1996;33(4):333–336.
3.
AwanSNLawsonLL. The effect of anchor modality on the reliability of vocal severity ratings. J Voice. 2009;23(3):341–352.
4.
BaylisALMunsonBMollerKT. Perceptions of audible nasal emission in speakers with cleft palate: a comparative study of listener judgments. Cleft Palate Craniofac J. 2011;48(4):399–411.
5.
BaylisALChapmanKLWhitehillTL, Group TA. Validity and reliability of visual analog scaling for assessment of hypernasality and audible nasal emission in children with repaired cleft palate. Cleft Palate Craniofac J. 2015;52(6):660–670.
6.
BorgG. Physical performance and perceived exertionDissertation Lund, Sweden: Studia Psychologica et Paedagogica, Series altera, Investigationes XI: Lund , Sweden: Lund University; 1962.
7.
BorgG. A category scale with ratio properties for intermodal and interindividual comparisons. In: GeisslerHGPetzoldP, eds. Psychophysical Judgment and the Process of Perception. Berlin, Germany: VEB Deutscher Verlag der Wissenschaften; 1982.
8.
BorgG. Borg’s Perceived exertion and pain scales. Champaign, IL: Human Kinetics; 1998.
9.
BorgE. On Perceived Exertion and its Measurement. Stockholm, Sweden: Stockholm University; 2007. Dissertation.
10.
BorgE. Placing verbal descriptors on a ratio scale. Presented at the 27th Annual Meeting of the International Society for Psychophysics; October 2011; Ra’anana, Israel.
11.
BorgGBorgE. A new generation of scaling methods: level-anchored ratio scaling. Psychologica. 2001;28:15–45. https://www.uc.pt/fpce/psychologica/indexes/ind28
12.
BrancampTULewisKEWattersonT. The relationship between nasalance scores and nasality ratings obtained with equal appearing interval and direct magnitude estimation scaling methods. Cleft Palate Craniofac J. 2010;47(6):631–637.
13.
BrunnegårdKLohmanderA. A cross-sectional study of speech in 10-year-old children with cleft palate: results and issues of rater reliability. Cleft Palate Craniofac J. 2007;44(1):33–44.
14.
BrunnegårdKLohmanderAvan DoornJ. Untrained listeners’ ratings of speech disorders in a group with cleft palate: a comparison with speech and language pathologists’ ratings. Int J Lang Commun Disord. 2009;44(5):656–674.
15.
CastickSKnightRASellD. Perceptual judgments of resonance, nasal airflow, understandability, and acceptability in speakers with cleft palate: ordinal versus visual analogue scaling. Cleft Palate Craniofac J. 2017;54(1):19–31.
16.
ChanKMYiuEM. The effect of anchors and training on the reliability of perceptual voice evaluation. J Speech Lang Hear Res. 2002;45(1):111–126.
17.
CicchettiDV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6:284–290.
18.
EadieTLBaylorCR. The effect of perceptual training on inexperienced listeners’ judgments of dysphonic voice. J Voice. 2006;20(4):527–544.
19.
EadieTLKapsner-SmithM. The effect of listener experience and anchors on judgments of dysphonia. J Speech Lang Hear Res. 2011;54(2):430–447.
20.
EfronB. Bootstrap methods: another look at the jackknife. Ann Stat. 1979;7:1–26.
21.
GerrattBRKreimanJAntonanzas-BarrosoNBerkeGS. Comparing internal and external standards in voice quality judgments. J Speech Hear Res. 1993;36(1):14–20.
22.
GranqvistS. The visual sort and rate method for perceptual evaluation in listening tests. Logoped Phoniatr Vocol. 2003;28(3):109–116.
23.
HuttersBHenningssonG. Speech outcome following treatment in cross-linguistic cleft palate studies: methodological implications. Cleft Palate Craniofac J. 2004;41(5):544–549.
24.
IwarssonJPetersenNR. Effects of consensus training on the reliability of auditory perceptual ratings of voice quality. J Voice. 2012;26(3):304–312.
25.
KentRD. Hearing and believing: some limits to the auditory-perceptual assessment of speech and voice disorders. Am J Speech Lang Pathol. 1996;5:7–23.
26.
KeuningKHWienekeGHDejonckerePH. The intrajudge reliability of the perceptual rating of cleft palate speech before and after pharyngeal flap surgery: the effect of judges and speech samples. Cleft Palate Craniofac J. 1999;36(4):328–333.
27.
KreimanJGerratBRKempsterGBErmanABerkeGS. Perceptual evaluation of voice quality: review, tutorial, and a framework for future research. J Speech Hear Res. 1993;36(1):21–40.
28.
KreimanJGerrattBR. Sources of listener disagreement in voice quality assessment. J Acoust Soc Am. 2000;108(4):1867–1876.
29.
LeeABrownSGibbonFE. Effect of listeners’ linguistic background on perceptual judgements of hypernasality. Int J Lang Commun Disord. 2008;43(5):487–498.
30.
LeeAWhitehillTLCioccaV. Effect of listener training on perceptual judgement of hypernasality. Clin Linguist Phon. 2009;23(5):319–334.
31.
LohmanderAWilladsenEPerssonCHenningssonGBowdenMHuttersB. Methodology for speech assessment in the Scandcleft project-an international randomized clinical trial on palatal surgery: experiences from a pilot study. Cleft Palate Craniofac J. 2009;46(4):347–362.
32.
LohmanderAPerssonCWilladsenELundeborgIAlaluusuaSAuknerRBauABoersMBowdenMDaviesJ. Scandcleft randomised trials of primary surgery for unilateral cleft lip and palate: 4. Speech outcomes in 5-year-olds—velopharyngeal competency and hypernasality. J Plast Surg Hand Surg. 2017;51(1):27–37.
33.
MollK. ‘Objective’’ measures of nasality [Letter to the editor.] Cleft Palate Craniofac J. 1964;1:371–374.
34.
MoonJBKuhenDPHuismanJJ. Measurement of velopharyngeal closure force during vowel production. Cleft Palate Craniofac J. 1994;31(5):356–363.
35.
NeelyGLjunggrenGSylvénCBorgG. Comparison between the visual analogue scale (VAS) and the category ratio scale (CR-10) for the evaluation of leg exertion. Int J Sports Med. 1992;13(2):133–136.
OliveiraACASFScarmagnaniRHFukushiroAPYamashitaR. The influence of listener training on the perceptual assessment of hypernasality. Codas. 2016;28:141–148.
38.
SagalPBorgG. The range principle and the problem of other minds. Br J Philos Sci. 1993;44:477–491.
39.
ShrivastavRSapienzaCMNandurV. Application of psychometric theory to the measurement of voice quality using rating scales. J Speech Lang Hear Res. 2005;48(2):323–335.
40.
SideridisGDSimosP. What is the actual correlation between expressive and receptive measures of vocabulary? Approximating the sampling distribution of the correlation coefficient using the bootstrapping method. Int J Educ Psychol Assess. 2010;5:117–133.
41.
StevensSSGalanterEH. Ratio scales and category scales for a dozen perceptual continua. J Exp Psychol. 1957;54:377–411.
42.
StevensSS. Psychophysics: introduction to its perceptual, neural and social prospects. New York, NY: Wiley; 1975.
43.
SvenssonE. Concordance between ratings using different scales for the same variable. Stat Med. 2000;19(24):3483–3496.
44.
WattersonTLewisKAllordMSulprizioSO’NeillP. Effect of vowel type on reliability of nasality ratings. J Commun Disord. 2007;40:503–512.
45.
WewersMELoweNK. A critical review of visual analogue scales in the measurement of clinical phenomena. Res Nurs Health. 1990;13(4):227–236.
46.
WhitehillTLLeeASYChunJC. Direct magnitude estimation and interval scaling of hypernasality. J Speech Lang Hear Res. 2002;45(1):80–88.
47.
ZraickRILissJM. A comparison of equal-appearing interval scaling and direct magnitude estimation of nasal voice quality. J Speech Lang Hear Res. 2000;43(4):979–988.
