Past research has demonstrated that there are cognitive processing costs associated with comprehension of speech generated by text-to-speech synthesizers, relative to comprehension of natural speech. This finding has important performance implications for the many applications that use such systems. The purpose of this study was to ascertain whether certain characteristics of synthetic speech slow on-line, real-time cognitive processing. Whereas past research has focused on the phonemic acoustic structure of synthetic speech, we manipulated prosodic, syntactic, and semantic cues in a task requiring participants to recall sentences spoken either by a human or by one of two speech synthesizers. The findings were interpreted to suggest that inappropriate prosodic modeling in synthetic speech was the major source of a performance differential between natural and synthetic speech. Prosodic cues, along with others, guide the parsing of speech and provide redundancy. When these cues are absent or inaccurate, the additional burden placed on working memory may exceed its capacity, particularly in time-limited, demanding tasks. Actual or potential applications of this research include improvement of text-to-speech output systems in warning systems, feedback devices in aerospace vehicles, educational and training modules, aids for the handicapped, consumer products, and technologies designed to increase the functional independence of older adults.
Get full access to this article
View all access options for this article.
References
1.
Benoit, C., Grice, M., & Hazan, V. (1996). The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using semantically unpredictable sentences. Speech Communication, 18, 381–392.
2.
Blaauw, E. (1994). The contribution of prosodic boundary markers to the perceptual difference between read and spontaneous speech. Speech Communication, 14, 359–375.
3.
Collier, R., & Hart, J. T. (1975). The role of intonation in speech perception. In A. Cohen & S. G. Nooteboom (Eds.), Structure and process in speech perception (pp. 107–121. New York: Springer-Verlag.
4.
Cooper, W. E., & Paccia-Cooper, J. (1980). Syntax and speech. Cambridge, MA: Harvard University Press.
5.
Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671–684.
6.
Darwin, C. J. (1975). On the dynamic use of prosody in speech perception. In A. Cohen & S. G. Nooteboom (Eds.), Structure and process in speech perception (pp. 178–194. New York: Springer-Verlag.
7.
Delogu, C., Conte, S., & Sementina, C. (1998). Cognitive factors in the evaluation of synthetic speech. Speech Communication, 24, 153–168.
8.
Duffy, S. A., & Pisoni, D. B. (1992). Comprehension of synthetic speech produced by rule: A review and theoretical interpretation. Language and Speech, 35, 351–389.
9.
Egan, J. P. (1948). Articulation testing methods. Laryngoscope, 58, 995–991.
10.
Greene, B. G., Logan, J. S., & Pisoni, D. B. (1986). Perception of synthetic speech produced by rule: Intelligibility of eight text-to-speech systems. Behavior Research Methods, Instruments, and Computers, 18, 100–107.
11.
Greene, B. G., Manous, L. M., & Pisoni, D. B. (1984). Perceptual evaluation of DECtalk: A final report on version 1.8. Research in Speech Perception (Progress Report No. 10). Bloomington: Indiana University Speech Research Laboratory.
12.
Grosjean, F. (1983). How long is the sentence? Prediction and prosody in the on-line processing of language. Linguistics, 21, 501–529.
13.
Haggard, M. (1975). Understanding speech understanding. In A. Cohen & S. G. Nooteboom (Eds.), Structure and process in speech perception (pp. 3–15. New York: Springer-Verlag.
14.
Hoover, J., Reiche, J., Van Tasell, D., & Cole, D. (1987). The intelligibility of synthesized speech: Echo II versus Votrax. Journal of Speech and Hearing Research, 30, 425–431.
15.
House, A. S., Williams, C. E., Hecker, M. H. L., & Kryter, K. (1965). Articulation-testing methods: Consonantal differentiation with a closed-response set. Journal of the Acoustical Society of America, 37, 158–166.
16.
Institute of Electrical and Electronics Engineers. (1969). IEEE recommended practice for speech quality measurements (IEEE No. 297). New York: Author.
17.
Jarvella, R. J. (1971). Syntactic processing of connected speech. Journal of Verbal Learning and Verbal Behavior, 10, 409–416.
18.
Liberman, A. M., Cooper, F. S., Shankweiler, D. P., & Studdert-Kennedy, M. (1967). Perception of the speech code. Psychological Review, 74, 431–461.
19.
Logan, J. S., Greene, B. G., & Pisoni, D. B. (1989). Segmental intelligibility of synthetic speech produced by rule. Journal of the Acoustical Society of America, 86, 566–581.
20.
Luce, P. A. (1982). Comprehension of fluent synthetic speech produced by rule. Journal of the Acoustical Society of America, 71, UU11.
21.
Luce, P. A., Feustel, T. C., & Pisoni, D. B. (1983). Capacity demands in short-term memory for synthetic and natural speech. Human Factors, 25, 17–32.
22.
Manous, L. M., & Pisoni, D. B. (1984). Effects of signal duration on the perception of natural and synthetic speech. Research on Speech Perception (Progress Report No. 10). Bloomington: Indiana University Speech Research Laboratory.
23.
Martin, J. G. (1972). Rhythmic (hierarchical) versus serial structure in speech and other behavior. Psychological Review, 79, 487–509.
24.
Martin, J. G. (1975). Rhythmic expectancy in continuous speech perception. In A. Cohen & S. G. Nooteboom (Eds.), Structure and process in speech perception (pp. 161–176. New York: Springer-Verlag.
25.
Meltzer, R. H., Martin, J. G., Mills, C. B., Imhoff, D. L., & Zohar, D. (1975). Reaction time to temporally displaced phoneme targets in continuous speech. Journal of Experimental Psychology: Human Perception and Performance, 2, 277–290.
26.
Miller, G. A., Heise, G. A., & Lichten, W. (1951). The intelligibility of speech as a function of the context of the test materials. Journal of Experimental Psychology, 41, 329–335.
27.
Mirenda, P., & Beukelman, D. R. (1987). A comparison of speech synthesis intelligibility with listeners from three age groups. Augmentative and Alternative Communication, 3, 120–128.
28.
Nusbaum, H. C., Francis, A. L., & Henley, A. S. (1995). Measuring the naturalness of synthetic speech. International Journal of Synthetic Speech, 1, 7–19.
29.
Nusbaum, H. C., & Pisoni, D. B. (1985). Constraints on the perception of synthetic speech generated by rule. Behavior Research Methods, Instruments, & Computers, 17, 235–242.
30.
Paris, C. R., Gilson, R. D., Thomas, M. H., & Silver, N. C. (1995). Effect of synthetic voice intelligibility upon speech comprehension. Human Factors, 37, 335–340.
31.
Pavlovic, C. V. (1987). Derivation of primary parameters and procedures for use in speech intelligibility predictions. Journal of the Acoustical Society of America, 82, 413–422.
32.
Pisoni, D. B. (1981). Speeded classification of natural and synthetic speech in a lexical decision task. Journal of the Acoustical Society of America, 70, S98.
33.
Pisoni, D. B., & Hunnicutt, S. (1980). Perceptual evaluation of MITalk: The MIT unrestricted text-to-speech system. In 1980 IEEE International Conference on Acoustics, Speech, and Signal Processing (pp. 572–575). New York: Institute of Electrical and Electronics Engineers.
34.
Pisoni, D. B., Manous, L. M., & Dedina, M. J. (1987). Comprehension of natural and synthetic speech: Effects of predictability on the verification of sentences controlled for intelligibility. Computer Speech and Language, 2, 303–320.
35.
Pisoni, D. B., Nusbaum, H. C., & Greene, B. G. (1985). Perception of synthetic speech generated by rule. Proceedings of the IEEE, 73, 1665–1676.
36.
Ralston, J. V., Pisoni, D. B., Lively, S. E., Greene, B. G., & Mullenix, J. W. (1991). Comprehension of synthetic speech produced by rule: Word monitoring and sentence-by-sentence listening times. Human Factors, 33, 471–491.
37.
Sanderman, A. A., & Collier, R. (1997). Prosodic phrasing and comprehension. Language and Speech, 40, 391–409.
38.
Simpson, C. A., & Marchionda-Frost, K. (1984). Synthesized speech rate and pitch effects on intelligibility of warning messages for pilots. Human Factors, 26, 509–517.
39.
Simpson, C. A., & Williams, D. H. (1980). Response time effects of alerting tone and semantic context for synthesized voice cockpit warnings. Human Factors, 22, 319–330.
40.
Slowiaczek, L. M., & Nusbaum, H. C. (1985). Effects of speech rate and pitch contour on the perception of synthetic speech. Human Factors, 27, 701–712.
41.
Stine, E. L., & Wingfield, A. (1987). Process and strategy in memory for speech among younger and older adults. Psychology and Aging, 2, 272–279.
42.
Terken, J. (1993). Synthesizing natural-sounding intonation for Dutch: Rules and perceptual evaluation. Computer Speech and Language, 7, 27–48.
43.
Terken, J., & Lemeer, G. (1988). Effects of segmental quality and intonation on quality judgments for texts and utterances. Journal of Phonetics, 16, 453–457.
44.
Wingfield, A. (1975). The intonation-syntax interaction: Prosodic features in perceptual processing of sentences. In A. Cohen & S. G. Nooteboom (Eds.), Structure and process in speech perception (pp. 146–160. New York: Springer-Verlag.
