EduSpeak ® : A speech recognition and pronunciation scoring toolkit for computer-aided language learning applications

Abstract

SRI International’s EduSpeak® system is a software development toolkit that enables developers of interactive language education software to use state-of-the-art speech recognition and pronunciation scoring technology. Automatic pronunciation scoring allows the computer to provide feedback on the overall quality of pronunciation and to point to specific production problems. We review our approach to pronunciation scoring, where our aim is to estimate the grade that a human expert would assign to the pronunciation quality of a paragraph or a phrase. Using databases of nonnative speech and corresponding human ratings at the sentence level, we evaluate different machine scores that can be used as predictor variables to estimate pronunciation quality. For more specific feedback on pronunciation, the EduSpeak toolkit supports a phone-level mispronunciation detection functionality that automatically flags specific phone segments that have been mispronounced. Phone-level information makes it possible to provide the student with feedback about specific pronunciation mistakes.Two approaches to mispronunciation detection were evaluated in a phonetically transcribed database of 130,000 phones uttered in continuous speech sentences by 206 nonnative speakers. Results show that classification error of the best system, for the phones that can be reliably transcribed, is only slightly higher than the average pairwise disagreement between the human transcribers.

Keywords

automatic pronunciation scoring computer aided language learning mispronunciation detection

Get full access to this article

View all access options for this article.

References

Bernstein, J. , Cohen, M. , Murveit, H. , Rtischev, D. , & Weintraub, M. ( 1990). Automatic evaluation and training in English pronunciation . Proceedings of ICSLP 90, 1185-1188. Kobe, Japan.

Bernstein, J. , De Jong, J.H.A.L. , Pisoni, D. , & Townshend, B. ( 2000) Two experiments on automatic scoring of spoken language proficiency. Proceedings of InSTILL2000: Integrating Speech Technology in Learning, 57-61. University of Abertay Dundee, UK.

Bialystok, E. , & Hakuta, K. ( 1994) In other words, the science and psychology of second-language acquisition. New York: Basic Books .

Bratt, H. , Neumeyer, L. , Shriberg, E. , & Franco, H. ( 1998). Collection and detailed transcription of a speech database for development of language learning technologies. Proceedings of ICSLP 98, 1539-1542. Sydney, Australia.

Breiman, L. , Friedman, J. , Olson, R. , & Stone, C. ( 1984). Classification and regression trees. The Waxworks & Brooks/Cole Statistics/Probability Series. Belmont, CA: Wadsworth International.

Buntine, W. , & Caruana, R. ( 1992). Introduction to IND V. 2.1 and recursive partitioning. Moffet Field, CA: NASA Ames Research Center.

Digalakis, V. ( 1992). Algorithm development in the Autograder project. SRI International Internal Communication.

Digalakis, V. , & Neumeyer, L. ( 1996). Speaker adaptation using combined transformation and Bayesian methods. IEEE Transactions on Speech and Audio Processing , 4(4), 294-300.

Digalakis, V. , Monaco, P. , & Murveit, H. ( 1996). Genones: Generalised mixture tying in continuous hidden Markov model-based speech recognizers. IEEE Transactions on Speech and Audio Processing, 4(4), 281-289.

10.

Franco, H. , & Neumeyer, L. ( 1998). Calibration of machine scores for pronunciation grading . Proceedings of ICSLP, 98, 2631-2634, Sydney, Australia.

11.

Franco, H. , Neumeyer, L. , Kim, Y. , & Ronen, O. ( 1997). Automatic pronunciation scoring for language instruction . Proceedings of ICASSP 97, 1471-1474. Munich, Germany.

12.

Franco, H. , Neumeyer, L. , Digalakis, V. , & Weintraub, M. ( 1999a). Automatic scoring of pronunciation quality. Speech Communication, 30, 83-93.

13.

Franco, H. , Neumeyer, L. , Digalakis, V. , & Ronen, O. ( 1999b). Combination of machine scores for automatic grading of pronunciation quality. Speech Communication, 30, 121-130.

14.

Franco, H. , Neumeyer, L. , Ramos, M. , & Bratt, H. ( 1999c). Automatic detection of phone-level mispronunciations for language learning. Proceedings of Eurospeech 99, 2, 851-854. Budapest, Hungary.

15.

Gauvain, J.L. , & Lee, C.H. ( 1994). Maximum a posteriori estimation for multivariable Gaussian mixture observatioins of Markov models. IEEE Transactions on Speech and Audio Processing, 2(2) 291-298.

16.

Kim, Y. , Franco, H. , & Neumeyer, L. ( 1997). Automatic pronunciation scoring of specific phone segments for language instruction. Proceedings of EUROSPEECH 97, 649-652. Rhodes, Greece.

17.

Neumeyer, L. , Franco, H. , Weintraub, M. , & Price, P. ( 1996). Automatic text-independent pronunciation scoring of foreign language student speech. Proceedings of ICSLP 96, 1457-1460. Philadelphia, PA.

18.

Precoda, K. Halverson, C. , Franco, H. ( 2000) Effect of speech recognition-based pronunciation feedback on second language pronunciation ability. Proceedings of InSTILL2000: Integrating Speech Technology in Learning, 102-105. University of Abertay Dundee, UK.

19.

Rao Gadde, V.R. ( 2000). Modeling word duration. Proceedings of the Sixth International Conference on Spoken Language Processing (ICSLP) 2000, 1, 601-604.

20.

Ronen, O. , Neumeyer, L. , & Franco, H. ( 1997). Automatic detection of mispronunciation for language instruction. Proceedings of EUROSPEECH, 645-648. Rhodes , Greece.

21.

Siegel, S. , & Castellan, N. John Jr. (1988). Nonparametric statistics for the behavioral sciences (2nd ed.). New York: McGraw-Hill.

22.

Witt, S. , & Young, S. ( 1997). Language learning based on non-native speech recognition . Proceedings of EUROSPEECH 97, 633-636. Rhodes, Greece.

23.

Witt, S. , & Young, S. ( 1998). Performance measures for phone-level pronunciation teaching in CALL. Proceedings of the Workshop on Speech Technology in Language Learning, 99-102. Marholmen, Sweden.