Technological developments in automatic speech recognition (ASR) technology over the last couple of decades have resulted in ASR being considered for use in an ever-increasing number of applications. However, a number of physical and psychological environments are still deemed inappropriate for this technology. Several of the stressors creating adverse environments are considered here, in conjunction with their effects on human speech production, human performance in general, and the implications for ASR system design. We propose that a combination of continuing technological developments, together with the application of system engineering and human factors, will lead to ensuring the attainment of robust ASR performance in a number of environments previously considered as high risk.
Get full access to this article
View all access options for this article.
References
1.
Armstrong, J. W., & Poock, G. A. (1981a). Effect of operator mental task loading on voice recognition system performance (Tech. Report NPS55-81-016). Monterey, CA: Naval Postgraduate School.
2.
Armstrong, J. W., & Poock, G. A. (1981b). Effect of task duration on voice recognition system performance (Tech. Report NPS55-81-017). Monterey, CA: Naval Postgraduate School.
3.
Baber, C. (1991). Speech technology in control room systems: A human factors perspective. Chichester, England: Ellis Horwood.
4.
Baber, C. (1995). The effects of workload on the use of speech recognition systems. In Proceedings of NATO/ESCA Workshop on Speech Under Stress (pp. 75–82). Lisbon: European Speech Communication Association.
5.
Baber, C., Stammers, R. B., & Taylor, R. G. (1991). An experimental assessment of automatic speech recognition in high cognitive workload situations in control rooms. In Proceedings of the 11th International Ergonomics Association Conference (pp. 833–835). London: Taylor & Francis.
6.
Baber, C., Usher, D. M., Stammers, R. B., & Taylor, R. G. (1992). Feedback requirements for automatic speech recognition in the process control room. International Journal of Man-Machine Studies,37, 703–719.
7.
Berkun, M. M. (1964). Performance decrement under psychological stress. Human Factors,6, 21–30.
8.
Bond, Z. S., Moore, T. J., & Anderson, T. R. (1987). The effects of high sustained acceleration on the acoustic-phonetic structure of speech: A preliminary investigation. Journal of the American Voice I/O Society,4, 1–19.
9.
Bond, Z. S., Moore, T. J., & Gable, B. (1989). Some acoustic phonetic characteristics of speech produced in noise and while wearing an oxygen mask. Journal of the Acoustical Society of America,85, 907–917.
10.
Broadbent, D. E. (1971). Decision and stress. London: Academic.
11.
Coleman, R. F., & Williams, R. (1979). Identification of emotional states using perceptual and acoustic analysis. In V. Lawrence & B. Weinberg, (Eds.), Transcript of the 8th Symposium: Care of the professional voice (pp. 7–13. New York: Voice Foundation.
12.
Cung, H. M., & Normandin, Y. (1993). Noise adaptation algorithms for robust speech recognition. Speech Communication,12, 267–276.
13.
Darwin, C. (1872). The expression of emotions in man and animals. London: Penguin.
14.
Driskell, J. E., & Salas, E. (1991). Overcoming the effects of stress on military performance: Human factors, training and selection strategies. In R. Gal & A. D. Mangelsdorff, (Eds.), Handbook of military psychology (pp. 183–193. New York: Wiley.
15.
Fairbanks, C., & Hoaglin, L. W. (1941). An experimental study of the durational characteristics of the voice during the expression of emotion. Speech Monographs,8, 85–90.
16.
Fairbanks, G., & Provonost, W. (1939). An experimental study of the pitch characteristics of the voice during the expression of emotion. Speech Monographs,6, 87–104.
17.
Fitts, P. M., & Seeger, C. M. (1953). S-R compatibility: Spatial characteristics of stimulus and response codes. Journal of Experimental Psychology,46, 199–210.
18.
Fonagy, I. (1978). A new method of investigating the perception of prosodic features. Language and Speech,21, 34–49.
19.
Frankish, C. R., & Jones, D. M. (1987). Parcel sorting by speech recognition: A case study in vocabulary design. In Proceedings of the European Conference on Speech Technology (pp. 197–201). Edinburgh: CEP Consultants.
20.
Frankish, C. R., Jones, D. M., & Hapeshi, K. C. (1990). Maintaining recognition accuracy during data entry tasks using speech input. In E. J. Lovesey (Ed.), Contemporary ergonomics (pp. 445–453. London: Taylor & Francis.
21.
Frankish, C. R., & Noyes, J. M. (1990). Sources of human error in data entry tasks using speech input. Human Factors,32, 697–716.
22.
Furness, T. A., & Lewis, C. H. (1978, March). Helmet-mounted display reading performance under whole-body vibration. Paper presented to the U.K. Informal Group on Human Response to Vibration, Silsoe, U.K.
23.
Gagon, L. (1993). A state based noise reduction approach for non-stationary additive interference. Speech Communication,12, 213–219.
24.
von Gierke, H. E., McCloskey, K., & Albery, W. B. (1991). Military performance in sustained acceleration and vibration environments. In R. Gal & A. D. Mangelsdorff, (Eds.), Handbook of military psychology (pp. 336–364. New York: Wiley.
25.
Graham, R., & Baber, C. (1993). User stress in ASR. In E. J. Lovesey (Ed.), Contemporary ergonomics (pp. 463–469. London: Taylor & Francis.
26.
Griffin, M. J. (1990). Handbook of human vibration. London: Academic.
27.
Hapeshi, K. C., & Jones, D. M. (1989). The ergonomics of automatic speech recognition interfaces. International Review of Ergonomics,2, 251–290.
28.
Haslegrave, C. M. (1991). Auditory environment and noise assessment. In J. R. Wilson & E. N. Corlett, (Eds.), Evaluation of human work (pp. 406–439. London: Taylor & Francis.
29.
Helms, R. E. (1986). Voice control of mobile telephones. In Proceedings of Speech Tech '86 (pp. 126–130). New York: Media Dimensions.
30.
Hockey, G. R. J. (1979). Stress and cognitive components of skilled performance. In V. Hamilton & D. M. Warburton, (Eds.), Human stress and cognition: An information processing approach (pp. 141–178. Chichester, England: Wiley.
31.
Hornick, R. J. (1962). Effects of whole body vibration in three directions upon human performance. Journal of Engineering Psychology,1, 93–101.
32.
Johnston, M. E. (1979). A preliminary investigation of the effect of 2-20 Hz. whole body vibration on spoken numbers (Tech. Memorandum FS 152). Farnborough, England: Royal Aircraft Establishment.
33.
Kotlyar, G. M., & Morozov, V. P. (1976). Acoustical correlates of the emotional content of vocalized speech. Journal of Acoustics of the Academy of Sciences of the USSR,22, 208–211.
34.
Kuroda, I., Fujiwara, O., Okamura, N., & Utsuki, N. (1976). Method for determining pilot stress through analysis of voice communication. Aviation, Space and Environmental Medicine,47, 528–533.
35.
Lane, H. L., Tranel, B., & Sisson, C. (1970). Regulation of voice communication by sensory dynamics. Journal of the Acoustical Society of America,47, 618–624.
36.
Leeks, C. (1986). Operation of a speech recognizer under whole body vibration (Tech. Memorandum FDS(F) 634). Farnborough, England: Royal Aircraft Establishment.
37.
Levelt, W. J. M. (1993). Speaking: From intention to articulation. Cambridge: MIT Press.
38.
Lewis, C. H., & Griffin, M. J. (1978). Predicting the effects of dual-frequency vertical vibration on continuous manual control performance. Ergonomics,21, 637–650.
39.
Lewis, C. H., & Griffin, M. J. (1980). Predicting the effects of vibration frequency and axis, and seating conditions on the reading of numeric displays. Ergonomics,23, 485–501.
40.
Lieberman, P., & Blumstein, S. E. (1988). Speech physiology, speech perception, and acoustic phonetics. Cambridge: Cambridge University Press.
41.
Lieberman, P., & Michaels, S. B. (1962). Some aspects of fundamental frequency and envelope amplitude as related to the emotional content of speech. Journal of the Acoustical Society of America,34, 922–927.
42.
Little, R., & Cowan, R. (1986). A flight evaluation of voice interaction as a component of an integrated helicopter avionics system (Tech. Memorandum FD(B) 637). Farnborough, England: Royal Aircraft Establishment.
43.
Malkin, F. J., & Dennison, T. (1986). The effect of helicopter vibration on the accuracy of a voice recognition system. In Proceedings of the National Aerospace and Electronics Conference (NAECON) (pp. 813–817). Piscataway, NJ: Institute of Electrical and Electronics Engineers.
44.
McCauley, M. E. (1984). Human factors in voice technology. In F. A. Muckler (Ed.), Human factors review (pp. 131–166. Santa Monica, CA: Human Factors and Ergonomics Society.
45.
McCauley, M. E., Root, R. W., & Muckler, F. A. (1982). Training evaluation of an automated air intercept controller training system (Tech. Report NAVTRAEQUIPCEN 81-C-0055-1). Orlando, FL: Naval Training Equipment Center.
46.
Moore, T. J. (1989). Speech technology in the cockpit. In R. S. Jensen (Ed.), Aviation psychology (pp. 50–65. Aldershot, England: Gower Technical.
47.
Moore, T. J., & Bond, Z. S. (1987). Acoustic-phonetic changes in speech due to environmental stressors: Implications for speech recognition in the cockpit. In Proceedings of the 4th International Symposium on Aviation Psychology (pp. 77–83). Columbus, OH: Aviation Psychology Laboratory.
48.
Moore, T. J., & McKinley, R. L. (1986). Research on speech processing for military avionics. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 1331–1335). Santa Monica, CA: Human Factors and Ergonomics Society.
49.
Mountford, S. J., & North, R. A. (1980). Voice entry for reducing pilot workload. In Proceedings of the Human Factors Society 29th Annual Meeting (pp. 185–189). Santa Monica, CA: Human Factors and Ergonomics Society.
50.
National Research Council. (1984). Automatic speech recognition in severe environments. Washington, DC: National Research Council Committee on Computerized Speech Recognition Technologies.
51.
Niwa, S. (1971). Changes of voice characteristics in urgent situations (Tech. Report 11, 246–251). Japan: Japan Air Self Defence Force, Aeromedical Laboratory.
52.
Noyes, J. M., Baber, C., & Frankish, C. R. (1992). Using automatic speech recognition in industrial applications. Journal of the American Voice I/O Society,13, 1–17.
53.
Noyes, J. M., & Frankish, C. R. (1992). Speech recognition technology for individuals with disabilities. Augmentative and Alternative Communication,8, 297–303.
54.
Oborne, D. J. (1987). Ergonomics at work. Chichester, England: Wiley.
55.
Pallett, D. S. (1985). Performance assessment of automatic speech recognizers. Journal of the National Bureau of Standards,90, 1–17.
56.
Paul, D. B., Lippman, R. P.Chen, Y., & Weinstein, C. J. (1986). Robust HMM-based techniques for recognition of speech produced under stress and in noise. Paper presented at the Defence Advanced Research Projects Association Strategic Computing Program Speech Recognition Workshop, Palo Alto, CA.
57.
Rajaskaran, P. K., Doddington, G. R., & Picone, J. W. (1986). Recognition of speech under stress. In Proceedings of the International Conference on Acoustic, Speech and Signal Processing (pp. 733–736). New York: Institute of Electrical and Electronics Engineers.
58.
Reed, L. (1985). Military applications of voice technology.. Speech Technology,2(4), 42–50.
59.
Rehsoft, C. (1984). Voice recognition at the Ford warehouse in Cologne. In Proceedings of the 1st International Conference on Speech Technology (pp. 103–112). Amsterdam: North-Holland.
60.
Roessler, R., & Lester, J. W. (1976). Voice predicts affect during psychotherapy. Journal of Nervous and Mental Disease,163, 166–176.
61.
Rollins, A. M. (1985). Speech recognition and manner of speaking in noise and quiet. In Proceedings of Human Factors in Computing Systems II (pp. 197–200). Amsterdam: North-Holland.
62.
Ruehl, H. W., Dobler, S., Weith, J., Meyer, P., Noll, A., Hamer, H. H., & Piotrowski, H. (1991). Speech recognition in the noisy car environment. Speech Communication,10, 11–22.
63.
Sanders, M. S., & McCormick, E. J. (1987). Human factors in engineering and design. New York: McGraw-Hill.
64.
Scherer, K. R. (1986). Vocal affect expression: A review and a model for future research.. Psychological Bulletin,99(2), 143–165.
65.
Schoenberger, R. W. (1974). An investigation of human information processing during whole body vibration. Aerospace Medicine,45, 143–153.
66.
Simonov, P. V., & Frolov, M. V. (1973). Utilisation of human voice for estimation of man's emotional stress and state attention. Aerospace Medicine,44, 256–258.
67.
Simpson, C. A. (1986). Speech variability effects on recognition accuracy associated with concurrent task performance by pilots. Ergonomics,29, 1343–1357.
68.
Simpson, C. A., McCauley, M. E., Roland, E. F., Ruth, J. C., & Williges, B. H. (1985). System design for speech recognition and generation. Human Factors,27, 115–143.
69.
Smith, A. (1985). The effects of noise on the processing of global shape and local detail. Psychological Research,47, 103–108.
70.
Smith, A. (1988). Noise, S-R compatibility and hand dominance. Scandinavian Journal of Psychology,30, 185–206.
71.
Starr, A. F. (1993). Is control by voice the right answer for the avionics environment. In C. Baber & J. M. Noyes, (Eds.), Interactive speech technology (pp. 85–97. London: Taylor & Francis.
72.
Taylor, M. (1986). Voice input applications in aerospace. In G. Bristow (Ed.), Electronic speech recognition (pp. 322–337. London: Collins.
73.
Utsuki, N., & Okamura, N. (1976). Relationship between emotional state and fundamental frequency of speech (Tech. Report 16, 179–188). Japan: Japan Air Self Defence Force, Aeromedical Laboratory.
74.
Van Summers, W., Pisoni, D. B., Bernacki, R. H., Pedlow, R. I., & Stokes, M. A. (1988). Effects of noise on speech production: Acoustic and perceptual analyses. Journal of the Acoustical Society of America,84, 917–928.
75.
Varga, A. P., & Moore, R. K. (1990). Hidden Markov model decomposition of speech and noise. In Proceedings of the International Conference on Acoustic, Speech and Signal Processing (pp. 845–848). New York: Institute of Electrical and Electronics Engineers.
76.
Vensko, G. (1985). A speech recognizer for tactical platforms. In Proceedings of Voice I/O Systems Applications Conference (pp. 17–20). Chicago: American Voice Input/Output Society.
77.
Watt, B. (1968). Habitability. American Documentation,14, 1–23.
78.
Wetterlind, P., & Johnston, W. L. (1986). An emergency command recognizer for voiced system control. In Proceedings of the 24th Annual Symposium of the SAFE Association (pp. 181–184). Van Nuys, CA: SAFE Association.
79.
Wickens, C. D., & Liu, Y. (1988). Codes and modalities for multiple resources: A success and some qualifications. Human Factors,30, 599–616.
80.
Wickens, C. D., Sandry, D. L., & Vidulich, M. (1983). Compatibility and resource competition between modalities of input, central processing, and output. Human Factors,25, 227–240.
81.
Wickens, C. D., Stokes, A. F., Barnett, B., & Hyman, F. (1991). The effects of stress on pilot judgment in a MIDIS simulator. In O. Svenson & J Maule (Eds.), Time pressure and stress in human judgment and decision making (pp. 271–292. Cambridge: Cambridge University Press.
82.
Wickens, C. D., Vidulich, M., & Sandry-Garza, D. (1984). Principles of S-C-R compatibility with spatial and verbal tasks: The role of display location and voice interactive display control interfacing. Human Factors,26, 533–542.
83.
Wickens, C. D., & Weingartner, A. (1985). Process control monitoring: The effect of spatial and verbal ability and concurrent task demand. In R. E. Eberts & C. G. Eberts, (Eds.), Trends in ergonomics/human factors (pp. 25–32. Amsterdam: North-Holland.
84.
William, C. E., & Stevens, K. N. (1969). On determining the emotional state of pilots during flight: An exploratory study. Aerospace Medicine,40, 1369–1372.
85.
Williams, C. E., & Stevens, K. N. (1972). Emotions and speech: Some acoustic correlates. Journal of the Acoustical Society of America,52, 1238–1250.
86.
Zarembo, L. A. (1986). The usefulness of voice recognition on the floor of the New York Stock Exchange. In Proceedings of Speech Tech '86 (pp. 69–72). New York: Media Dimensions.
