The Effects of Background Noise on User Experience and Performance of Mixed Reality Voice Dictation

Abstract

From education to manufacturing and medicine, the Microsoft HoloLens 2 mixed reality headset has been increasingly adopted. Text entry is a core functionality of the HoloLens 2, enabling communication and search features. The device tracks hand and finger movements interacting with a virtual keyboard but is slow and cumbersome. Speech-to-text dictation is a promising alternative requiring less physical effort and time. However, the HoloLens 2 dictation performance in environmental settings with varying noise levels is unclear. This study investigated the text input accuracy (WER), speed (WPM), and user perceptions of the HoloLens 2 dictation across three background noise levels (40 dB, 55 dB, and 70 dB). The 70 dB condition resulted in inferior outcomes for text input speed, perceived usability, mental demand, temporal demand, and effort. Users also reported lengthy processing delays in the 70 dB condition, while accuracy remained consistent (2–5% WER) across all noise conditions.

Get full access to this article

View all access options for this article.

References

American Academy of Audiology (2010). Level of noise in decibels (dB). https://audiology-web.s3.amazonaws.com/migrated/Noise Chart_Poster-8.5x11.pdf_5399b289427535.32730330.pdf

Bangor

Kortum

P. T.

Miller.

J. T

. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123. https://uxpajournal.org/determining-what-individual-sus-scoresmean-adding-an-adjective-rating-scale/

Borg

(1998). Borg’s Perceived Exertion and Pain Scales. Human Kinetics.

Brooke

(1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4–7.

Derby

J. L.

Rarick

C. T.

Chaparro

B. S.

(2019). Text input performance with a mixed reality head-mounted display (HMD). Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 1476–1480. https://doi.org/10.1177/1071181319631279

Derby

J. L.

Rickel

E. A.

Harris

K. J.

Lovell

J. A.

Chaparro

B. S.

(2020). “We Didn’t Catch That!” Using voice text input on a mixed reality headset in noisy environments. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 64(1), 2102-2106. https://doi.org/10.1177/1071181320641509

Evans

Miller

Pena

M. I.

MacAllister

Winer

E. H.

(2017). Evaluating the Microsoft HoloLens through an augmented reality assembly application. Proceedings of Degraded Environments: Sensing, Processing, and Display 2017. https://doi.org/10.1117/12.2262626

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9

Hoover

(2018). An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task [Doctoral dissertation, Iowa State University]. CORE. https://lib.dr.iastate.edu/etd/16378

10.

Kennedy

R. S.

Lane

N. E.

Berbaum

K. S.

Lilienthal

M. G.

(1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203–220. https://doi.org/10.1207/s15327108ijap0303_3

11.

Kumar

R. P.

Pelanis

Bugge

Brun

Palomar

Aghayan

D. L.

…Elle

O. J.

(2020). Use of mixed reality for surgery planning: Assessment and development workflow. Journal of Biomedical Informatics: X, 8, 100077. https://doi.org/10.1016/j.yjbinx.2020.100077

12.

Lewis

J. R.

Sauro

(2018). Item benchmarks for the system usability scale. Journal of Usability Studies, 13(3), 158–167. https://uxpajournal.org/item-benchmarks-system-usability-scalesus/

13.

MacKenzie

I. S.

Soukoreff

R. W.

(2003). Phrase sets for evaluating text entry techniques. CHI’ 03 Extended Abstracts on Human Factors in Computing Systems, 754–755. https://doi.org/10.1145/765891.765971

14.

Microsoft. (2022). HoloLens 2. https://www.microsoft.com/enus/hololens

15.

Newnham

(2017). Microsoft HoloLens by example: Create immersive augmented reality experiences. Packt Publishing Ltd.

16.

OSHA (2013). OSHA Technical Manual (OTM). https://www.osha.gov/otm

17.

Park

Bokijonov

Choi

(2021). Review of Microsoft HoloLens applications over the past five years. Applied Sciences, 11(16), 7259. https://doi.org/10.3390/app11167259

18.

Ruan

Wobbrock

J. O.

Liou

Landay

(2016). Speech is 3x faster than typing for English and mandarin text entry on mobile devices. arXiv preprint arXiv:1608.07323. https://hci.stanford.edu/research/speech/paper/speech_paper.pdf

19.

Shelstad

W. J.

Benedict

J. D.

Smith

J. K.

Momo

Guliuzo

J. D.

Reuss

N. R.

…Chaparro

B. S.

(2019, January). User experience evaluation of the Microsoft HoloLens for student-related tasks [Poster Presentation]. Human Factors and Applied Psychology Student Conference, Orlando, FL.

20.

Sinlapanuntakul

Derby

J. L.

Chaparro

B. S.

(2022). Understanding the effects of mixed reality on video game satisfaction, enjoyment, and performance. Simulation & Gaming, 53(3), 237–252. https://doi.org/10.1177/10468781221094473

21.

Smith

A. L.

Chaparro

B. S.

(2015). Smartphone text input method performance, usability, and preference with younger and older adults. Human factors, 57(6), 1015–1028. https://doi.org/10.1177/0018720815575644

22.

Strange

(2019). Microsoft’s HoloLens 2 team answers more questions about biometric security, audio, and hand tracking. Next Reality. https://HoloLens.reality.news/news/microsofts-HoloLens -2-team-answers-more-questions-about-biometricsecurity-audio-hand-tracking-0194712/

23.

Texas A&M University EHS (2022). Noise thermometer. https://ehs.tamu.edu/programs/hearing-conservation/

24.

Vovk

Wild

Guest

Kuula

(2018). Simulator sickness in augmented reality training using the Microsoft HoloLens. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–9. https://doi.org/10.1145/3173574.3173783

25.

Vrellis

Delimitros

Chalki

Gaintatzis

Bellou

Mikropoulos

T. A.

(2020). Seeing the unseen: User experience and technology acceptance in augmented reality science literacy. 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT), 333–337. https://doi.org/10.1109/ICALT49669.2020.00107