Background. Understanding how sound functions on informational and emotional levels within video games is critical to understanding player experience of games. User interface sounds, such as player-character health, are a pivotal component of gameplay across many video game genres, yet have not been studied in detail.
Method. To address this research gap in user interface sounds, we present two studies: The first study examines the impact of the presence or absence of player-health sounds on player experience. The second study explores the impact of the types of sound used to indicate player health. We use mixed methods with qualitative and physiological measures.
Results. Our results reveal that despite the presence of visual cues, sound is still important to game design for conveying health-related information and that the type of sound affects player experience.
BlattnerM. M.SumikawaD. A.GreenbergR. M. (1989). Earcons and icons: Their structure and common design principles. Human–Computer Interaction, 4(1), 11-44. doi:10.1016/0003-6870(90)90160-y
2.
BradleyM. M. (1994). Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49-59. doi:10.1016/0005-7916(94)90063-9
3.
BrewsterS. A. (1998). Using non-speech sounds to provide navigation cues. ACM Transactions on Computer-Human Interaction (TOCHI), 5(3), 224-259. doi:10.1145/292834.292839
4.
BrewsterS. A.WrightP. C.EdwardsA. D. N. (1993). An evaluation of earcons for use in auditory human-computer interfaces. In Proceedings of the INTERACT ‘93 and CHI ‘93 Conference on Human Factors in Computing Systems (CHI ‘93) (pp. 222–227). New York, NY, USA: Association of Computing Machinery (ACM). doi:10.1145/169059.169179
5.
CollinsK. (2007). An introduction to the participatory and non-linear aspects of video games audio. In HawkinsS.RichardsonJ. (Eds.), Essays on sound and vision (pp. 263-298). Helsinki, Finland: Helsinki University Press.
6.
CollinsK. (2008). Game sound: An introduction to the history, theory, and practice of video game music and sound design. Cambridge, MA: MIT Press.
7.
CollinsK. (Ed.). (2011). From Pac-Man to pop music: Interactive audio in games and new media. Aldershot, UK: Ashgate.
8.
DixonM. J.HarriganK. A.SantessoD. L.GraydonC.FugelsangJ. A.CollinsK. (2014). The impact of sound in modern multiline video slot machine play. Journal of Gambling Studies, 30(4), 913-929. doi:10.1007/s10899-013-9391-8
9.
EndsleyM. R. (2001). Designing for situation awareness in complex systems. Proceedings of the Second Intenational Workshop on Symbiosis of Humans, Artifacts and Environment, Kyoto, Japan.
10.
EndsleyM. R.JonesD. B. (2016). Designing for situation awareness: An approach to user-centered design. Cleveland, OH: CRC Press.
11.
FagerholtE.LorentzonM. (2009). Beyond the HUD-user interfaces for increased player immersion in FPS games (Master’s thesis). Chalmers University of Technology, Gothenburg, Sweden.
12.
GarciaJ. M. (2012). From heartland values to killing prostitutes: An overview of sound in the video game grand theft auto liberty city stories. In Audio Mostly 2006: A Conference on Sound in Games, AM 2006-Proceedings (pp. 22-25). Proceedings of the Audio Mostly Conference- a Conference on Sound in Games, October 11-12, 2006, Piteå, Sweden. Published online: http://audiomostly.com/wp-content/uploads/2013/12/amc_proceedings_low.pdf
13.
GarnerT.GrimshawM.NabiD. A. (2010). A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game. In Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound (AM ’10) (Article 10). New York, NY: Association of Computing Machinery (ACM). doi:10.1145/1859799.1859809
14.
GaverW. W. (1986). Auditory icons: Using sound in computer interfaces. Human-Computer Interaction, 2(2), 167-177. doi:10.1207/s15327051hci0401_3
15.
GoldsteinE. B. (1989). Sensation and perception. Pacific Grove, CA: Brooks/Cole.
16.
GoodwinS. N. (2009). 3-D sound for 3-D games-beyond 5.1. In AES 35th International Conference, February 11-13, 2009 (pp.1-8), London, UK.
17.
GrimshawM.SchottG. (2008). A conceptual framework for the analysis of first-person shooter audio and its potential use for game engines. International Journal of Computer Games Technology, 2008, Article 5. doi:10.1155/2008/720280
18.
HébertS., S. J. (2005). Physiological stress response to video-game playing: The contribution of built-in music. Life Sciences, 76(20), 2371-2380. doi:10.1016/j.lfs.2004.11.011
19.
JohnsonD.WilesJ. (2003). Effective affective user interface design in games. Ergonomics, 46(13-14), 1332-1345. doi:10.1080/00140130310001610865
20.
JørgensenK. (2007). On transdiegetic sounds in computer games. Northern Lights: Film & Media Studies Yearbook, 5(1), 105-117. doi:10.1386/nl.5.1.105_1
21.
JørgensenK. (2008). Left in the dark: Playing computer games with the sound turned off. In CollinsK. (Ed.), From Pac-Man to pop music (pp. 163-176). Aldershot, UK: Ashgate.
22.
JørgensenK. (2010). Time for new terminology? Diegetic and nondiegetic sounds in computer games revisited. In GrimshawM. (Ed.), Game sound technology and player interaction: Concepts and developments (pp. 78-97). Hershey, PA: IGI Global.
23.
JørgensenK. (2012). Between the game system and the fictional world a study of computer game interfaces. Games and Culture, 7(2), 142-163. doi:10.1177/1555412012440315
KramerG. (1994). An introduction to auditory display. In KramerG. (Ed.), SFI Studies in the Sciences of Complexity, Proceedings. Auditory display: Sonification, audification and auditory interfaces (Santa Fe Institute Studies in the Sciences of Complexity Proceedings), July 5, 1994 (pp. 1-77). Addison-Wesley: Westview Press.
MacaulayC.CrerarA. (1998). Observing the workplace soundscape: Ethnography and auditory interface design. Presented at ICAD 98, The Fifth International Conference on Auditory Display, November 1-4, 1998. University of Glasgow, UK.
29.
MandrykR. L.NackeL. E. (2016). Biometrics in gaming and entertainment technologies. In MitraS.GofmanM. (Eds.), Biometrics in a data driven world: Trends, technologies, and challenges (pp. 197-224). Boca Raton, FL: CRC Press.
30.
McAuleyE.DuncanT.TammenV. V. (1989). Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: A confirmatory factor analysis. Research Quarterly for Exercise and Sport, 60(1), 48-58. doi:10.1080/02701367.1989.10607413
31.
McGookinD. K.BrewsterS. A. (2004). Understanding concurrent earcons: Applying auditory scene analysis principles to concurrent earcon recognition. ACM Transactions on Applied Perception, 1(2), 130-155. doi:10.1145/1024083.1024087
32.
McGregorI.LeplatreG.CrerarA.BenyonD. (2006, June20-23). Sound and soundscape classification: Establishing key auditory dimensions and their relative importance. Proceedings of the 12th International Conference on Auditory Display, London, England.
33.
MillerD.PareckiA.DouglasS. A. (2007). Finger dance: A sound game for blind people. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 253-254). New York, NY, USA: Association of Computing Machinery (ACM). doi:10.1145/1296843.1296898
34.
NackeL. E. (2015). Games user research and physiological game evaluation. In BernhauptR. (Ed.), Game user experience evaluation (pp. 63-86). Cham (ZG), Switzerland: Springer. doi:10.1007/978-3-319-15985-0_4
35.
NackeL. E.GrimshawM. (2011). Player-game interaction through affective sound. In Game Sound Technology and Player Interaction: Concepts and Developments (pp. 264-285). Hershey, PA: IGI Global. doi:10.4018/978-1-61692-828-5.ch013
36.
NackeL. E.GrimshawM. N.LindleyC. A. (2010). More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game. Interacting with Computers, 22(5), 336-343. doi:10.1016/j.intcom.2010.04.005
37.
NackeL. E.WehbeR. R.StahlkeS. N.NoguieraP. A. (2016). Games of the heart and mind: Affective ludology and the development of emotionally aware player experiences. In PerronB.SchröterF. (Eds.), Video games and the mind: Essays on cognition, affect and emotion (pp. 105-125). Jefferson, NC: McFarland.
38.
NgP.NesbittK. (2013). Informative sound design in video games. In Proceedings of The 9th Australasian Conference on Interactive Entertainment: Matters of Life and Death (IE ‘13) (Article 9, 9 pages). New York, NY, USA: ACM. doi:10.1145/2513002.2513015
39.
OakleyI.BrewsterS. A.GrayP. D. (2000). Communicating with Feeling. In BrewsterS. A.Murray-SmithR. (Eds.)., Proceedings of the First International Workshop on Haptic Human-Computer Interaction (pp. 61-68). London, UK: Springer-Verlag.
40.
PaoL. Y.LawrenceD. A. (1998). Synergistic visual/haptic computer interfaces. In Proceedings of Japan/USA/Vietnam Workshop on Research and Education in Systems, Computation, and Control Engineering (pp. 155-162). doi:10.1109/mcg.2004.60
41.
PerronB. (2004). Sign of a threat: The effects of warning systems in survival horror games. In ClarkeA. (Ed.), COSIGN 2004 ProceedingsSeptember 14–16, 2004 (pp. 132-141). Croatia: University of Split. Retrieved from http://www.cosignconference.org/downloads/papers/perron_cosign_2004.pdf
42.
RobbJ. (2015). Beyond the noise: An exploration of informative sound design in video games (Master’s thesis). University of Ontario Institute of Technology.
43.
RyanR. M.RigbyC. S.PrzybylskiA. (2006). The motivational pull of video games: A self-determination theory approach. Motivation and Emotion, 30(4), 344-360. doi:10.1007/s11031-006-9051-8
44.
SkalskiP.LangeR.TamboriniR. (2006). Mapping the way to fun: The effect of video game interfaces on presence and enjoyment. In BrackenC. C. (Ed.), Proceedings of the Ninth Annual International Workshop on Presence. August 24-26, 2006, Cleveland, Ohio, USA: Temple University. Retrieved from http://astro.temple.edu/~lombard/ISPR/Proceedings/2006/P2006proceedings.pdf
45.
WatsonW.ClarkL. A.TellegenA. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063-1070. doi:10.1037/0022-3514.54.6.1063
46.
WelchR. B.WarrenD. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88(3), 638-667. doi:10.1037/0033-2909.88.3.638
47.
WhartonA.CollinsK. (2011). Subjective measures of the influence of music customization on the video game play experience: A pilot study. Game Studies: The International Journal of Computer Game Research, 11(2). Retrieved from http://gamestudies.org/1102/articles/wharton_collins
48.
WolfM. J. P. (1997). Inventing space: Toward a taxonomy of on-and off-screen space in video games. Film Quarterly, 51(1), 11-23. doi:10.2307/1213527
