Although a number of researchers consider serious games as effective teaching/learning tools, the literature is fragmented when it comes to the factors that shape the users’ experience and views. We are currently lacking a comprehensive tool that simultaneously examines their effectiveness while contrasting user views. The study is an attempt to fill this gap. It reports the development and validation of a scale which initially included seventy-two items belonging to thirteen factors. A total of 542 university students played two serious games and the aforementioned questionnaire was administered to them. The exploratory and confirmatory factor analysis revealed that twelve factors and fifty-three items should be retained. The final version of the Serious Games Evaluation Scale demonstrated satisfactory reliability and validity. The factorial structure of the questionnaire and its implications for research and practice are also discussed.
Abt, C.C. (1970). Serious Games. New York: Viking Press.
2.
All, A., Castellar, E.P.N. & Van Looy, J. (2015). Towards a conceptual framework for assessing the effectiveness of digital game-based learning. Computers & Education, 88, 29–37. doi:10.1016/j.compedu.2015.04.012.
3.
Bartlett, M.S., Hager, J.C., Ekman, P. & Sejnowski, T.J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36(2), 253–263. doi:10.1017/S0048577299971664.
4.
Blumberg, F.C., Almonte, D.E., Anthony, J.S. & Hashimoto, N. (2013). Serious games: What are they? What do they do? Why should we play them? In K.E.Dill (Ed.), The Oxford Handbook of Media Psychology (pp. 334–351). Oxford University Press.
5.
Brockmyer, J.H., Fox, C.M., Curtiss, K.A., McBroom, E., Burkhart, K.M. & Pidruzny, J.N. (2009). The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology, 45(4), 624–634. doi:10.1016/j.jesp.2009.02.016.
6.
Brooke, J. (1996). SUS – a quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4–7.
7.
Brown, T.A. (2014). Confirmatory Factor Analysis for Applied Research (2nd ed.). New York, NY: Guilford Press.
8.
Buck, R., Khan, M., Fagan, M. & Coman, E. (2018). The User Affective Experience Scale: A measure of emotions anticipated in response to pop-up computer warnings. International Journal of Human–Computer Interaction, 34(1), 25–34. doi:10.1080/10447318.2017.1314612.
9.
Calderón, A. & Ruiz, M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, 87, 396–422. doi:10.1016/j.compedu.2015.07.011.
10.
Cattell, R.B. (1978). The Scientific Use of Factor Analysis. New York: Plenum. doi:10.1007/978-1-4684-2262-7.
11.
Cheng, M.T. & Annetta, L. (2012). Students’ learning outcomes and learning experiences through playing a Serious Educational Game. Journal of Biological Education, 46(4), 203–213. doi:10.1080/00219266.2012.688848.
12.
Cheng, M.T., Lin, Y.W. & She, H.C. (2015). Learning through playing Virtual Age: Exploring the interactions among student concept learning, gaming performance, in-game behaviors, and the use of in-game characters. Computers & Education, 86, 18–29. doi:10.1016/j.compedu.2015.03.007.
13.
Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T. & Boyle, J.M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661–686. doi:10.1016/j.compedu.2012.03.004.
14.
Costello, A.B. & Osborne, J.W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 1–9.
15.
Couceiro, R.M., Papastergiou, M., Kordaki, M. & Veloso, A.I. (2013). Design and evaluation of a computer game for the learning of Information and Communication Technologies (ICT) concepts by physical education and sport science students. Education and Information Technologies, 18(3), 531–554. doi:10.1007/s10639-011-9179-3.
16.
Davis, F.D., Bagozzi, R.P. & Warshaw, P.R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982–1003. doi:10.1287/mnsc.35.8.982.
17.
de Freitas, S. (2018). Are games effective learning tools? A review of educational games. Journal of Educational Technology & Society, 21(2), 74–84.
18.
DeVellis, R.F. (2016). Scale Development: Theory and Applications (Vol. 26). Sage Publications.
19.
Erhel, S. & Jamet, E. (2019). Improving instructions in educational computer games: Exploring the relations between goal specificity, flow experience and learning outcomes. Computers in Human Behavior, 91, 106–114. doi:10.1016/j.chb.2018.09.020.
20.
Ermi, L. & Mäyrä, F. (2005). Fundamental components of the gameplay experience: Analysing immersion. Worlds in Play: International Perspectives on Digital Games Research, 37(2), 37–53.
21.
Faizan, N., Löffler, A., Heininger, R., Utesch, M. & Krcmar, H. (2019). Classification of evaluation methods for the effective assessment of simulation games: Results from a literature review. International Journal of Engineering Pedagogy, 9(1), 19–33. doi:10.3991/ijep.v9i1.9948.
22.
Finney, S.J. & DiStefano, C. (2013). Nonnormal and categorical data in structural equation modeling. In G.R.Hancock and R.O.Mueller (Eds.), Structural Equation Modeling: A Second Course (2nd ed., pp. 269–314). Charlotte, NC: IAP.
23.
Fokides, E. & Atsikpasi, P. (2018). Factors affecting primary school students’ learning outcomes when using MUVEs. Development and validation of a scale. In J.Y.Qian (Ed.), Integrating Multi-User Virtual Environments in Modern Classrooms (pp. 185–206). Hershey, PA: IGI Global. doi:10.4018/978-1-5225-3719-9.ch009.
24.
Fokides, E., Atsikpasi, P., Kaimara, P. & Deliyannis, I. (2019). Serious games: Which factors players consider important for their learning and gaming experience? Manuscript submitted for publication.
25.
Fu, F.L., Su, R.C. & Yu, S.C. (2009). EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52(1), 101–112. doi:10.1016/j.compedu.2008.07.004.
26.
Garris, R., Ahlers, R. & Driskell, J.E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467. doi:10.1177/1046878102238607.
27.
Gilbert, L. & Gale, V. (2007). Principles of E-Learning Systems Engineering. Elsevier. doi:10.1533/9781780631196.
28.
Girard, C., Ecalle, J. & Magnan, A. (2013). Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning, 29(3), 207–219. doi:10.1111/j.1365-2729.2012.00489.x.
29.
Hair, J.F., Black, W.C., Babin, B.J. & Anderson, R.E. (2010). Multivariate Data Analysis: International Version (7th ed.). New Jersey: Pearson.
30.
Hancock, G.R. (2001). Rethinking construct reliability within latent variable systems. In Structural Equation Modeling: Present and Future (pp. 195–216). Scientific Software International.
31.
Hersh, M. & Leporini, B. (2018). Serious games, education and inclusion for disabled people. British Journal of Educational Technology, 49(4), 587–595. doi:10.1111/bjet.12650.
32.
Hu, L.T. & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. doi:10.1080/10705519909540118.
33.
Huang, W.D., Johnson, T.E. & Han, S.H.C. (2013). Impact of online instructional game features on college students’ perceived motivational support and cognitive investment: A structural equation modeling study. The Internet and Higher Education, 17, 58–68. doi:10.1016/j.iheduc.2012.11.004.
34.
Huang, W.H., Huang, W.Y. & Tschopp, J. (2010). Sustaining iterative game playing processes in DGBL: The relationship between motivational processing and outcome processing. Computers & Education, 55(2), 789–797. doi:10.1016/j.compedu.2010.03.011.
35.
Ifenthaler, D., Eseryel, D. & Ge, X. (2012). Assessment for game-based learning. In Assessment in Game-Based Learning (pp. 1–8). New York, NY: Springer. doi:10.1007/978-1-4614-3546-4_1.
36.
IJsselsteijn, W., De Kort, Y.A.W. & Poels, K. (2013). The Game Experience Questionnaire. Eindhoven: Technische Universiteit Eindhoven. Retrieved from https://pure.tue.nl/ws/files/21666907/GaMediume_Experience_Questionnaire_English.pdf.
37.
Ivory, J.D. & Kalyanaraman, S. (2007). The effects of technological advancement and violent content in video games on players’ feelings of presence, involvement, physiological arousal, and aggression. Journal of Communication, 57(3), 532–555. doi:10.1111/j.1460-2466.2007.00356.x.
38.
Kaiser, H.F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151. doi:10.1177/001316446002000116.
Khan, A. & Webster, J. (2017). Digital game narrative quality: Developing a measure. In Proceedings of the Thirty-Eighth International Conference on Information Systems, Seoul.
41.
Khenissi, M.A., Essalmi, F. & Jemni, M. (2015). Comparison between serious games and learning version of existing games. Procedia – Social and Behavioral Sciences, 191, 487–494. doi:10.1016/j.sbspro.2015.04.380.
42.
Kiili, K. (2005). Digital game-based learning: Towards an experiential gaming model. The Internet and Higher Education, 8(1), 13–24. doi:10.1016/j.iheduc.2004.12.001.
43.
Kline, R.B. (2005). Principles and Practice of Structural Equation Modeling (2nd ed.). New York: Guilford Press.
44.
Koeffel, C., Hochleitner, W., Leitner, J., Haller, M., Geven, A. & Tscheligi, M. (2010). Using heuristics to evaluate the overall user experience of video games and advanced interaction games. In R.Bernhaupt (Ed.), Evaluating User Experience in Games (pp. 233–256). London: Springer. doi:10.1007/978-1-84882-963-3_13.
45.
Lallemand, C., Gronier, G. & Koenig, V. (2015). User experience: A concept without consensus? Exploring practitioners’ perspectives through an international survey. Computers in Human Behavior, 43, 35–48. doi:10.1016/j.chb.2014.10.048.
46.
Lamb, R.L., Annetta, L., Firestone, J. & Etopio, E. (2018). A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while using serious educational games, serious games, and simulations. Computers in Human Behavior, 80, 158–167. doi:10.1016/j.chb.2017.10.040.
47.
Marsh, T. (2011). Serious games continuum: Between games for purpose and experiential environments for purpose. Entertainment Computing, 2(2), 61–68. doi:10.1016/j.entcom.2010.12.004.
48.
Martens, R., Bastiaens, T. & Kirschner, P.A. (2007). New learning design in distance education: The impact on student perception and motivation. Distance Education, 28(1), 81–93. doi:10.1080/01587910701305327.
49.
Matsunaga, M. (2010). How to factor-analyze your data right: Do’s, don’ts, and how-to’s. International Journal of Psychological Research, 3(1), 97–110. doi:10.21500/20112084.854.
50.
McDonald, R.P. & Ho, M.R. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. doi:10.1037/1082-989X.7.1.64.
51.
Mortara, M., Catalano, C.E., Bellotti, F., Fiucci, G., Houry-Panchetti, M. & Petridis, P. (2014). Learning cultural heritage by serious games. Journal of Cultural Heritage, 15(3), 318–325. doi:10.1016/j.culher.2013.04.004.
52.
Muratet, M., Viallet, F., Torguet, P. & Jessel, J. (2009). Une ingénierie pour jeux sérieux [Engineering for serious games]. In Proceedings of Conférence EIAH-Atelier Jeux Sérieux (pp. 53–63). France: Le Mans.
53.
Muthén, B. & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189. doi:10.1111/j.2044-8317.1985.tb00832.x.
54.
Norris, A.E. & Aroian, K.J. (2004). To transform or not transform skewed data for psychometric analysis: That is the question!Nursing Research, 53(1), 67–71. doi:10.1097/00006199-200401000-00011.
55.
Novak, T.P., Hoffman, D.L. & Yung, Y.-F. (2000). Measuring the customer experience in online environments: A structural modeling approach. Marketing Science, 19(1), 22–42. doi:10.1287/mksc.19.1.22.15184.
56.
O’Brien, S. (2016). Why User Experience Design Is Critical to Driving and Maintaining User Engagement and Motivation for Online and Mobile Educational Tools?Proceedings of the Future of Education Conference (Vol. 328). Libreriauniversitaria Edizioni.
57.
O’Connor, B.P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, and Computers, 32, 396–402. doi:10.3758/BF03200807.
Phan, M.H., Keebler, J.R. & Chaparro, B.S. (2016). The development and validation of the Game User Experience Satisfaction Scale (GUESS). Human Factors, 58(8), 1217–1247. doi:10.1177/0018720816669646.
60.
Pinelle, D., Wong, N. & Stach, T. (2008). Heuristic evaluation for games: Usability principles for video game design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1453–1462). New York: ACM. doi:10.1145/1357054.1357282.
61.
Raubenheimer, J. (2004). An item selection procedure to maximise scale reliability and validity. SA Journal of Industrial Psychology, 30(4), 59–64. doi:10.4102/sajip.v30i4.168.
62.
Ravyse, W.S., Blignaut, A.S., Leendertz, V. & Woolner, A. (2017). Success factors for serious games to enhance learning: A systematic review. Virtual Reality, 21(1), 31–58. doi:10.1007/s10055-016-0298-4.
63.
Rebelo, F., Noriega, P., Duarte, E. & Soares, M. (2012). Using virtual reality to assess user experience. Human Factors, 54(6), 964–982. doi:10.1177/0018720812465006.
64.
Selwyn, N. (1997). Students’ attitudes toward computers: Validation of a computer attitude scale for 16–19 education. Computers & Education, 28, 35–41. doi:10.1016/S0360-1315(96)00035-8.
65.
Serrano-Laguna, Á., Manero, B., Freire, M. & Fernández-Manjón, B. (2018). A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes. Multimedia Tools and Applications, 77(2), 2849–2871. doi:10.1007/s11042-017-4467-6.
66.
Shi, Y.R. & Shih, J.L. (2015). Game factors and game-based learning design model. International Journal of Computer Games Technology, 2015, 11 pp. doi:10.1155/2015/549684.
67.
Squire, K. & Jenkins, H. (2003). Harnessing the power of games in education. Insight, 3(1), 5–33.
68.
Steiner, C., Hollins, P., Kluijfhout, E., Dascalu, M., Nussbaumer, A., Albert, D. & Westera, W. (2015). Evaluation of serious games: A holistic approach. Retrieved from http://dspace.ou.nl/bitstream/1820/6139/1/RAGE_ICERI2015_final.pdf.
69.
Tabachnick, B.G. & Fidell, L.S. (2007). Using Multivariate Statistics. Boston: Pearson Education Inc.
70.
Tamborini, R., Bowman, N.D., Eden, A., Grizzard, M. & Organ, A. (2010). Defining media enjoyment as the satisfaction of intrinsic needs. Journal of Communication, 60(4), 758–777. doi:10.1111/j.1460-2466.2010.01513.x.
71.
Usability-First (2009). Playability definition. Retrieved from http://www.usabilityfirst.com/glossary/playability/.
72.
Voida, A. & Greenberg, S. (2012). Console gaming across generations: Exploring intergenerational interactions in collocated console gaming. Universal Access in the Information Society, 11(1), 45–56. doi:10.1007/s10209-011-0232-1.
73.
Wang, L., Fan, X. & Willson, V.L. (1996). Effects of nonnormal data on parameter estimates and fit indices for a model with latent and manifest variables: An empirical study. Structural Equation Modeling: A Multidisciplinary Journal, 3(3), 228–247. doi:10.1080/10705519609540042.
74.
Witmer, B.G. & Singer, M.J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), 225–240. doi:10.1162/105474698565686.
75.
Worthington, R.L. & Whittaker, T.A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. doi:10.1177/0011000006288127.
76.
Wu, B.F. & Lin, C.H. (2018). Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE Access, 6, 12451–12461. doi:10.1109/ACCESS.2018.2805861.
77.
Zeng, Z., Pantic, M., Roisman, G.I. & Huang, T.S. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58. doi:10.1109/TPAMI.2008.52.
