Researchers have called for additional empirical studies associated with video games. However, every game is novel in terms of mechanics, content, context, and agency; known variables may be operationalized differently based on the game involved. It is incumbent upon researchers to leverage or create the best tools when extracting data from games. This article models instrument development of a scale intended to catalog users’ actions in a novel context (i.e., the game League of Legends) using a mixed methods approach. Specifically, this work outlines data collection and validation strategies using an online social news aggregation, rating, and discussion resource (i.e., Reddit), involving multiple cycles to elicit expert input and queries to generate consensus from expert gamers, followed by analysis of responses and an exploratory factor analysis for scale construction. Reddit provided three unique functions relative to the process: (a) access to experts who informed the development of scale items, (b) a social space and mechanism to validate scale items, and (c) the opportunity to capture data necessary to establish the psychometric properties of the instrument. Findings associated with scale development (i.e., item generation, theoretical and psychometric analyses) are presented. Overall, implications for instrument development in continually evolving contexts are discussed.
BaekY. (2017) Game-based learning: Theory, strategies and performance outcomes, Hauppauge, NY: Nova Science.
3.
BarabS.ThomasM.DodgeT.CarteauxR.TuzunH. (2005) Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research & Development53(1): 86–107.
4.
BediouB.AdamsD. M.MayerR. E.TiptonE.GreenC. S.BavelierD. (2018) Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills. Psychological Bulletin144(1): 77–110.
BernackiM. L.WalkingtonC. (2018) The role of situational interest in personalized learning. Journal of Educational Psychology110(6): 864–881. doi:10.1037/edu0000250.
7.
CarrollJ. M.CampbellR. L. (1989) Artifacts as psychological theories: The case of human-computer interaction. Behaviour & Information Technology8(4): 247–256.
8.
Carroll, M., Schrader, P. G., McCreery, M. P., Bacos, C. A., Head, D., & Fiorentini, J. (2018). Virtual reality research as a complex system: A developmental model using established theoretical frameworks. Las Vegas, NV: Association for the Advancement of Computing in Education.
9.
ChappellD.EatoughV.DaviesM.GriffithsN. (2006) EverQuest —It’s just a computer game right? An interpretative phenomenological analysis of online gaming addiction. International Journal of Mental Health and Addiction4(3): 205–216.
10.
ClarkD. B.Tanner-SmithE. E.KillingsworthS. S. (2016) Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research86(1): 79–122. doi:10.3102/0034654315582065.
11.
Collins, K. M. T. (2015). Validity in multimethod and mixed research. In S. N. Hesse-Biber, & R. B. Johnson, (Eds.), The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry. Retrieved from https://doi.org/10.1093/oxfordhb/9780199933624.013.17.
12.
ConnollyT. M.BoyleE. A.MacArthurE.HaineyT.BoyleJ. M. (2012) A systematic literature review of empirical evidence on computer games and serious games. Computers & Education59: 661–686.
13.
CostelloA. B.OsborneJ. W. (2005) Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation10(7): 1–9.
14.
CreswellJ. W.Plano ClarkV. L. (2017) Designing and conducting mixed methods research, 3rd ed. Thousand Oaks, CA: Sage.
15.
DixA. (2009) Human-computer interaction. In: LiuL.ÖzsuM. T. (eds) Encyclopedia of database systems, Boston, MA: Springer, pp. 1327–1331.
EganD. E. (1988) Individual differences in human-computer interaction. In: HelandaerM. (ed) Handbook of human-computer interaction, New York, NY: Elsevier, pp. 543–568. doi:10.1016/B978-0-444-70536-5.50029-4.
18.
FerdigR. E. (2008) Handbook of research on effective electronic gaming in educationVols. I, II, IIIHershey, PA: Information Science Reference.
19.
FettersM. D. (2016) “Haven’t We Always Been Doing Mixed Methods Research?”: Lessons learned from the development of the horseless carriage. Journal of Mixed Methods Research10(1): 3–11. doi:10.1177/1558689815620883.
20.
FongJ.KwanI. (2003) Effective e-learning by using HCI and interactivity on data modeling. International Journal of Computer Processing of Oriental Languages16(4): 293–310.
21.
GeX.IfenthalerD. (2017) Designing engaging educational games and assessing engagement in game-based learning. In: ZhengR.GardnerM. K. (eds) Handbook of research on serious games for educational applications, Hershey, PA: IGI Global, pp. 255–272.
22.
GeeJ. P. (2003) What video games have to teach us about learning and literacy, New York, NY: Palgrave/St. Martin’s Press.
23.
GentileD. A.LynchP. J.LinderJ. R.WalshD. A. (2004) The effects of violent video game habits on adolescent hostility, aggressive behaviors, and school performance. Journal of Adolescence27: 5–22.
24.
GirardC.EcalleJ.MagnanA. (2013) Serious games as new educational tools: How effective are they? a meta-analysis of recent studies. Journal of Computer Assisted Learning29(3): 207–219.
25.
GoodmanC. M. (1987) The Delphi technique: A critique. Journal of Advanced Nursing12: 729–734.
26.
Grime, M. M., & Wright, G. (2016). The Delphi method. In N. Balakrishnan, T. Colton, B. Everitt, W. Piegorsch, F. Ruggeri., & J. L. Teugels (Eds.), Wiley statsref: Statistics reference online (Vol. 1). Hoboken, NJ: Wiley. doi:10.1002/9781118445112.stat07879.
27.
HenryN.GoodellH.ElmqvistN.FeketeJ. D. (2007) 20 Years of four HCI conferences: A visual exploration. International Journal of Human-Computer Interaction23(3): 239–285. doi:10.1080/10447310701702402.
28.
HilpertJ.MarchandG. (2018) Complex systems research in educational psychology: Aligning theory and method. Educational Psychologist53(3): 185–202.
IfenthalerD.EseryelD.GeX. (2012) Assessment in game-based learning: Foundations, innovations, and perspectives, New York, NY: Springer.
31.
JohnsonR. B.OnwuegbuzieA. J. (2004) Mixed methods research: A research paradigm whose time has come. Educational Researcher33(7): 14–26. doi:10.3102/0013189x033007014.
32.
KafaiY. B. (1996) Electronic play worlds: Gender differences in children’s construction of video games. In: KafaiY.ResnickM. (eds) Constructivism in practice: Designing, thinking, and learning in a digital world, Mahwah, NJ: Erlbaum, pp. 97–123.
33.
KarrayF.AlemzadehM.SalehJ. A.ArabM. N. (2008) Human-computer interaction: Overview on state of the art. International Journal on Smart Sensing and Intelligent Systems1: 137–159.
34.
KeF. (2009) A qualitative meta-analysis of computer games as learning tools. In: FerdigR. (ed) Handbook of research on effective electronic gaming in educationVol. IHershey, PA: Information Science Reference, pp. 1–32.
35.
LeechN. L.OnwuegbuzieA. J. (2009) A typology of mixed methods research designs. Quality & Quantity43: 265–275.
Loh, C. S., Sheng, Y., & Ifenthaler, D. (Eds.). (2015). Serious games analytics: Methodologies for performance measurement, assessment, and improvement. New York, NY: Springer.
38.
LundT. (2012) Combining qualitative and quantitative approaches: Some arguments for mixed methods research. Scandinavian Journal of Educational Research56(2): 155–165. doi:10.1080/00313831.2011.568674.
39.
MayerR. E. (2015) On the need for research evidence to guide the design of computer games for learning. Educational Psychologist50(4): 349–353. doi:10.1080/00461520.2015.1133307.
40.
McCreeryM. P.KrachS. K.SchraderP. G.BooneR. (2012) Defining the virtual self: Personality, behavior and the psychology of embodiment. Computers in Human Behavior28(3): 976–983.
41.
McCreeryM. P.SchraderP. G.KrachS. K.BooneR. (2013) A sense of self: The role of presence in virtual environments. Computers in Human Behavior29(4): 1635–1640.
42.
McCreery, M. P., Schrader, P. G., Krach, K., Lafferiere, J. R., Bacos, C. A., & Fiorentini, J. P. (in press). Examining designed experiences: A walkthrough for understanding video games as performance assessments. In D. Ifenthaler and Y. J. Kim (Eds.), Game-based Assessment Revisited. New York, NY: Springer.
43.
McCreeryM. P.VallettD.ClarkC. (2015) Social interaction in a virtual environment: Examining sociospatial interactivity and social presence using behavioral analytics. Computers in Human Behavior51: 203–206.
44.
MorgadoF. F. R.MeirelesJ. F. F.NevesC. M.AmaralA. C. S.FerreiraM. E. C. (2017) Scale development: Ten main limitations and recommendations to improve future research practices. Psicologia: Reflexão e Crítica30(3): 1–20.
45.
MorganG. A.GriegoO. V. (1998) Easy use and interpretation of SPSS for windows: Answering research questions with statistics, Mahwah, NJ: Lawrence Erlbaum.
46.
PellasN.KazanidisI.KonstantinouN.GeorgiouG. (2016) Exploring the educational potential of three-dimensional multi-user virtual worlds for STEM education: A mixed-method systematic literature review. Education and Information Technologies22(5): 2235–2279. doi:10.1007/s10639-016-9537-2.
47.
Salmons, J. E. (2015). Conducting multimethod and mixed methods research online. Oxford Handbooks Online. In S. N. Hesse-Biber, & R. B. Johnson, (Eds.), The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry. Retrieved from https://doi.org/10.1093/oxfordhb/9780199933624.013.32.
48.
SalomonG.PerkinsD. N.GlobersonT. (1991) Partners in cognition: Extending human intelligence with intelligent technologies. Educational Researcher20: 2–9.
49.
SchraderP. G.DenizH.KeiltyJ. (2016) Breaking SPORE: Building instructional value in science education using a commercial, off-the shelf game. Journal of Learning and Teaching in Digital Age1(1): 32–46.
50.
SchraderP. G.LawlessK. A. (2007) Dribble files: Methodologies to evaluate learning and performance in complex environments. Performance Improvement46(1): 40–48.
51.
SchraderP. G.McCreeryM. (2007) The acquisition of skill and expertise in massively multiplayer online games. Educational Technology Research & Development56(5–6): 557–574. doi:10.1007/s11423-007-9055-4.
52.
SchraderP. G.McCreeryM. (2012) Are all games the same? Examining three frameworks for assessing learning from, with, and in games. In: IfenthalerD.EseryelD.GeX. (eds) Assessment in game-based learning: Foundations, innovations, and perspectives, New York, NY: Springer, pp. 11–28.
53.
Schrader, P. G., McCreery, M. P., Carroll, M. C., Head, D., & Lafferiere, J. R. (in press). Assessing learning from, with, and in games revisited: A heuristic for emerging methods and commercial-off-the-shelf games. In D. Ifenthaler, & Y. J. Kim (Eds.), Game-based assessment revisited. New York, NY: Springer.
54.
SchraderP. G.McCreeryM. P.VallettD. B. (2017) Performance in situ. In: BaekY. (ed) Game-based learning: Theory, strategies and performance outcomes, Hauppauge, NY: Nova Science Publishers, pp. 421–437.
55.
ShafferD. W. (2006) How computer games help children learn, New York, NY: Palgrave Macmillan.
56.
Shute, V. J., Ke, F., & Wang, L. (2017). Assessment and adaptation in games. In P. Wouters, & H. van Oostendorp (Eds.), Instructional techniques to facilitate learning and motivation of serious games (pp. 59–78). Switzerland: Springer International Publishing. doi:10.1007/978-3-319-39298-1_4.
57.
SquireK. D. (2006) From content to context: Video games as designed experiences. Educational Researcher35(8): 19–29.
58.
TabachnickB. G.FidellL. S. (2007) Using multivariate statistics (5th ed.), Upper Saddle River, NJ: Pearson Allyn & Bacon.
59.
Tashakkori, A., Teddlie, C., & Johnson, B. (2015). Mixed Methods. International encyclopedia of the social & behavioral sciences (2nd ed., Vol. 15). Amsterdam, the Netherlands: Elsevier. doi:10.1016/B978-0-08-097086-8.10550-1.
60.
TavakolM.DennickR. (2011) Making sense of Cronbach’s alpha. International Journal of Medical Education2: 53–55.
61.
ThangaratinamS.RedmanC. W. E. (2005) The Delphi technique. Education: The Obstetrician & Gynaecologist7: 120–125.
62.
van MerrienboerJ. J. G.KirschnerP. A. (2018) Ten steps to complex learning: A systematic approach to four-component instructional design, 3rd ed. New York, NY: Routledge.
63.
van MerrienboerJ. J. G.SluijsmansD. A. (2009) Toward a synthesis of cognitive load theory, four-component instructional design, and self-directed learning. Educational Psychology Review29: 5566.
64.
VogelJ. J.VogelD. S.Cannon-BowersJ.MuseK.WrightM. (2006) Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research34(3): 229–243. doi:10.2190%2FFLHV-K4WA-WPVQ-H0YM.
65.
WorthingtonR. L.WhittakerT. A. (2006) Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist34(6): 806–838.
66.
WoutersP.Van NimwegenC.Van OostendorpH.Van der SpekE. D. (2013) A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology105(2): 249–265. doi:10.1037/a0031311.
YeeN. (2008) Maps of digital desires: Exploring the topography of gender and play in online games. In: KafaiY. B.HeeterC.DennerD.SunJ. Y. (eds) Beyond Barbie and Mortal Kombat: New perspectives on gender and gaming, Cambridge, MA: MIT Press, pp. 156–183.
69.
YoungM. F.SlotaS.CutterA. B.JaletteG.MullinG.LaiB.YukhymenkoM. (2012) Our princess is in another castle: A review of trends in serious gaming for education. Review of Educational Research82(1): 61–89.
70.
YuxinM.WilliamsD.PrejeanL.RichardC. (2007) A research agenda for developing and implementing educational computer games. British Journal of Educational Technology38(3): 513–518. doi:10.1111/j.1467-8535.2007.00714.x.
71.
ZellnerM.CampbellS. (2015) Planning for deep-rooted problems: What can we learn from aligning complex systems and wicked problems?Planning Theory & Practice16(4): 457–478.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
