In this paper, we present a five-stage framework for developing online experimental cognitive tasks using the Unity game engine. As researchers increasingly conduct cognitive performance research online, there is a need to accurately capture complex behavioral and performance data. Game engines like Unity offer a solution, providing extensive control over stimulus presentation while being able to collect detailed behavioral data. However, the perceived complexity of using these tools may prevent researchers from adopting them as experimental testbeds. This framework helps to address the learning curve associated with utilizing game engines for research purposes by covering topics such as design considerations, data management, and hosting created experiments online with the goal of empowering human factors researchers and practitioners alike to leverage Unity or other game engines for online cognitive assessment and experimentation.
AbbeyJ. D.MeloyM. G. (2017). Attention by design: Using attention checks to detect inattentive respondents and improve data quality. Journal of Operations Management, 53, 63–70. https://doi.org/10.1016/j.jom.2017.06.001
2.
BrayT. (2014). The JavaScript Object Notation (JSON) data interchange format (RFC 7159). Internet Engineering Task Force. https://doi.org/10.17487/RFC7159
3.
BrookesJ.WarburtonM.AlghadierM.Mon-WilliamsM.MushtaqF. (2020). Studying human behavior with virtual reality: The Unity Experiment Framework. Behavior Research Methods, 52, 455–463. https://doi.org/10.3758/s13428-019-01242-0
4.
CrumpM. J.McDonnellJ. V.GureckisT. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLOS ONE, 8(3), e57410. https://doi.org/10.1371/journal.pone.0057410
5.
DamaševičiusR.MaskeliūnasR.BlažauskasT. (2023). Serious games and gamification in healthcare: A meta-review. Information, 14(2), 10. https://doi.org/10.3390/info14020110
DouglasB. D.EwellP. J.BrauerM. (2023). Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. PLOS ONE, 18(3), e0279720. https://doi.org/10.1371/journal.pone.0279720
8.
FinleyA.PenningrothS. (2015). Online versus in-lab: Pros and cons of an online prospective memory experiment. Advances in Psychology Research, 113, 135–162.
9.
GagnéN.FranzenL. (2023). How to run behavioural experiments online: Best practice suggestions for cognitive psychology and neuroscience. Swiss Psychology Open, 3(1). https://doi.org/10.5334/spo.34
10.
GriffinM.MartinoR. J.LoSchiavoC.Comer-CarruthersC.KrauseK. D.StultsC. B.HalkitisP. N. (2022). Ensuring survey research data integrity in the era of internet bots. Quality & Quantity, 56(4), 2841–2852. https://doi.org/10.1007/s11135-021-01234-4
JacksonK. M.ShawT. H.HeltonW. S. (2023a). The effects of dual-task interference on visual search and verbal memory. Ergonomics, 66(1), 125–135. https://doi.org/10.1080/00140139.2022.2138505
13.
JacksonK.ShawT.HeltonW. (2023b). Evaluating the dual-task decrement within a simulated environment: Word recall and visual search. Applied Ergonomics, 106, 103861. https://doi.org/10.1016/j.apergo.2022.103861
14.
MuszyńskiM. (2023). Attention checks and how to use them: Review and practical recommendations. PsyArXiv. https://doi.org/10.31234/osf.io/xyz (replace with correct DOI/preprint URL if available)
15.
RaghavendraA. T. (2010). Event-based data collection engine for serious games. In LeeY.HaradaH.NakatsuH. (Eds.), Entertainment computing—ICEC 2010: 9th International Conference, ICEC 2010, Seoul, Korea, September 8–11, 2010, Proceedings (Vol. 9, pp. 294–301). Springer. https://doi.org/10.1007/978-3-642-15399-0_29
16.
RoddJ. M. (2024). Moving experimental psychology online: How to obtain high quality data when we can’t see our participants. Journal of Memory and Language, 134, 104472. https://doi.org/10.1016/j.jml.2023.104472
17.
ScharingerC.PrislanL.BerneckerK.NinausM. (2023). Gamification of an n-back working memory task: Is it worth the effort? An EEG and eye-tracking study. Biological Psychology, 179, 108545. https://doi.org/10.1016/j.biopsycho.2023.108545
18.
SharpW. H.JacksonK. M.ShawT. H. (2023). The frequency of positive and negative interactions influences relationship equity and trust in automation. Applied Ergonomics, 108, 103961. https://doi.org/10.1016/j.apergo.2023.103961
VidakisN.BarianosA. K.TrampasA. M.PapadakisS.KalogiannakisM.VassilakisK. (2020). In-game raw data collection and visualization in the context of the “ThimelEdu” educational game. In SotiriadisS.MaciaszekL. A.FillH. G. (Eds.), Computer supported education: 11th International Conference, CSEDU 2019, Heraklion, Crete, Greece, May 2–4, 2019, Revised selected papers (Vol. 11, pp. 629–646). Springer. https://doi.org/10.1007/978-3-030-58459-7_30
