Virtual reality (VR) provides an immersive medium for information processing. Gestures offer a natural and intuitive means of translating cognitive processes into physical actions, making them a promising interaction method to support information processing in VR. However, how gestures reflect and map abstract cognitive tasks to physical interactions remains underexplored. To address this gap, this study explored the mental models by which end-users design gestural interactions to support information processing in VR. Using a gesture elicitation method, 8 participants created 445 gestures representing 19 cognitive processes in Bloom’s taxonomy. Five categories of mental models were identified: linguistic-symbolic, spatial-manipulative, metaphoric, social-conventional, and traditional graphical user interface-derived. Furthermore, the study found that the cognitive process categories affected the mental model adoption, with higher-order cognitive processes prompting more human-like interaction. These findings suggest the potential for developing more natural interactions for cognitive tasks and offer guidance for designing gestural interactions in VR.
AlibaliM. W.BoncoddoR.HostetterA. B. (2014). Gesture in reasoning: An embodied perspective. In ShapiroL.SpauldingS. (Eds.), The Routledge handbook of embodied cognition (pp. 150–159). Routledge.
AndersonL. W.KrathwohlD. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of bloom’s taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc.
4.
BloomB. S.EngelhartM. D.FurstE. J.HillW. H.KrathwohlD. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain (pp. 1103–1133). Longman.
5.
BürknerP.-C. (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80, 1–28. https://doi.org/10.18637/jss.v080.i01
6.
BurnsA.XiongC.FranconeriS.CairoA.MahyarN. (2020). How to evaluate data visualizations across different levels of understanding. arXiv. https://doi.org/10.48550/arXiv.2009.01747
LisleL.DavidsonK.GitreE. J. K.NorthC.BowmanD. A. (2021). Sensemaking strategies with immersive space to think [Conference session]. IEEE Virtual Reality and 3D User Interfaces (VR), Lisboa, Portugal (pp. 529–537). IEEE. https://doi.org/10.1109/VR50410.2021.00077
11.
LutzS. T.HuittW. G. (2003). Information processing and memory: Theory and applications. Educational Psychology Interactive, 1, 17.
12.
MoranA.GadepallyV.HubbellM.KepnerJ. (2015). Improving big data visual analytics with interactive virtual reality [Conference session]. IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA (pp. 1–6). IEEE. https://doi.org/10.1109/HPEC.2015.7322473
13.
MorrisM. R.WobbrockJ. O.WilsonA. D. (2010). Understanding users’ preferences for surface gestures [Conference session]. Proceedings of Graphics Interface 2010, Ottawa, ON (pp. 261–268). Canadian Information Processing Society.
14.
PereiraA.WachsJ. P.ParkK.RempelD. (2015). A user-developed 3-D hand gesture set for human–computer interaction. Human Factors, 57(4), 607–621. https://doi.org/10.1177/0018720814559307
15.
SahuP.CogswellM.Rutherford-QuachS.DivakaranA. (2021). Comprehension based question answering using bloom’s taxonomy. arXiv. https://doi.org/10.48550/arXiv.2106.04653
16.
SkulmowskiA.ReyG. D. (2018). Embodied learning: Introducing a taxonomy based on bodily engagement and task integration. Cognitive Research: Principles and Implications, 3(1), 6. https://doi.org/10.1186/s41235-018-0092-9
17.
VarelaF. J.ThompsonE.RoschE. (2017). The embodied mind: Cognitive science and human experience (revised ed.). MIT Press.
18.
VatavuR. D.WobbrockJ. O. (2015). Formalizing agreement analysis for elicitation studies: New measures, significance test, and toolkit. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1325–1334).
19.
VuleticT.DuffyA.HayL.McTeagueC.CampbellG.GrealyM. (2019). Systematic literature review of hand gestures used in human computer interaction interfaces. International Journal of Human-Computer Studies, 129, 74–94. https://doi.org/10.1016/j.ijhcs.2019.03.011
20.
VuleticT.DuffyA.McTeagueC.HayL.BriscoR.CampbellG.GrealyM. (2021). A novel user-based gesture vocabulary for conceptual design. International Journal of Human-Computer Studies, 150, Article 102609. https://doi.org/10.1016/j.ijhcs.2021.102609
21.
WangY.WangY.LiX.ZhaoC.MaN.GuoZ. (2023). A comparative study of the typing performance of two mid-air text input methods in virtual environments. Sensors, 23(15), Article 6988.
22.
WobbrockJ. O.MorrisM. R.WilsonA. D. (2009). User-defined gestures for surface computing [Conference session]. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA (pp. 1083–1092). ACM Press.
23.
WuH.LuoW.PanN.NanS.DengY.FuS.YangL. (2019). Understanding freehand gestures: A study of freehand gestural interaction for immersive VR shopping applications. Human-Centric Computing and Information Sciences, 9(1), Article 43. https://doi.org/10.1186/s13673-019-0204-7
24.
WuH.WangY.QiuJ.LiuJ.ZhangX. (2019). User-defined gesture interaction for immersive VR shopping applications. Behaviour & Information Technology, 38(7), 726–741. https://doi.org/10.1080/0144929X.2018.1552313
