Scenario-based Group Usability Testing as a Mixed Methods Approach to the Evaluation of Three-Dimensional Virtual Learning Environments

Abstract

Although virtual reality became popular technology whose application is recognized in various domains, the field generally still lacks a widespread culture of usability. This is also evident when considering environments intended for learning, specifically virtual learning environments (VLEs). According to our findings, it is clear that there is a growing need for systematic evaluation approach to help with the design and development of usable learner-centered VLE solutions. After comprehensive introductory background and state of the art in the field, this article provides an insight into Scenario-based Group Usability Testing (ScerGUT), a mixed methods approach to the evaluation of three-dimensional VLEs which integrates several different methods of usability testing with measurements of educational value. While the majority of the existing work has made use of usually one single usability assessment technique, ScerGUT employs a number of methods putting in focus users and user testing. To examine efficiency and applicability of the approach, empirical validation is conducted as a case study of particular VLE. The contribution of the article is twofold: (a) ScerGUT as a mixed methods approach to the evaluation of VLEs, which brings new scientific value and could help other researchers and (b) ScerGUT’s application to a particular VLE, which brings quantitative and qualitative results, thus providing an insight into ease of use and educational value of specific VLE.

Keywords

virtual learning environments user interface usability evaluation approach use case

Get full access to this article

View all access options for this article.

References

Alsumait

Al-Osaimi

(2010) Usability heuristics evaluation for child E-learning applications. Journal of Software 5(6): 654–661.

Andreas

Tsiatsos

Terzidou

Pomportsis

(2010) Fostering collaborative learning in Second Life: Metaphors and affordances. Computers & Education 55(2): 603–615.

Ardito, C., Costabile, M., Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., & Rossano, V. (2006). An approach to usability evaluation of e-learning applications. Universal Access in the Information Society, 4(3), 270–283.

Bach

Scapin

D. L.

(2010) Comparing inspections and user testing for the evaluation of virtual environments. International Journal of Human–Computer Interaction 26(8): 786–824.

Bailey, B. (2006). Getting the complete picture with usability testing. Retrieved from https://www.usability.gov/get-involved/blog/2006/03/complete-picture-with-usability-testing.html.

Bernérus, A., & Zhang, J. (2010). A peek at the position of pedagogical aspects in usability evaluation of E-learning system—A literature review of usability evaluation of E-learning system conducted since 2000 (University of Gothenburg, Report No. 2010:085). Retrieved from https://gupea.ub.gu.se/bitstream/2077/23482/1/gupea_2077_23482_1.pdf.

Bolchini

Garzotto

(2008) Quality and potential for adoption of web usability evaluation methods: An empirical study on MILE+. Journal of Web Engineering 7(4): 299–317.

Bowman

Gabbard

Hix

(2002) A survey of usability evaluation in virtual environments: Classification and comparison of methods. Presence: Teleoperators and Virtual Environments 11(4): 404–424.

Brooke

(2013) SUS: A retrospective. Journal of Usability Studies 8(2): 29–40.

10.

Calvo-Ferrer

J. R.

(2017) Educational games as stand-alone learning tools and their motivational effect on L2 vocabulary acquisition and perceived learning gains. British Journal of Educational Technology 48(2): 264–278.

11.

Chen

C. J.

Lau

S. Y.

Teh

C. S.

(2015) A feasible group testing framework for producing usable virtual reality learning applications. Virtual Reality 19(2): 129–144.

12.

Chittaro

Ranon

(2007) Web3D technologies in learning, education and training: Motivations, issues, opportunities. Computers & Education 49: 3–18.

13.

Davis

F. D.

(1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3): 319–340.

14.

Dede

Ketelhut

D. J.

Ruess

(2002) Motivation, usability, and learning outcomes in a prototype museum-based multi-user virtual environment. In: P. Bell R. Stevens T. Satwicz (eds) Keeping learning complex: The proceedings of the Fifth International Conference of the Learning Sciences, Mahwah, NJ: Erlbaum, pp. 530–531.

15.

de Noyelles

Hornik

S. R.

Johnson

R. D.

(2014) Exploring the dimensions of self-efficacy in virtual world learning: Environment, task, and content. MERLOT Journal of Online Learning and Teaching 10(2): 255–271.

16.

Doubleday

E. G.

O’Loughlin

V. D.

Doubleday

A. F.

(2011) The virtual anatomy laboratory: Usability testing to improve an online learning resource for anatomy education. Anatomical Sciences Education 4(6): 318–326.

17.

Drettakis

Roussou

Reche

Tsingos

(2007) Design and evaluation of a real-world virtual environment for architecture and urban planning. Presence: Teleoperators and Virtual Environments 16(3): 318–332.

18.

Ewais

De Troyer

(2014) Authoring adaptive 3D virtual learning environments. International Journal of Virtual and Personal Learning Environments 5(1): 1–19.

19.

Firestorm (2018). Retrieved from http://www.firestormviewer.org/.

20.

Granić

(2008) Experience with usability evaluation of e-learning systems. Universal Access in the Information Society 7(4): 209–221.

21.

Granić

Ćukušić

(2011) Usability testing and expert inspections complemented by educational evaluation: A case study of an e-learning platform. Educational Technology & Society 14(2): 107–123.

22.

Granić

Maratou

Mettouris

Papadopoulos

G. A.

Xenos

(2015) Personalized context-aware recommendations in 3D virtual learning environments. TOJET: The Turkish Online Journal of Educational Technology 14: 396–406.

23.

Greenberg, S., & Buxton, B. (2008, April 5–10). Usability evaluation considered harmful (some of the time). In CHI’08 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy (pp. 111–120). New York, NY: ACM Press.

24.

Herrero, R., Bretón-López, J., Farfallini, L., Quero, S., Miralles, I., Baños, R., & Botella, C. (2015). Acceptability and satisfaction of an ICT-based training for university teachers. Educational Technology & Society, 18(4), 498–510.

25.

International Organization for Standardization (1998) Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability (ISO9241–11:1998(E)), Geneva, Switzerland: Author.

26.

Jarmon

Traphagan

Mayrath

Trivedi

(2009) Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life. Computers & Education 53(1): 169–182.

27.

Johnson

Adams Becker

Cummins

Estrada

Freeman

Hall

(2016) NMC horizon report: 2016 higher education edition, Austin, TX: The New Media Consortium.

28.

Kokua. (2018). Retrieved from http://blog.kokuaviewer.org/.

29.

Lanzilotti

Ardito

Costabile

De Angeli

(2006) eLSE methodology: A systematic approach to the e-learning systems evaluation. Educational Technology & Society 9(4): 42–53.

30.

Marsh

Wright

(1999) Co-operative evaluation of a desktop virtual reality system. In: M. Harrison S. Smith (ed) Proceedings of the workshop on user centered design and implementation of virtual environments, York, England: University of York, pp. 99–108.

31.

Mayhew

D. J.

(1999) The usability engineering lifecycle, San Francisco, CA: Morgan Kaufmann.

32.

Mehlenbacher, B., Bennett, L., Bird, T., Ivey, M., Lucas, J., Morton, J., & Whitman, L. (2005). Usable E-learning: A conceptual model for evaluation and design. In Proceedings of HCI International 2005: 11th International Conference on Human–Computer Interaction (Volume 4—Theories, Models, and Processes in HCI) (pp. 1–10). Las Vegas, NV: Mira Digital P.

33.

Merchant

Goetz

E. T.

Keeney-Kennicutt

Kwok

Cifuentes

Davis

T. J.

(2012) The learner characteristics, features of desktop 3D virtual reality environments, and college chemistry instruction: A structural equation modeling analysis. Computers & Education 59(2): 551–568.

34.

Mettouris, C., Maratou, V., Vučkovic, D., Papadopoulos G. A., & Xenos, M. (2015). Information security awareness through a virtual world: An end-user requirements analysis. In Proceedings of the 5th International Conference on Information Society and Technology, ICIST 2015, Kopaonik, Serbia (pp. 273–278). Society for Information Systems and Computer Networks.

35.

Monahan

McArdle

Bertolotto

(2008) Virtual reality for collaborative e-learning. Computers & Education 50(4): 339–1353.

36.

MOODLE. (2018). Retrieved from https://moodle.org/

37.

Nielsen

(1994) Heuristic evaluation. In: J. Nielsen R. L. Mack (ed) Usability inspection methods, New York, NY: John Wiley & Sons, pp. 25–62.

38.

OpenSimulator. (2018). Retrieved from http://opensimulator.org/

39.

Plano Clark

V. L.

Creswell

J. W.

(2015) Understanding research: A consumer’s guide, 2nd ed. Upper Saddle River, NJ: Pearson Education.

40.

Rosli, D. I., Chen, J. C., & Teh, C. S. (2008, June 23–26). Interface design usability testing of virtual reality (VR)-based learning environment. In The 5th International Conference on Information Technology and Applications, ICITA, Queensland, Australia.

41.

Rusu, C., Muñoz, R., Roncagliolo, S., Rudloff, S., Rusu, V., & Figueroa, A. (2011, August 21–27). Usability heuristics for virtual worlds. In The Third International Conference on Advances in Future Internet, AFIN, Nice, France.

42.

Schwebel

D. C.

McClure

L. A.

Severson

(2014) Usability and feasibility of an Internet-based virtual pedestrian environment to teach children to cross streets safely. Virtual Reality 18(1): 5–11.

43.

Scullion

Baxter

Stansfield

(2015) UNITE: Enhancing students’ self-efficacy through the use of a 3D virtual world. Journal of Universal Computer Science 21(12): 1635–1653.

44.

SLOODLE. (2018). Retrieved from http://www.sloodle.org/docs/.

45.

Squires

Preece

(1999) Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting With Computers 11(5): 467–483.

46.

Ssemugabi

de Villiers

(2010) Effectiveness of heuristic evaluation in usability evaluation of e-learning applications in higher education. South African Computer Journal 45: 26–39.

47.

Stanney, K. M., Mollaghasemi, M., & Reeves, L. (2000). Development of MAUVE, the multi-criteria assessment of usability for virtual environment system (Final report). Orlando, FL: Naval Air Warfare Center, Training Systems Division, 8/00.

48.

Stanney

K. M.

Mollaghasemi

Reeves

Breaux

Graeber

D. A.

(2003) Usability engineering of virtual environments (VEs): Identifying multiple criteria that drive effective VE system design. International Journal of Human–Computer Studies 58(4): 447–481.

49.

Sutcliffe

A. G.

Kaur

K. D.

(2000) Evaluating the usability of virtual reality user interfaces. Behaviour & Information Technology 19(6): 415–426.

50.

Tromp

J. G.

Steed

Wilson

J. R.

(2003) Systematic usability evaluation and design issues for collaborative virtual environments. Presence: Teleoperators and Virtual Environments 12(3): 241–267.

51.

Tselios

Avouris

Komis

(2008) The effective combination of hybrid usability methods in evaluating educational applications of ICT: Issues and challenges. Education and Information Technologies 13(1): 55–76.

52.

Uldall-Espersen

Frøkjær

Hornbæk

(2008) Tracing impact in a usability improvement process. Interacting With Computers 20(1): 48–63.

53.

V-ALERT. (2016). Virtual world for awareness and learning on information security (Project Number 543224-LLP-1-2013-1-GR-KA3-KA3MP). Retrieved from http://v-alert.eu/.

54.

Villanueva, R., Moore, A., & Wong, B. L. W. (2004). Usability evaluation of non-immersive, desktop, photo-realistic virtual environments. Presented at the 16th Annual Colloquium of the Spatial Information Research Centre (SIRC 2004: A Spatio-temporal Workshop) (pp. 139–146). Dunedin, New Zealand.

55.

Virvou

Katsionis

(2008) On the usability and likeability of virtual reality games for education: The case of VR-ENGAGE. Computers & Education 50(1): 154–178.

56.

Vosinakis S., & Koutsabasis, P. (2011). A framework for problem-based learning activities in virtual worlds. In Proceedings of the 3rd Conference on Informatics in Education (CIE).

57.

Vosinakis

Koutsabasis

(2013) Interaction design studio learning in virtual worlds. Virtual Reality 17(1): 59–75.

58.

Wang

S. H.

Lin

M. C.

Liao

C. W.

(2014) A virtual experiential learning and students’ ill-structured problem-solving ability. Interacting With Computers 26(4): 334–347.

59.

Yilmaz

R. M.

Reisoglu

Topu

F. B.

Karakus

Goktas

(2015) The development of a criteria list for the selection of 3D virtual worlds to design an educational environment. Croatian Journal of Education 17(4): 1037–1069.

60.

Zaharias

(2009) Usability in the context of e-learning: A framework augmenting ‘traditional’ usability constructs with instructional design and motivation to learn. International Journal of Technology and Human Interaction 5(4): 38–61.

61.

Zaharias

Poylymenakou

(2009) Developing a usability evaluation method for e-learning applications: Beyond functional usability. International Journal of Human–Computer Interaction 25(1): 75–98.