Developing a Self-Report-Based Sequential Analysis Method for Educational Technology Systems

Abstract

The development of a usability evaluation method for educational systems or applications, called the self-report-based sequential analysis, is described herein. The method aims to extend the current practice by proposing self-report-based sequential analysis as a new usability method, which integrates the advantages of self-report in survey research (e.g., questionnaires and interviews) and lag sequential analysis to evaluate educational systems or applications and further understand students’ operational process patterns of behaviors toward the systems’ interfaces. An empirical study using a drill-based game as a platform was conducted to test the effectiveness of the self-report-based sequential analysis method. The results showed the critical mediators and behavioral transfer patterns among the different area of interests during the students’ overall operational process behaviors. For example, the health points and badges as rewards served as critical mediators between the rewards and feedback and health points and countdown timer, respectively. The self-report-based sequential analysis helped explore the participants’ operational process behaviors in the area of interests, and why behaviors shift on area of interests that can assist to diagnose the inappropriate arrangement of area of interests. Suggestions are provided based on the results.

Keywords

usability testing behavioral pattern analysis interactive learning environment sequential analysis

Get full access to this article

View all access options for this article.

References

Alessi

S. M.

Trollip

S. R.

(2001) Multimedia for learning: Methods and development, Boston, MA: Allyn and Bacon.

Ardito

Costabile

M. F.

De Marsico

Lanzilotti

Levialdi

Roselli

Rossano

(2006) An approach to usability evaluation of e-learning applications. Universal Access in the Information Society 4(3): 270–283.

Bakeman

(1986) Observing interaction: An introduction to sequential analysis, New York, NY: Cambridge University Press.

Bakeman

Gottman

J. M.

(1997) Observing interaction: An introduction to sequential analysis, 2nd ed. Cambridge, England: Cambridge University Press.

Bakeman

Quera

(1995) Analyzing interaction: Sequential analysis with SDIS and GSEQ, New York, NY: Cambridge University Press.

Carver

C. S.

Scheier

M. E.

(1981) Attention and self-regulation: A control-theory approach to human behavior, New York, NY: Springer-Verlag.

Cooke

N. J.

Salas

Cannon-Bowers

J. A.

Stout

(2000) Measuring team knowledge. Human Factors 42: 151–173.

Creswell

J. W.

(2007) Qualitative inquiry and research design: Choosing among five traditions, 2nd ed. Thousand Oaks, CA: Sage.

Desurvire, H., Caplan, M., & Toth, J.A. (2004, April). Using heuristics to evaluate the playability of games. In Proceedings of the CHI ’04 Extended Abstracts on Human Factors in Computing Systems (pp. 1509–1512). New York, NY: ACM Press. doi:10.1145/985921.986102.

10.

Ebling, M. R., & John, B. E. (2000). On the contributions of different empirical data in usability testing. In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods and Techniques (pp. 289–296). New York, NY: ACM Press. doi:10.1145/347642.347766.

11.

Environmental Protection Department. (2012). Garbage classification and resource recycling. Retrieved from http://envir.epb.tpc.gov.tw/recycle1/page1-2.asp.

12.

Fukuyama, K. (2000). Effectiveness of social systems sustained by residents’ participation: A collection system of classified resource and rubbish. In Proceedings of the 2000 IEEE International Conference on Systems, Man and Cybernetics (pp. 506–511). Tennessee, USA. doi:10.1109/ICSMC.2000.885043.

13.

Gottman

J. M.

Levenson

R. W.

(1985) A valid procedure for obtaining self-report of affect in marital interaction. Journal of Consulting and Clinical Psychology 53: 151–160.

14.

Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006). The validity of the stimulated retrospective thinkaloud method as measured by eye tracking. In Proceedings of the CHI ’06 SIGCHI Conference on Human Factors in Computing Systems (pp. 1253–1262). New York, NY: ACM Press. doi:10.1145/1124772.1124961.

15.

Henderson

R. D.

Smith

M. C.

Podd

Varelaalvarez

(1995) A comparison of the four prominent user-based methods for evaluating the usability of computer software. Ergonomics 38: 2030–2044.

16.

Hilbert

Redmiles

D. F.

(2000) Extracting usability information from user interface events. ACM Computing Surveys 32(4): 384–421.

17.

Hoeft

Watson

C. L.

Kesler

S. R.

Bettinger

K. E.

Reiss

A. L.

(2008) Gender differences in the mesocorticolimbic system during computer game-play. Journal of Psychiatric Research 42(4): 253–258.

18.

Hornbæk

(2006) Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies 64: 79–102.

19.

Hou

H. T.

(2010) Exploring the behavioural patterns in project-based learning with online discussion: Quantitative content analysis and progressive sequential analysis. Turkish Online Journal of Educational Technology 9(3): 52–60.

20.

Hou

H. T.

(2012) Exploring the behavioral patterns of learners in an educational massively multiple online role-playing game (MMORPG). Computers & Education 58(4): 1225–1233.

21.

Isbister

Schaffer

(2008) Game usability: Advancing the player experience, San Francisco, CA: Morgan Kaufman Publishers.

22.

Hou

H. T.

S. Y.

(2011) Analyzing the social knowledge construction behavioral patterns of an online synchronous collaborative discussion instructional activity using an instant messaging tool: A case study. Computers & Education 57: 1459–1468.

23.

Jacob

R. J.

Karn

K. S.

(2003) Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind 2(3): 573–605.

24.

Kali

Goodyear

Markauskaite

(2011) Researching design practices and design cognition: Contexts, experiences and pedagogical knowledge-in-pieces. Learning, Media, and Technology 36(2): 129–149.

25.

Keller

J. M.

(2010) Motivational design for learning and performance: The ARCS model, New York, NY: Springer Verlag.

26.

Koskey

K. L. K.

Karabenick

S. A.

Woolley

M. E.

Bonney

C. R.

Dever

B. V.

(2010) Cognitive validity of students’ self-reports of classroom mastery goal structure: What students are thinking and why it matters. Contemporary Educational Psychology 35: 254–263.

27.

Kiili

(2005) Content creation challenges and flow experience in educational games: The IT-emperor case. The Internet and Higher Education 8: 183–198.

28.

Laitinen

(2006) Do usability expert evaluation and test provide novel and useful data for game development? Journal of Usability Studies 1(2): 64–75.

29.

Law, L.-C., & Hvannberg, E. T. (2002). Complementarity and convergence of heuristic evaluation and usability test: A case study of UNIVERSAL brokerage platform. In Proceedings of the Second Nordic Conference on Human-Computer Interaction (pp. 71–80). New York, NY: ACM Press. doi:10.1145/572020.572030.

30.

Lin, Y. C., Hsieh, Y. H., Hou, H. T., Yen Y. S., Chou, Y. S., & Chen, H. (2011, November). Designing a web-based drill game to improve learners resources classification abilities: A case study. Paper presented at the 19th International Conference on Computers in Education (ICCE2011), Chiang Mai, Thailand.

31.

Luik

(2007) Characteristics of drills related to development of skills. Journal of Computer Assisted Learning 23: 56–68.

32.

Nielsen

(1993) Usability engineering, San Diego, CA: Academic Press.

33.

Nørgaard, M., & Rau, J. (2007). User testing in the combat zone. In The Proceedings of Workshop of the International Conference on Advances in Computer Entertainment Technology (ACE’ 07), Toronto, Ontario.

34.

Norman

D. A.

Draper

S. W.

(1986) User centered system design: New perspectives on human-computer interaction, Hillsdale, NJ: Lawrence Erlbaum.

35.

Patton

M. Q.

(2002) Qualitative research and evaluation methods, 3rd ed. Thousand Oaks, CA: Sage.

36.

Rasmussen

Pejtersen

A. M.

Goodstein

L. P.

(1994) Cognitive systems engineering, New York, NY: Wiley.

37.

Schell

(2008) The art of game design: A book of lenses, Burlington, MA: Morgan Kaufmann Publishers.

38.

Scoresby

Shelton

B. E.

(2011) Visual perspectives within educational computer games: Effects on presence and flow within virtual immersive learning environments. Instructional Sceince 39: 227–254.

39.

Squires

(1999) Usability and educational software design: Special issue of interacting with computers. Interacting with Computers 11(5): 463–466.

40.

Sung

Y.-T.

Chang

K.-E.

Lee

Y.-H.

W.-C.

(2008) Effects of a mobile electronic guidebook on visitors’ attention and visiting behaviors. Educational Technology & Society 11(2): 67–80.

41.

Terlecki

M. S.

Newcombe

N. S.

Little

(2008) Durable and generalized effects of spatial experience on mental rotation: Gender differences in growth patterns. Applied Cognitive Psychology 22: 996–1013.

42.

Thomas

Macredie

(1994) Games and the design of human-computer interfaces. Educational and Training Technology International 31(2): 134–142.

43.

Tsai

M. J.

Hou

H. T.

Lai

M. L.

Liu

W. Y.

Yang

F. Y.

(2012) Visual attention for solving multiple-choice science problem: An eye-tracking analysis. Computers & Education 58(1): 375–385.

44.

Tullis

Albert

(2010) Measuring the user experience: Collecting, analyzing, and presenting usability metrics, San Francisco, CA: Morgan Kaufmann Publishers.

45.

van den Haak

M. J.

De Jong

M. D. T.

Schellens

P. J.

(2003) Retrospective vs. concurrent think aloud protocols: Testing the usability of an online library catalogue. Behaviour& Information Technology 22(5): 339–351.

46.

van Someren

M. W.

Barnard

Y. F.

Sandberg

J. A. C.

(1994) The think aloud method: A practical guide to modeling cognitive processes, London, England: Academic Press.

47.

Wang

L. C.

Chen

M. P.

(2010) The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning. Innovations in Education and Teaching International 47(1): 49–52.

48.

Wiebe, E. N., & Annetta, L. A. (2007). Animation and narration: Using eye-tracking to understand visual attention distribution. Paper presented at the Annual meeting of the American Educational Research Association, Chicago, IL.