The development of a usability evaluation method for educational systems or applications, called the self-report-based sequential analysis, is described herein. The method aims to extend the current practice by proposing self-report-based sequential analysis as a new usability method, which integrates the advantages of self-report in survey research (e.g., questionnaires and interviews) and lag sequential analysis to evaluate educational systems or applications and further understand students’ operational process patterns of behaviors toward the systems’ interfaces. An empirical study using a drill-based game as a platform was conducted to test the effectiveness of the self-report-based sequential analysis method. The results showed the critical mediators and behavioral transfer patterns among the different area of interests during the students’ overall operational process behaviors. For example, the health points and badges as rewards served as critical mediators between the rewards and feedback and health points and countdown timer, respectively. The self-report-based sequential analysis helped explore the participants’ operational process behaviors in the area of interests, and why behaviors shift on area of interests that can assist to diagnose the inappropriate arrangement of area of interests. Suggestions are provided based on the results.
AlessiS. M.TrollipS. R. (2001) Multimedia for learning: Methods and development, Boston, MA: Allyn and Bacon.
2.
ArditoC.CostabileM. F.De MarsicoM.LanzilottiR.LevialdiS.RoselliT.RossanoV. (2006) An approach to usability evaluation of e-learning applications. Universal Access in the Information Society4(3): 270–283.
3.
BakemanR. (1986) Observing interaction: An introduction to sequential analysis, New York, NY: Cambridge University Press.
4.
BakemanR.GottmanJ. M. (1997) Observing interaction: An introduction to sequential analysis, 2nd ed. Cambridge, England: Cambridge University Press.
5.
BakemanR.QueraV. (1995) Analyzing interaction: Sequential analysis with SDIS and GSEQ, New York, NY: Cambridge University Press.
6.
CarverC. S.ScheierM. E. (1981) Attention and self-regulation: A control-theory approach to human behavior, New York, NY: Springer-Verlag.
7.
CookeN. J.SalasE.Cannon-BowersJ. A.StoutR. (2000) Measuring team knowledge. Human Factors42: 151–173.
8.
CreswellJ. W. (2007) Qualitative inquiry and research design: Choosing among five traditions, 2nd ed. Thousand Oaks, CA: Sage.
9.
Desurvire, H., Caplan, M., & Toth, J.A. (2004, April). Using heuristics to evaluate the playability of games. In Proceedings of the CHI ’04 Extended Abstracts on Human Factors in Computing Systems (pp. 1509–1512). New York, NY: ACM Press. doi:10.1145/985921.986102.
10.
Ebling, M. R., & John, B. E. (2000). On the contributions of different empirical data in usability testing. In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods and Techniques (pp. 289–296). New York, NY: ACM Press. doi:10.1145/347642.347766.
Fukuyama, K. (2000). Effectiveness of social systems sustained by residents’ participation: A collection system of classified resource and rubbish. In Proceedings of the 2000 IEEE International Conference on Systems, Man and Cybernetics (pp. 506–511). Tennessee, USA. doi:10.1109/ICSMC.2000.885043.
13.
GottmanJ. M.LevensonR. W. (1985) A valid procedure for obtaining self-report of affect in marital interaction. Journal of Consulting and Clinical Psychology53: 151–160.
14.
Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006). The validity of the stimulated retrospective thinkaloud method as measured by eye tracking. In Proceedings of the CHI ’06 SIGCHI Conference on Human Factors in Computing Systems (pp. 1253–1262). New York, NY: ACM Press. doi:10.1145/1124772.1124961.
15.
HendersonR. D.SmithM. C.PoddJ.VarelaalvarezH. (1995) A comparison of the four prominent user-based methods for evaluating the usability of computer software. Ergonomics38: 2030–2044.
16.
HilbertD.RedmilesD. F. (2000) Extracting usability information from user interface events. ACM Computing Surveys32(4): 384–421.
17.
HoeftF.WatsonC. L.KeslerS. R.BettingerK. E.ReissA. L. (2008) Gender differences in the mesocorticolimbic system during computer game-play. Journal of Psychiatric Research42(4): 253–258.
18.
HornbækK. (2006) Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies64: 79–102.
19.
HouH. T. (2010) Exploring the behavioural patterns in project-based learning with online discussion: Quantitative content analysis and progressive sequential analysis. Turkish Online Journal of Educational Technology9(3): 52–60.
20.
HouH. T. (2012) Exploring the behavioral patterns of learners in an educational massively multiple online role-playing game (MMORPG). Computers & Education58(4): 1225–1233.
21.
IsbisterK.SchafferN. (2008) Game usability: Advancing the player experience, San Francisco, CA: Morgan Kaufman Publishers.
22.
HouH. T.WuS. Y. (2011) Analyzing the social knowledge construction behavioral patterns of an online synchronous collaborative discussion instructional activity using an instant messaging tool: A case study. Computers & Education57: 1459–1468.
23.
JacobR. J.KarnK. S. (2003) Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind2(3): 573–605.
24.
KaliY.GoodyearP.MarkauskaiteL. (2011) Researching design practices and design cognition: Contexts, experiences and pedagogical knowledge-in-pieces. Learning, Media, and Technology36(2): 129–149.
25.
KellerJ. M. (2010) Motivational design for learning and performance: The ARCS model, New York, NY: Springer Verlag.
26.
KoskeyK. L. K.KarabenickS. A.WoolleyM. E.BonneyC. R.DeverB. V. (2010) Cognitive validity of students’ self-reports of classroom mastery goal structure: What students are thinking and why it matters. Contemporary Educational Psychology35: 254–263.
27.
KiiliK. (2005) Content creation challenges and flow experience in educational games: The IT-emperor case. The Internet and Higher Education8: 183–198.
28.
LaitinenS. (2006) Do usability expert evaluation and test provide novel and useful data for game development?Journal of Usability Studies1(2): 64–75.
29.
Law, L.-C., & Hvannberg, E. T. (2002). Complementarity and convergence of heuristic evaluation and usability test: A case study of UNIVERSAL brokerage platform. In Proceedings of the Second Nordic Conference on Human-Computer Interaction (pp. 71–80). New York, NY: ACM Press. doi:10.1145/572020.572030.
30.
Lin, Y. C., Hsieh, Y. H., Hou, H. T., Yen Y. S., Chou, Y. S., & Chen, H. (2011, November). Designing a web-based drill game to improve learners resources classification abilities: A case study. Paper presented at the 19th International Conference on Computers in Education (ICCE2011), Chiang Mai, Thailand.
31.
LuikP. (2007) Characteristics of drills related to development of skills. Journal of Computer Assisted Learning23: 56–68.
32.
NielsenJ. (1993) Usability engineering, San Diego, CA: Academic Press.
33.
Nørgaard, M., & Rau, J. (2007). User testing in the combat zone. In The Proceedings of Workshop of the International Conference on Advances in Computer Entertainment Technology (ACE’ 07), Toronto, Ontario.
34.
NormanD. A.DraperS. W. (1986) User centered system design: New perspectives on human-computer interaction, Hillsdale, NJ: Lawrence Erlbaum.
35.
PattonM. Q. (2002) Qualitative research and evaluation methods, 3rd ed. Thousand Oaks, CA: Sage.
36.
RasmussenJ.PejtersenA. M.GoodsteinL. P. (1994) Cognitive systems engineering, New York, NY: Wiley.
37.
SchellJ. (2008) The art of game design: A book of lenses, Burlington, MA: Morgan Kaufmann Publishers.
38.
ScoresbyJ.SheltonB. E. (2011) Visual perspectives within educational computer games: Effects on presence and flow within virtual immersive learning environments. Instructional Sceince39: 227–254.
39.
SquiresD. (1999) Usability and educational software design: Special issue of interacting with computers. Interacting with Computers11(5): 463–466.
40.
SungY.-T.ChangK.-E.LeeY.-H.YuW.-C. (2008) Effects of a mobile electronic guidebook on visitors’ attention and visiting behaviors. Educational Technology & Society11(2): 67–80.
41.
TerleckiM. S.NewcombeN. S.LittleM. (2008) Durable and generalized effects of spatial experience on mental rotation: Gender differences in growth patterns. Applied Cognitive Psychology22: 996–1013.
42.
ThomasP.MacredieR. (1994) Games and the design of human-computer interfaces. Educational and Training Technology International31(2): 134–142.
43.
TsaiM. J.HouH. T.LaiM. L.LiuW. Y.YangF. Y. (2012) Visual attention for solving multiple-choice science problem: An eye-tracking analysis. Computers & Education58(1): 375–385.
44.
TullisT.AlbertW. (2010) Measuring the user experience: Collecting, analyzing, and presenting usability metrics, San Francisco, CA: Morgan Kaufmann Publishers.
45.
van den HaakM. J.De JongM. D. T.SchellensP. J. (2003) Retrospective vs. concurrent think aloud protocols: Testing the usability of an online library catalogue. Behaviour& Information Technology22(5): 339–351.
46.
van SomerenM. W.BarnardY. F.SandbergJ. A. C. (1994) The think aloud method: A practical guide to modeling cognitive processes, London, England: Academic Press.
47.
WangL. C.ChenM. P. (2010) The effects of game strategy and preference-matching on flow experience and programming performance in game-based learning. Innovations in Education and Teaching International47(1): 49–52.
48.
Wiebe, E. N., & Annetta, L. A. (2007). Animation and narration: Using eye-tracking to understand visual attention distribution. Paper presented at the Annual meeting of the American Educational Research Association, Chicago, IL.
