Sage Journals: Discover world-class research

Abstract

This project aimed to develop an engaging formative assessment that simulates the social and dialogic processes that are often missing in traditional instruction and assessment of argumentation. In this article, we describe the design of an argumentative reasoning task within a scenario-based assessment enhanced with game elements (Seaball—Semester at Sea). This Seaball task requires students to classify foods as “junk food” or not, by identifying relevant evidence to support their decisions and evaluating reasoning errors in others’ arguments. Data from a small-scale study of 42 middle school students revealed that students performed significantly better on questions requiring selection of appropriate evidence than questions requiring identification of logical fallacies. Additionally, students’ total scores were significantly correlated with external measures of argumentation skills, and with teacher’s ratings of their argumentative writing skills. Survey indicated that students perceived the game-enhanced scenario-based assessment as engaging. Implications for formative assessment are discussed.

Keywords

scenario-based assessment argumentation game features game-based assessment logical fallacies

Get full access to this article

View all access options for this article.

References

Argubot Academy (2014) Mars Generation One: Argubot Academy. Retrieved from http://www.gamesforchange.org/play/mars-generation-one-argubot-academy/.

Deane, P., & Song, Y. (2014). A Case Study in Principled Assessment Design: Designing Assessments to Measure and Support the Development of Argumentative Reading and Writing Skills, Spanish Journal of Educational Psychology (Psicologia Educativa), 20, 99–108.

Deane, P., & Song, Y. (2015). The key practice, ‘Discuss and Debate Ideas’: Conceptual framework, literature review, and provisional learning progressions for argumentation. (Research Report No. RR-15-33). Princeton, NJ: Educational Testing Service.

Song, Y., Sparks, J. R., Brantley, J. W., Appel, C., Vezzu, M. E., & Jackson, G. T. (2015, April). Assessing argumentation skills using conversations: Compared constrained- and open-response formats in game-based assessment. Poster presented at the annual meeting of American Educational Research Association, Chicago, Illinois.

Song, Y., & Sparks, J. R. (Under Review). Building an innovative formative assessment to gather evidence about middle school studentsã argumentation skills. Educational Technology Research and Development.

Bennett

R. E.

(2010) Cognitively based assessment of, for, and as learning: A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives 8: 70–91.

Burstein

Tetreault

Madnani

(2013) The e-rater automated essay scoring system. In: Shermis

M. D.

Burstein

(eds) Handbook of automated essay evaluation: Current applications and new directions, New York, NY: Routledge, pp. 55–67.

Council of Chief State School Officers & National Governors Association (2010) Common core state standards for English language arts and literacy in history/social studies, science, and technical subjects. Retrieved from www.corestandards.org/the-standards/ELA-Literacy.

Deane, P., Sabatini, J. P., Feng, G., Sparks, J., Song, Y., Fowles, M., . . . Foley, C. (2015). Key practices in the English language arts: Linking learning theory, assessment, and instruction (Research Report 15–17). Princeton, NJ: Educational Testing Service.

10.

Deane

Williams

Weng

Trapani

C. S.

(2013) Automated essay scoring in innovative assessments of writing from sources. The Journal of Writing Assessment 6: 40–56.

11.

Dede

(2007) Transforming education for the 21st century: New pedagogies that help all students attain sophisticated learning outcomes. (Commissioned by the NCSU Friday Institute). . Retrieved from http://tdhahwiki.wikispaces.com/file/view/Dede_21stC-skills_semi-final.pdf.

12.

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: Defining “gamification.” Paper presented at the 15th International Academic MindTrek Conference, Tampere, Finland. Retrieved from https://www.cs.auckland.ac.nz/courses/compsci747s2c/lectures/paul/definition-deterding.pdf.

13.

Ferretti

R. P.

Lewis

W. E.

Andrews-Weckerly

(2009) Do goals affect the structure of students’ argumentative writing strategies? Journal of Educational Psychology 101: 577–589.

14.

Ferretti

R. P.

MacArthur

C. A.

Dowdy

N. S.

(2000) The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normally achieving peers. Journal of Educational Psychology 93: 694–702.

15.

Finn, B. (2015). Measuring motivation in low-stakes assessments (Research Report 15-19). Princeton, NJ: Educational Testing Service.

16.

Heilman, M., & Madnani, N. (2015). The impact of training data on automated short answer scoring performance. In: J. Tetreault, Y. Labs; J. Burstein, Educational Testing Service; Claudia Leacock, McGraw-Hill Education/CTB (Eds.), Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 81−85). Stroudsburg, PA: Association for Computational Linguistics.

17.

Hillocks

Jr. (2002) The testing trap: How state writing assessments control learning, New York, NY: Teachers College Press.

18.

iCivics Inc. (2015). iCivics. Retrieved from http://www.icivics.org/.

19.

Klopfer

Osterweil

Salen

(2009) Moving learning games forward, Cambridge, MA: The Education Arcade.

20.

Kuhn

(1991) The skills of argument, Cambridge, England: Cambridge University Press.

21.

Kuhn

(2009) The importance of learning about knowing: Creating a foundation for development of intellectual values. Child Development Perspectives 89: 112–117.

22.

Kuhn

Crowell

(2011) Dialogic argumentation as a vehiche for developing young adolescents’ thinking. Psychological Science 22: 545–552.

23.

Kuhn

Iordanou

Pease

Wirkala

(2008) Beyond control of variables: What needs to develop to achieve skilled scientific thinking? Cognitive Development 23: 435–451.

24.

Liu

O. L.

Bridgeman

Adler

R. M.

(2012) Measuring learning outcomes in higher education: Motivation matters. Education Researcher 41: 352–362.

25.

Liu

O. L.

Rios

J. A.

Borden

(2015) The effects of motivational instruction on college students’ performance on low-stakes assessment. Educational Assessment 20: 79–94.

26.

McNamara

D. S.

Jackson

G. T.

Graesser

A. C.

(2010) Intelligent tutoring and games (ITaG). In: Baek

Y. K.

(ed.) Gaming for classroom-based learning: Digital role-playing as a motivator of study, Hershey, PA: IGI Global, pp. 44–65.

27.

Messick

(1994) The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher 23(2): 13–23.

28.

Mislevy

R. J.

Oranje

Bauer

von Davier

A. A.

Hao

Corrigan

John

(2014) Psychometric considerations in game-based assessment, Redwood City, CA: GlassLab.

29.

Mislevy

R. J.

Steinberg

L. S.

Almond

R. G.

(2003) On the structure of educational assessment (with discussion). Measurement: Interdisciplinary Research and Perspective 1: 3–62.

30.

Northwest Evaluation Association. (2015). Measures of academic progress (MAP) interim assessments for Grades K-12. Retrieved from https://www.nwea.org/content/uploads/2014/07/Introduction-to-MAP-Brochure-SEPT15.pdf.

31.

Nussbaum

E. M.

(2008) Collaborative discourse, argumentation, and learning: Preface and literature review. Contemporary Educational Psychology 33: 345–359.

32.

Nussbaum

M. E.

Kardash

C. M.

(2005) The effects of goal instructions and text on the generation of counterarguments during writing. Journal of Educational Psychology 97: 157–169.

33.

Persky

H. R.

Daane

M. C.

Jin

(2003) The nation’s report card: Writing 2002, Washington, DC: Government Printing Office. (U.S. Department of Education Publication No. NCES 2003-529).

34.

Sabatini, J., O’Reilly, T., & Deane, P. (2013). Preliminary reading literacy assessment framework: Foundation and rationale for assessment and system design (Research Report 13-30). Princeton, NJ: Educational Testing Service.

35.

Shaffer

D. W.

(2006) How computers help children learn, Basingstoke, England: Palgrave Macmillan.

36.

Sheehan

O’Reilly

(2012) The case for scenario-based assessments of reading competency. In: Sabatini

J. P.

O’Reilly

Albro

E. R.

(eds) Reaching an understanding: Innovations in how we view reading assessment, Lanham, MD: Rowman & Littlefield Education, pp. 19–33.

37.

Shermis, M. D. & Burstein, J. (Eds.) (2003) Automated essay scoring: A cross-disciplinary perspective, Mahwah, NJ: Erlbaum.

38.

Shute

V. J.

(2008) Focus on formative feedback. Review of Educational Research 78: 153–189.

39.

Shute

V. J.

Kim

Y. J.

(2011) Does playing the World of Goo facilitate learning? In: Dai

D. Y.

(ed.) Design research on learning and thinking in educational settings: Enhancing intellectual growth and functioning, New York, NY: Routledge Books, pp. 359–387.

40.

van Eemeren

F. H.

Grootendorst

Henkemans

F. S.

(1996) Fundamentals of argumentation theory: A handbook of historical backgrounds and contemporary developments, Mahwah, NJ: Erlbaum.

41.

van Merrienboer

J. J. G.

Sweller

(2005) Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review 17: 147–178.

42.

van Rijn

P. W.

Graf

E. A.

Deane

(2014) Empirical recovery of argumentation learning progressions in scenario-based assessments of English language arts. Spanish Journal of Educational Psychology (Psicologia Educativa) 20: 109–115.

43.

Vygotsky

L. S.

(1978) Mind in society: The development of higher psychological processes, Cambridge, MA: Harvard University Press.

44.

Weinstock

Neuman

Tabak

(2004) Missing the point or missing the norms? Epistemological norms as predictors of students’ ability to identify fallacious arguments. Contemporary Educational Psychology 29: 77–94.

45.

Wouters

van Nimwegen

van Oostendorp

van der Spek

E. D.

(2013) A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology 105: 249.

46.

Zapata-Rivera

Bauer

(2011) Exploring the role of games in educational assessment. In: Mayrath

Clarke-Midura

Robinson

Shraw

(eds) Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research, Charlotte, NC: Information Age, pp. 147–169.

47.

Zhang, M., & Deane, P. (2015). Development and deployment of automated scoring model for CR items in CBAL writing (Final Project Report). Princeton, NJ: Educational Testing Service.

Measuring Argumentation Skills Through a Game-Enhanced Scenario-Based Assessment

Abstract

Keywords

Get full access to this article

References