With the pressing need for accountability in higher education, standardized outcomes assessments have been widely used to evaluate learning and inform policy. However, the critical question on how scores are influenced by students’ motivation has been insufficiently addressed. Using random assignment, we administered a multiple-choice test and an essay across three motivational conditions. Students’ self-report motivation was also collected. Motivation significantly predicted test scores. A substantial performance gap emerged between students in different motivational conditions (effect size as large as .68). Depending on the test format and condition, conclusions about college learning gain (i.e., value added) varied dramatically from substantial gain (d = 0.72) to negative gain (d = −0.23). The findings have significant implications for higher education stakeholders at many levels.
ArumR.ChoE.KimJ.RoksaJ. (2012). Documenting uncertain times: Post-graduate transitions of the academically adrift cohort. Brooklyn, NY: Social Science Research Council.
2.
ArumR.RoksaJ. (2011). Academically adrift: Limited learning on college campuses. Chicago, IL: University of Chicago Press.
BantaT. (2008). Trying to clothe the emperor. Assessment Update, 20, 3-4, 16-17.
5.
BarryC. L.HorstS. J.FinneyS. J.BrownA. R.KoppJ. (2010). Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing, 10(4), 342-363.
6.
BaumertJ.DemmrichA. (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16, 441-462.
7.
BelcheirM. J. (2002). Academic profile results for selected nursing students (Report No. 2002-05). Boise, ID: Boise State University.
8.
BraunH.KirschI.YamamotoK. (2011). An experimental study of the effects of monetary incentives on performance on the 12th grade NAEP reading assessment. Teachers College Record, 113, 2309-2344.
9.
ColeJ. S.OsterlindS. J. (2008). Investigating differences between low- and high-stakes test performance on a general education exam. The Journal of General Education, 57, 119-130.
10.
DuckworthA. L.QuinnP. D.LynamD. R.LoeberR.Stouthamer-LoeberM. (2011). Role of test motivation in intelligence testing. Proceedings of the National Academy of Sciences, 108, 7716-7720.
11.
Educational Testing Service. (2010). Market research of institutions that use outcomes assessment. Princeton, NJ: Author.
12.
HaladynaT. M.DowningS. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23, 17-27.
13.
HendelD. D. (1991). Evidence of convergent and discriminant validity in three measures of college outcomes. Educational and Psychological Measurement, 51, 351-358.
KuhG. D.IkenberryS. O. (2009). More than you think, less than we need: Learning outcomes assessment in American higher education. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.
16.
KuhG. D.KinzieJ.BuckleyJ. A.BridgesB. K.HayekJ. C. (2006). What matters to student success: A review of the literature (Report commissioned for the National Symposium on Postsecondary Student Success: Spearheading a Dialog on Student Success). Washington, DC: National Postsecondary Education Cooperative.
17.
LakinJ.ElliottD.LiuO. L. (in press). Investigating the impact of ELL status on higher education outcomes assessment. Educational and Psychological Measurement.
18.
LiuO. L. (2008). Measuring learning outcomes in higher education using the Measure of Academic Proficiency and Progress (MAPP™) (ETS Research Report Series RR-08-047). Princeton, NJ: Educational Testing Service.
19.
LiuO. L. (2011a). An overview of outcomes assessment in higher education. Educational Measurement: Issues and Practice, 30, 2-9.
20.
LiuO. L. (2011b). Value-added assessment in higher education: A comparison of two methods. Higher Education, 61, 445-461.
21.
LivingstonS. A.AntalJ. (2010). A case of inconsistent equatings: How the man with four watches decides what time it is. Applied Measurement in Education, 23(1), 49-62.
22.
MarrD. (1995). Validity of the academic profile. Princeton, NJ: Educational Testing Service.
23.
MessickS. (1995). Validity of psychological assessment: Validation of references from persons’ responses and performances on scientific inquiry into score meaning. American Psychologist, 50, 741-749.
O’NeilH. F.SugrueB.BakerE. L. (1995/1996). Effects of motivational interventions on the National Assessment of Educational Progress mathematics performance. Educational Assessment, 3, 135-157.
27.
PintrichP. R.SchunkD. H. (2002). Motivation in education: Theory, research, and applications (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
28.
SteedleJ. (2011). Selecting value-added models for postsecondary institutional assessment. Assessment and Evaluation in Higher Education, 1-16.
29.
SundreD. L. (1997, April). Differential examinee motivation and validity: A dangerous combination. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
30.
SundreD. L. (1999, April). Does examinee motivation moderate the relationship between test consequences and test performance?Paper presented at the annual meeting of the American Educational Research Association, Montreal.
31.
SundreD. L.KitsantasA. L. (2004). An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance?Contemporary Educational Psychology, 29(1), 6-26.
32.
SundreD. L.WiseS. L. (2003, April). Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of tests. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
33.
U.S. Department of Education. (2006). A test of leadership: Charting the future of American higher education (Report of the commission appointed by Secretary of Education Margaret Spellings). Washington, DC: Author.
34.
Voluntary System of Accountability. (2008). Information on learning outcomes measures. Author.
35.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1-17.
36.
WiseS. L.DeMarsC. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19-38.
37.
WiseS. L.DeMarsC. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15, 27-41.
38.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163-183.
39.
WiseV. L. (2004). The effects of the promise of test feedback on examinee performance and motivation under low-stakes testing conditions (Unpublished doctoral dissertation). University of Nebraska-Lincoln, Lincoln, NE.
40.
WiseV. L.WiseS. L.BholaD. S. (2006). The generalizability of motivation filtering in improving test score validity. Educational Assessment, 11(1), 65-83.
41.
WolfL. F.SmithJ. K. (1995). The consequence of consequence: Motivation, anxiety, and test performance. Applied Measurement in Education, 8, 227-242.
