In this article, the change in examinee effort during an assessment, which we will refer to as persistence, is modeled as an effect of item position. A multilevel extension is proposed to analyze hierarchically structured data and decompose the individual differences in persistence. Data from the 2009 Program of International Student Achievement (PISA) reading assessment from N = 467,819 students from 65 countries are analyzed with the proposed model, and the results are compared across countries. A decrease in examinee effort during the PISA reading assessment was found consistently across countries, with individual differences within and between schools. Both the decrease and the individual differences are more pronounced in lower performing countries. Within schools, persistence is slightly negatively correlated with reading ability; but at the school level, this correlation is positive in most countries. The results of our analyses indicate that it is important to model and control examinee effort in low-stakes assessments.
AbdelfattahH. (2010). The relationship between motivation and achievement in low-stakes examinations. Social Behavior and Personality, 38, 159–167.
2.
BarryC. L.HorstS. J.FinneyS. J.BrownA. R.KoppJ. P. (2010). Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing, 10, 342–363.
3.
ButlerJ.AdamsR. J. (2007). The impact of differential investment of student effort on the outcomes of international studies. Journal of Applied Measurement, 8, 279–304.
4.
ColeJ. S.BerginD. A.WhittakerT. A. (2008). Predicting student achievement for low stakes tests with effort and task value. Contemporary Educational Psychology, 33, 609–624.
5.
DebeerD.JanssenR. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50, 164–185.
6.
DebeerD.JanssenR.De BoeckP. (2013, July). Modeling missing-data processes: A tree-based IRT approach. Paper presented at the International Meeting of the Psychometric Society, Arnhem, The Netherlands.
7.
EcclesJ. (1983). Expectancies, values, and academic behaviors. In SpenceJ. T. (Ed.), Achievement and achievement motives (pp. 75–146). San Francisco, CA: Freeman.
8.
EklöfH.PavešičB. J.GrønmoL. S. (2014). A cross-national comparison of reported effort and mathematics performance in TIMSS. Advanced, Applied Measurement in Education, 27, 31–45.
9.
GlasC. A. W.PimentelJ. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68, 907–922.
10.
GoegebeurY.De BoeckP.WollackJ. A.CohenA. S. (2008). A speeded item response model with gradual process change. Psychometrika, 73, 65–87.
11.
HaladynaT. M.DowningS. M. (2004). Construct–irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23, 17–27.
12.
HartigJ.BuchholzJ. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54, 418–431.
13.
HohensinnC.KubingerK. D.ReifM.Holocher-ErtlS.KhorramdelL.FrebortM. (2008). Examining item-position effects in large-scale assessment using the Linear Logistic Test Model. Psychology Science Quarterly, 50, 391–402.
14.
HolmanR.GlasC. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58, 1–17.
15.
KamataA.BauerD. J.MiyazakiY. (2008). Multilevel measurement modeling. In O’ConnellA. A.McCoachD. B. (Eds.), Multilevel modeling of educational data (pp. 345–388). Charlotte, NC: Information Age Publishing.
16.
KongX. J.WiseS. L.BholaD. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67, 606–619.
17.
KunterM.SchümerG.ArteltC.BaumertJ.KliemeE.NeubrandM.PrenzelM.SchiefeleU.SchneiderW.StanatP.TillmannK.-J.WeiβM. (2002). German scale handbook for PISA 2000. Berlin, Germany: Max-Planck-Institut für Bildungsforschung.
18.
LiuO. L.BridgemanB.AdlerR. M. (2012). Measuring learning outcomes in higher education: Motivation matters. Educational Researcher, 41, 352–362.
19.
MeyerJ. P. (2010). A mixture Rasch model with item response time components. Applied Psychological Measurement, 34, 521–538.
20.
MeyersJ. L.MillerG. E.WayW. D. (2009). Item position and item difficulty change in an IRT-based common item equating design. Applied Measurement in Education, 22, 38–60.
21.
Organization for Economic Cooperation and Development. (2012). PISA 2009. Technical Report. Paris, France: Author.
22.
PohlS.GräfeL.RoseN. (2014). Dealing with omitted and not-reached items in competence tests. Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74, 423–452.
23.
RaudenbushS. W.BrykA. S.CongdonR. (2004). HLM 6 for Windows [Computer software]. Skokie, IL: Scientific Software International.
24.
RaudenbushS. W.BrykA. S.CongdonR. (2013). HLM 7 for Windows [Computer software]. Skokie, IL: Scientific Software International.
25.
RiosJ. A.LiuO. L.BridgemanB. (in press). Identifying unmotivated examinees on student learning outcomes assessment: A comparison of two approaches. New Directions for Institutional Research.
26.
SilmG.MustO.TaehtK. (2013). Test-taking effort as a predictor of performance in low-stakes tests. Trames, 17, 433–448.
27.
SteedleJ. T. (2014). Motivation filtering on a multi-institution assessment of general college outcomes. Applied Measurement in Education, 27, 58–76.
28.
SundreD. L.MooreD. L. (2002). The Student Opinion Scale: A measure of examinee motivation. Assessment Update, 14, 8–9.
29.
SundreD. L.WiseS. L. (2003). “Motivation filtering”: An exploration of the impact of low examinee motivation on the psychometric quality of tests. Paper presented at the National Council on Measurement in Education Annual Conference, Chicago, IL.
30.
SwerdzewskiP. J.HarmesJ. C.FinneyS. J. (2011). Two approaches for identifying low-motivated students in a low-stakes assessment context. Applied Measurement in Education, 24, 162–188.
31.
WaskiwiczR. A. (2011). Pharmacy students’ test-taking motivation-effort on a low-stakes standardized test. American Journal of Pharmaceutical Education, 75, 1–8, Article 41.
32.
WigfieldA.EcclesJ. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81.
33.
WiseS. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19, 95–114.
34.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1–17.
35.
WiseS. L.DeMarsC. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.
36.
WiseS. L.DeMarsC. E.(2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15, 27–41.
37.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.
38.
WiseS. L.PastorD. A.KongX. J. (2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education, 22, 185–205.
39.
WiseV. L.WiseS. L.BholaD. S.(2006). The generalizability of motivation filtering in improving test score validity. Educational Assessment, 11, 65–83.
40.
WolfL. F.SmithJ. K. (1995). The consequence of consequence: Motivation, anxiety, and test performance. Applied Measurement in Education, 8, 227–242.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
