In low-stakes testing contexts, there are minimal personal consequences associated with examinee performance. Examples include assessments administered for research, program evaluation, test development, and international comparisons (e.g., Programme for International Student Assessment [PISA]). Because test-taking motivation can suffer in low-stakes conditions, the Student Opinion Scale (SOS) is commonly administered to measure test-taking effort and how personally important the examinee feels it is to do well on the test. Although popular, studies of the scale’s internal validity yield conflicting findings. The present study critically evaluates the creation of the SOS and considers its factor structure across six samples of college students differing in their college experience level and version of the SOS administered. Because findings only support the internal validity of the effort subscale, further study and development of the importance subscale is recommended.
American Educational Research AssociationAmerican Psychological AssociationNational Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
2.
BandalosD. L.FinneyS. J. (2019). Factor analysis: Exploratory and confirmatory. In HancockG. R.StapletonL. M.MuellerR. O. (Eds.), The reviewer’s guide to quantitative methods in the social sciences (pp. 98–122). Routledge. https://doi.org/10.4324/9781315755649
3.
BaumertJ.DemmrichA. (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology in Education,16(3), 441–462. https://doi.org/10.1007/BF03173192
4.
DeMarsC. E.BashkovB. M.SochaA. B. (2013). The role of gender in test-taking motivation under low-stakes conditions. Research & Practice in Assessment, 8, 69–82.
FinneyS. J.MyersA. J.MathersC. E. (2018). Test instructions do not moderate the indirect effect of perceived test importance on test performance in low-stakes testing contexts. International Journal of Testing, 18, 297–322. https://doi.org/10.1080/15305058.2017.1396466
ImmekusJ.C.McGeeD. (2016). The measurement invariance of the student opinion scale across English and non-English language learner students within the context of low- and high-stakes assessments. Frontiers in Psychology, 7(1352), 1–10. https://doi.org/10.3389/fpsyg.2016.01352
12.
JorgensenT. D.PornprasertmanitS.SchoemannA. M.RosseelY. (2021). semTools: Useful tools for structural equation modeling. R package version 0.5-5. https://CRAN.R-project.org/package=semTools
13.
KnektaE.EklöfH. (2015). Modeling the test-taking motivation construct through investigation of psychometric properties of an expectancy-value-based questionnaire. Journal of Psychoeducational Assessment, 33(7), 662–673. https://doi.org/10.1177/0734282914551956
14.
KunterM.SchümerG.ArteltC.BaumertJ.KliemeE.NeubrandM.PrenzelM.SchiefeleU.SchneiderW.StanatP.TillmannK.-J.WeiβM. (2002). German scale handbook for PISA 2000.Max-Planck-Institut für Bildungsforschung.
15.
LikertR. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 55.
16.
McDonaldR. P. (1999). Test theory: A unified treatment. Erlbaum.
17.
MotlR.ConroyD.HoranP. (2000). The social physique anxiety scale: An example of the potential consequence of negatively worded items in factorial validity studies. Journal of Applied Measurement, 1(4), 327–345.
18.
MyersA. J.FinneyS. J. (2021). Does it matter if examinee motivation is measured before or after a low-stakes test? A moderated mediation analysis. Educational Assessment, 26(1), 1–19. https://doi.org/10.1080/10627197.2019.1645591
19.
NunnallyJ. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
20.
PenkC.RichterD. (2017). Change in test-taking motivation and its relationship to test performance in low-stakes assessment. Educational Assessment, Evaluation and Accountability, 29(1), 55–79. https://doi.org/10.1007/s11092-016-9248-7
21.
PerkinsB. A.PastorD. A.FinneyS. J. (2021). Between- versus within-examinee variability in test-taking effort and test emotions during a low-stakes test. Applied Measurement in Education, 34(4), 285–300. https://doi.org/10.1080/08957347.2021.1987905
22.
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
23.
RhemtullaM.Brosseau-LiardP. E.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315
24.
RiosJ. (2021). Improving test-taking effort in low-stakes group-based educational testing: A meta-analysis of interventions. Applied Measurement in Education, 34(2), 85–106. https://doi.org/10.1080/08957347.2021.1890741
25.
RosseelY. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
26.
SessomsJ.FinneyS. J. (2015). Measuring and modeling change in examinee effort in low-stakes tests across testing occasions. International Journal of Testing, 15(4), 356–388. https://doi.org/10.1080/15305058.2015.1034866
27.
SilmG.MustO.TähtK.PedasteM. (2020). Does test-taking motivation predict test results in a high-stakes context?Educational Research & Evaluation, 26(7–8), 387–413. https://doi.org/10.1080/13803611.2021.1949355
28.
SilmG.PedasteM.TähtK. (2020). The relationship between performance and test-taking effort when measured with self-report or time-based instruments: A meta-analytic review. Educational Research Review, 31(2). https://doi.org/10.1016/j.edurev.2020.100335
29.
SundreD. L. (1996, April). The role of examinee motivation in assessment: Celebrity, scene stealer, or cameo? [Paper presentation]. National Council on Measurement in EducationConference, New York, NY, United States.
30.
SundreD. L. (1997, March). Differential examinee motivation and validity: A dangerous combination. [Paper presentation]. American Educational Research Association Conference, Chicago, IL, United States.
31.
SundreD. L. (1999, April). Does examinee motivation moderate the relationship between test consequences and test performance? [Paper presentation]. American Educational Research Association Conference, Montreal, Canada.
SwerdzewskiP. J.HarmesJ. C.FinneyS. J. (2011). Two approaches for identifying low-motivated students in a low-stakes assessment context. Applied Measurement in Education, 24(2), 162–188. https://doi.org/10.1080/08957347.2011.555217
34.
ThelkA. D.SundreD. L.HorstS. J.FinneyS. J. (2009). Motivation matters: Using the Student Opinion Scale to make valid inferences about student performance. Journal of General Education, 58(3), 129–151. https://doi.org/10.1353/jge.0.0047
35.
WiseS. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95–114. https://doi.org/10.1207/s15324818ame1902_2
36.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://doi.org/10.1207/s15326977ea1001_1
37.
WiseS. L.DeMarsC. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15(1), 27–41. https://doi.org/10.1080/10627191003673216
38.
WolfL. F.SmithJ. K. (1993). The effects of motivation and anxiety on test performance [Paper presentation]. American Educational Research Association, Atlanta, Georgia.
39.
WolfL. F.SmithJ. K.BirnbaumM. E. (1995). Consequence of performance, test motivation, and mentally taxing items. Applied Measurement in Education, 8(4), 341–351. https://doi.org/10.1207/s15324818ame0804_4
40.
YuC.-Y.MuthénB. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. [Unpublished doctoral dissertation]. University of California. http://www.statmodel.com/download/Yudissertation.pdf
41.
ZilberbergA.FinneyS. J.MarshK. R.AndersonR. (2014). The role of students’ attitudes and test-taking motivation on the validity of college institutional accountability tests: A path analytic model. International Journal of Testing, 14(4), 360–384. https://doi.org/10.1080/15305058.2014.928301
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
