Restricted accessReview articleFirst published online 2019-10
Identifying Naturally Occurring Direct Assessments of Social-Emotional Competencies: The Promise and Limitations of Survey and Assessment Disengagement Metadata
Social-emotional learning (SEL) is gaining increasing attention in education policy and practice due to growing evidence that related constructs are strongly predictive of long-term academic achievement and attainment. However, the work of educators to support SEL is hampered by a lack of available, unbiased measures of related competencies. In this study we conducted a literature review to investigate whether assessment metadata (typically data relevant to how students behave on a test or survey) can provide information on SEL constructs. Implications of this new source of SEL data for practice, policy, and research are discussed.
AbediJ. (2002). Standardized achievement tests and English language learners: Psychometrics issues. Educational Assessment, 8(3), 231–257.
2.
AllensworthE. (2013). The use of ninth-grade early warning indicators to improve Chicago schools. Journal of Education for Students Placed at Risk (JESPAR), 18(1), 68–83.
3.
AlmlundM.DuckworthA. L.HeckmanJ. J.KautzT. D. (2011). Personality psychology and economics. Cambridge, MA: National Bureau of Economic Research.
4.
AndersonR.TheirM.PittsC. (2017). Interpersonal and intrapersonal skill assessment alternatives: Self-reports, situational-judgment tests, and discrete-choice experiments. Learning and Individual Differences, 53, 47–60.
5.
AndersonK. P.RitterG. W. (2018). Examining disparities in student discipline: Mapping inequities from infractions to consequences. Peabody Journal of Education, 93(2), 161–173.
6.
BalfanzR.HerzogL.Mac IverD. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist, 42(4), 223–235.
7.
BanduraA. (1997). Self-efficacy: The exercise of control. New York, NY: Macmillan.
8.
BardsleyN. (2008). Dictator game giving: Altruism or artefact?Experimental Economics, 11(2), 122–133.
9.
BarryC. L.FinneyS. J. (2016). Modeling change in effort across a low-stakes testing session: A latent growth curve modeling approach. Applied Measurement in Education, 29(1), 46–64.
10.
BarryC. L.HorstS. J.FinneyS. J.BrownA. R.KoppJ. P. (2010). Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing, 10(4), 342–363.
11.
BelfieldC.BowdenA. B.KlappA.LevinH.ShandR.ZanderS. (2015). The economic value of social and emotional learning. Journal of Benefit-Cost Analysis, 6(3), 508–544.
12.
BorghansL.SchilsT. (2012). The leaning tower of Pisa: Decomposing achievement test scores into cognitive and noncognitive components. Retrieved from http://www.sole-jole.org/13260.pdf.
13.
BrieschA. M.ChafouleasS. M. (2009). Review and analysis of literature on self-management interventions to promote appropriate classroom behaviors (1988–2008). School Psychology Quarterly, 24(2), 106.
14.
ChengA. (2015). Like teacher, like student: Teachers and the development of student noncognitive skills. Fayetteville, AR: University of Arkansas.
15.
ChengA.HittC. (2018). Hard work and soft skills: The attitudes, abilities, and character o students in career and technical education. Washington, DC: American Enterprise Institute.
16.
ChengA.ZamarroG. (2018). Measuring teacher non-cognitive skills and its impact on students: Insight from the Measures of Effective Teaching Longitudinal Database. Economics of Education Review, 64, 251–260.
17.
ChengA.ZamarroG.OrriensB. (2018). Personality as a predictor of unit nonresponse in an internet panel. Sociological Methods & Research, 0049124117747305.
18.
CurranP. G. (2015). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.
19.
DeAngelisC. A. (2019). Does private schooling affect non-cognitive skills?International evidence based on test and survey effort on PISA. Washington, DC: Center for Educational Freedom, Cato Institute.
20.
DemarsC. E. (2007). Changes in rapid-guessing behavior over a series of assessments. Educational Assessment, 12(1), 23–45.
21.
DamianL. E.Negru-SubtiricaO.PopE. I.BabanA. (2016). The costs of being the best: Consequences of academic achievement on students’ identity, perfectionism, and vocational development. In MontgomeryA.KehoeI. (Eds.), Reimagining the purpose of schools and educational organisations (pp. 173–188). Cham, ZG: Springer. https://doi.org/10.1007/978-3-319-24699-4_13.
22.
DuckworthA. L.YeagerD. S. (2015). Measurement matters assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251.
23.
DuckworthA. L.QuinnP. D. (2009). Development and validation of the Short Grit Scale (GRIT–S). Journal of Personality Assessment, 91(2), 166–174.
24.
DweckC.WaltonG. M.CohenG. L. (2011). Academic tenacity: Mindset and skills that promote long-term learning. Seattle, WA: Bill & Melinda Gates Foundation.
25.
DweckC. (2006). Mindset: The new psychology of success. New York, NY: Random House.
26.
FalkA.HeckmanJ. J. (2009). Lab experiments are a major source of knowledge in the social sciences. Science, 326(5952), 535–538.
27.
FarringtonC. A.RoderickM.AllensworthE.NagaokaJ.KeyesT. S.JohnsonD. W.BeechumN. O. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance—A critical literature review. Chicago, IL: Chicago Consortium on School Research at the University of Chicago.
28.
GallaB. M.PlummerB. D.WhiteR. E.MeketonD.D’MelloS. K.DuckworthA. L. (2014). The Academic Diligence Task (ADT): Assessing individual differences in effort on tedious but important schoolwork. Contemporary Educational Psychology, 39(4), 314–325.
29.
GershensonS.HoltS. B.PapageorgeN. W. (2016). Who believes in me? The effect of student-teacher demographic match on teacher expectations. Economics of Education Review, 52, 209–224.
30.
GregoryA.FergusE. (2017). Social and emotional learning and equity in school discipline. Future of Children, 27(1), 117–136.
31.
HeckmanJ.VytlacilE. (2001). Identifying the role of cognitive ability in explaining the level of and change in the return to schooling. Review of Economics and Statistics, 83(1), 1–12.
32.
HedengrenD.StratmannT. (2012). The dog that didn’t bark: What item nonresponse shows about cognitive and non-cognitive ability. Unpublished Manuscript, George Mason University, Fairfax, VA. Available from https://ssrn.com/abstract=2194373. doi:10.2139/ssrn.2194373
33.
HernándezM.HershaffJ. (2014). Skipping questions in school exams: The role of socio-emotional skills on educational outcomes. Department of Economics and Ford School of Public Policy, University of Michigan, Ann Arbor, MI.
34.
HittC. E. (2016). Just filling in the bubbles: Using careless answers patterns on surveys as a proxy measure of noncognitive skills. Fayetteville, AR: Department of Education Reform, University of Arkansas.
35.
HittC.TrivittJ.ChengA. (2016). When you say nothing at all: The predictive power of student effort on surveys. Economics of Education Review, 52, 105–119.
36.
JacksonJ. J.WoodD.BoggT.WaltonK. E.HarmsP. D.RobertsB. W. (2010). What do conscientious people do? Development and validation of the Behavioral Indicators of Conscientiousness (BIC). Journal of Research in Personality, 44(4), 501-511.
37.
JonesD. L. (2011). Academic dishonesty: Are more students cheating?Business Communication Quarterly, 74(2), 141–150.
38.
KassenboehmerS. C.SchurerS. (2018). Survey item-response behavior as an imperfect proxy for unobserved ability: Theory and application. Bonn, Germany: Institute for the Study of Labor
39.
KongX. J.WiseS. L.BholaD. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67, 606–619.
40.
KraftM. A. (2019). Teacher effects on complex cognitive skills and social-emotional competencies. Journal of Human Resources, 54(1), 1–36.
41.
KraftM. A.BlazarD.HoganD. (in press). The effect of teaching coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research.
42.
KranzK. M. (2003). Development of the alcohol and other drug self-efficacy scale. Research on Social Work Practice, 13(6), 724-741.
43.
KrosnickJ. A.NarayanS. S.SmithW. R. (1996). Satisficing in surveys: Initial evidence. In BravemanM. T.SlaterJ. K. (Eds.), Advances in survey research (pp. 29–44). San Francisco, CA: Jossey-Bass.
44.
KyllonenP. C.KellH. (2018). Ability tests measure personality, personality tests measure ability: Disentangling construct and method in evaluating the relationship between personality and ability. Journal of Intelligence, 6(3), 1–26.
45.
LeeY. H.JiaY. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-Scale Assessments in Education, 2(1), 8.
46.
LugtigP. J.2014. Panel attrition: Separating stayers, fast attriters, gradual attriters, and lurkers. Sociological Methods and Research, 43(4), 699–723.
47.
MarcusB.SchutzA. (2005). Who are the people reluctant to participate in research? Personality correlates of four different types of nonresponse as inferred from self- and observer ratings. Journal of Personality, 73(4), 959–984.
48.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437.
49.
MillerD. M.LinnR. L. (2000). Validation of performance-based assessments. Applied Psychological Measurement, 24(4), 367–378.
50.
MischelW.EbbesenE. B.Raskoff ZeissA. (1972). Cognitive and attentional mechanisms in delay of gratification. Journal of Personality and Social Psychology, 21(2), 204.
51.
NicholsS. L.BerlinerD. C. (2007). Collateral damage: How high-stakes testing corrupts America’s schools. Cambridge, MA: Harvard Education Press.
52.
PiedmontR. L.McCraeR. R.RiemannR.AngleitnerA. (2000). On the invalidity of validity scales: Evidence from self-reports and observer ratings in volunteer samples. Journal of Personality and Social Psychology, 78(3), 582.
53.
Robinson-CimpianJ. P. (2014). Inaccurate estimation of disparities due to mischievous responders: Several suggestions to assess conclusions. Educational Researcher, 43(4), 171–185.
54.
RiosJ. A.GuoH.MaoL.LiuO. L. (2016). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not?International Journal of Testing, 1–31.
55.
RiosJ. A.LiuO. L.BridgemanB. (2014). Identifying low-effort examinees on student learning outcomes assessment: A comparison of two approaches. New Directions for Institutional Research, 2014(161), 69–82.
56.
ShavelsonR. J.BaxterG. P.PineJ. (1991). Performance assessment in science. Applied Measurement in Education, 4(4), 347–362.
57.
SkibaR. J.HornerR. H.ChungC.RauschM. K.MayS. L.TobinT. (2011). Race is not neutral: A national investigation of African American and Latino disproportionality in school discipline. School Psychology Review, 40(1), 85–107.
58.
SolandJ. (2019). Can response process metadata provide insight into hard-to-measure constructs? Initial evidence from an international achievement test. Educational Measurement: Issues and Practice. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1111/emip.12260
59.
SolandJ.JensenN.KeysT.WolkE.BiS. (2019). Is low test motivation a sign of disengagement from school? Implications for measurement and practice. Educational Assessment, 24(2), 119–134.
60.
SolandJ.HamiltonL. SStecherB. M. (2013). Measuring 21st century competencies. Santa Monica, CA: RAND Corporation.
61.
SolandJ. (2018a). Are achievement gap estimates biased by differential student test effort? Putting an important policy metric to the test. Teachers College Record, 120(12).
62.
SolandJ. (2018b). The achievement gap or the engagement gap? Investigating the sensitivity of gaps estimates to test motivation. Applied Measurement in Education, 31(4), 312–323.
63.
SteeleC. M.AronsonJ. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 69(5), 797.
64.
SwerdzewskiP. J.HarmesJ. C.FinneyS. J. (2009). Skipping the test: Using empirical evidence to inform policy related to students who avoid taking low-stakes assessments in college. The Journal of General Education, 58(3), 167–195.
65.
WattsT. W.DuncanG. J.QuanH. (2018). Revisiting the marshmallow test: A conceptual replication investigating links between early delay of gratification and later outcomes. Psychological Science, 29(7), 1159–1177.
66.
WestM. R.BuckleyK.KrachmanS. B.BookmanN. (2018). Development and implementation of student social-emotional surveys in the CORE Districts. Journal of Applied Developmental Psychology, 55, 119–129. doi:10.1016/j.appdev.2017.06.001
67.
WiseS. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252.
68.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183.
69.
ZamarroG.ChengA.ShakeelM. D.HittC. (2018). Comparing and validating measures of non-cognitive traits: Performance task measures and self-reports from a nationally representative internet panel. Journal of Behavioral and Experimental Economics, 72, 51–60.
70.
ZamarroG.HittC.MendezI. (2016). When students don’t care: Reexamining international differences in achievement and non-cognitive skills. Fayetteville, AR: The Department of Education Reform, University of Arkansas.
71.
ZamarroG.NicholsM.DuckworthA.D’MelloS. (2018). Further validation of survey effort measures of conscientiousness: Results from a sample of high school students. Fayetteville, AR: The Department of Education Reform, University of Arkansas.
