The purpose of this study was to evaluate the internal consistency reliability, validity, and diagnostic accuracy of Social, Academic, and Emotional Behavior Risk Screener–Teacher Rating Scale (SAEBRS) scores. Teachers (n = 68) universally screened 1,242 elementary students using two measures: the SAEBRS and the Behavioral and Emotional Screening System (BESS). Multilevel analyses indicated that although SAEBRS scores were internally consistent at the overall level, reliability suffered for certain SAEBRS scores at the between-group (classroom) level. Multilevel correlational analyses revealed moderate-to-large and statistically significant relations between SAEBRS and BESS scores at the overall, between-group, and within-group levels. Follow-up Fisher’s z tests revealed a pattern of convergent and discriminant relations in accordance with theory-driven expectations. Receiver operating characteristic (ROC) curve analyses supported the diagnostic accuracy of each SAEBRS scale. Further examination of findings relative to prior research suggested SAEBRS diagnostic accuracy is moderated by the type of criterion measure under consideration.
BlieseP. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In BollenK.LongJ. (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.
2.
BossuytP. M.ReitsmaJ. B.BrunsD. E.GatsonisC. A.GlasziouP. P.IrwigL. M.. . . LijmerJ. G. (2003). The STARD statement for reporting studies of diagnostic accuracy: Explanation and elaboration. Clinical Chemistry, 49, 7–18.
3.
BruhnA. L.Woods-GrovesS.HuddleS. (2014). A preliminary investigation of emotional and behavioral screening practices in K–12 schools. Education and Treatment of Children, 37, 611–634.
4.
BurnsM. K.Riley-TillmanT. C.VanDerHeydenA. (2012). RTI applications: Vol. 1. Academic and behavioral interventions. New York, NY: Guilford Press.
5.
CookC. R.RasetshwaneK. B.TruelsonE.GrantS.DartE. H.CollinsT. A.SpragueJ. (2011). Development and validation of the Student Internalizing Behavior Screener: Examination of reliability, validity, and classification accuracy. Assessment for Effective Intervention, 36, 71–79.
6.
CurranP. J.WestS. G.FinchJ. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1, 16–29.
7.
DeverB. V.MaysK. L.KamphausR. W.DowdyE. (2012). The factor structure of the BASC-2 Behavioral and Emotional Screening System teacher form, child/adolescent. Journal of Psychoeducational Assessment, 30, 488–495.
8.
DrummondT. (1994). The Student Risk Screening Scale (SRSS). Grants Pass, OR: Josephine County Mental Health Program.
9.
EklundK.KilgusS. P.von der EmbseN.BeardmoreM.TannerN. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: A multilevel approach. Psychological Assessment, 29, 486–499.
10.
ElliottS. N.GreshamF. M. (2008). Social Skills Improvement System: Performance Screening Guide. Minneapolis, MN: Pearson.
11.
GreshamF. M.ElliottS. N. (2008). Social Skills Improvement System: Rating Scales. Bloomington, MN: Pearson.
12.
HintzeJ. M.SilberglittB. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34, 372–386.
13.
JonssonA.SvingbyG. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130–144.
14.
KamphausR. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6, 89–97.
15.
KamphausR. W.ReynoldsC. R. (2007). BASC-2 Behavioral and Emotional Screening System. Minneapolis, MN: Pearson.
16.
KilgusS. P.ChafouleasS. M.Riley-TillmanT. C. (2013). Development and initial validation of the Social and Academic Behavior Risk Screener for elementary grades. School Psychology Quarterly, 28, 210–226.
17.
KilgusS. P.EklundK. (2016). Consideration of base rates within universal screening for behavioral and emotional risk: A novel procedural framework. School Psychology Forum, 10, 120–130.
18.
KilgusS. P.EklundK.von der EmbseN. P.TaylorC.SimsW. A. (2016). Psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39.
19.
KilgusS. P.MetheS. A.MagginD. M.TomasulaJ. L. (2014). Curriculum-Based Measurement Oral Reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52, 377–405.
20.
KilgusS. P.SimsW. A.von der EmbseN. P.Riley-TillmanT. C. (2015). Confirmation of models for interpretation and use of the Social and Academic Behavior Risk Screener (SABRS). School Psychology Quarterly, 30, 335–352.
21.
KilgusS. P.SimsW. A.von der EmbseN. P.TaylorC. N. (2016). Technical adequacy of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) in an elementary sample. Assessment for Effective Intervention, 42, 46–59.
22.
KilgusS. P.von der EmbseN. P. (2014). Social, Academic, and Emotional Behavior Risk Screener (Unpublished technical manual).
23.
LaneK. L.MenziesH. M.OakesW. P.KalbergJ. R. (2012). Systematic screenings of behavior to support instruction: From preschool to high school. New York, NY: Guilford Press.
24.
LaneK. L.OakesW. P.EnnisR. P.HirschS. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have Tier 2 and Tier 3 needs?Preventing School Failure: Alternative Education for Children and Youth, 58, 171–182.
25.
LipseyM. W.PuzioK.YunC.HebertM. A.Steinka-FryK.ColeM. W.. . . BusickM. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: U.S. Government Printing Office. Retrieved from http://ies.ed.gov/ncser/pubs/20133000/
26.
MillerF. G.CohenD.ChafouleasS. M.Riley-TillmanT. C.WelshM. E.FabianoG. A. (2015). A comparison of measures to screen for social, emotional, and behavioral risk. School Psychology Quarterly, 30, 184–196.
27.
PetscherY.KimY.FoormanB. R. (2011). The importance of predictive power in early screening assessments: Implications for placement in the response to intervention framework. Assessment for Effective Intervention, 36, 158–166.
28.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: SAGE.
29.
ReschlyA. L.BuschT. W.BettsJ.DenoS. L.LongJ. D. (2009). Curriculum-Based Measurement Oral Reading as an indicator of reading achievement: A meta-analysis of the correlational evidence. Journal of School Psychology, 47, 427–469.
30.
RevelleW. (2015). Procedures for psychological, psychometric, and personality research (R package Version 1.4.5) [Computer software manual]. Retrieved from http://personality-project.org/
31.
RobinX.TurckN.HainardA.TibertiN.LisacekF.SanchezJ. C.MüllerM. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 1–8.
32.
SteigerJ. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251.
33.
StreinerD. L.CairneyJ. (2007). What’s under the ROC? An introduction to receiver operating characteristics curves. The Canadian Journal of Psychiatry [La Revue canadienne de psychiatrie], 52, 121–128.
34.
SugaiG.Lewis-PalmerT.ToddA.HornerR. H. (2001). The School-Wide Evaluation Tool (SET). Eugene: University of Oregon.
35.
von der EmbseN. P.PendergastL.KilgusS. P.EklundK. R. (2016). Evaluating the applied use of a mental health screener: Structural validity of the Social, Academic, and Emotional Behavior Risk Screener. Psychological Assessment, 28, 1265–1275.
36.
WhitcombS. A.MerrellK. W. (2013). Behavioral, social, and emotional assessment of children and adolescents (4th ed.). New York, NY: Routledge.
