As a follow-up to Horner et al., this study focuses on the internal consistency and validity of the School-wide Evaluation Tool (SET) at all school levels. Analyzing SET data from 833 elementary, 264 middle, and 93 high schools, the authors focused on (a) describing commonalities and differences in SET data across the school levels, (b) assessing the SET's internal consistency at all school levels, and (c) examining the SET's validity as the extent to which SET scores collected by external observers correlated with Team Implementation Checklist (TIC) scores reflecting internal self-assessments. Results indicated that overall, the SET performs best in elementary schools, shows less cohesion in middle and high schools, and highly correlates with TIC scores. Based on these results, the authors formulated a number of recommendations to improve the utility of SET data for research and implementation decisions across school levels and implementation phases.
Barrett, S., Bradshaw, C., & Lewis-Palmer, T. (2008). Maryland state-wide PBIS initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions, 10(2), 105-114.
2.
Bohanon, H., Fenning, P., Carney, K.L., Minnis-Kim, M.J., Anderson-Harriss, S., Moroz, K.B., et al. (2006). Schoolwide application of positive behavior support in an urban high school: A case study. Journal of Positive Behavior Interventions, 8(3), 131-145.
3.
Bohanon, H., Fenning, P., Flannery, K.B., Borgmeier, C., & Malloy. ( 2008). High school environments. In G. Sugai, R. Horner, G. Dunlap, & W. Sailor (Eds.), Handbook of positive behavior support: A volume in the Issues in Clinical Child Psychology series (pp. 581-602). New York: Springer.
4.
Bradshaw, C., Reinke, W., Brown, L., Bevans, K., & Leaf, P. ( 2008). Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 31(1), 1-26.
5.
Carr, E.G., Taylor, J.C., & Robinson, S. ( 1991). The effects of severe behavior problems in children on the teaching behavior of adults. Journal of Applied Behavior Analysis, 24, 523-535.
6.
Cortina, J. ( 1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104.
7.
Cronbach, L. ( 1951). Coefficient alpha and the internal structure of tests . Psychometrika, 16(3), 297-334.
8.
Cronbach, L.J., & Shavelson, R.J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391-418.
9.
Crone, D.A., & Horner, R.H. ( 2003). Building positive behavior support systems in schools: Functional behavioral assessment. New York: Guilford.
10.
Crone, D.A., Horner, R.H., & Hawken, L.S. ( 2004). Responding to problem behavior in schools: The behavior education program. New York: Guilford .
11.
Ervin, R.A., Schaughency, E., Goodman, S.D., McGlinchey, M.T., & Matthews, A. ( 2006). Merging research and practice agendas to address reading and behavior schoolwide. School Psychology Review, 35, 198-223.
12.
Flannery, B., Sugai, G., & Anderson, C. (in press). High school school-wide positive behavior support: Early lessons learned. Journal of Positive Behavior Interventions.
13.
Galloway, R., Panyan, M., Smith, C., & Wessendorf, S. (2008). Systems change with school-wide positive behavior support: Iowa’s work in progress. Journal of Positive Behavior Interventions, 10(2), 129-135.
14.
Gersten, R., Fuchs, L.S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. ( 2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149-164.
15.
Greenwood, C.R. ( 1996). The case for performance-based models of instruction . School Psychology Quarterly, 11, 283-296.
16.
Hatcher, L. ( 1994). SAS: Principal component analysis. In A step-by-step approach to using SAS for factor analysis and structural equation modeling (pp. 1-56). Cary, NC: SAS Institute . Retrieved July 8, 2008, from http://support.sas.com/publishing/pubcat/chaps/55129.pdf
17.
Horner, R.H. ( 2000). Positive behavior supports. Focus on Autism and Other Developmental Disabilities, 15(2), 97-105.
18.
Horner, R.H., & Sugai, G. ( 2006). Policy brief: Scaling up effective educational innovations . Available from http://www.pbis.org
19.
Horner, R.H., Sugai, G., & Lewis-Palmer, T. (2005). School-wide positive behavior support evaluation template. Retrieved July 14, 2008 , from http://pbis.org/tools.htm
20.
Horner, R.H., Sugai, G., Lewis-Palmer, T., & Todd, A.W. ( 2001). Teaching school-wide behavioral expectations. Report on Emotional and Behavioral Disorders in Youth, 1(4) 77-79, 93-96.
21.
Horner, R.H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A., et al. (in press). A randomized, waitlist-controlled effectiveness trial assessing school-wide positive behavior support in elementary schools . Journal of Positive Behavior Interventions.
22.
Horner, R.H., Sugai, G., Todd, A.W., & Lewis-Palmer, T. (2005). School-wide positive behavior support. In L. Bambara & L. Kern (Eds.), Individualized supports for students with problem behaviors: Designing positive behavior plans (pp. 359-390). New York: Guilford.
23.
Horner, R.H., Todd, A.W., Lewis-Palmer, T., Irvin, L.K., Sugai, G., & Boland, J.B. ( 2004). The School-Wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1), 3-12.
24.
Kern, L., & Manz, P. ( 2004). A look at current validity issues of schoolwide behavior support. Behavioral Disorders, 30(1), 47-59.
25.
Lago-Delello, E. ( 1998). Classroom dynamics and the development of serious emotional disturbance. Exceptional Children, 64, 479-492.
26.
Lewis, T.J., & Sugai, G. ( 1999). Effective behavior support: A systems approach to proactive schoolwide management. Focus on Exceptional Children, 31, 1-24.
27.
Messick, S. ( 1988). The once and future issues of validity: Assessing the meaning and consequences of measurement. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 33-45). Hillsdale, NJ: Lawrence Erlbaum.
28.
Messick, S. ( 1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 13-23.
29.
Murphy, J., Beck, L.G., Crawford, M., Hodges, A., & McGaughy C.L. ( 2001). The productive high school: Creating personalized academic communities. Thousand Oaks, CA: Corwin.
30.
Muscott, H.S., Mann, E.L., & LeBrun, M.R. (2008). Positive behavioral interventions and supports in New Hampshire: Effects of large-scale implementation of schoolwide positive behavior support on student discipline and academic achievement. Journal of Positive Behavior Interventions, 10, 190-205.
31.
Nakasato, J. ( 2000). Data-based decision making in Hawaii’s behavior support effort. Journal of Positive Behavior Interventions , 2(4), 247-251.
32.
Nelson, J.R., & Roberts, M. ( 2000). Ongoing reciprocal teacher- student interactions involving disruptive behaviors in general education classrooms. Journal of Emotional and Behavioral Disorders, 8(1), 27-37.
33.
Nersesian, M., Todd, A., Lehmann, J., & Watson, J. ( 2000). Schoolwide behavior support through district-level system change. Journal of Positive Behavior Interventions, 2(4), 244-247.
34.
Nunnally, J.C. ( 1967). Psychometric theory (2nd ed.). New York: McGraw-Hill.
35.
Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
36.
Sadler, C., & Sugai, G. (in press). Effective behavior and instructional support: A district model for early identification and prevention of reading and behavior. Journal of Positive Behavior Interventions.
Scott, T.M., & Barrett, S.B. ( 2004). Using staff and student time engaged in disciplinary procedures to evaluate the impact of schoolwide PBS. Journal of Positive Behavior Interventions, 6, 21-28.
39.
Siskin, L.S. ( 1994). Realms of knowledge: Academic departments in secondary schools. Washington, DC: Falmer .
40.
Spearman, C. ( 1910). Correlation calculated with faulty data. British Journal of Psychology, 3, 271-295.
41.
Sprague, J.R., & Horner, R.H. ( 2006). Schoolwide positive behavioral supports. In S. R. Jimerson & M. J. Furlong (Eds.), The handbook of school violence and school safety (pp. 413-427). Mahwah, NJ: Lawrence Erlbaum.
42.
Sugai, G., & Horner, R.H. ( 2002). The evolution of discipline practices: School-wide positive behavior supports. In J. K. Luiselli & C. Diament (Eds.), Behavior psychology in the schools: Innovations in evaluation, support, and consultation (pp. 23-50). New York: Haworth.
43.
Sugai, G., & Horner, R.H. ( 2004). School-wide positive behavior supports: Achieving and sustaining effective learning environments for all students. In W. Heward (Ed.), Focus on behavior analysis in education: Achievements, challenges, and opportunities (pp. 90-102). Columbus, OH: Prentice.
44.
Sugai, G., Horner, R.H., Dunlap, G., Hieneman, M., Lewis, T.J., Nelson, C.M., et al. (2000). Applying positive behavioral support and functional behavioral assessment in schools. Journal of Positive Behavioral Interventions, 2, 131-143.
45.
Sugai, G., Horner, R.H., & Gresham, F. ( 2002). Behaviorally effective school environments. In M. R. Shinn, H. M. Walker, & G. Stoner (Eds.), Interventions for academic and behavior problems II: Preventive and remedial approaches (pp. 315-350). Silver Spring, MD: National Association for School Psychologists.
46.
Sugai, G., Horner, R.H., & Lewis-Palmer, T. (2002). Team implementation checklists . Eugene: University of Oregon.
47.
Sugai, G., Horner, R.H., & Todd, A. ( 2000) Effective Behavior Support Self-Assessment Survey (EBS-SAS) . Eugene: University of Oregon.
48.
Sugai, G., Lewis-Palmer, T., & Hagan-Burke, S. (2004). School-wide discipline and instructional classroom management: A systems approach. In E. J. Kame’enui & C. Darch (Eds.), Instructional classroom management (2nd ed., pp. 218-248). White Plains, NY: Longman.
49.
Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R.H. ( 2001). The School-Wide Evaluation Tool (SET). Eugene: University of Oregon.
50.
Thompson, B., Diamond, K., McWilliam, R., Snyder, P., & Snyder, S. ( 2005). Evaluating the quality of evidence from correlational research for evidence-based practice. Exceptional Children , 71, 181-194.
51.
Tobin, T.J. ( 2006a, March). Positive behavior support in high schools: Use of team implementation checklists. Paper presented at the third international conference of the Association for Positive Behavior Support Reno, NV.
52.
Tobin, T.J. ( 2006b). Use of the Team Implementation Checklist in regular and alternative high schools. Unpublished manuscript, University of Oregon, College of Education, Educational and Community Supports , Eugene, OR. Retrieved July 11, 2008, from http://uoregon.edu/~ttobin/alt_tic.pdf
53.
Tobin, T., Sugai, G., & Colvin, G. ( 1996). Patterns in middle school discipline records. Journal of Emotional and Behavioral Disorders, 4(2), 82-94.
54.
Todd, A., Lewis-Palmer, T., Horner, R.H., Sugai, G., Sampson, N., & Phillips, D. ( 2005). School-Wide Evaluation Tool implementation manual. Eugene: University of Oregon.
55.
Van Acker, R., Grant, S., & Henry, D. ( 1996). Teacher and student behavior as a function of risk for aggression. Education and Treatment of Children, 19, 316-334.
56.
Walker, H.M., Horner, R.H., Sugai, G., Bullis, M., Sprague, J.R., Bricker, D., et al. (1996). Integrated approaches to preventing antisocial behavior patterns among school-age children and youth. Journal of Emotional and Behavioral Disorders, 4, 194-209.
57.
Wehby, J., Symons, F., Canale, J., & Go, F. ( 1998). Teaching practices in classrooms for students with emotional and behavioral disorders: Discrepancies between recommendations and observations . Behavioral Disorders, 24(1), 51-56.
