School Climate,Student Engagement,and Academic Achievement: A Latent Variable,Multilevel Multi-Informant Examination

Abstract

This study tested the authoritative school climate theory that schools characterized by high structure and student support have greater levels of student engagement and that these factors are associated with higher academic achievement, as indicated by school graduation rates and school performance on state-mandated testing. The model was tested through a multilevel multi-informant structural model on a statewide sample of 60,441 students and 11,442 teachers in 298 high schools. Consistent with the authoritative school climate model, both structure and student support were associated with higher student engagement in schools. Moreover, student engagement was directly associated with academic achievement and operated as an intervening factor. Results provide new evidence that an authoritative school climate is associated with high school academic achievement.

a substantial body of research has found that students are more engaged in school and attain higher academic achievement in schools with a positive school climate (Thapa, Cohen, Guffey, & Higgins-D’Alessandro, 2013). For example, a meta-analysis of 78 published research articles concluded that “a positive school climate contributed to higher academic achievement and decreased the negative influence of poor SES [socioeconomic status] background characteristics and other risk factors on academic achievement” (Berkowitz, Moore, Astor, & Benbenishty, 2017, p. 33). Another review concluded that a positive school climate leads to higher academic achievement when it is characterized by high academic expectations and high-quality teacher-student relationships (Wang & Degol, 2016).

With so much supporting research, the improvement of school climate has become an important educational goal (Caskey, Cerna, Hanson, Polik, & Houten, 2016). For example, in 2014, the U.S. Department of Education issued guidelines for improving school climate (U.S. Departments of Education, 2014a) and awarded over $70 million in school climate transformation grants to 138 recipients in 38 states (U.S. Department of Education, 2014b). Its Office of Safe and Healthy Students has promulgated a compendium of school climate surveys and invested in the development of a national survey (American Institutes for Research, 2018). The 2015 Every Student Succeeds Act (ESSA; U.S. Department of Education, 2017) encourages schools to measure “school climate and safety” as a nonacademic indicator of school quality or student success. As might be expected, numerous educational programs have been developed with school climate goals such as improving student behavior and strengthening the quality of teacher-student relationships (National School Climate Center, 2017; O’Brennan & Bradshaw, 2013).

Across the numerous studies of school climate is the recurrent question of how school climate is linked to academic outcomes. What specific aspects of school climate are associated with student achievement, and what is the mechanism underlying this relationship? In identifying directions for further research, Wang and Degol (2016) asserted that future studies should conceptualize school climate as a multidimensional construct and identify what specific components of school climate are associated with student academic outcomes. They criticized previous studies for their reliance on a unidimensional model of school climate based on a single scale and single informant. Finally, they called for multilevel modeling procedures to support these more complex conceptualizations of school climate.

In their comprehensive analysis of school climate research, Astor and Benbenishty (2018) pointed out that school climate theory has been severely limited by a failure to construct conceptual models that identify mechanisms by which specific features of school climate are associated with student outcomes such as academic achievement. Similarly, Cornell and Huang (2018) argued that school climate should be regarded as a system of school characteristics that influence one another and are linked to meaningful student outcomes. They suggested that inherently interpersonal characteristics, such as the quality of teacher-student relationships, should be distinguished from personal characteristics such as motivation and engagement and those in turn should be distinguished from behavioral outcomes such as test performance or school attendance.

Research on student engagement has the potential to help construct a more complex and meaningful model of the association between school climate and achievement. Several studies have suggested that a positive school climate fosters greater student engagement in school, while other studies have found that engagement leads to greater learning and academic success (Archambault, Janosz, Fallu, & Pagani, 2009; Lawson & Masyn, 2015; Murray, 2009; Wang & Eccles, 2013). In essence, student engagement could serve as a valuable link between school climate and student achievement. This is an important formulation from a theoretical perspective because it contributes to a functional model of how school climate influences academic achievement, and from a practical perspective, it guides school authorities to gauge the impact of their school climate interventions on student engagement to achieve academic outcomes. If it can be established how the qualities of a school climate are transmitted to student outcomes, there will be direct practical implications for more effective school-based interventions at school and student levels. However, it is necessary to construct a testable working model of school climate that includes all three components.

Student engagement has long been recognized as a critical factor in student learning and achievement (Fredricks, Filsecker, & Lawson, 2016; Lawson & Masyn, 2015; Wang & Eccles, 2013) and was identified by the National Research Council (2003) as a critical goal of national school improvement efforts. Engagement is generally regarded as having behavioral, affective, and cognitive aspects (Fredricks et al., 2016). Students demonstrate engagement behaviorally by attending school and participating in school activities, affectively by feelings of pride and attachment to their school, and cognitively by engaging in studying and learning. However, studies of student engagement often find that a combined measure of overall engagement is a more robust predictor of student outcomes than individual components (Archambault et al., 2009). High engagement is consistently related to academic outcomes such as course grades and achievement test scores (Fredricks et al., 2016). During adolescence, students who become disengaged from school are more likely to exhibit problem behaviors such as substance use and delinquency and eventually drop out of school (Wang & Fredricks, 2014).

Although the research literature has accumulated substantive evidence that engagement is critical to student academic success, what remains less well understood is what factors affect engagement. Astor and Benbenishty (2018) point out that the field has focused too heavily on individual-level models of achievement motivation, failing to appreciate the influence of school-level factors and, in particular, the salutary effects of a positive school climate. They point to research on schools that serve a socioeconomically disadvantaged community but produce higher than expected academic achievement. They hypothesize that features of the school climate that encourage and support students are critical to their success. Such a model would be especially helpful to school administrators serving disadvantaged communities.

Previous studies have reported that student perceptions of their school climate are associated with their engagement (e.g., Mehta, Cornell, Fan, & Gregory, 2013; Wang & Eccles, 2013). In their study of a disadvantaged urban middle school sample, Wang and Eccles (2013) observed that several features of school climate were related to higher engagement. Notably, they emphasized the need for schools to be structured by clear expectations for student behavior and provide an emotionally supportive and caring school environment. However, there is a need to test more specifically whether student engagement mediates the relations between school climate and academic outcomes. Academic success in a high school is often measured by school-wide student performance on standardized tests and graduation rates, both of which are required by ESSA (U.S. Department of Education, 2017).

Despite the accumulation of studies pointing to the benefits of a positive school climate on student achievement and the nationwide push to evaluate and enhance school climate, there are important problems in its conceptualization and measurement. Reviews of school climate measures have lamented the amorphous nature of the construct, the absence of guiding theory, and the need for rigorous psychometric research (Johnson, 2009; Ramelow, Currie, & Felder-Puig, 2015; Wang & Degol, 2016). Concepts of school climate are often ambiguous and fail to specify how different components of school climate are related to one another or expected academic benefits (Rudasill, Snyder, Levinson, & Adelson, 2017; Wang & Degol, 2016). Further, measures of climate are often limited to the perspective of a single informant group such as students or teachers rather than integrating their perspectives and demonstrating cross-informant validity (Konold & Cornell, 2015b; Waasdorp, Pas, O’Brennan, & Bradshaw, 2011). Finally, school climate is inherently an organizational construct that should be evaluated at the school level but is typically measured at the individual level using measures validated solely at the individual level (Berkowitz et al., 2017; Konold & Cornell, 2015a).

Authoritative School Climate Model

The authoritative school climate (ASC) model presents a promising approach to identifying key features of school climate and their association with positive student outcomes (Gregory, Cornell, & Fan, 2011; Konold & Cornell, 2015a). This model was derived from Baumrind’s (1968) work on authoritative parenting that continues to guide a substantial body of child development research (Larzelere, Morris, & Harrist, 2013). Parenting research has identified two important dimensions of parenting: one dimension concerned with the parent’s high expectations and demands for the child and the other concerned with how warm and supportive the parent is toward the child. Authoritative parents provide a combination of high expectations (also called “demandingness”) and emotional support (also called “responsiveness”) for their children. Parents are less effective when they are highly structured and demanding but not supportive (authoritarian), emotionally supportive but lacking in structure (permissive), or lacking in both structure and support (disengaged or neglectful).

The ASC model was developed to apply concepts of high expectations and supportive relationships to schools (Gill, Ashton, & Algina, 2004; Gregory & Cornell, 2009; Pellerin, 2005). Although there is no expectation that an authoritative school climate is fully concordant with authoritative parenting, there are parallels that help to organize research on school climate. The two key dimensions of an authoritative school climate are structure (or high expectations) and support. Structure in the ASC model is operationalized to include high expectations in both academic and disciplinary domains. Teachers in authoritative schools have high academic expectations for their students and expect them to work hard and learn a lot. In schools with high discipline expectations, students experience school rules as strict but fair. Students perceive that rules are applied in the same way to all students, and they have a chance to explain when accused of doing something wrong.

The high disciplinary expectations of an authoritative school must be distinguished from a zero tolerance approach to school discipline in which students are punished harshly for violation of a rule regardless of the circumstances and whether the action was intentional or unintentional (American Psychological Association Zero Tolerance Task Force, 2008). Previous studies have reported that schools with characteristics of an authoritative school climate have lower suspension rates than other schools (Catizone, Cornell, & Konold, 2018; Gregory et al., 2010; Huang & Cornell, 2018). The lower rates of school suspension in schools with an authoritative school climate are especially noteworthy because they are independent of other student and school demographic characteristics and extend across racial/ethnic groups, supporting the potential to help schools reduce the high rates of suspension for Black and Hispanic students (Huang & Cornell, 2018).

Support is characterized both by adult respect for students and students being willing to seek help. The authoritative model posits that a healthy school climate requires both high expectations and a supportive environment. School climate research has generally supported the idea that school structure and support deserve a central role in research on school climate. For example, Johnson’s (2009) review of 25 studies concluded that “schools with less violence tend to have students who are aware of school rules and believe they are fair” and “have positive relationships with their teachers” (p. 451).

Research using the authoritative school climate model has found positive associations with academic outcomes. Pellerin (2005) found that high schools using authoritative practices had less truancy and fewer dropouts than schools using an authoritarian approach. An analysis of NELS data determined that authoritative schools, characterized as both demanding and responsive, had higher levels of student engagement (Gill et al., 2004). Lee (2012) reported that an authoritative school climate was associated with higher student engagement and reading achievement. Although they did not explicitly use an authoritative conceptual framework, Wang and Eccles (2013) found that “school structure support” (defined as the clarity and consistency of teacher expectations) and “teacher emotional support” (defined as level of care and support from teachers) were associated with greater behavioral, emotional, and cognitive engagement.

Two studies using the Authoritative School Climate Survey found that schools characterized by high expectations for students and supportive teacher-student relationships had more positive academic outcomes. In the first study, academic adjustment was measured by student engagement in school, course grades, and educational aspirations for high school graduation and college attendance (Cornell, Shukla, & Konold, 2016). Hypotheses were tested in separate statewide samples of 423 middle schools and 323 high schools. The same pattern of findings was found in both samples: Both higher disciplinary structure and student support were associated with higher student engagement in school, higher course grades, and higher educational aspirations at the student level. At the school level, higher disciplinary structure was associated with higher engagement, and higher student support was associated with higher engagement and grades in both samples.

A second study of 315 high schools found that when students perceive their teachers as supportive, high academic expectations were associated with lower dropout rates (Jia, Konold, & Cornell, 2016). These analyses controlled for school demographics of school enrollment size, percentage of low-income students, percentage of minority students, and urbanicity. From this brief summary, it is clear that multiple studies using a variety of measures have supported an authoritative model of school climate.

Multiple Informant Perspectives

When measurement is a matter of perspective or perception, the use of ratings from multiple informants is considered best practice as a means for capturing a variety of viewpoints that might add value to assessing a trait (Bauer et al., 2013). Assessments of school climate could rely on perceptions of both students and teachers, but their perceptions of the school environment are likely to differ somewhat as a function of their roles. For example, teacher reports of school climate have been characterized as capturing the context in which students develop, and student reports are believed to be more closely tied to their personal experiences of these contexts (Wang & Degol, 2016). Similarly, Ramsey, Spira, Parisi, and Rebok (2016) point out that student experiences are linked to their schools through expectations that are established and applied by teachers. Variations in student and teacher perspective might also be attributed to their use of different normative frameworks and differences in opportunities to observe student interactions with others. Teachers interact with students largely in the classroom, whereas students have many peer interactions outside of the classroom in less structured situations.

Ratings from both teachers and students have been found to contribute to the validity of school climate estimates. For example, latent variable, multilevel multitrait-multimethod research examined student and teacher ratings of school climate dimensions by disentangling the variance in observed variable ratings that could be attributed to individual informant influences, school-level informant influences, and trait influences. Controlling for these non-trait informant effects, analyses revealed that ratings obtained by both students and teachers were strong indicators of school climate traits in both middle schools (Konold & Cornell, 2015b) and high schools (Konold & Shukla, 2017).

Despite recognition that student and teacher perceptions are important indicators of school functioning (Cohen, McCabe, Michelli, & Pickeral, 2009; Gase et al., 2017), investigators have been slow to incorporate both perspectives. In their review of school climate research, Wang and Degol (2016) found that only 17% of the school climate studies incorporated the perspectives of different informant types and that 50% of studies only considered student reports. In another review, Berkowitz et al. (2017) noted “a tendency to dismiss climate reports of teachers” (p. 26) and reported 77% of school climate studies relied solely on the perspectives of either students (64%) or teachers (13%), with only 6% incorporating reports by both students and teachers.

School-Level Focus

School climate is a school-level construct intended to characterize the school as a whole, but most research on school climate examines student-level effects. Ideally, studies should conduct multilevel analyses that consider both school and individual effects. One review reported that approximately 60% of published studies of school climate and achievement used single-level statistical models (Berkowitz et al., 2017). The most obvious and well-known consequence of failing to model the nested nature of informants within schools is violation of the independence assumption and its impact on estimated standard errors (Raudenbush & Bryk, 2002).

A related concern is that most school climate scales are developed with individual-level analyses as though they were measures of individual student traits rather than school characteristics. Constructs like school climate may have different meaning for individuals (e.g., students and teachers) versus the school itself (Bliese, 2000; Muthén, 1991). Observed ratings of school climate obtained from individual informants are likely to be influenced by both lower (e.g., peers) and higher (e.g., school culture) level influences (Reise, Ventura, Nuechterlein, & Kim, 2005). Perceptions of school climate quality also vary among individuals (e.g., students or teachers) with similar roles in a given school (Konold & Cornell, 2015b). These differences in perspective could arise as a function of their personalities or their immediate peer groups (Wang & Degol, 2016). Examination of within-school informant commonalities allows for the varying perspectives to be evaluated for convergence across informants for purposes of obtaining a more robust assessment of climate at the school level (Dedrick & Greenbaum, 2011; Konold & Cornell, 2015b).

Present Study

The purpose of the present study was to evaluate the authoritative school climate model in relationship to standardized high school academic outcomes. Specifically, the study hypothesized that schools characterized by high levels of structure and student support would be associated with greater student engagement and that engagement would mediate the relationships of structure and support with achievement. As reviewed here, previous studies have examined components of this model, showing that school climate influences engagement and that engagement is an important predictor of achievement, but have not integrated them into a more complex substantive model (see Cornell et al., 2016; Konold & Shukla, 2017). Notably, this model tests the mediating role of student engagement in explaining the association between a positive school climate and academic success. Engagement has been widely recognized as a critical factor in student academic achievement and can be conceptualized as having both an affective and cognitive component (Gase et al., 2017; Mehta et al., 2013; Yang, Sharkey, Reed, Chen, & Dowdy, 2018). Students engage affectively by liking their school, being proud to be a student at their school, and feeling they belong at their school and cognitively by finishing their homework, learning as much as they can, and placing importance on earning good grades. Previous research conceptualizes student engagement as a student characteristic that is a proximal outcome and not a component of school climate. For this reason, student engagement has been identified as an important outcome of efforts to improve school climate (Fredricks et al., 2016). Other research has focused on student engagement as an important factor in student willingness to work hard and be successful in school. This study contributes to the field by linking together these bodies of work and testing whether engagement serves as a mediator between school climate and achievement.

From a theoretical perspective, this study tested a tripartite model linking school climate, individual characteristics, and behavioral outcomes. Specifically, the study examined how high adult expectations for students and supportive adult-student relationships at the school climate level are associated with student engagement, which in turn mediates relations with academic achievement. This is a correlational study that cannot demonstrate causal effects, but it lays the foundation for intervention studies that could provide stronger evidence that school climate improvement has an impact on student achievement by increasing student engagement.

In addition to its substantive focus, this study goes beyond prior investigations of the relationships between school climate and achievement in its methodology in several ways. First, the current study integrates the perspectives of both students and teachers in a large statewide sample of 298 high schools. Although previous studies have relied principally on student reports, the use of both teacher and student informants adds comprehensiveness and credibility to the assessment of school climate. Studies that incorporated more than one perspective typically did so through some form of observed score aggregation across informants to create a single composite or conducted analyses separately by informant type (Enticott, Boyne, & Walker, 2009; Kearney & Peters, 2013; Kumar, Stern, & Anderson; 1993; Van Bruggen, Lilien, & Kacker, 2002; Vaughn & Hoza, 2013). The current examination uses observed variable reports obtained from students and teachers as indicators to create latent variables that capture common sources of trait variance across informant types. While this approach has been described to understand the role of school climate on teasing and bullying in schools (Konold & Shukla, 2017), we focus on the role of school climate in relation to standardized measures of academic performance.

Second, the current study was based on a multitrait-multimethod (MT-MM) design in which multiple school climate traits were measured through the use of multiple student and teacher informants (i.e., methods) within each school. This allowed for disaggregation of school climate trait variance from that which could be attributed to non–trait based variance related to informant method effects and other sources of unreliability, using indicators generated by a correlated trait—correlated method latent analysis (CT-CM; Lance, Noble, & Scullen, 2002). Further, this method of analysis allows for an evaluation of various aspects of the measurement model underlying the trait factors.

Third, the multilevel nature of students and teachers reporting on their schools was also taken into consideration through recently described multilevel methods for CT-CM latent variable models. These models allow for the control of non-trait method effects at the levels of both the informant and the school (Bleidorn & Peters, 2011; Carretero-Dias, Eid, & Ruch, 2011; Koch, Schultze, Burrus, Roberts, & Eid, 2015). Even though the focus of the current study was on school-level relationships, a multilevel analytic strategy allowed for non-trait informant-based sources of variance in observed score ratings to be controlled at the individual level prior to aggregating these ratings to the school and to further control for these influences on climate trait factors at the school level (Konold, 2018)

Fourth, the construct of academic achievement investigated in the current study is based on graduation rates and Standards of Learning examinations completed by all students. These measures go beyond most school climate studies that rely on student self-reported grades, which are susceptible to inflation, or classroom grades assigned by teachers, which can vary in rigor. First administered nearly two decades ago, annual school performance on the Standards of Learning exams has been a criterion for school accreditation and funding in Virginia since 2006 (Virginia Department of Education, 2018).

Fifth, the current investigation acknowledges through design that reports of school climate can be influenced by a variety of demographic contextual factors that are beyond school control (Gottfredson, Gottfredson, Payne, & Gottfredson, 2005; Konold, 2018). These factors include school size (Koth, Bradshaw, & Leaf, 2008), family socioeconomic status (SES) (Khoury-Kassabri, Benbenishty, Astor, & Zeira, 2004), population density (rural to urban locations; Abel & Sewell, 1999), and race. For example, non-White students (De Pedro, Gilreath, & Berkowitz, 2016) and those from lower SES backgrounds (Khoury-Kassabri et al., 2004) have been found to report more negative school climates.

Method

This study was conducted using a statewide sample of students and teachers from 320 high schools who completed the Virginia Secondary School Climate Survey (Cornell et al., 2016). The survey was administered anonymously online.

Participants and Settings

All 322 Virginia public schools serving a general education high school population were eligible to complete a statewide school climate survey. The school participation (N = 320) rate of 99.3% was achieved with the cooperation of the Virginia Department of Education and the Virginia Department of Criminal Justice Services, who endorsed the survey and encouraged participation.

Of the 322 schools eligible for participation, teacher surveys were obtained from 302 schools, and student surveys were received from 320 schools, representing school-level participation rates of 93.8% and 99.3%, respectively. Surveys were received from N = 11,442 teachers in these 298 schools, with an average of N = 39 teachers reporting on each of their respective schools. The teachers were predominately female (66.9%) with teaching experience of more than 10 years (61.1%), 6 to 10 years (18.6%), 3 to 5 years (12.0%), and 1 to 2 years (8.3%). Their race/ethnicity was 83.2% White, 8.3% Black, 3.4% Hispanic, 1.3% Asian, 0.2% American Indian or Alaska Native, 0.1% Native Hawaiian or Pacific Islander, with an additional 3.4% identified with more than one race. To preserve teacher anonymity, no other demographic information was collected.

A total of N = 68,951 students completed the survey. Approximately 80% of the surveys were completed between 7.8 and 21.8 minutes. To improve data quality (Wise, 2017), a multistage screening procedure resulted in the removal of students (2.4%) for completing the survey too rapidly (i.e., less than 6 minutes). Following an established screening procedure to identify students who admit not being truthful (Cornell, Klein, Konold, & Huang, 2012; Jia, Konold, & Cornell, & Huang, 2016), an additional 6.7% were removed for responses to two validity questions (“I am telling the truth on this survey” and “How many of the questions on the survey did you answer truthfully?”). See the technical report (Cornell et al., 2018) for additional information and description of sampling procedures.

The resulting analytic sample of N = 60,441 students (51.2% female) were from 298 different schools, with an average of N = 202 students reporting on each of their respective schools. Students were distributed across 9th (27.4%), 10th (25.9%), 11th (24.7%), and 12th (22.0%) grades. The race/ethnicity breakdown of students was 54.6% White, 17.6% Black, 11.8% Hispanic, 5.9% Asian, 1.0% American Indian or Alaska Native, 0.5% Native Hawaiian or Pacific Islander, with an additional 8.7% reporting more than one race. Approximately 23.1% of the students reported speaking a language other than English at home, and parent education was reported in the categories of completed postgraduate studies (25.1%), completed a four-year college degree (26.5%), completed a two-year college or technical education degree (14.3%), graduated from high school (26.6%), and did not graduate from high school (7.5%).

Measures

Students completed their survey questions in classrooms under teacher or school staff supervision, and both groups followed a standard set of instructions. The surveys consisted of 100 items that included the measures of Structure, Support, and Engagement examined in the present study. The contextual variables (i.e., the percentage of students receiving free and reduced priced meals, school enrollment size, the percentage of White students in the school, and population density) and the academic outcomes (i.e., school graduation rates and Standards of Learning test scores) were obtained from the Virginia Department of Education.

Items on the student and staff surveys covered largely similar content, with wording differences to accommodate differences in roles and/or differences in perspective. For example, on the Engagement scale, students responded to “I like this school,” whereas teachers responded to “Students generally like this school.” See Appendix for a complete list of the items comprising the four observed variable scales used in the current study. Recent multilevel confirmatory factor analyses (CFA) and structural models of these items and scales revealed good psychometric properties when examined on the basis of student (Konold & Cornell, 2015a) and teacher (Huang & Cornell, 2016) responses.

Structure was evaluated through items that align with the dimensions of Disciplinary Structure and Academic Expectations. Previous multilevel CFA of these items in high school samples revealed student-level standardized pattern coefficients that ranged from .36 to .93 and school-level estimates that ranged from .65 to .99, with average student- and school-level reliability estimates of .75 and .91, respectively (Konold & Cornell, 2015a). Standardized pattern coefficients based on a sample of teacher responses ranged from .63 to .82, with school-level values ranging from .61 to 1.0 and average teacher- and school-level reliability estimates of .74 and .81, respectively (Huang & Cornell, 2016).

Support measured the perception that teachers and other school staff members are supportive through scales labeled Respect for Students and Students’ Willingness to Seek Help. Prior research employing these items revealed that high schools characterized by higher levels of support had less bullying and peer victimization as reported by ninth-grade students and their teachers (Gregory et al., 2010). In addition, previous multilevel CFAs of these items in high school samples revealed that student-level standardized pattern coefficients ranged from .36 to .87 and school-level estimates that ranged from .67 to 1.0, with average student- and school-level reliability estimates of .80 and .85, respectively (Konold & Cornell, 2015a). Standardized pattern coefficients based on a sample of teacher responses ranged from .54 to .92, with school-level values ranging from .60 to .96 and average teacher- and school-level reliability estimates of .86 and .79, respectively (Huang & Cornell, 2016).

Student Engagement in school was measured with six items that were derived in part from the Commitment to School scale (Thornberry, Lizotte, Krohn, Farnworth, & Jang, 1991). The scale taps into both affective and cognitive aspects of student engagement. Student-level standardized pattern coefficients ranged from .68 to .93, and school-level estimates ranged from .35 to 1.0, with average student- and school-level reliability estimates of .80 and .84, respectively (Konold & Cornell, 2015a). Standardized pattern coefficients based on a sample of teacher responses ranged from .54 to .93, with school-level values ranging from .76 to 1.0 and average teacher and school-level reliability estimates of .78 and .94, respectively (Huang & Cornell, 2016).

Standards of Learning (SOL) achievement tests are administered annually in the state of Virginia to determine whether students and schools meet state requirements for achievement in English, mathematics, history, and science. These state-mandated subject tests are intended to measure student learning and achievement. They were developed using test blueprints, item development specifications, review committees, field testing, and item banking. These procedures were used to limit item bias and ensure appropriate item difficulty and content coverage. English reading tests are administered in Grades 6 through 8 and 11, and English writing tests are administered in Grades 8 and 11. Students take mathematics tests in Grades 6 through 8 and at the end of Algebra I, geometry, and Algebra II. Science tests are administered at the end of Grade 8 and biology, chemistry, and earth science courses. Because fewer students take some of the more advanced courses (e.g., Algebra II and chemistry), the current study focused on the six most commonly administered SOL subject exams completed in high school: English reading, English writing, Algebra I, geometry, earth science, and biology. For all tests, student scaled scores range from 0 to 600, with scores above 400 considered to be “pass/proficient” and scores above 500 considered to be “pass/advanced.” The most current technical report available from the 2015 test administration cycle indicates that reliability (Cronbach’s alpha) for all assessments are >.80 overall and by gender and for Black and White subgroups (Virginia Department of Education, n.d.-b). Although individual student-level scores were not available, mean scores by school were obtained from Virginia Department of Education (n.d.-a).

Analytic Plan

The full structural model is shown in Figure 1. The factors of School Structure, Student Support, and Student Engagement were derived from both students and teachers. These school-level factors were estimated through multilevel correlated trait–correlated method procedures to account for the fact that students and teachers were nested within schools (i.e. multilevel) and that reports from multiple informants were available to serve as indicators of their respective factors (Konold & Shukla, 2017; Lance et al., 2002).

Figure 1.

Full multilevel multitrait-multimethod model with school-level authoritative school climate structural linkages.

Informant effects, unrelated to the traits being measured, were explicitly estimated at both the individual (i.e., Level 1, left side of Figure 1) and school (i.e., Level 2, bottom right side of Figure 1) levels, with substantive trait effects of primary interest estimated at the school level. The observed variables (enclosed in boxes) were modeled to be directly influenced by Level 1 student and teacher method effects (left side of Figure 1). All raw score scales were separately standardized for students and teachers (M = 50, SD = 10) to aid interpretation. The Level 1 method factors estimate the unique individual perspectives of informants. These are included in the model to control for unique non-trait sources of variance (e.g., halo or horn effects) in informant ratings that could contaminate school-level trait estimates when aggregated at the school level. Covariances between these method factors were fixed to zero (Eid et al., 2008). Estimated school-level true (T) scores for each measured trait obtained by students (s) and teachers (t) are shown in circles (Tstr,s to Twtsh,t). True scores represent the expected value (or average) of school ratings across raters within a school (Carretero-Dios et al., 2011). These true scores are modeled to be directly influenced by school-level traits and school-level method effects (right side of Figure 1). School-level trait and method factors represent the common shared effects of informants within schools (i.e., across informants within schools), where trait and method factor covariances were fixed to zero (Eid et al., 2008).

The structural components of the substantive theoretical model are illustrated in Figure 2. The school-level trait factors of structure and student support were modeled to be influenced by school and community contextual variables, have a direct influence on both student engagement, and be indirectly associated with academics. Moreover, student engagement was modeled to have a direct influence on academics. Measures of school climate were obtained by informants midway through the academic year, and student achievements on the academic measures were obtained at the end of the year. The model was estimated through full information maximum likelihood in Mplus version 8.0.

Figure 2.

Substantive school-level structural elements from the full multilevel multitrait-multimethod model.

Statistical tests of mediating effects have received increased attention in recent years due to the typically asymmetric sampling distributions of the product terms that are used to evaluate whether they are statistically greater than zero (Darlington & Hayes, 2017). One common approach to overcoming this problem involves the use of bias-corrected bootstrap confidence intervals (see e.g., Lau & Cheung, 2012). However, this procedure is not currently available in Mplus for multilevel structural models. As a result, we used a Monte Carlo–based parametric bootstrap approach that makes no assumptions about the sampling distribution of the product term (Preacher, Zyphur, & Zhang, 2010). R code for estimating 95% confidence intervals for the indirect effects were generated through use of the Monte Carlo Method for Assessing Mediation (MCMAM; Selig & Preacher, 2008) software tool. Confidence intervals for the indirect effects were obtained on the basis of asymptotic variances and covariances of the unstandardized parameter estimates of the direct effects surrounding a given mediating effect and were based on 20,000 replications.

Four measures of fit were considered in evaluating model quality: Tucker-Lewis Index (TLI), Comparative Fit Index (CFI), root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR; Browne & Cudeck, 1993; Hu & Bentler, 1995). These four measures generally range between 0 and 1.0. Values of .90 or greater (Bentler & Bonett, 1980) or .95 or greater (Hu & Bentler, 1999) on the first two measures are often taken as evidence of good fitting models. Alternatively, smaller RMSEA and SRMR values support better fitting models.

Results

Intraclass correlations (σ²_B / [σ²_B + σ²_W]; Muthén, 1991) reflect the degree to which variability in individual informant ratings can be explained by the schools in which they reside. Values ranged from .03 to .08 across the student measures and from .06 to .28 across the teacher measures. School-level descriptive statistics for all observed variables are shown in Table 1. Measures of fit for the structural model in Figure 1 without the community contextual variables were favorable (CFI = .981, TLI = .976, RMSEA = .018, SRMR_L1 = .018, SRMR_L2 = .067). Although inclusion of these contextual variables resulted in an increase in the SRMR_L2 (= .176), values of this size are not necessarily reflective of poor fit, particularly when the quality of the measurement model is strong (McNeish & Hancock, 2018; Miller et al., 2018). Completely standardized factor coefficients for the measurement portion of the structural model are shown in Figure 1, and structural linkages among the latent variables are shown in Figure 2. This format was adopted to provide a clearer presentation of results.

Table 1

School-Level Observed Variable Means and Standard Deviation

	Students		Teachers
Observed Variables	M	SD	M	SD
Structure	50.0	10.0	50.0	10.0
Student engagement	50.0	10.0	50.0	10.0
Respect for students	50.0	10.0	50.0	10.0
Willingness to seek help (WTSH)	50.0	10.0	50.0	10.0
		Schools
		Mean	SD
Percentage graduation rate		91.24	8.14
English Reading SOL (Grade 11)		442.12	12.38
English Writing SOL (Grade 11)		449.07	27.33
Algebra I SOL		415.21	13.13
Geometry		427.52	19.22
Earth science		434.43	16.95
Biology		439.42	17.36

Note. SOL = Virginia State-Mandated Standards of Learning Tests.

Method, Trait, and Behavioral Outcome Factors

Informant-based method effects were modeled at both the individual (i.e., Level 1) and school (i.e., Level 2) levels of analysis to remove systematic variance attributable to students and teachers that was unrelated to the trait being measured. The unique Level 1 method effects of student and teacher raters were modeled to directly influence their individual observed ratings (left side of Figure 1). These method effects are unique in that they are not shared among other raters within a school and model the degree to which different students or different teachers vary in their ratings of the same trait (Nussbeck, Eid, Geiser, Courvoisier, & Lischetzke, 2009). Completely standardized coefficients linking the unique method factors to their respective indicators were all statistically significant (ps < .001) and moderate to large in magnitude. Results also indicated statistically significant variation among the students (S² = 69.91) and teachers (S² = 45.21) with respect to the unique method factors. Contrasts between these Level 1 unique method variance estimates and Level 2 student (S² = 5.51) and teacher (S² = 16.03) method factor estimates obtained at the school level (see bottom of Figure 1) suggests that much of the unique method variance is likely due to rater differences that are a function of the schools in which they reside. In other words, much of the non-trait informant-based variation among N = 60,441 students and N = 11,442 teachers was captured at the school level.

The influence of common school-level method effects (i.e., effects that are common to all teachers or student within a school) were modeled to directly influence their respective school-level true score indicators (bottom right side of Figure 1). The resulting coefficients associated with these paths represent shared method effects at the school level among students or teachers. All coefficients were statistically significant (ps < .001) and reveal a shared common perspective among student raters and among teacher raters of their respective schools that is not captured by the traits being measured.

The common school-level trait factors of structure, student support, and student engagement are shown in the middle of Figure 1. These substantive trait factors were modeled to directly influence their respective true score indicators. Standardized trait factor loadings linking the average ratings of the schools by students and teachers with the common school-level traits they were designed to assess were all statistically significant (ps < .001). Notably, these estimates were free of measurement error and non-trait informant-based variance that can occur at both the individual and school levels. Lastly, the academic factor is shown on the right side of Figure 1. All standardized factor loadings were statistically significant (ps < .001) and large.

Substantive Linkages

Structural relationships among the school-level trait factors and the academic outcome factor (middle of Figure 1) that are of primary substantive interest are further illustrated in Figure 2 to more explicitly show how the school and community contextual variables were modeled. As expected, schools with higher percentages of students receiving free and reduced priced meals tended to have lower levels of structure (β = –.44, p < .001) and student support (β = –.48, p < .001). By contrast, the actual number of students enrolled in the school was not found to be related to structure (β = .05, p > .05) or student support (β = –.10, p > .01) in the school. Schools with higher percentages of White students were likely to report more structure (β = .39, p < .001) and student support (β = .44, p < .001), and schools located in more populous (less rural) communities also tended to report more structure (β = .43, p < .001) and student support (β = .37, p < .001). In combination, these contextual variables accounted for 53% of the variance in the structure of schools (R² = .53) and 56% of the variance in the student support of schools (R² = .56).

Schools with higher levels of structure (β = .41, p < .001) and student support (β = .52, p < .001) were likely to have more engaged students. Together, structure and support accounted for 65% of the variance in student engagement (R² = .65). Both structure and support were also found to have an indirect effect on academics through student engagement (β = .31 and .40, respectively, ps < .001), where the 95% Monte Carlo–based confidence intervals for the unstandardized indirect effects were 0.43 to 1.74 for structure and 1.62 to 13.32 for support. In addition, engagement was positively associated with the academic factor (β = .77, p < .001). In combination, the authoritative school climate model accounted for 77% of the variation in school-level academics (R² = .77).

Discussion

This study demonstrated a strong association between a positive school climate and school-level academic achievement that advances our understanding beyond previous studies by using a more specific conceptual model of school climate and a more rigorous multilevel, multitrait-multimethod analysis of latent variables using a large statewide sample of 298 high schools. Although some previous studies have demonstrated an association between school climate and student engagement and other studies have shown an association between student engagement and academic outcomes, the present study tested a more comprehensive model that demonstrated hypothesized linkages among school climate, student engagement, and academic outcomes.

From a theoretical perspective, these findings support a model of school climate as consisting of characteristics of the social environment reflected in the interpersonal interactions and relationships among students and adults in a school. A positive school climate is associated with individual student characteristics that in turn serve as mediators for student behavioral outcomes. Specifically, the study found that characteristics of an authoritative school climate in the form of high adult expectations for students and supportive adult-student relationships are linked to higher student engagement, which mediated relations with academic achievement. This tripartite model could be applied to other student outcomes such as prosocial behavior and absence of aggressive behavior.

From a practical perspective, this study points toward the potential value of school climate interventions as a means of increasing student engagement, leading toward greater learning and achievement. School authorities could monitor student engagement as a more proximal outcome of intervention efforts and indicator of whether more distal outcomes such as performance on end-of-year testing are likely. School climate interventions and related efforts to enhance student engagement could be especially valuable to overcome socioeconomic disadvantages that are linked with low achievement and high rates of school failure and dropout. This is a correlational study that cannot demonstrate causal effects, but it lays the foundation for experimental and longitudinal studies that could provide stronger evidence.

The hypothesized model also examined the effects of school-level demographics on authoritative school climate constructs and how the school climate constructs of structure and support were related to student engagement and academic achievement. Academic achievement was measured by school graduation rates and school-level performance on six standardized achievement tests. Schools with a more authoritative school climate, as indicated by high levels of structure and support, had higher levels of student engagement, with both direct and indirect effects on academic achievement. The authoritative model accounted for 65% of the school-level variance in student engagement and, including student engagement, accounted for 77% of the variance in academic achievement. These findings provide new evidence for understanding differences between schools in their social climate across schools that varied in school demographics such as school size, student poverty, and racial/ethnic composition. Moreover, these climate factors were strongly related to differences in academic achievement.

The present study overcomes some of the common limitations found in previous research. Despite general recognition that the use of multiple informants is considered best practice when measurement is based on informant ratings (Bauer et al., 2013), recent reviews of the school climate literature reveal that evaluations are typically made on the basis of either student or teacher reports (Berkowitz et al., 2017; Wang & Degol, 2016). Our analyses provide for a more comprehensive perspective of school-level climate that capitalized on the shared perspectives of students and teachers through methods that allowed for extraction of student and teacher communalities in their ratings (Dedrick & Greenbaum, 2011). Because students and teachers have markedly different roles and perspectives on school conditions that might influence their ratings, an analysis that uses both sources as indicators of school climate traits has the potential to provide a more accurate and discriminating characterization of the school.

The extent of common rater method effects (i.e., effects that are common to all teachers or students) that are unrelated to the school-level climate traits being assessed through ratings obtained from these informant types can be described in at least two ways. First, this influence was revealed through examination of the school-level informant method factor loadings linking the student and teacher factors to their respective average school-level ratings (see bottom right side of Figure 1). All coefficients were statistically significant and moderate to large in size for students and teachers. These completely standardized estimates for students ranged in value from .75 to .98 and from .36 to .98 for teachers. Second, method specificity coefficients measure the proportion of true-score variance that is attributable to common informant effects unrelated to the trait being measured (Eid et al., 2008). In other words, common specificity coefficients reflect the extent to which reports obtained by students and teachers are a specific type of informant and reveal a shared common perspective among students and teachers that is independent of the school traits being measured. Average estimates across measures were .50 for students and .38 for teachers. In the aggregate, these results suggest that ratings of school climate contain non-trait variance that can be attributed to the use of informants.

At the same time, agreement between students and teachers can also be examined in at least two ways. First, school-level consistency coefficients provide an estimate of the proportion of true-score variance that is accounted for by the school climate trait factors and reflect the extent to which reports obtained by different informants are consistent (Nussbeck et al., 2009). Across measures, average consistency estimates were .50 for students and .62 for teachers. Standardized factor loadings for the trait factors were also statistically significant and revealed cross-informant patterns that were similar to those reported elsewhere when the focus was on scales comprised only of common items (Konold, 2018).

Second, comparisons between standardized trait factor loadings for students and teachers across measures revealed that teacher ratings were somewhat better indicators than student ratings of their respective traits for measures of support and engagement. By contrast, student ratings of structure were more strongly associated with that trait factor than teacher ratings (see Figure 1). Student reports may be more influenced by their personal experiences and peer interactions, whereas teachers may be more likely to take a holistic view of the school. This might give them a generally more discriminating view of support than students, whose responses might largely reflect their individual experiences with teachers and other adults at school. This difference in perspective might explain why the loadings for school support were higher for teachers (.86 for respect for students, .63 for willingness to seek help) than for students (.59 and .22, respectively). It may also help explain why loadings for engagement were higher for teachers (.93) than for students (.66). One also might expect that teachers are consistently more prone to perceive school discipline as fair than are students, which could make teacher ratings less discriminating. Accordingly, the loading for school structure was higher for students (.56) than teachers (.22). By using both teacher and student sources as indicators of school climate traits, these analyses have the potential to provide a more accurate and discriminating characterization of the school.

Many studies of school climate are limited by a lack of independence between independent and dependent variables. For example, Cornell et al. (2016) found that student perceptions of school climate were associated with student reports of their own course grades and educational aspirations. However, shared method effects might inflate the associations between school climate and academic measures when they are measured by the informants on the same survey. Similarly, Gase et al. (2017) found that student perceptions of school climate were consistently associated with student reports of well-being but found few associations using independent measures of staff reports of school climate or school administrative measures of school climate. The present study used academic outcomes—school-level performance on standardized achievement tests and school graduation rates—that were obtained independently of student and teacher reports of school climate.

Student demographics such as school size, family income, racial/ethnic composition, and urbanicity are frequently associated with school climate (Berkowitz et al., 2017; Gottfredson et al., 2005; Khoury-Kassabri et al., 2004; Konold, 2018). In this study, three of the four demographic variables, excepting school size, were associated with the school climate measures of structure and support. Consistent with previous research, schools with students from low-income families and minority backgrounds and located in more densely populated urban communities were independently associated with less favorable school climates. Controlling for the influence of these demographic variables on the school climate trait factors, schools with higher levels of both structure and support had higher levels of student engagement. Structure was directly associated with academic achievement, but the relation between support and academic achievement was fully mediated by student engagement. These findings are important because they suggest that even in schools serving disadvantaged student populations, school climate is associated with student engagement and academic achievement. Several studies have suggested that a positive school climate can buffer the negative impact of poverty on achievement (Battistich Solomon, Kim, Watson, & Schaps, 1995; Wang & Holcombe, 2010).

It is important to acknowledge the limitations of this study. This is a cross-sectional, correlational study that cannot provide strong evidence of a causal relationship or isolate the direction of effects. There is a general assumption across most school climate research that a positive school climate will facilitate student engagement in school and that this will enhance academic achievement. However, Benbenishty, Astor, Roziner, and Wrabel (2016) tested the causal direction of the association between school climate and school academic performance using a cross-lagged panel autoregressive model with the California Healthy Kids Survey; contrary to their hypotheses, they found support for the opposite direction of causal effects, with higher academic performance leading to an improved school climate rather than the other way around. More longitudinal studies are needed to test whether and how improvements in school climate lead to higher school academic performance.

Concept of School Climate

The nationwide movement to improve school climate makes it critical to define the construct clearly so that it can be measured accurately. The U.S. Department of Education (2013) described school climate broadly as “a multi-faceted concept that describes the extent to which a school community creates and maintains a safe school campus, a supportive academic, disciplinary, and physical environment, and respectful, trusting, and caring relationships throughout the school community” (p. 2). Critics have pointed out that definitions of school climate are so broad that they “encompass just about every feature of the school environment that impacts cognitive, behavioral, and psychological development” (Wang & Degol, 2016, p. 3). Rudasill and colleagues (2017) described research on school climate as “a chaotic conceptual landscape” (p. 7) because definitions often fail to distinguish what school climate is and what it is not. One of the contributions of this study was to test a more specific and conceptually coherent model of school climate based on authoritative school climate theory. The measures of structure and support are key qualities identified across multiple studies and are clearly within the boundaries of multiple conceptualizations of school climate.

School climate is a metaphorical term that needs a clearer conceptual foundation (Cornell & Huang, 2018). By comparison, the meteorological climate of a city refers to the patterns of weather that characterize the area and distinguish it from other areas. Analogously, the concept of school climate refers to the patterns of daily social interactions in the school that distinguish it from other schools. The school’s climate should be distinguishable from other elements of the school environment, such as the condition of the building and the demographics of its students. Otherwise, the term school climate means little more than “the school.” In their ecological systems model of school climate, Rudasill et al. (2017) contended that certain school features should be defined as outside the domain of school climate, although they may influence it. The present study demonstrated the value of distinguishing school climate from student engagement and achievement and advances the idea of school climate as a system that affects other aspects of the school and its stakeholders. Under this model, a positive school climate leads students to be more engaged in school and results in higher academic performance by the school. This formulation does not exclude the notion of reciprocal effects or feedback loops, such as the academic success of a school leading its students and staff to feel proud of themselves and place greater value on learning and teaching. Longitudinal studies and intervention trials will make it possible to elucidate these systemic qualities.