Abstract
Although American schools are required to meet civic education goals of preparing students to become active and informed citizens, high-quality civic opportunities (e.g. service learning and volunteering) are consistently less available to youth of color who are typically enrolled in schools located in high-poverty communities. The purpose of this study is to evaluate efficacy of the Word Generation program to improve students’ self-reported civic engagement (
More than half of eligible young Americans (aged 18–29 years) did not vote in the 2012 elections, which raises concerns about the political health of this country (The Center for Information and Research on Civic Learning & Engagement (CIRCLE), 2013). Adler and Goggin (2005) explain that fulfilling one’s civic duty to vote is connected to a broader field of civic engagement, which includes the community context (e.g. volunteering and helping a neighbor) and electoral context (e.g. voting, donating money to political candidates, and protesting). As much as family, friends, and media can influence youth to become active in their communities, schools can also play a pivotal role in providing civic learning opportunities to adolescents (Gibson and Levine, 2003). In the landmark report called
Literature
Civic engagement among youth
A 2006 report by the CIRCLE published results from civic engagement surveys completed by 1300 American youth aged 15–25 years (Lopez et al., 2006). The report indicates that Hispanic youth lagged behind their White, Asian, and African American counterparts in various civic engagement activities including volunteering for a non-political group, engaging in community solving, and voicing their political opinions (e.g. contacted an elected official and joined a boycott). The 2006 CIRCLE report also suggests that Asian American youth were behind other racial groups in several key areas of civic engagement: volunteering for a charity and discussing current events with friends and family. High-quality civic learning opportunities are also consistently less available to ethnic minority youth (Levinson, 2012). In a study of students’ experiences (
The Word Generation (WG) program
The Word Generation (WG) program is a cross-content literacy program delivered at the classroom or grade level that instructs students to learn five all-purpose academic words, which are embedded in brief passages covering a different controversial issue each week (Snow et al., 2009). The program features controversial topics based on a broad range of political and science-based controversies, such as whether the government should fund stem cell research or animal testing. As part of the WG program, teachers in the four main content areas – English language acquisition (ELA), social studies, science, and math – present materials related to the controversial issue of the week and explore academic language that is embedded in the curriculum through discourse and writing.
For 15 minutes a day, teachers and students engage in program activities related to vocabulary instructions and classroom discussions (see Figure 1 on sequence of weekly activities). On Monday, the ELA teacher presents a text that starts with a narrative about a girl who was accidentally shot and paralyzed, but for whom stem cell research might offer a cure; the rest of the brief text presents arguments

Sample weekly schedule of the Word Generation program.
Various studies have been conducted on quasi-experimental and experimental trials of the WG program that was implemented in several urban school districts across the United States. A broad range of research topics has been explored, including writing outcomes (Mancilla-Martinez, 2010), vocabulary development (Lawrence et al., 2015), and differentiated effects for English language learners (Hwang et al., 2015). This is the first study to assess whether enrollment in the WG program can improve students’ self-reported civic engagement and voting interest.
Controversial issues as a basis for classroom discussion
Few studies have assessed the effects of using controversial issues in the classroom (Hess and Posselt, 2002; Venville and Dawson, 2010). One study (Venville and Dawson, 2010) examined a 10-week intervention designed to improve students’ argumentation skills by engaging in classroom debates on the moral dilemmas associated with genetic engineering. In the quasi-experimental study, 10th grade students participated in treatment (
Exposure to issues-centered curricula can improve students’ civic engagement
Alex Romeo Lin (2015) published a review documenting the last 25 years of research on civic education programs and highlighted four issues-centered curricula – CityWorks, Kids Vote, Project Citizen, and Student Voices – that have been tested in various research contexts (Hartry and Porter, 2004; McDevitt and Chaffee, 2000). Classroom discussions of various controversial topics are usually encouraged as part of the curriculum. Compared to evaluations of the other issues-centered curricula (e.g. Cityworks, Kids Vote and Project Citizen), evaluations on the Student Voices (SV) program are notable for its quasi-experimental design (with comparison groups) that involves multiple research sites (Feldman et al., 2007; Pasek et al., 2006; Syvertsen et al., 2009). Syvertsen et al. (2009) examined the effects of the SV program on high school students who enrolled in social studies classrooms that participated in treatment (
Assessing ethnic minority youth’s experience with issues-centered curricula
This section reviews current understandings about how students of color may respond to issues-centered curricula in schools. Compared to youth from other ethnic backgrounds, classroom supports for discussions of controversial issues is particularly important for Hispanic youth. One study (Torney-Purta et al., 2007) examined school and neighborhood factors that contribute to the civic engagement gap between Hispanic (
Methods
Research design
One urban school district located in the West Coast of the United States participated in an evaluation study of the WG program. The Strategic Education Research Partnership (SERP) recruited leaders from the school district, who then invited their school leadership teams to participate in the randomized study. School district leaders informed parents and students about the study. Students obtained parental consent to participate and release information on their academic and psychological performances for research purposes. In the first phase of the randomization process, state accountability data were used to rank schools on several school-level variables based on student enrollment and prior academic achievement. Each school received general composite scores computed from state accountability data that were used to rank each school on several school-level variables based on the percentage of students identified as ethnic minorities, low-income status, English language learners, as well as the school’s prior academic achievement. The composite scores were then used to determine each school’s ranking. Each sequential pair of schools was assigned into dyads and the final process of randomization occurred within each dyad. As a result, six schools were assigned to the treatment and six schools into the control conditions. Teachers in the treatment conditions were required to implement the WG program; whereas teachers in the control conditions did not use the WG program and participated in “business as usual” instructions (Lawrence et al., 2015).
Preliminary analysis indicates a consistently high level of implementation during specific times that observers were present; however, students’ completion rates of workbooks varied and tended to decrease precipitously toward the end of the year. Observers found that teachers implemented the Day 1 program activities in a “smooth and efficient” manner in 83% of the 96 observed class periods. In 89% of the observation periods, observers found that teachers and students brought their workbooks to class. Students’ workbook completion rate varied among schools from 44.4% to 89.9%. Furthermore, workbook completion rate decreased over time from the start in Week 1 (82%), midpoint at Week 12 (69%), and at the end in Week 24 (57%). These preliminary analyses suggest that program implementation was quite consistent in the beginning, but may be noticeably inconsistent throughout the school year.
Participants
Although the study analyzes an equal number of schools that were randomly assigned to treatment and control conditions, the analytic sample contains more treatment students (
Grade level contributions by treatment conditions (analytic sample).
WG: Word Generation.
Pseudonyms were used to protect the schools’ identity.
Table 2 indicates that the sample analyzed in this study has slightly more females (52.3%) than males (47.7%). Nearly two-thirds of the participants were from low-income background (63.9%), as indicated by their eligibility in the Free and Reduced Lunch (FRL) program at their respective schools. A substantial number of students come from extremely diverse ethnic background: Asian (54.2 %), Hispanic (19.4%), and African American (6.9%). Among language minority students, there were quite a number of students classified as Redesignated (44.2%) and Limited English Proficient (12.6%).
Demographic variables of students by treatment status.
Language Status – not listed were seven students who were categorized as “pending”; among Asian students, 2240 (78%) were identified as Chinese; primary language measured by the most frequent language used at home; 629 (63.5%) of Hispanic students indicate Spanish as their primary language; 5169/5786 (89.3%) of the students had demographic data.
Procedures
Classroom observations were conducted in all major content-area classrooms (ELA, math, science, and social studies) across treatment and control schools. Observers attended training sessions that instructed on using standardized codes to rate conversational moves observed among students and teachers in video footage of classroom discussion. In the classroom evaluation phase, participating teachers were randomized and a schedule was created identifying specific observation periods to be conducted on a given day.
Prior to test administration, we conducted cognitive lab interviews with middle school students enrolled in the school district to develop meaningful survey items appropriate to the context of youth from diverse population. The data were analyzed to develop appropriate survey items on civic engagement. During the two testing days in May 2012, students in treatment and control conditions completed a number of civic engagement measures. At the time of assessment, students in the treatment group were essentially at the point of fully completing the program.
Students also completed the Gates-MacGinitie reading comprehension assessment in Fall 2011 and Spring 2012. The test is a 48-item assessment used to measure students’ overall reading comprehension. The test includes 14 short multiple-paragraph passages from a wide range of domains including narrative, autobiographical, biology, and history to measure comprehension at the passage level. For the purposes of evaluating the WG program, the school district provided demographic data with information on students’ gender, grade level, ethnicity, school status (language minority status, GATES, and special education), and eligibility in the FRL status program. Contribution rates of demographic data were quite high among students (
Measures
Classroom observations
Student surveys
District data
For all analyses, demographic variables were included for students’ treatment/control assignment, ethnic-racial identity, gender, and socioeconomic status. Eligibility in the FRL program at school was used as a proxy indicating students from low-income background. Regarding students’ designated school status, students were coded as whether they participated in a Special Education and Gifted and Talented Education (GATE) program. Language minority students consist of students receiving additional ELA at school (Limited English Proficiency) and students who achieved a minimal English proficiency and do not have additional English support (Redesignated). Students’ academic ability measure is based on students’ pretest scores (Fall 2011) on the Gates-MacGinitie reading comprehension.
Data analytic plan
This section provides an overview of how the data were prepared and then discusses the analytical strategies used to answer each of the research questions.
Data preparation
This study replicates the approach used in a past WG evaluation (Lawrence et al., 2015), which involves creating a composite of the four rubrics used to measure the quality of classroom discussion and later transformed into a weighted school-level discussion quality ratings. This process facilitates comparisons of scores across classrooms that accounts for different content areas observed:
Components of the civic engagement scaled score.
SD: standard deviation.
Analytic strategy
Hypothesis 1 is tested by analyzing classroom observation data used to determine whether there are significant differences in classroom discussion between treatment and control conditions. Before discussing the statistical test used to answer this research question, it is important to clarify the approach of analyzing classroom discussion quality at the observation level similar to a previous WG evaluation (Lawrence et al., 2015). At the core of implementing the WG program are grade level teaching teams of ELA, science, social studies, and math teachers. However, there was an uneven distribution of observations conducted across content areas. For example, if a teaching-team observation score contained only social studies with no ratings for math and science, then this score could not be generalized to the other content areas classrooms associated with this particular teaching team. Thus, the analysis considers analyzing classroom discussion quality at the observational level, rather than teaching-team level.
Hypothesis 2 is answered by determining whether participating in the WG program has a significant impact on students’ self-reported civic engagement and voting interest. This analysis utilizes regression analyses with adjusted standard errors that accounts for clustering at the teaching-team grade level. Including a cluster option (“vce”) in the regression command (STATA software) assumes that observations are clustered within grade level teaching teams, which takes into account that observations may be correlated (Murnane and Willett, 2010). The hypothesized regression model is:
where
Hypothesis 3 is tested by analyzing interaction terms added to the model to determine whether students’ ethnic identification moderates the relationship between program participation and self-reported civic engagement. The hypothesized model consists of determining whether the moderator (racial ethnicity) alters the relationship strength between the predictor (WG program participation) and the outcome (civic engagement). A moderator effect is present if the interaction term is found to be statistically significant (Bennett, 2000; Frazier et al., 2004).
Results
Table 4 presents classroom discussion ratings in the 139 observations sampled across content-area classrooms. Although control classrooms received higher discussion scores in science (Δ = 0.24,
Classroom discussion quality rating by treatment status and content-area observed with
SD: standard deviation; ELA: English language acquisition.
Discussion quality rating is an average score (out of a possible total of 4) based on (1) teacher talk moves (out of 5), (2) substantive contributions (out of 5), (3) support for participation (out of 3), and (4) student engagement (out of 3).
Hypothesis 1 is tested by analyzing the weighted total scores of classroom discussion in treatment and control schools. Classrooms using the WG program received higher overall weighted scores on classroom discussion quality than those in the control schools. Based on
The first section is descriptive analyses based on comparing mean scores of students’ civic engagement (Table 5) and voting interest (Table 6) between those in treatment and control schools. Among sixth and seventh graders, students in the treatment schools reported significantly higher civic engagement scores than those in control schools (see Table 5). In regards to students across different ethnic identification, Asian and African American students who participated in the WG program were found to have higher self-reported civic engagement than their counterparts in the control schools. According to Table 6, seventh and eighth students who participated in the WG program reported higher voting interest than their counterparts in the control schools. African American students reported higher self-reported voting interest than their counterparts in the control schools; this was not found to be the case for Asian and Hispanic students. White students in treatment schools reported higher civic engagement and voting interest in comparison with their counterparts in control schools. The next analysis is used to answer the study’s research question, which consists of estimating the WG program’s effects on students’ self-reported civic engagement while taking into account possible confounding variables.
Comparison of students’ self-reported civic engagement standardized scores by treatment status organized by grade level, gender, ethnicity, and school status with
SD: standard deviation.
Civic engagement is based on a 3-item measure relating to students’ reported frequency of helping their friend, school, and community on a Likert scale of 1 “rarely” to 5 “always”; FRL refers to free and reduced lunch status.
Comparison of students’ voting interest (unstandardized scores) by treatment status organized by grade level, gender, ethnicity, and school status with
SD: standard deviation.
Voting interest is a single item measure that reflect interest in voting when student reaches legal age on a scale of 1 “not at all interested” to “extremely interested”; FRL refers to free and reduced lunch status.
Table 7 presents results from regression analyses with predicted effects on students’ self-reported civic engagement (Outcomes 1A-1E) and voting interest (Outcomes 2A-2E). The table presents models with the following characteristics: demographic and school covariates (Model A), covariates with treatment variable (Model B), covariates with political interest (Model C), covariates with treatment and political interest (Model D), and inclusion of interactions (Model E).
Regression models predicting students’ self-reported civic engagement (scaled score) and voting interest (single measure) with standardized coefficients and adjusted standard errors in parentheses.
Standard error adjusted for 36 teaching-team clusters (in parentheses); Coefficients for Other-Non-White, Southeast Asians, Primary Language status not shown for the sake of space; FRL refers to eligibility in the Free and Reduced Lunch program; Academic ability based on Gates-MacGinitie reading comprehension; LEP refers to Limited English Proficient; RFEP refers to Redesignated.
Sixth graders used as reference group.
Male used as reference group.
White students used as reference group.
English proficient students used as reference group.
Model 1A indicates that students identified as Hispanic are less likely to report being civically engaged (
Models 2A–2E measure the extent to which treatment and various covariates are related to students’ voting interest. Students identified as Asians and Hispanics report significantly lower interest in voting than their Caucasian counterparts. On a side note, Academic ability and political interest is strongly related to students’ voting interest. Model 2D indicates that the WG program has no effect on students’ voting interest. In sum, these results support mixed support of Hypothesis 2: WG program improved students’ self-reported civic engagement, but not voting interest.
According to Table 7, Model 1E predicts students’ self-reported civic engagement and includes the interaction terms between treatment and various indicators of students’ ethnic background. No significant interactions were found between treatment and all ethnic identifications (i.e. Asian, African American, and Hispanic). Model 2E predicts students’ voting interest and also includes the same interaction terms examined in Model 1E. No significant interactions were found between treatment and ethnic identifications. These findings do not support Hypothesis 3 that Asian, African American, and Hispanic students gain additional benefits on their self-reported civic engagement and voting interest.
Discussion
Increasingly concerned about the national declines in civic engagement among adolescents, experts in the civic education field have been looking for innovative ways of connecting students with their school, community, and beyond (Carnegie Corporation and CIRCLE, 2003). Whereas past evaluations found that comprehensively designed curricula incorporating a broad range of civic activities can improve students’ civic engagement (Feldman et al., 2007; McDevitt and Chaffee, 2002), a modest intervention primarily focused on literacy development can achieve similar goals. Trained observers generally found higher levels of classroom discussion quality in classrooms using the WG program in comparison with those in control schools. More importantly, this study provides support for the central research question that participating in the WG program was effective in improving student’s self-reported civic engagement, though not for voting interest. These findings are particularly noteworthy because program effects were supported in the context of a randomized trial. Further exploration of these treatment effects indicate that certain ethnic groups did not gain additional benefits. This section provides implications on how the findings from this study contribute to relevant theories and research.
This study found that implementing the WG program led to more active and enriching discussions in classrooms than those in the control schools. Treatment classrooms had higher classroom discussion scores in math, ELA, and social studies, although not for science courses. These findings suggest that controversial issues can motivate classroom discussion, but may depend on the content-area context in which it is used. Further inquiry is needed to examine how classroom discussions of controversial issues are approached differently in science courses. Nonetheless, these findings provide promising evidence that modest school support of an issues-centered curricula can make a difference in improving classroom discussion quality.
Results from the study support past research showing that students identified as Hispanic and Asian are less inclined to engage in civic activities (Lopez et al., 2006). These findings affirm the need to help youth from these populations to improve their exposure to civic learning opportunities. Fortunately, there is support for the central research question that the WG program can improve students’ self-reported civic engagement. The WG program instructs students to engage in classroom activities that build on learning and engaging in discussions of controversial issues. Controversial issues play a critical role in informing adolescents about pressing issues that are relevant to their communities and beyond (National Council for the Social Studies (NCSS), 2007). Although the WG program does not explicitly support or provide students with community action tools to address social issues, the program exposes students to controversial issues on a daily basis that helps them develop social awareness. In turn, these program activities can help students build confidence in their capacity to become active citizens. One explanation is that controversial issues have a mobilizing function that helps students understand not only the significance but also the procedural knowledge associated with taking actions to resolve community issues (Scheufele, 2002).
Despite positive findings on the WG program’s impact on civic engagement, the program does not seem to have any impacts on students’ voting interest. These results suggest that getting younger adolescents (middle school students) to be interested in voting may be less feasible, in comparison with older adolescents (high school students). According to Hooghe and Wilkenfeld (2008), older adolescents are closer to the legal voting age and more likely to receive concrete information about voting registration procedures in their social studies courses. Despite being enrolled in the WG program, students may not be aware of when and where to vote because voting registration procedures are not usually communicated until high school (Callahan et al., 2008). Nonetheless, this finding considers how the WG program may be limited in its capacity to improve students’ voting interest.
Although the WG program was found to have positive effects on students’ self-reported civic engagement, students from particular ethnic minority backgrounds did not seem to gain additional benefits. This non-significant finding should not be interpreted to suggest that the program does not work for students from diverse backgrounds (Huey and Polo, 2008; Miranda et al., 2005). Rather, these results suggest that the WG program may have equally positive effects on students identified as ethnic minorities and Caucasian (Lochman and Wells, 2003). However, these findings suggest that possible modifications may be needed to ensure that the WG program is culturally adaptive and sensitive to the learning needs of ethnic minorities. (Huey and Polo, 2008).
Limitations
Although the WG program was observed to have a number of positive outcomes on students’ self-reported civic engagement, several limitations in this study are acknowledged. First, pretest surveys were not administered to students. Pretest data are useful in understanding students’ self-reported civic engagement at baseline that also providing more precise treatment effects. The study was also limited to examining a broad civic engagement measure that lacks specificity on whether students participated in certain civic activities, such as contacting public officials, donating money for specific causes, and volunteering in community services (e.g. soup kitchen). Another limitation was relying on students’ self-reported civic engagement. Although students’ self-reported civic engagement is considered a modestly reliable predictor of their future behaviors (Kahne and Middaugh, 2008), it is still possible that students may inaccurately report their civic engagement (McDevitt and Kiousis, 2006). Students may be overconfident in how often they help their communities because of biases based on social desirability or trying to appear as good citizens (Bouffard-Bouchard et al., 1991). Finally, there are other instruments that can be used to assess a broad range of civic outcomes. For example, a standard civic knowledge assessment relies on a battery of items testing whether students know how a bill becomes a law and the three branches of government (Galston, 2007). Addressing these limitations is critical to estimating more precise treatment effects as well as improving the generalizability of this study’s findings.
Conclusion and implications
In summary, the study provides evidence that exposing students to learning and discussing controversial issues through the WG program can positively impact students’ self-reported civic engagement. Civic engagement is important for adolescents in the broad context of developing awareness and responsibility of improving the community, as well as eventually fulfilling one’s civic duty to participate and vote in the national elections (Althof and Berkowitz, 2006; Carnegie Corporation and CIRCLE, 2003). This study makes a number of contributions that inform how schools can develop teaching strategies and curricula content used to improve students’ civic engagement.
A serious concern is that students enrolled in diverse and urban school districts have fewer opportunities to practice civic activities in comparison with students enrolled in more privileged schools (Ballard et al., 2014; CIRCLE, 2013). For example, schools located in high-poverty areas have fewer opportunities for service learning and community service outreach (Hart and Atkins, 2002). One reason explaining this trend is that school administrators are constantly pressured to ensure that students excel on standardized math and ELA assessments, which leaves less attention to improving social studies courses (Burroughs et al., 2005). The current study reports that ELA, science, and math teachers can also share in the responsibility of helping students to becoming informed and active citizens. Indeed, the implications of this study suggest that the lack of civic learning opportunities in highly diverse urban school districts can be addressed by considering a modest intervention requiring a reasonably low amount of instructional time and resources.
Footnotes
Funding
This work was supported by Grant Number R305A090555, Word Generation: An Efficacy Trial from the Institute of Educational Sciences, US Department of Education (Catherine Snow, PI).
