Abstract
In fall 2013, the National Center for Education Statistics (NCES) hosted a group of researchers to provide input into its program of longitudinal surveys. Organized by the National Academy of Education with a grant from the Institute of Education Sciences, the Workshop to Examine Current and Potential Uses of NCES Longitudinal Surveys by the Education Research Community addressed issues of new designs, new measures, persisting issues, and technologies that could introduce new approaches. Findings from the workshop have appeared in AERA Open and on the National Academy of Education’s website and have been discussed at the American Educational Research Association annual meeting. This paper provides a background to the workshop, highlights key findings, and discusses the responses of NCES to researcher input in the context of constraints that limit NCES’s ability to introduce change. These issues deserve widespread attention in light of the importance of NCES longitudinal surveys as a resource for education research.
In the fall of 2013, the National Academy of Education (NAEd) convened a group of researchers to provide input to NCES on the future of its longitudinal studies program. Papers presented at the workshop addressed such topics as new designs for NCES surveys, new measures that NCES surveys should consider, persisting issues that warrant more attention in NCES surveys, and technologies that could introduce new approaches to NCES data collections. Four of the papers presented at the workshop have been published in AERA Open (Espelage, 2015; Moore, Lippman, & Ryberg, 2015; Muller, 2015; Warren, 2015), as well as a fifth paper that emerged in response to the others (Schneider, Saw, & Broda, 2016). Other papers are available on the NAEd website. 2 Table 1 lists the original papers presented at the workshop.
Papers Presented at the Workshop to Examine Current and Potential Uses of National Center for Education Statistics (NCES) Longitudinal Surveys by the Education Research Community
My purpose for writing is to discuss key insights from the workshop, focusing primarily on the papers those that have appeared in AERA Open. I will place these insights in the context of past efforts to provide input from researchers to NCES, and I will consider the response from NCES to researcher input. My reflections are based on my experience as a researcher using longitudinal surveys and my role as chair of the NAEd-NCES workshop. 3
Background to the NAEd-NCES Workshop
The NCES longitudinal survey program consists of more than 20 different surveys, most of which are still active with new surveys planned or currently in the field, ongoing follow-ups, or new waves of existing surveys. The surveys cover all age levels from birth through postsecondary education and beyond, and also include surveys directed at K−12 school and district officials, teachers, postsecondary institutions, and households. Perhaps the best known and most widely used surveys are those of secondary school students, beginning with the high school class of 1972 and repeated (with modifications) for the classes of 1980, 1982, 1992, 2002, and 2013, with several follow-ups for each survey. Although many items are repeated, there have also been substantial changes over time in the design of the secondary school surveys, particularly in when the first wave of the surveys were collected, whether in eighth, ninth, 10th, or 12th grade. Another widely used survey series begins with children in birth or in kindergarten and has continued through eighth grade, making it possible, at a few points in time, to construct synthetic cohorts that run from kindergarten through the postsecondary years. 4
Although not the only constituency for NCES longitudinal surveys, the research community is certainly an important one, and researchers have had formal occasions to provide input at least since 1963 (U.S. Department of Health, Education, and Welfare, Office of Education, 1963). A 1986 National Academy of Sciences Panel to Evaluate the National Center for Education Statistics was heavily critical, citing problems with data quality, timeliness and accuracy of reporting, and management challenges. In contrast, a 1995 report concluded that “ten years later, with the direction provided by the [1986] panel, strong leadership at NCES, and support from . . . the larger education community, NCES is much stronger and has become a widely respected statistical agency” (Hoachlander, Griffith, & Ralph, 1995, p. 13).
The 1995 report was the result of a “futures conference” in which NCES staff and outside researchers gathered to discuss new developments in education research and their implications for NCES. Deliberations and commissioned papers for the conference focused on the implications of major issues and trends in education as they affected NCES; advances in research methodology, in data collection and analysis, and in technology that were relevant to NCES surveys; and technological advances that could improve dissemination of NCES products (Hoachlander et al., 1995). Since that time, NCES has periodically convened internal task forces and standing panels to address specific topics, such as the National Assessment of Educational Progress (NAEP), confidentiality across data collection and analysis efforts, the use of computerized adaptive testing, effect size, longitudinal data integration, and nonresponse bias analysis, among other topics, as well as technical working groups to guide specific surveys. However, nearly two decades had passed since the last formal occasion for broad-based input from researchers on the longitudinal surveys program as a whole.
Insights From the NAEd-NCES Workshop
The NAEd’s effort to provide new input to NCES was motivated by recognition on all sides that changing times created new challenges and new opportunities for the longitudinal surveys.
New Measures of Student Experiences and Outcomes
One focus of the NAEd-NCES workshop was on trends and issues in the wider society that have bearing on the priorities and activities of NCES. For example, educational psychologist Dorothy Espelage argued at the workshop that NCES should expand its current assessment of school violence and bullying, given the prevalence of this concern and the growing body of research on the negative effects of violence and victimization. Current longitudinal surveys have few items that yield evidence on these issues. Espelage (2015) proposed survey questions that address the power relationship between victims and perpetrators, effects on academic achievement, and violence outside the school. Adding such questions would allow researchers to examine how bullying behavior changes across grade spans and how it manifests in different student subgroups. Espelage’s paper is particularly helpful in that it provides a conceptual framework for thinking about violence issues in the context of research on the school as a social setting, guidance to researchers on what can be learned with existing surveys measures, and advice to NCES on ways to extend and improve those measures.
Another paper focused on expanding assessments of student outcomes in areas other than academic achievement. Whereas current federal educational policy overwhelmingly emphasizes learning outcomes, recent research has increasingly pointed to the importance of other types of outcomes, referred to as noncognitive skills, social-emotional learning, grit, academic mind-set, and other labels (e.g., Farrington et al., 2012; Duckworth & Yeager, 2015). Social psychologist Kristen Moore, demographer Lisa Lippman, and sociologist Renee Ryberg made the case for expanded measures in these domains in their paper on “improving outcome measures other than achievement,” calling for indicators of self-regulation, motivation, persistence, social skills, physical health, and healthy relationships with family and peers (Moore et al., 2015). Assessment of these personal characteristics could lead to a richer understanding of student growth and development as well as educational progress. Indicators of well-being, for example, could be added to multivariate analyses of factors increasing the risk of dropout, creating a richer understanding of these factors. They could also be used to assess workforce readiness, as character strengths and interpersonal skills are keys to success in the modern workplace. Like Espelage (2015), Moore and colleagues (2015) made the case for new measures in part on the basis of subgroup differences that may be masked with more commonplace indicators. They offered specific constructs and indicators whose importance has been demonstrated in prior research and whose inclusion in NCES surveys could enhance the value of the surveys overall by giving them more analytic nuance and predictive capacity.
Other topics that call for new measures of students’ experiences and outcomes included the increased salience of immigration (Rumbaut, 2014), video and observational evidence of classroom instruction (Grossman, 2014), and developments in science, technology, engineering and mathematics education (Secada, 2014).
New Study Designs
Advances in technology and in the widespread availability of administrative data have created new opportunities for researchers to expand the range of data they examine and therefore broaden and deepen the research questions they can ask. NCES pioneered the linkage of survey and administrative data by obtaining high school and college transcript records for participants in its secondary and postsecondary longitudinal surveys, beginning with the National Longitudinal Survey of the High School Class of 1972 (Sebring et al., 1987). The enactment of the No Child Left Behind legislation in 2002 led nearly all states (with considerable federal funding) to create longitudinal data systems that link students and schools over time and in some cases follow students out of the state’s K−12 education system and into postsecondary education and the workforce (Data Quality Campaign, 2015). In a workshop paper, economist Susanna Loeb (2014) urged NCES to explore the possibility of linking NCES nationally representative surveys to state longitudinal data. This recommendation was well aligned with NCES plans, as the latest longitudinal survey, the High School Longitudinal Study (HSLS) of 2009, includes representative samples for 10 states (Ingels et al., 2011). However, NCES staff reported at the workshop that concerted efforts over several years to link individual-level achievement data from state records to NCES survey participants had been unsuccessful due to bureaucratic and legal barriers.
In addition to linking to state education data, it is theoretically possible to link NCES survey data to administrative data in a variety of domains, such as personnel records, tax records, and census records. At the workshop, economist Susan Dynarski (2014) discussed the administrative hurdles that confronted efforts to create such linkages, including resolution of sensitive issues around data security and privacy that arise when such comprehensive data sets are made available to researchers. Clearly, this is not an area in which NCES can act on its own, although it can play an active role within the federal statistics community. It is noteworthy, however, that President Obama recently signed a bipartisan act to establish a Commission on Evidence-Based Policy, which would be charged with exploring the feasibility of linking data from many federal government agencies, including education, to advance the possibility of evaluating and improving government programs (Evidence-Based Policymaking Commission Act of 2015).
Other advances rest on statistical and technological innovations. At the workshop and subsequently in her AERA Open contribution, sociologist Chandra Muller (2015) explained how such innovations create new opportunities for measuring key dimensions of school context. Arguing that prior research had been insufficiently attentive to heterogeneity in how students experience the context of their schools, Muller demonstrated that new data collection methods that take advantage of technological innovations could better reflect the reality that students encounter. She recommended the use of experience sampling methods, which query students at random times on what they are doing and how they are feeling, to gather time-specific data on students’ ongoing experiences as well as biological data collected through devices that students could wear (like a Fitbit), all made possible by technological advances. Muller also recommended increasing the number of students surveyed in each school to afford better estimates of within-school heterogeneity of students’ experiences and outcomes. Recent NCES surveys, such as the HSLS, include 25 students per school, enough to capture some degree of heterogeneity but not enough for social network methods that Muller has applied with other survey studies (e.g., Frank, Muller, & Mueller, 2013).
Taking into account these ideas and others presented at the workshop, sociologist John Robert Warren (2015) presented a radical plan for reshaping the entire series of longitudinal surveys. He argued that NCES surveys are not frequent enough to gauge short-term trends or effects of policies. Warren pointed to the development of census surveys as offering a response to this concern. Census surveys evolved from a once-per-decade administration (which still exists, of course), to more frequent surveys of one out of six households, to the current American Community Survey, which is administered annually to one out of 100 households. Warren proposed that NCES could similarly change the design of some of its longitudinal studies from a large-sample survey initiated once per decade to a smaller-sample survey that is initiated more frequently. Warren also points to the General Social Survey (GSS) conducted by the National Opinion Research Center as an example of innovation and response to researchers’ needs. The GSS has a body of questions that are asked annually, but it also allows “supplemental modules” of questions that research teams can add. The modules are decided upon through a competitive process, and researchers must reimburse GSS. Building on the contributions of Loeb (2014) and Dynarski (2014), Warren also advocated for linkages with state and federal administrative data, recognizing the legal and political hurdles.
Also in response to the workshop as a whole, sociologists Barbara Schneider, Guang Saw, and Michael Broda (2016) offered a counterpoint to Warren’s call for more frequent surveys of smaller samples. Although recognizing the concern with the infrequency of surveys, they emphasized the value of the current design for longitudinal investigations, particularly those that capture the transition from one level of schooling to another or from schooling to the workplace. They also pointed to the importance of large within-school samples to decompose variation into within- and between-school components, a point also made by Muller (2015). Another concern with more frequent surveys was whether that would increase the burden on schools and thus raise levels of resistance to NCES surveys, a perennial problem in social survey research generally. Nonetheless, Schneider and colleagues also recommended a series of modifications, such as oversampling of specialized school contexts, such as charter schools and homeschooling; embedding experiments within national surveys (discussed in more detail by Cook, 2014); increased linkages to administrative data as recommended by Loeb (2014) and Dynarski (2014); and the use of new technologies and the development of new measures as recommended by the authors discussed above.
NCES Responses to Researcher Input
When considering how NCES responded to the workshop and subsequent papers, three points stand out. First, input from researchers is important for helping NCES to adapt to societal trends, challenges, and opportunities. Second, the leadership at NCES expressed openness to input from the research community in preparation for, during, and moving forward from the workshop. Third, political, bureaucratic, and resource constraints place limits on the extent to which NCES can respond to this productive input despite the openness.
Balancing Continuity and Change
Stable measurement is an essential quality in longitudinal cohort surveys, such as those administered by NCES, as it enables researchers to compare current cohorts with those of the past. Yet there are countervailing pressures for change, as new issues emerge that must be addressed to lend insight into current problems. As early as 1986, NCES was cautioned to avoid “conceptual obsolescence” by adapting its surveys for changing times (Levine, 1986, p.20). A key challenge for NCES is to balance the need for repeated surveys on the same topics using the same designs to allow for repeated measures within and across cohorts with the need to introduce new designs, measures, and issues so that those using the data can respond to contemporary policy issues. It is especially important—and perhaps most challenging—to stay abreast of new ways of thinking about the education system that may result in policy-relevant modifications to future surveys, such as oversampling immigrant students or specialized schools.
Researchers who analyze data from longitudinal surveys are well positioned to provide input to NCES by contrasting emerging theoretical and policy issues with the data available in current surveys. Since the last formal, broad-based occasion for researcher input, much has changed in education and in the wider society, leading to new demands and new opportunities for NCES longitudinal surveys.
Openness to Change
NCES staff members were highly responsive to the workshop presentations and follow-up activities. Over a dozen NCES staff members attended the workshop, including the commissioner at the time, Jack Buckley, the deputy commissioner (and now acting commissioner), Peggy Carr, and long-time survey leaders, such as Jeff Owings and Chris Chapman. NCES provided feedback to authors as they worked on their papers to ensure they had up-to-date information, and NCES staff members, including Commissioner Buckley, responded to each presentation with questions and comments. At a public forum at the annual American Educational Research Association (AERA) meeting of 2015, Chris Chapman responded to a panel of workshop presenters, and Acting Commissioner Peggy Carr continued the dialogue with the research community at the 2016 meeting. A follow-up meeting also occurred between NCES staff members and NAEd staff and members. These encounters reflect a pattern of significant engagement of NCES staff with recommendations from a key constituency for their products.
Specific reactions to suggestions emerging at the workshop and codified in the papers varied by domain. Some of the recommendations were viewed as highly desirable but difficult to achieve. These included Loeb’s (2014) encouragement to link NCES data to state longitudinal data systems. Other ideas were viewed as valuable in principle but involved high costs that NCES staff were not confident they could fund, such as Muller’s (2015) advice to increase within-school sample sizes and Warren’s (2015) idea to implement a series of smaller annual education surveys with longitudinal follow-up. NCES staff expressed interest in Cook’s (2014) insights on the possibility of embedding randomized experimental interventions within the longitudinal surveys. A plausible example discussed at the workshop was the notion of implementing social psychological experiments that support growth mind-sets, which can be implemented through brief writing exercises (Yeager & Dweck, 2012), as part of a national longitudinal survey. This would combine the internal validity of an experiment with the external validity of a national survey. Moore et al.’s (2015) recommendations for new indicators of outcomes other than test scores were seen as straightforward developments in the manner that NCES surveys have always progressed, adding new indicators as they become salient in research. However, due to cost and respondent burden, the length of the surveys cannot be extended, so new indicators generally mean that old measures must be cut. Similarly, Espelage’s (2015) exposition on the need to expand the capacity to study bullying and victimization elicited a positive response, with concern with what may need to be dropped to fit in a new series of questions. With that said, these seemed to be the most likely areas for modifications to the NCES surveys, based on responses at the workshop and subsequently. Indeed, the latest (2011) version of the Early Childhood Longitudinal Study has incorporated items on peer victimization along the lines of those recommended by Espelage and measures of self-regulation, persistence, social skills, physical health, and social relationships consistent with the recommendations of Moore and colleagues.
Constraints on Responses to Researcher-Driven Change
Despite the openness to change, the agency operates within a number of constraints that limit its ability to respond to researcher-driven initiatives and recommendations. These include political, bureaucratic, and resource constraints.
Political constraints
Like any government agency, NCES must navigate its political waters. Most of the political attention on NCES has focused on the NAEP, the complex sample survey designed to measure the status and trends in student performance in Grades 4, 8, and 12, rather than on the longitudinal surveys that were the focus of this workshop. As long as sensitive survey questions are avoided, the longitudinal surveys seem to escape political manipulation. However, one of the recommendations at the workshop was to include information about respondent sexuality to enable researchers to examine victimization by sexual orientation in experiences of violence and bullying in schools (Espelage, 2015). In the past, efforts to include questions about teen sexuality and sexual behavior in government surveys have often been blocked by political concerns (e.g., Alexander, 1998; “U.S. Teen Survey Faulted,” 1991). Yet attitudes toward sexual diversity have changed, and the time may have arrived to enable important research on this subject. Although specific concerns were not raised at the workshop about the feasibility of including questions on sexuality, participants viewed this as a sensitive issue that warranted caution in considering whether and how to address this gap in NCES surveys.
Another aspect of the political context of NCES may affect its operations in a different way. Instituted in 1867, the collection of U.S. education statistics predated the U.S. Department of Education by more than a century. Over time, however, education statistics has become less of an autonomous operation, as NCES was incorporated into the Department of Education and more recently into IES. A bill to reauthorize the legislation that created IES, called the Strengthening Education through Research Act (SETRA), would take this process further, giving responsibility for selecting the commissioner of education statistics to the director of IES (instead of the president) and making clear that NCES is fully under the authority of IES. AERA and several other organizations have gone on record opposing these changes, stating,
NCES has the responsibility for collecting data and statistics on the conditions of education in the U.S. and the role of providing the statistical data and resulting products that can help shape priorities for research. It is simply not possible (or advisable) for the same person (in this case, the Director of IES) to determine what data should be obtained and how that information should be used to set the research agenda. These two functions are separate and should remain separable.
5
Despite the efforts of AERA and others to maintain presidential appointment of the NCES commissioner as well as NCES autonomy over data priorities, budget, and publications, those responsibilities are ceded to IES in SETRA, which may well become law by the time this article appears. Whether these changes will affect the NCES longitudinal survey program remains to be seen.
Bureaucratic constraints
Sociologists have long regarded the U.S. education system as a complex, loosely coupled bureaucratic environment (Bidwell, 1965, 2001). When it comes to data collection and data linkages, a bureaucratic decision-making process plays a large role. The extent to which this may operate as a constraint was especially evident in discussions about linking NCES longitudinal surveys to state education data (Loeb, 2014; Schneider et al., 2016). In this case, NCES was out ahead of the research community, having planned nearly a decade ago to add state-representative samples for 10 states to a national longitudinal survey of high school students, the HSLS of 2009. This would enable NCES to link data to state education data for the representative samples. It offered both the research possibilities and the state incentives outlined by Loeb (2014), and the state education agencies were willing to pursue the linkages. Unfortunately, as Commissioner Buckley revealed at the workshop, ultimately not a single state provided its data for linkage with the national survey. Despite agreement at the political level, NCES and its state counterparts were unable to resolve the bureaucratic barriers to linking state and federal data.
This experience is disheartening on many levels. It prevents states from gaining richer insight into the school experiences and performance of their students and prevents analyses of policies that vary within and among states and could inform further policymaking. These absences are especially poignant now that the No Child Left Behind Act, which consolidated substantial power over education policy at the federal level, has been replaced by the Every Student Succeeds Act, which has returned much of the authority over education programs and policies to states. Although the bureaucratic challenges to linking state and federal education data remain unresolved, the climate for data linkages may be improving, as evident in discussions of a Commission on Evidence-Based Policymaking, which would examine the merits and feasibility of linking federal data sets. However, even if bureaucratic barriers to linking different federal data sets together could be overcome, that may not extend to states, where most of the education data reside.
Resource constraints
Concern about whether NCES has the funds it needs to carry out its mission go back at least to 1986 (Levine, 1986) and probably farther back than that. Each of the recommendations from researchers carries some cost, ranging from staff and contractor time to consider and potentially implement survey revisions to more expensive options, such as larger samples, more frequent surveys, and innovative methodologies, like experience sampling. Yet NCES is hard-pressed to meet its current aims with available resources. As displayed in Table 2, the NCES budget line for statistics, which includes the longitudinal surveys, was cut by $5 million between 2012 and 2013, a consequence of the “sequester” that affected budgets across the federal government. After 2 years of level budgets, the 2016 budget will finally restore funds taken away in 2013 and provide a small increase. The statistics line will go up to $112 million, an increase of 3% over the 2012 amount (and 8.7% above the 2015 amount). Although insufficient to cover cost increases since 2012, the 2016 budget at least moves NCES in a positive direction.
Changes in National Center for Education Statistics (NCES) Funding, 2012 to 2016
Note. Figures are in millions of dollars. FY = fiscal year.
Source. Committee for Education Funding, http://blogs.edweek.org/edweek/campaign-k-12/Omnibus%20Funding%20Table.pdf.
Increases in the statistics line are intended to support new waves of existing surveys, such as a new School Survey on Crime and Safety. Consequently, securing funds for survey improvements and innovations will be difficult for NCES, but at least that will be more feasible in a time of rising rather than declining budgets.
Conclusions
The NAEd workshop was not intended to evaluate the performance of NCES or the performance of its longitudinal studies program. To the extent that evaluative statements came up, they were invariably positive. Many speakers prefaced their remarks with comments along the lines of Warren (2015, p. 1), who stated, “NCES’s collection of longitudinal student surveys has long been an enormously valuable resource for academic and applied research on education. As a nation, we are fortunate to have them, and even maintaining the status quo would be valuable.” Indeed, the recommendations for improvement were premised on the agency’s demonstrated ability to collect and release valid, reliable, and timely data on our nation’s education system.
After a long gap in formal opportunities for researcher input, the present is an especially important time for such comments. The last decade has witnessed strong emphasis on improving the internal validity of research conclusions supported by the research divisions of IES. By contrast, NCES’s longitudinal studies program is mainly about external validity, as it allows researchers to characterize the education system as a whole and to identify key dimensions of difference at multiple levels, including geographic differences and, depending on the particular survey, variation by district, school, classroom, and student. NCES surveys allow generalization to known populations; that is their chief virtue and that distinguishes them from most other research supported by IES. It is especially valuable to emphasize and build on the strengths of NCES surveys at a time when much of the attention has been drawn elsewhere, especially to the development, implementation, and results of randomized interventions.
Participants in the NAEd workshop argued that changing times called for a fresh look at the design and content of NCES surveys. Despite the arguments mustered in favor of such revisions, change will likely be modest, due to NCES’s need to balance continuity and change and to the external constraints that pose challenges to change. With that said, some of the proposed changes are neither especially burdensome nor blocked by external constraints, such as new measures that will allow researchers to consider new populations, new outcomes, and new indicators of context that have emerged in recent years. An ongoing dialogue with researchers of the sort sparked by this workshop can help ensure that the NCES longitudinal surveys will continue to serve as an essential resource for researchers and policymakers in the future.
Footnotes
Acknowledgements
The author is grateful for contributions from members of the steering committee of the National Academy of Education’s Workshop to Examine Current and Potential Uses of NCES Longitudinal Surveys by the Education Research Community, from the scholars who presented papers at the workshop and from Susan Fuhrman (former president), Michael Feuer (current president), Gregory White (executive director), and Judie Ahn (former project director) of the National Academy of Education. Chris Chapman, Pascal Forgione, and Emerson Elliott provided helpful comments on an earlier version of this paper. The workshop was funded by the U.S. Department of Education, Institute of Education Sciences (Grant No. R305U120002) to the National Academy of Education. This paper reflects the views of the author and not necessarily those of the supporting agencies.
1.
For example, a Google Scholar search on “High School and Beyond” yields 14,300 items; discounting for an estimated 12% that do not actually refer to the National Center for Education Statistics (NCES) survey still results in a total of 12,584 items. A search on “NELS 88” yields 5,910 items and for “National Educational Longitudinal Study” delivers 1,420. A total of 5,360 items emerge from a search on “Schools and Staffing Survey,” and 1,360 are returned for “Baccalaureate and Beyond.”
3.
The workshop was supported by a grant from the Institute of Education Sciences, which houses NCES, to National Academy of Education (NAEd) executive director Gregory White. I led the workshop along with a steering committee that included Steven Barnett, Laura Desimone, Pascal Forgione, Pat Goldsmith, Jennifer Lee, Sean Reardon, and Barbara Schneider. Forgione is a former school superintendent and former commissioner of NCES, and the others are leading scholars with substantial experience analyzing NCES longitudinal surveys. Former NAEd staff member Judie Ahn provided able support for the workshop and follow-up activities.