Abstract
Research indicates that educational, economic, and life success reflect children’s nonacademic as well as academic competencies. Therefore, longitudinal surveys that assess educational progress and success need to incorporate nonacademic measures to avoid omitted variable bias, inform development of new intervention strategies, and support mediating and moderating analyses. Based on a life course model and a whole child perspective, this article suggests constructs in the domains of child health, emotional/psychological development, educational achievement/attainment, social behavior, and social relationships. Four critical constructs are highlighted: self-regulation, agency/motivation, persistence/diligence, and executive functioning. Other constructs that are currently measured need to be retained, including social skills, positive relationships, activities, positive behaviors, academic self-efficacy, educational engagement, and internalizing/emotional well-being. Examples of measures that are substantively and psychometrically robust are provided.
Keywords
Educational success is critical in a modern industrial economy. Academic outcomes, such as degree attainment and academic test scores, are often assessed because they are widely known to predict occupation, income, health, and other outcomes. However, nonacademic attributes of the individual, such as social competence and perseverance, are important as well (Heckman & Kautz, 2013; Levin, 2012). Nonacademic attributes have been defined as “the personal attributes not thought to be measured by IQ tests or achievement tests” (Heckman & Kautz, 2013, p. 10). They also contribute to or undermine educational attainment, labor market success, health, behavior, and earnings (Almlund, Duckworth, Heckman, & Kautz, 2011; Durlak et al., 2011; Heckman & Kautz, 2013; Lippman, Ryberg, et al., 2014; Steinberg, 2005). In addition, nonacademic attributes or competencies are intrinsically important to child and youth development in their own right (Bornstein, Davidson, Keyes, Moore, & The Center for Child Well-Being, 2003; Peterson and Seligman, 2004). This article uses the term nonacademic rather than noncognitive in recognition that all these attributes require cognition.
In this article, we focus on nonacademic constructs that have been included or excluded from surveys conducted by the National Center for Education Statistics (NCES). In particular, we highlight several excluded constructs that research suggests would be valuable additions to any survey assessing educational and life success. The nonacademic constructs that we consider most critical for success are
self-regulation (included in data collection for some but not all ages);
agency/motivation;
persistence/diligence; and
executive functioning.
In addition, a number of important nonacademic attributes, including social and emotional behaviors, have already been included in educational surveys conducted by the NCES (see Table 1, in italics), and we highlight several that research suggests are particularly important to retain:
Constructs for Consideration in Longitudinal Surveys
Note. Constructs in italics have been included in National Center for Education Statistics longitudinal surveys in the past in some format.
Before kindergarten.
Including kindergarten.
social skills/social competence;
positive relationships with family and peers;
physical health and special health care needs;
activities, such as sports, art, and music;
positive behaviors, such as volunteering and environmental stewardship;
academic self-efficacy;
educational engagement; and
internalizing/emotional well-being.
Defining a Nonacademic Outcome
Nonacademic “attributes” go by many labels, including noncognitive attributes, soft skills, socioemotional learning competencies, and character skills and, as noted, can be seen as encompassing those individual attributes other than the ones assessed by achievement and IQ tests (Heckman & Kautz, 2013). In addition, there are aspects of individual health and emotional and social development, including status and behaviors, that are important to measure, as they are related to education outcomes and are important indicators of well-being in their own right. This is a large category. This article considers individual nonacademic attributes and health, social, and emotional outcomes, outside of education. These “outcomes” are not, of course, outcomes in any final sense. Rather, they represent well-being at a point in time, which reflects influences experienced up to that time; moreover, they will, in turn, affect future outcomes.
We recognize that many of the competencies described as noncognitive or nonacademic actually encompass cognitive and academic elements, making these terms somewhat inappropriate, albeit ones that are in common use at this time. We primarily use the term nonacademic in preference to noncognitive, as few of the constructs that we discuss lack a cognitive element.
Nonacademic attributes can encompass competencies, subjective feelings, attitudes, and values. Moreover, well-being comprises elements both positive and negative (Lippman, Ryberg, et al., 2014; Moore & Lippman, 2005; National Research Council & Institute of Medicine, 2002). Federal surveys and indicator systems have a history of measuring and reporting on negative child and youth outcomes, such as mortality, crime, violence, and adolescent childbearing, although the NCES and the National Institute of Child Health and Human Development have been exceptions to this pattern. In addition to recognizing the importance of negative outcomes (Baumeister, Bratslavsky, Finkenauer, & Vohs, 2001), a balanced portrait of childhood requires positive measures as well as negative measures (Moore & Lippman, 2005).
Priority is given to those characteristics of a child or an adolescent that have been found to have a strong influence on educational success, although additional factors that are intrinsically important for children’s development and well-being are also discussed in this article. Although they represent a primary domain of child well-being, discussion of academic outcomes, such as educational attainment and cognitive or achievement test scores, is not covered here.
Another important contributor to educational success—children’s contexts and environments (e.g., family processes, school practices, and neighborhood characteristics)—is also excluded from this article. While much research confirms the effects of family structure, economic factors, school characteristics, and climate, neighborhood, and family processes on educational outcomes (Ainsworth, 2002; Crosnoe, Mistry, & Elder, 2004; Ginther & Pollak, 2004; Lippman et al., 2013; Ripski & Gregory, 2009), the present focus is on measures that are needed in NCES surveys to assess outcomes at the level of the individual child. Measures of children’s contexts and environment are inputs that affect children’s development and would generally be considered independent variables in analyses of children’s development.
However, this article does consider children’s relationships. The relationships between children/youth and family members, peers, and other adults in their environments have not historically been identified as a uniquely primary domain of child well-being. However, two of the current authors identify relationships as a domain of child and adolescent well-being in their work (Lippman, Moore, & McIntosh, 2009, 2011), since such relationships represent the interaction of children and adolescents with others and so reflect the qualities, behaviors, and attitudes of those individuals. Unfortunately, the importance of children’s relationship quality is often overlooked in national surveys. The quality of relationships is an extremely important aspect of well-being. In fact, children and adolescents often identify relationships as the most important aspect of their well-being (National Economic and Social Council, 2009). Moreover, positive relationships have a powerful influence on educational outcomes and child development in general (Blum & Rinehart, 1997; Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002; Klem & Connell, 2004; National Scientific Council on the Developing Child, 2004). In addition, positive relationships are a primary predictor of life satisfaction (Dew & Huebner, 1994; Heubner, Gilman, & Laughlin, 1999; Oberle, Schonert-Reichl, & Zumbo, 2011). This category can include relationships with parents, teachers, siblings, peers, and others, such as a coach or mentor. Like other outcomes and competencies, relationships function both as independent variables and as dependent variables.
The Importance of Nonacademic Outcomes
Research that incorporates nonacademic outcomes serves a variety of purposes for researchers, practitioners, and policy makers.
Providing an Accurate Understanding of Child Development and Educational Progress
When a rich array of variables is available for analysis, a more complete understanding of the etiology of development is possible. For example, when all we know about a student is his or her grades, it is likely that tutoring programs will represent the most obvious approach to improving educational success. Understanding the rich array of factors that affect educational progress (Princiotta, Ryberg, et al., 2014) makes it more clear that there are a myriad of factors that affect educational outcomes. Thus, a broader and likely more effective approach to education is suggested (Moore, Terzian, et al., 2014).
Avoiding Omitted Variable Bias
Another way to view the importance of noncognitive outcomes is through the lens of omitted variable bias. This bias occurs when researchers leave out independent variables that are related to outcomes of interest. This omission can lead to incorrect estimates of the magnitude of the effects of academic predictors or misattribution of causality (Angrist & Pischke, 2009; Barreto & Howland, 2005). For example, given research indicating that physical and mental health problems increase the risk of dropout, failing to include a measure of physical and mental health in a multivariate analysis of longitudinal survey data is likely to produce estimates of the magnitude of other variables that are larger than what they would be if variables measuring physical and mental health were included in the model. Thus, policy makers might get a false sense of the magnitude of those variables that are included in an equation.
Assessing Workforce Readiness
Also, to fully describe the readiness of youth in the United States for workforce participation and success, it is not enough to measure whether youth have a degree or a certificate (Gutman & Schoon, 2013; Heckman & Kautz, 2013; Levin, 2012). Beyond degrees, it is necessary to have various character strengths (e.g., diligence, empathy, self-control, tolerance, and openness to new experiences) and soft skills (e.g., timeliness and attentiveness) to be successful in the labor market (America’s Promise Alliance, 2006, 2007; Guerra, Modecki, & Cunningham, 2014; Heckman & Kautz, 2013; Levin, 2012; Lippman, Atienza, Rivers, & Keith, 2008).
Informing Intervention Approaches
To develop effective intervention efforts, it is valuable to identify potential core components from basic research studies (Child Trends, 2013; Embry, Lipsey, Moore, & McCallum, 2013). Specifically, with longitudinal data that include a rich array of variables and cognitive and nonacademic attributes, researchers can assess whether and how health, emotional/psychological, and behavioral outcomes affect educational success. For example, externalizing and bullying behavior have been found to be related to lower educational engagement and thus academic attainment (Becker & Luthar, 2002; Bridgeland, DiIulio, & Morison, 2006; Davis & Jordan, 1994; Lau & Roeser, 2002; Wentzel, 1998). This suggests that externalizing and bullying behavior are good targets for intervention programs that seek to improve educational outcomes. Including other such constructs in surveys could further inform development of future interventions.
New research can also inform improvement of existing intervention approaches. For example, Child Trends recently completed analyses of the National Education Longitudinal Survey to identify predictors of high school dropout for a study of integrated student services. In a multivariate analysis, the most powerful predictor of dropout—with a larger odds ratio than eighth-grade math scores—is being a teen parent or expecting a child in 10th grade (Moore, Terzian, et al., 2014). This information highlights a behavioral risk factor—teen parenthood—that is a recognized risk factor for school dropout. This understanding has informed the development of programs to address adolescent pregnancy and parenting. Future efforts to improve educational outcomes will need the next generation of this kind of information to identify the as-yet-unmeasured malleable factors that will need to be targeted (e.g., 21st-century communication, information, and media skills for using new and evolving technologies).
Analyses of Subgroups, Interaction Effects, and Mediators
Data on nonacademic outcomes can allow program providers and policy makers to identify particular subgroups in need of assistance or intervention. For example, adolescents with health conditions, substance abuse problems, and behavior problems likely represent subgroups (moderators) for whom the predictors of educational success differ from the findings based on all students.
A comparable argument can be made for the value of data on nonacademic outcomes to assess the mediators or pathways by which various programs, experiences, and policies affect well-being. For example, mental and physical health might mediate the effect of curriculum interventions on educational outcomes (Boccanfuso, Moore, & Whitney, 2010). In addition, interaction effects may occur such that school characteristics matter most or only for students with particular character or emotional characteristics. Research on such complex effects requires rich data to identify subgroups, mediating variables, and interaction effects.
The Intrinsic Importance of Child Well-Being
Thirty years ago, the notion that children’s well-being, broadly construed, was intrinsically important was just beginning to gain currency, with the publication of the State of the Child report by UNICEF in 1979 and other monitoring efforts of child well-being (Lippman, 2007). Today, the value of including broad surveys measuring subjective well-being and social–emotional learning is accepted (Deke & Haimson, 2006; Lippman et al., 2008; Lotkowski, Robbins, & Noeth, 2004; Rosen, Glennie, Dalton, Lennon, & Bozick, 2010).
Conceptual Frameworks That Underlie the Identification and Selection of Constructs
Two perspectives inform this article: the life course model and the whole child perspective. A database that will be relevant for multiple disciplines and for understanding complex processes will reflect these perspectives, at the least.
Life Course Model
The life course model posits that an outcome at any one stage of development is generally an input at the next stage of development, and this perspective recognizes how lives are organized over time and across contexts, from birth until death (Elder & Shanahan, 2006). Researchers seek to examine how development unfolds over time (ideally, in fact, before birth, beginning with the circumstances of pregnancy). Interestingly, Cunha, Heckman, and Schennach (2010) found that noncognitive skills (their terminology) are malleable throughout childhood, whereas cognitive skills are more malleable in early childhood. They argue that fostering noncognitive skills during adolescence is more effective than fostering cognitive skills.
Also, many policy makers, taxpayers, and parents care about how children negotiate the transitions into and through school and from high school into postsecondary education and training, emerging adulthood, and entry into the workforce. Accordingly, we need to consider nonacademic measures from early childhood through high school, recognizing that these nonacademic skills are crucial to educational and workforce outcomes (K. Duckworth et al., n.d.; Dweck, Walton, & Cohen, 2011; Gutman & Schoon, 2013; Levin, 2012). This line of research is important because it highlights the malleability of nonacademic outcomes throughout the school years (Cunha et al., 2010).
Whole Child Perspective
The whole child perspective posits that research and monitoring studies need to encompass physical development and safety, psychological and emotional development, social development and behavior, and cognitive development and approaches to learning (Weissman & Hendrick, 2013; Zaff et al., 2003; Zigler & Bishop-Josef, 2006). Analyses based on this widely accepted perspective will include variables from multiple domains, as independent or dependent variables or as control variables, to understand development.
Child Trends has assessed a variety of perspectives where researchers and practitioners focus on varied outcomes (Moore, Theokas, et al., 2008). The particular categories and labels vary, but similar broad domains tend to be widely employed (Bornstein et al., 2003; Federal Interagency Forum on Child and Family Statistics, 2013; Lippman et al., 2009, 2011; Moore & Theokas, 2008; Moore, Theokas, et al., 2008; Moore, Vandivere, Atienza, & Thiot, 2008; Zigler & Bishop-Josef, 2006). As noted, we are also adding a fifth category, relationships, because relationships have been identified as a critical element of child well-being and a critical antecedent of educational and life success (Bornstein et al., 2003; Lippman et al., 2009). The domains composing a whole child perspective are
educational achievement and cognitive attainment,
health and safety,
emotional and psychological well-being,
social behavior, and
social relationships.
Methodological Reflections
A number of issues regarding measurement should be considered in concert with the constructs. A few of these considerations are provided here, including the importance of using multiple reporters and repeated and rigorous measures.
Multiple Reporters
Having multiple reporters can be valuable (De Los Reyes, Thomas, Goodman, & Kundey, 2012). Information about a child’s behavior, knowledge, attitudes, and values is more accurate if it comes directly from the child or adolescent, if possible. Nevertheless, the perspective of the parent, a teacher, or other observer is also useful. For example, a child might be the best informant about his or her subjective emotional well-being and risky behaviors; however, a teacher could report on how frequently the child fights or disrupts the classroom, and a parent can report on a child’s behavior and activities in the home. The child can be a primary informant from about age 8 or 9 going forward.
Multiple Methods
Data collection should be multimethod as well as multi-informant. Numerous approaches have been employed. In addition to traditional surveys, in both hard copy or electronic form, data can be provided by assessments, administrative records, and observations. Biological data are also being collected in many studies funded by the National Institutes of Health, but the cost and difficulty of collecting saliva, urine, or blood samples are substantial and seem less necessary for educational studies.
Brief Repeated Measures
While many would argue for lengthy instruments with strong psychometrics, brief repeated measures can be a powerful strategy in a longitudinal survey. Moore, Halle, Vandivere, and Mariner (2002) looked at the Behavior Problems Index in the National Longitudinal Survey of Youth, which has 28 items, and compared the effectiveness of that scale with 3-item subsets from it measured over time. It turns out that a short scale measured over time is just as strong as a long scale measured once.
Also, the National Survey of Children’s Health contains extremely short scales and indices. For example, the Educational Engagement Scale has two items. It might be better if it were a little longer (to assess all elements of the educational engagement construct), but even this two-item scale is related to other outcomes in the ways that would be expected (Moore, Kinghorn, & Bandy, 2011). This is quite important because space is always a constraint in any data collection effort.
Rigorous Measurement
Some contend that it is not possible to rigorously measure nonacademic constructs, particularly subjective and positive constructs, with validity and reliability. In practice, though, income is actually a much harder construct to measure. Many positive constructs can be measured with reliability and validity. In the following section, selected findings from Child Trends’ work on defining and measuring flourishing are presented to highlight the feasibility of measuring important nonacademic outcomes (Lippman, Moore, et al., 2014).
Potential Constructs for Inclusion
On the basis of these perspectives, potential constructs can be identified. Specifically, through the life course perspective, we have identified constructs across the stages of childhood: preschool (ages, 0–5 years), childhood (ages, 6–11 years), and adolescence (ages, 12–17 years). Reflecting the whole child perspective, constructs are identified within each of the five domains noted above. Given the focus on education, we emphasize constructs that assess approaches to learning. To organize our work and presentation, we suggest subdomains within each domain. This strategy implies the grid presented in Table 1.
To put these in the context of what the NCES has measured at some point, we note in italics constructs that have been included in NCES longitudinal surveys in the past in some format. For example, the NCES has fielded the Social Skills Rating Scale in the Early Childhood Longitudinal Study–Kindergarten Cohort (ECLS-K). Nevertheless, there are many other opportunities for inclusion of varied nonacademic outcome measures across the longitudinal program addressing each age group.
For specific information on the types of measures included in each construct, see the appendix (Table A1).
Physical Health, Development, and Safety
Physical health and safety are prerequisites for healthy child development. This domain includes health status, chronic health conditions, and health risk and promotion behaviors. Health risk behaviors vary by age and can include screen time, substance use, and risky sexual behaviors. Health promotion behaviors, however, include a healthy diet, exercise, safety, and sufficient sleep (Bornstein et al., 2003; Conner, 2003). For example, a series of questions on children’s diets is included in the ECLS-K. While these constructs are addressed in many health surveys and studies, it is important to include health constructs in educational surveys because health affects educational outcomes (Zaff et al., 2003).
Psychological and Emotional Development
This domain captures positive and negative aspects of psychological and emotional development. The importance of these outcomes has received increasing attention (National Research Council & Institute of Medicine, 2009) because internalizing and externalizing problems undermine development. Internalizing includes depression, anxiety, suicidality, and loneliness. Externalizing refers to negative emotions such as ongoing anger and frustration, as well as acting-out behaviors, such as fighting and bullying.
Despite the importance of these negative emotions, positive as well as negative measures are needed to provide a balanced perspective on outcomes (Lippman, Moore, et al., 2014; Moore, 1997; Moore & Halle, 2001). Positive social–emotional development includes having a positive self-concept, emotional competence, empathy, hope, goal orientation, academic self-efficacy, internal locus of control, intrinsic motivation, self-regulation, and life satisfaction. Developing an identity is an especially important task in the adolescent age range.
Coping skills are another positive subdomain of psychological and emotional development; this construct includes self-management (including persistence, motivation, initiative, time management, and high expectations). Given that nearly every child faces adversity to some degree, coping skills are very important for a developing child. (There is a question in the National Survey of Children’s Health about whether the child “bounces back,” which captures a coping orientation.) Self-regulation is also widely recognized as a very important skill for healthy development across age spans (Lerner et al., 2011). There are a variety of measures available for social–emotional development. For example, the Forum on Child and Family Statistics (http://www.childstats.gov) is currently creating a compendium of measures of social–emotional development in early childhood.
Social Development and Behavior
The social domain includes subdomains of social competence, activity engagement, positive social behaviors, and negative social behaviors. Clearly, social competence, the ability to collaborate and cooperate, and a prosocial orientation, such as tolerance or appreciation for the many differences that characterize our diverse country, represent an important skill set.
Activity engagement has been found strongly related to educational outcomes, but the type of activity has varied implications for nonacademic outcomes (Barber, 2005). It is important to encompass a variety of types of activities, such as clubs, sports, and religious activities, because students have diverse interests and participate in different types of activities. Accordingly, items asking about these activities are important to retain in NCES longitudinal surveys.
Positive social behaviors include volunteering or community service, for which the NCES has a history of collecting data, and environmental stewardship, which represents an issue that is going to be very important for the next generation. A scale to assess environmental stewardship has been created (Lippman, Moore, et al., 2014).
Negative behaviors include externalizing behaviors, such as bullying and fighting, as well as substance use and early sexual activity for adolescents. As noted, according to Child Trends’ analysis of the National Education Longitudinal Survey, having a baby is by far the largest predictor of dropout. Students are almost three times as likely to drop out if they have a baby by 10th grade (Moore, Terzian, et al., 2014). Students who have a child in high school are also less likely to complete postsecondary education. According to original analyses of the Beginning Postsecondary Students data, students who have a child during their first year of postsecondary education are also significantly less likely to graduate (Princiotta, Lippman, et al., 2014). Negative behaviors such as delinquency, substance abuse, and disciplinary problems in school (already collected by the NCES; see Table A1) need to continue to be collected because of their association with academic outcomes.
Cognitive Development and Education
The cognitive development and education domain refers, of course, to academic achievement and attainment (which are not shown in the table or addressed in this article), but it also includes the skills, attitudes, and behaviors, as well as the underlying executive functioning, that promote learning, problem solving, and educational success in educational and work settings. Educational engagement refers to how students are cognitively, emotionally, and behaviorally engaged in their learning, including cognitive interest in the work, behaviors such as attending class prepared, and emotional attachment to school and teachers (Fredricks, Blumenfeld, Friedel, & Paris, 2005; Furlong et al., 2003; Goslin, 2003; National Research Council & Institute of Medicine, 2004). It also has been positively linked to achievement, higher grades and academic competence, and higher expectations for current and future success in school.
School connectedness refers to students’ sense of belonging at school, as fostered through relationships with other students and staff, and it is related to achievement, expectation for school success, and subjective value of school (Gregory & Weinstein, 2004; van Ryzin et al., 2009, in Lippman, Ryberg, et al., 2014). A positive academic self-concept, or positive ideas about one’s academic abilities, is associated with educational engagement, grades, test scores, and educational expectations (Lau & Roeser, 2002; Mau & Bikos, 2000; van Ryzin et al., 2009, in Lippman, Ryberg, et al., 2014).
Learning skills and attitudes are important to measure to reflect how students learn—what goes on behind the scenes within a student. Curiosity expands learning to be interactive and includes the desire to learn more about a subject (Kashdan, 2009; Wentworth & Witryol, 2003). Problem solving is a cognitive skill that students may formally learn in an educational setting or more informally. It is described as developing or planning a sequence of actions to provide varied ways to solve a problem (Smith, 2003). These skills can be viewed as mediators and can be assessed with questionnaire items or with observational approaches. Measures would be useful when trying to understand why children achieve or do not achieve.
Relationships
As discussed above, relationships are critical to children’s well-being and healthy development. Relationships can be between a child and his or her family (parents, siblings), peers (friends, classmates), other caring adults, or, for some, a spiritual being. In subsequent sections, we discuss potential measures.
The NCES has included measures of relationships in a number of its surveys, including the High School Longitudinal Study, the National Education Longitudinal Survey, the Educational Longitudinal Study, and the Early Childhood Longitudinal Study series. These measures include activities and communication with peers, parents, and teachers that relate to school engagement. However, more could be done in measuring relationship quality since rigorous measures are now available—for example, measures of peer relationship quality developed by Child Trends (Lippman, Moore, et al., 2014). The Early Childhood Longitudinal Study–2011 Kindergarten Cohort has taken a large step in this direction by including measures of parent–child relationship quality.
Measures of sibling relationships are surprisingly lacking. The longest relationship that most people have is with a sibling, and yet we know surprisingly little about siblings and sibling relationships (Volling & Blandon, 2005).
Peers, obviously, can be supportive. They are often viewed as negative, but, actually, most of the effects of peers are positive (Bearman & Bruckner, 1999). For school, naturally, relationships with staff members—not just teachers but staff in general—can be important (McNeely, 2005). In the community, positive relationships with unrelated adults can similarly be valuable.
An analysis with the National Survey of Children’s Health of a single item—whether adolescents have an adult outside of their home who knows them and cares about them—is related to every child outcome examined except one (Murphey, Bandy, Schmitz, & Moore, 2013).
Regarding the macrosystem, relationships to larger entities—such as the political system, religious organizations, social media, and sports teams or players—may be important to some youth, although building brief reliable measures represents a substantial challenge.
Data Gaps
There are constructs, of course, that lack good measures. In middle childhood, measures of play and curiosity are needed. For adolescents, self-regulation is still really important, as well as social behaviors. In addition, rigorous measures of soft skills and life skills for young adults are needed.
The federal government has a unique opportunity to test the importance of promising nonacademic factors for educational success on a large, nationally representative sample. A recent review of the literature (see Lippman, Ryberg, et al., 2014) identified a number of promising nonacademic factors that may be related to educational success. These factors have been found to relate to nonacademic outcomes, such as prosocial behaviors, delinquency, and depression, but they have not yet been proven to be related to educational outcomes. They include optimism and emotional stability.
Adapted from Lippman, Ryberg, et al. (2014), Table 2 provides a summary of research indicating the extent to which varied nonacademic measures are significant predictors of other outcomes for middle school– and high school–aged children and youth. The research summarized in this table represents 85 of the most rigorous studies available. To be included in the table, studies must have met at least two of the following criteria: a sample size of at least 200, controls for demographic variables, random sampling, and a longitudinal design with a follow-up of at least 1 year (Lippman, Ryberg, et al., 2014).
Positive and Protective Factors in Adolescent Well-Being and Associations With Varied Outcomes
Source. Adapted from Lippman, Ryberg, et al. (2014).
Note. Prosocial = prosocial behavior, social contribution; Positive = positive youth development; Life = life satisfaction, well-being, self-esteem; Academic = academic achievement/school attachment; Relation = positive relationships with peers and/or adults; Substance = substance abuse; Delinq = delinquency; Violence = violence, aggression, externalizing behavior; Depress = depression, internalizing behavior; Risky = risky sexual behaviors.
Priority Constructs for Consideration
From the many potential constructs, we highlight the following:
self-regulation,
agency/motivation,
persistence/diligence, and
executive functioning.
Self-regulation includes the control of disruptive emotions as well as the production and regulation of positive emotions. Self-regulation (also referred to as self-control or effortful control) is generally defined as the ability to focus attention, manage emotions, and control behaviors (Halle et al., 2014). It encompasses “the ability to flexibly activate, monitor, inhibit, persevere and/or adapt one’s behavior, attention, emotions and cognitive strategies in response to direction from internal cues, environmental stimuli and feedback from others, in an attempt to attain personally-relevant goals” (Moilanen, 2007, p. 835, in Lerner et al., 2011, p. 4). Measures designed to assess children’s self-regulation might include adult-reported items, such as “Child keeps working at something until he/she is finished” or “Child interrupts others when they are speaking.” Self-regulation is more well studied than other recommended constructs, and its relationship to education as well as other outcomes is well established. Table 2 summarizes its relationship to multiple outcomes in other rigorous longitudinal and cross-sectional studies of youth, including academic achievement, prosocial behaviors, substance use, delinquency, depression, and positive youth development in general.
Agency/motivation, or the willpower to get something done, needs to be coupled with the necessary self-perception or self-concept or the belief that one can accomplish it. Snyder (2005) referred to these two pieces as critical to an overall perception that one’s goals can be met. Snyder called this construct “hope”; however, based on cognitive interviews with youth and available literature, these appear to be better described as aspects of goal orientation. The Flourishing Children Project (Lippman, Moore, et al., 2014) developed measures of goal orientation that can be recommended for longitudinal surveys of youth (see below for the items). Inclusion of an item on the ability to make viable plans is key. This scale was found in regression analyses with sociodemographic controls to be positively related to grades and negatively related to smoking, fighting, and depression.
Literature on persistence/diligence as well as reliability/grit/tenacity was already strong at the time of a 2008 review as predictors of college and workforce readiness (Lippman et al., 2008). The literature and applications to schooling continue to grow in strength, and specific aspects of these constructs, such as “grit” and “growth mind-set,” have been studied in relationship to academic achievement and attainment (see, e.g., A. L. Duckworth, Kirby, Tsukayama, Berstein, & Ericcson, 2011; Dweck et al., 2011). Child Trends has developed a scale of diligence and reliability for consideration for the NCES’s longitudinal surveys (see below). A national pilot study found that diligence and reliability were related in cross-sectional analyses to higher grades and to less smoking, delinquency, and depression and are therefore good candidates for further fielding.
From early childhood onward, executive functioning is critical to measure and monitor over time in education longitudinal surveys, as it underlies so many other aspects of social, emotional, and cognitive development. Executive functioning involves cognitive processes, including working memory, attention, and inhibitory control for the purposes of planning and executing problem solving and goal-directed activity. Strong evidence has emerged underscoring that the development of executive function skills is a crucial contributor to the development of cognitive and social capacities (Center on the Developing Child, 2011). Executive functioning differs from self-regulation in that it focuses primarily on the processes required for the conscious control of thought, emotion, and action rather than the control itself.
What Might Measures Based on These Kinds of Constructs Look Like?
Because the importance of these constructs has been recognized, measurement work has been underway, and yet measures are still evolving. Measures are being developed and/or adapted from small-scale studies—for example, for use in administration in national studies of large and representative populations of children and youth. However, measures are not necessarily available for all age groups, and evidence of predictive validity is sometimes scarce. Below we provide several promising examples of relevant measures. The first three constructs are from the Flourishing Children Project (Lippman, Moore, et al., 2014): diligence and reliability, initiative taking, and goal orientation. The fourth construct highlighted here is executive functioning.
The purpose of Child Trends’ Flourishing Children Project was to develop short, robust, and usable scales for 19 positive child well-being constructs. Many of these constructs had not been widely or well measured before the scales were developed. The constructs that had been measured were based on long, unwieldy scales. The goal was to develop scales that would work with a diverse group of adolescents and their parents and could be used cost-effectively. To accomplish this goal, 3 years were invested in developing measures.
Child Trends developed initial items and then conducted three rounds of cognitive interviews with adolescents across the country to ensure that items were relevant and salient to them. When strong items were developed, they were tested in a nationally representative web-based survey with adolescents aged 12 to 17 years and their parents, and substantive and psychometric analyses were performed. A selection of the measures—those most highly related to educational outcomes—is presented below. Full results are available in Flourishing Children: Defining and Testing Indicators of Positive Development (Lippman, Moore, et al., 2014). 1
Diligence and Reliability
Diligence and reliability are defined as performing tasks with thoroughness and effort from start to finish where one can be counted on to follow through on commitments and responsibilities. It includes working hard or with effort, having perseverance and performing tasks with effort from start to finish, and being able to be counted on (see Box 1).
Diligence and Reliability
The Adolescent Diligence and Reliability Scale is composed of seven items on a frequency scale. Adolescents are asked how often the following happen:
“Do you work harder than others your age?”
“Do you do as little work as you can get away with?”
“Do you finish the tasks you start?”
“Is it hard for you to finish the tasks you start?”
“Do you give up when things get difficult?”
“Can people count on you to get tasks done?”
“Do you do the things that you say you are going to do?”
There is a corresponding parent scale, with the same seven items oriented to parents. For example, parents are asked, “How often does your child work harder than others his/her age?”
These scales exhibit relatively strong psychometric properties. Both scales have Cronbach’s alphas >.75: .79 for adolescents and .89 for parents. The comparative fit index and Tucker-Lewis index are above the .95 threshold for both scales, and the root mean square error of approximation is less than the .085 threshold for adolescents while it is .086 for parents (Hu & Bentler, 1998, 1999). The distributions of adolescent and parent responses cover the continuum of possibilities, but the parent distribution is positively skewed. This is expected, as positive items are generally highly positively skewed.
To test concurrent validity, the relationship between the scale score and outcomes in the areas of health, education, social behavior, and emotional health was examined while controlling for a variety of demographic variables—including teen gender, age, and race; household income and size; parental education, marital status, home ownership, and employment; and metropolitan area and region of residence. Diligence and reliability were related to each outcome. That is, diligent and reliable adolescents are less likely to get into fights, smoke, and report being depressed and are more likely to earn high grades.
Initiative Taking
Initiative taking is defined as the practice of initiating an activity toward a specific goal by adopting the following characteristics: reasonable risk taking and openness to new experiences, drive for achievement, innovativeness, and willingness to lead (see Box 2; Knight, 1921; McClelland, 1961; Zhao & Seibert, 2006).
Initiative Taking
The Adolescent Initiative-Taking Scale is composed of four items with the following prompt: “Please indicate how much these statements describe you.”
“I am willing to risk failure to reach my goals.”
“When I work at something, I care about doing my best.”
“I like coming up with new ways to solve problems.”
“I am a leader, not a follower.”
The parent version of this scale is made up of corresponding items worded for parents, such as “My child is willing to risk failure to reach his/her goals.”
The psychometrics for the adolescent and parent scales are good. Cronbach’s alpha is .70 for the adolescent scale and .73 for the parent scale. The comparative fit index and Tucker-Lewis index are >.95, and the root mean square error of approximation is <.085 for each scale. The distribution of the adolescent responses is positively skewed but covers the full range of responses. This is viewed as a strong distribution because positive survey items are generally highly positively skewed.
Regarding concurrent validity, students who take initiative are less likely to smoke and report being depressed and are more likely to have good grades. There was no relationship between initiative taking and fighting, however.
Goal Orientation
Goal orientation is defined as children’s motivation and ability to make viable plans and take action toward desired goals.
The goal orientation scales (see Box 3) have high Cronbach’s alphas: .88 for adolescents and .93 for parents. The comparative fit index, Tucker-Lewis index, and root mean square error of approximation make the cutoffs for a strong fit on both scales. The concurrent validity shows that goal orientation is related to all outcomes—fighting, smoking, depression, and grades—in the expected directions.
Goal Orientation
This scale uses two response scales. Five items ask the respondent how much the statements describe him or her, from not at all like me to exactly like me:
“I develop step-by-step plans to reach my goals.”
“I have goals in my life.”
“If I set goals, I take action to reach them.”
“It is important to me that I reach my goals.”
“I know how to make my plans happen.”
Two items use a frequency scale:
“How often do you make plans to achieve your goals?”
“How often do you have trouble figuring out how to make your goals happen?”
The parent version of this scale includes seven corresponding items.
Executive Functioning
Executive functioning refers to the cognitive processes that underlie planning and execution of problem solving and goal-directed activities. These include working memory, attention, and inhibitory control. This important capacity has been given considerable attention during early childhood and, increasingly, at older ages as well. The ECLS-K includes two measures: one focused on cognitive flexibility (Dimensional Change Card Sort) and one on working memory (Woodcock–Johnson III). The Early Childhood Longitudinal Study–2011 Kindergarten Cohort included a measure of effortful control (Children’s Behavior Questionnaire). A related measure of executive functioning was included in the National Children’s Study (Children’s Behavior Questionnaire–Very Short Form). Going forward, future federal surveys could consider fielding the Behavior Rating Inventory of Executive Function–Preschool Version for preschoolers and the Childhood Executive Function Inventory for children aged 4 to 15 years. Each of these measures is described below.
Measures Already Fielded in National Surveys
The Dimensional Change Card Sort (Zelazo, 2006; see also the NIH Toolbox on the web) was used in the ECLS-K to assess one aspect of executive functioning: children’s cognitive flexibility. It is easily administered, either with cards or electronically; it takes <5 minutes; and it can be used with children of varied ages, as well as with adults. The task involves sorting a series of test pictures that vary across two dimensions (e.g., shape and color). The child is asked to match these test pictures (e.g., yellow balls and blue trucks) to a target picture, first according to one dimension (e.g., color) and then, after a number of trials, according to the other dimension (e.g., shape). Scoring is based on a combination of accuracy and reaction time. In the ECLS-K, the Dimensional Change Card Sort was administered as a physical card sort in kindergarten and first grade and as an electronic card sort, which allows for the capture of response time, beginning in second grade.
The ECLS-K included the Numbers Reversed subtest of the Woodcock–Johnson III Tests of Cognitive Abilities as a measure of working memory (Mather & Woodcock, 2001). This is also a straightforward test to administer; it involves asking children to repeat increasingly long series of dictated digits in reversed order.
Rothbart’s Temperament Questionnaires–The Children’s Behavior Questionnaire was developed in 2001 and was used in the Early Childhood Longitudinal Study–2011 Kindergarten Cohort study. The Children’s Behavior Questionnaire is a highly differentiated assessment of temperament for children aged 3 to 7 years. The measure has 195 items containing 15 scales composing three factors—surgency/extraversion, negative affectivity, and effortful control—capturing elements of executive functioning.
Rothbart’s Children’s Behavior Questionnaire–Very Short Form targets children who are 3 to 7 years old, and it includes an Executive Function subscale, which was adapted and shortened for the National Children’s Study.
Measures for Consideration for Future National Surveys
The Behavior Rating Inventory of Executive Function–Preschool Version assesses executive functioning in children aged 2 to 5 years. It contains 63 items for a parent or teacher questionnaire. The items are organized into five subscales: 16 items in the inhibit category, 10 in shift, 10 in emotional control, 17 in working memory, and 10 in plan/organize. There is also a Behavior Rating Inventory of Executive Function to assess executive functioning for older children.
Additionally, for older children, the Childhood Executive Function Inventory is a rating instrument for parents and teachers that can be used to measure executive functioning in children aged 4 to 15 years. The measure contains 24 items with 2 additional optional questions. It is divided into four subscales tapping inhibition, regulation, working memory, and planning. Factor analyses revealed only two factors—inhibition and working memory. As yet, this measure has not been used in a large-scale study.
Both of these measures earned a ranking of “strong” on multiple criteria in Child Trends’ ongoing review of existing measures of executive function.
Conclusion and Recommendations
The goal of this article is to provide conceptual and empirical justification for the inclusion of nonacademic outcome measures in longitudinal education surveys. To this end, specific rationales for several research-based constructs to be measured are suggested, and examples of how these rigorous measures might be developed and used are provided as well. Finally, recognizing that space is at a premium in surveys, those nonacademic constructs that are most critical are highlighted as predictors of educational outcomes and as developmental outcomes in their own right. We provide examples of robust measures to show that these constructs can be rigorously assessed.
Because of the importance of these constructs, where there are gaps in measures or in the extent to which effective measures are available for varied age groups and other subgroups, it would be worthwhile to invest in developing and testing measures that are substantively and psychometrically robust across social and demographic groups.
Footnotes
Appendix
Table A1 provides a survey of nonacademic outcomes contained in NCES longitudinal surveys, from 1988 to the present day. The following surveys were reviewed to compile Table A1:
Entries in the table are organized by developmental domains, age of child, and survey. The age of the child and type of reporter are presented in parentheses after each outcome.
Acknowledgements
This paper was initially prepared for the National Academy of Education’s Workshop to Examine Current and Potential Uses of NCES Longitudinal Surveys by the Education Research Community, held on November 5-6, 2013, in Washington, DC. This paper was supported with funding from the Department of Education Institute of Education Sciences under grant # R305U120002 and with internal support from Child Trends. This paper reflects the view(s) of the authors and not necessarily those of the U.S. Department of Education.
Notes
Authors
KRISTIN ANDERSON MOORE is senior scholar at Child Trends. Her research interests include positive youth development programs, adolescent development, and public policy. She publishes in academic journals as well as formats that reach practitioners and policy makers (e.g., research briefs, testimony, webinars). She and Laura H. Lippman codirected the Flourishing Children Project.
LAURA H. LIPPMAN is a senior research scientist at Child Trends. Her research interests include college and workforce readiness, school climate, and family contributors to education. She has published books, chapters, peer-reviewed articles, and public reports on these and other topics and codirected the Flourishing Children Project with Dr. Kristin Anderson Moore.
RENEE RYBERG is a research analyst at Child Trends and a doctoral student at the University of North Carolina at Chapel Hill. Her work focuses on adolescent development, social stratification, and positive youth development in an international comparative framework.