The Structure of Student Engagement in Community College Student Success Programs: A Quantitative Activity Systems Analysis

Abstract

Community colleges increasingly implement various student success programs, including 1st-year seminars, college skills courses, learning communities, and orientation, in an effort to boost degree completion. However, it is unclear how success programs’ curricular designs may contribute to these and associated student outcomes. Such inquiry is limited, in part, by the lack of methodological frameworks for program impact heterogeneity research. This study proposes a new conceptualization of nominally different student success programs as instances of a broader activity, which also provides a way to operationalize their curricular structures in comparable ways. Second, to briefly illustrate this approach, the study leverages matched program and student data to investigate how variations in student engagement—an emergent intermediate outcome for fostering successful college going—are related to variation in program design. Findings reveal that structural and underutilized curricular elements may be more impactful than skills-based curricula that are typically the organizing focus of these programs.

Keywords

community college high-impact practice engagement activity theory

The perennial challenge of fostering academic persistence and success of college students has taken on greater urgency in light of the emergent completion agenda in U.S. higher education that calls for greatly increasing the number of college graduates (Lee, Rawls, Edwards, & Menson, 2011; Lester, 2014). Community colleges in particular have been singled out for their perceived capacity to improve national completion rates at the same time that they are obliged to continue their mission of open access while upholding commitments to quality education (Humphreys, 2012; Lester, 2014).

In response to increased attention on student outcomes, and following on decades of research on student environments and effective practices in undergraduate education (Astin & Antonio, 2012; Mayhew et al., 2016), many have focused efforts on identifying particularly high-impact practices (HIPs) that “appear to engage participants at levels that elevate their performance across multiple engagement and desired-outcomes measures” (Kuh, 2008, p. 14). Among these HIPs, the type that arguably has received the most attention is organized courses or other time-limited, group-based interventions, typically designed to take place at the outset of students’ first college experiences (Hatch, 2016). Collectively, these are often called student success programs, or more precisely, 1st-year student success programs. They include orientation, 1st-year seminars, college skills courses, learning communities, and co-requisite or accelerated developmental education, among other variations. However, as I argue below, traditionally there has been no clear conceptual definition that explains why they are intuitively grouped together or that affords a way to study the relative impact of different designs, a question of interest to practitioners for whom too few research studies provide actionable information (Weiss, Bloom, & Brock, 2013).

Student success programs are premised on the notion that new college students stand to benefit from explicit instruction in how to develop skills, knowledge, and support networks shown to be critical for persistence, achievement, and completion (Robbins et al., 2004). These skills and knowledge are particularly salient for many community college students, who are often underprepared for college-level work, the first in their families to pursue college, or trying college out to determine if they feel they belong and are college material, despite previous invalidating experiences (Rendón & Muñoz, 2011). Another equally important notion, then, that provides a rationale for student success programs is that students stand to benefit from explicit enactment or rehearsal of behaviors and interactions that constitute often unspoken norms and expectations of college going as a form of learned social literacy (Gildersleeve, 2010; Hatch, Mardock-Uman, Garcia, & Johnson, in press). In short, the form of student learning in student success courses is as much an object of these programs’ designs as is the content knowledge of their curriculum, although the form is not often recognized as a concurrent learning objective.

Empirical evidence, as summarized in various literature reviews (Brownell & Swaner, 2009; Crisp & Taggart, 2013; Karp, 2011, 2016; Kuh, 2008; Kulik, Kulik, & Shwalb, 1983), bears out the proposition that student success programs provide the means, in form and content, to foster college persistence, academic achievement, learning gains, and completion. As a result, student success programs have been broadly implemented in public 2-year community colleges, according to national surveys, even though student participation rates remain relatively low overall (Center for Community College Student Engagement [CCCSE], 2012). Nonetheless, much of the evidence of their effectiveness is correlational or even anecdotal in nature, leading to unanswered questions about how or why some programs have an impact and therefore how to scale them up to benefit more students.

Many argue the evidence for student success program effectiveness in fostering student success can be mostly attributed to selection bias (Pike, Hansen, & Lin, 2011). Yet longitudinal and random-assignment research designs have shown this to not necessarily be the case; and in fact, there is evidence that participation can in some cases have a long-lasting effect (Rutschow, Cullinan, & Welbeck, 2012; Weiss et al., 2014). The difference between long-lasting, diminishing, or null effects, according to a study by Karp, Raufman, Efthimiou, and Ritze (2016), may ultimately depend in large part on the extent to which students actually enact and apply the metacognitive skills and knowledge that the student success programs’ curricula are built around. Fortunately, this phenomenon of metacognitive skills and knowledge application is readily operationalized through the notion of student engagement, which now is measured at hundreds of colleges yearly through the National Survey of Student Engagement (NSSE) and the Community College Survey of Student Engagement (CCSSE) (McCormick, Kinzie, & Gonyea, 2013). In these popular survey efforts, student engagement is defined as the extent to which students engage in educationally meaningful activities as a function of efforts of the institution to foster that activity (Kuh, Kinzie, Schuh, & Whitt, 2005) and typically measured through composite engagement benchmarks or indicators. A substantial body of research has shown that student engagement has an important influence on later positive student outcomes (McCormick et al., 2013). Evidence is also clear that participation in student success programs is related to higher levels of engagement (CCCSE, 2012). The question remains, though: Is it participation alone that matters? Or are there features of different student success programs that foster student engagement to different degrees?

A fundamental challenge to approaching this question is that the current state of art in program impact research suggests few methodological ways forward. Scholars (Bailey & Alfonso, 2005; Brownell & Swaner, 2009; Crisp & Taggart, 2013; Karp, 2016) point to the lack of multisite studies, the lack of substantive programmatic detail in research reports, and an almost exclusive reliance on analytical designs that, although showing the differential effects of participation in dichotomous terms, fail to explain the source of the variation of outcome measures (Weiss et al., 2013). Thus, researchers have produced little information that is useful to practitioners who are tasked with crafting experiences that make a difference for students whom community colleges readily admit but who typically can afford little time getting started right in their educational trajectory if they are to persist and achieve their goals (Calcagno, Crosta, Bailey, & Jenkins, 2007; Hatch & Garcia, 2017).

Dual Purposes of the Study

The purpose of this study is twofold. First, the purpose in straightforward terms is to understand how the curricular and programmatic structure of student success programs at community colleges is related with measures of student engagement. I employ the framework of activity theory, described in detail below, to conceptualize multiple instances of student success programs in comparable ways and operationalize their curricular and programmatic features so as to determine which are most closely related to engagement measures. By determining the relationship of content (design and curriculum) with form (the emergent patterns of student engagement), we avail ourselves of one way of looking into the black box of student success programs.

A secondary purpose, however, and one that is arguably more salient in the context of research literature in which this study is positioned, arises from the call for novel conceptual and methodological approaches to uncover evidence of program impact beyond predominant single-site inquiries that rely on dichotomous indicators to operationalize participation. The lengthier first portion of this study, therefore, is primarily a conceptual argument for understanding and operationalizing student success courses as instances of a broader type of activity. Consequently, I posit that they have a coherent form beyond various narrow curricular definitions, which warrants one way to conduct nuanced yet wide-scale research into variations of that form. Alas, the empirical portion of this study regarding how engagement is related to variations in these curricular and pedagogical settings addresses this conceptual purpose only in part, as it relies on secondary, quantitative data rather than a requisite multimethod approach utilizing detailed in situ case studies (Plewis & Mason, 2005). Thus, the study offers a starting point, or perhaps a counterpoint, for this kind of nuanced, multisite empirical research needed to advance this field of research.

The central research question guiding this study in narrow terms is, What is the relationship between student engagement and the design of student success programs in terms of their curricular and sociocultural elements? To accomplish the study’s dual purposes, however, this research question requires further elaboration in light of activity theory as a conceptual framework and its ontological linkages to engagement as a means to ascertain the student environment.

Research and Theories Guiding the Study

In this review of research literature and theories that inform the study, I present an overview of activity theory and a rationale for using it as a conceptual framework for studying student success programs. This is followed by an explanation of the sociocultural nature of student engagement theory and how it aligns well with activity theory as a way to understand how student success programs are structured to foster engagement. Because of the limited number of studies regarding student engagement in relation to student success programs, I draw on the literature from both the 2-year and 4-year sectors, especially in light of the shared provenance of the student engagement construct found in NSSE and CCSSE, respectively (McCormick et al., 2013).

Activity Theory and Defining Student Success Programs as Activity Systems

Activity theory, rather than being a theory proper in the narrow terms of a set of explanatory propositions, is an accommodating conceptual framework for understanding jointly individual and group behavior within particular social structures (Roth & Lee, 2007). Activity theory is one in a family of sociocultural and cultural-historical ecological frameworks stemming from Vygostky’s pioneering notion that a subject’s action in relation to some object is always mediated by an artifact or tool, whether concrete or abstract (as cited in Roth & Lee, 2007; Yamagata-Lynch, 2010).

The variant known as activity systems analysis, developed by Engeström (1987), extends mediated action to account also for additional environmental processes and factors. Activity systems analysis, like other activity theory variants, aims to make sense of complex systems of human work and labor to reveal their nature, inherent tensions, and ultimately, opportunities for institutional improvement. The framework is scalable and dialectical, adaptable to the study of transitory praxis (time-bounded instances of actions) and broader patterns of activity (Roth & Lee, 2007), such as in this case of planned courses and interventions. Activity systems analysis posits that an activity system, as depicted in Figure 1, consists of its participants, the object or motive of the activity, its mediating artifacts (e.g., instruments, tools, signs and symbols), the rules and social conventions that shape how participants carry out the activity, the community within which and for which the activity takes place, and the division of labor within the activity. Whereas the outcome of an activity system—that is, the resulting product or, in this case, desired and actual student outcomes—is external to the system itself, the object (i.e., learning objective, purpose, motive) is a defining aspect of the system. Indeed, activity theory posits that “the main thing that distinguishes one activity from another…is the difference between their objects [which] gives [them] a determined direction” (Leont’ev, 1978, p. 62).

Figure 1.

Diagram of a student success program as an activity system, adapted from Engeström (2010) and Roth and Lee (2007).

The salience of object-oriented activity is important to the question of student success programs because it provides a way to explain why researchers have intuitively grouped various types of student success programs together, despite the lack to date of conceptual justifications for those groupings. This is especially clear in literature reviews over decades (Bailey & Alfonso, 2005; Brownell & Swaner, 2009; CCCSE, 2012; Crisp & Taggart, 2013; Hatch, 2016; Karp, 2011, 2016; Kuh, 2008; Kulik et al., 1983), where scholars have loosely and variously gathered in studies on student success programs based not on conceptual understandings of programs’ organizing purposes. Rather, their justification has rested on the preponderance of published studies that name given interventions meant to support the success of students who arrive at college less than prepared or at risk for failure. Researchers’ reliance on nominal categories of programs with broad-reaching purposes has resulted in disjointed lines of research based on trends in names of student success interventions, making it challenging to draw conclusions as to how programs are related among themselves conceptually, let alone as to which of their features are effective, for whom, in what combination, and under what circumstances.

Different nominal categories of 1st-year student success programs are not completely arbitrary, of course, and cannot be discounted out of hand. Traditional definitions point to important distinguishing features, for instance, such as whether a program is skills focused (1st-year seminar), of relatively brief duration (orientation), or utilizes co-enrollment in multiple courses or activities (learning community, co-requisite developmental education). Yet, despite these idiosyncrasies, research shows that student success programs across college sectors share a large extent of curricular and programmatic features and that hybridization is more the rule than the exception (Hatch & Bohlig, 2016; Young & Hopp, 2014; Young & Keup, 2016). But even more pointedly, despite their particulars, 1st-year student success programs invariably all share common, fundamental objectives, which are to socialize entering students to college life and equip them with the self-regulatory skills, knowledge, and social and academic networks to succeed (Hatch, 2016; Robbins, Oh, Le, & Button, 2009).

Thus, as seen through the framework of activity theory, these practices are instances of a broader kind of activity. I call this concept a structured group socialization experience (SGSE), a term adapted from what CCCSE labeled structured group learning experiences (CCCSE, 2012) in light of their conceptualization through the framework of activity theory that focuses on their end goals of socialization toward a college-going literacy. Additional support for this conceptualization for SGSEs comes from two studies. One was a multiple case study by Reid, Reynolds, and Perkins-Auman (2014), one of the few of its kind that takes on the task of deriving a conceptual definition of student success programs—in this particular case, 1st-year seminars. Reid and colleagues found that regardless of their nominal objectives, the three 1st-year seminars in their sample tended to converge in practice on a common objective of learning and rehearsing self-regulatory skills. Furthermore, seminars were characterized by a triadic reciprocality among the participants’ cognitive factors, their interpersonal behavior, and the environment, findings that echo the triple-faceted sociocultural nature of activity systems (Figure 1). In a separate study (Hatch et al., in press), which entailed a multiple case study of four student success courses at community college campuses of varying size and locations, it was also found that in daily activity the participants’ objectives converged around community building and college-going rehearsal despite nominal and substantive differences in stated program objectives.

The terms student success program and student success course are often used interchangeably in the literature. This may be the case because the most common instances or manifestations of programs are time-limited, group-based interventions that are realized as courses, even though their effective mechanisms for student success should arguably be distributed throughout college (Karp, 2016). In this study, I use the broad term program, unless in reference to other authors’ studies, and with the understanding that most participants’ responses in the current data refer to courses.

Leveraging Activity Systems Analysis for Quantitative Inquiry

Engeström’s (1987, 2000) widely used activity system triangle (Figure 1) is a tool to reveal, depict, and give structural coherence to the social and material resources that are salient in an activity (Roth & Lee, 2007). Yamagata-Lynch (2010) points out that a reason the diagram is prevalent in activity systems analysis research is that it provides a method for communicating results in a manageable and meaningful manner. This matters to the problem at hand because it points to a way for cross-site and longitudinal comparisons of evidence regarding how student success programs work in all their complexities. Because the unit of analysis in activity systems analysis is mediated activity itself, rather than individuals or environments separately, the diagram provides a useful organizing framework for focusing the discussion on why and how systems work, not just whether they do for individual participants. As a heuristic tool then, the activity systems framework allows investigators to accomplish collaborative inquiry using multiple research approaches when studying multiple contexts of complex real-world human learning situations (Yamagata-Lynch, 2010).

Whereas the majority of research implementing activity theory is qualitative in nature, the activity systems analysis approach and activity systems heuristic triangle points to how activity theory can be leveraged in quantitative studies too, as illustrated in two studies by Atteberry and Bryk (2011) and Plewis and Mason (2005). Atteberry and Bryk, in their study of the link between coaching of literacy instruction among K–2 urban school teachers and changes in student learning, turned to an activity theory framework “to conceptualize how and why teachers’ engagement in professional development with school-based literacy coaches might vary from classroom to classroom and school to school” (Atteberry & Bryk, 2011, p. 358). Although not utilizing activity theory to its fullest extent to explore the cultural-historical evolution of literacy coaching as a practice, they nonetheless found activity theory to be useful for the purposes of cross-site analysis of similar practices in drawing attention to how individual agents approach their tasks as influenced by tools at their disposal and the social context. In this case, an activity systems framework provided a useful sociological grounding for exploratory work where a causal design was not warranted or possible. This is the approach used in this article, which should not be confused with an activity systems analysis proper. Rather, akin to prevailing approaches to quantitative studies of college choice, persistence, and attainment, for instance, nuanced factors entangled in complex sociocultural processes are selected and operationalized through a given framework, revealing evidence of variable relationships in addition to opportunities for further research given limits to quantitative measurement.

The Atteberry and Bryk (2011) article is in fact just one outcome of the much broader Consortium on Chicago School Research (CCSR) research effort created by Anthony Bryk at the Carnegie Foundation (University of Chicago Consortium on School Research, 2017). Although the conceptual framework of the project has evolved over time, activity theory and an activity systems framework were at the foundation of the multipronged CCSR research effort, including the Chicago Public Schools’ 5Essentials Survey and the CCSR model for the role of research in supporting urban school reform (A. Atteberry, personal communication, May 21, 2017). Thus, Atteberry and Bryk (2011) is an example of how quantitative inquiry can leverage activity systems analysis as a heuristic tool in support of a larger research agenda where nuanced findings are comparable across related and coordinated research efforts. In quantitative terms, the use of activity systems analysis ultimately takes the form of a conceptual framework providing guidance in selecting and operationalizing variables and interpreting findings.

Plewis and Mason (2005), in their methodological paper describing an approach to discovering “what works and why” in community-based programs to reduce juvenile criminal behavior, show how a quantitative activity systems analysis can be used to identify differential effects of various program design features. The rationale for their approach stems from the fact that although experimental or quasiexperimental methods bring forth evidence whether program participation has an impact, the program itself remains a black box—we simply do not know why the outcome was observed or not (Weiss et al., 2013). Plewis and Mason propose making program heterogeneity the express object of study, rather than differential effect of (non)participation. This is essentially the approach that Porter and Swing (2006) used to analyze the effect of 1st-year seminar program features on student persistence. Yet without data about the program design, Porter and Swing had to rely on group means of student-level data to describe courses, an approach that overlooks the difference in the intended curriculum and its resultant outcomes. The key to program impact heterogeneity research is to have data on multiple parallel programs and their participants to model the program effects while accounting for individual factors in a multilevel fashion. This is the approach adopted for the current study. Short of a full mixed-methods approach that allows for extensive qualitative case studies and quantitization (Sandelowski, Voils, & Knafl, 2009) of those data, I rely on a survey of college officials in charge of student success programs in order to gather data in a systematic way on a large scale that can be similarly quantitized and merged with student-level data.

Previous Student Engagement–Student Success Program Research

To date there is a limited number of studies that investigate student success programs in relation to student engagement. In a broad sense of engagement as one form of psychosocial (motivational, emotional, and social) control factors, Robbins, Oh, et al. (2009) showed through a meta-analysis of student success programs in the 4-year and 2-year sector that participation in First-Year Experience-type courses on average have a meager effect on social engagement, which in turn mediates also to a limited degree student retention, but was unrelated to academic performance.¹ Zhao and Kuh (2004), using a definition and measurement of engagement akin to the one used in this current study, but for the 4-year sector, found that learning community participation was associated with higher engagement levels, in accordance with research by Inkelas and Weisman (2003) on living learning communities in 4-year colleges. Pike, Kuh, and McCormick (2011) studied this relationship in more depth and found that there was substantial variability in the relationship across colleges but that institutional characteristics accounted for as little as 30% of the variability between colleges. They specifically called for additional research accounting for “the character and structure of the learning community experience [that] can account for the unexplained variance in student engagement–learning community relationships” (Pike et al., 2011, p. 316). To date no research has investigated this relationship in the 2-year college sector, whether in regard to learning communities or student success programs more broadly. One notable exception to the reliance on dichotomization in program impact research is a study by Porter and Swing (2006), who used student responses to the First-Year Initiative survey at 45 different institutions, extrapolated to the college level, to understand the relative impact of various aspects of different 1st-year seminars on students’ intent to persist. The study accounted for campus engagement and peer connections (types of student engagement) but only as school-level independent variables instead of outcome variables, thus providing a methodological example for the current study though no directly comparable empirical results.

The Sociocultural Nature of Student Engagement Theory

In this study, I use the notion of student engagement as a way to unpack the black box of SGSEs. Student engagement, as described by McCormick et al. (2013) and Wolf-Wendel, Ward, and Kinzie (2009), is closely related to, and has developed alongside, concepts of student involvement (Astin, 1984) and integration (Tinto, 1993). One essential aspect of student engagement as most commonly used today (McCormick & McClenney, 2012) that is regularly overlooked in the literature is that engagement is conceptually a joint phenomenon existing at the intersection of individuals and institutions. Despite its name, it is not accurately a student-centric concept alone. Rather, engagement has two key components:

The first, is the amount of time and effort students put into their studies and other activities that lead to the experiences and outcomes that constitute student success. The second is the ways an institution allocates its human and other resources and organizes learning opportunities and services to encourage students to participate in and benefit from such activities. (Kuh et al., 2005, p. 9)

In other words, engagement is not something a student does or experiences but rather is the result of a lived reality that is co-constructed by students along with their peers, faculty members, and others, who all interact within colleges in a simultaneous specific and broad context. Whereas the constructs of involvement (the amount of physical and psychological energy a student devotes to his or her academic experience; Astin, 1984) and integration (the extent to which students come to share the attitudes and beliefs of their peers and faculty and the extent to which students adhere to the structural rules and requirements of the institution; Mayhew et al., 2016; Tinto, 1993) involve what students do, and therefore have implications for institutional action, engagement in contrast comprises institutional action—thus involving, not just implicating, what institutions do.²

From the perspective of sociocultural theory, of which activity theory is a variant, the conceptualization of student engagement as a dual student–institutional phenomenon takes on a renewed focus and inherent consistency. For example, in broad terms, engagement emphasizes the mesolevel of analysis that can be leveraged to work against stubborn regimes of educational practices (Trowler, 2005), which persist in part because of the misalignment of traditional macrolevel (sociological) and microlevel (psychological) conceptualizations of educational improvement. This is evident in how NSSE and CCSSE were developed in part as a response to the national discourse on college quality traditionally characterized by a capitalistic asset-based philosophy of reputation, resources, and selectivity (McCormick et al., 2013). Instead, engagement emphasizes behavior and environments, which jointly define achievement and are shaped by all stakeholders. Engagement theory, similar to activity theory, was first developed so that practitioners themselves—facilitated by researchers if appropriate—can improve their own practice, with an a priori assumption of a shared responsibility in the work (Engeström, 2000; Kuh et al., 2005). This rhetorical stance of engagement, if used in its strict sense, thereby requires an antideficit understanding of students’ role in the co-creation of meaningful educational environments and the institutional responsibility to be responsive to their students (cf. Harper & Quaye, 2014).

Another example of the sociocultural nature of engagement theory is that in this view, engagement is correctly conceptualized not as an outcome measure, as researchers sometimes inappropriately construe it, but rather as a kind of intermediate outcome (Astin & Antonio, 2012) that is a result of the sociocultural structuring of the college environment—not the (ultimate) end but an intermediate means to an end (Wolf-Wendel et al., 2009). Put in terms of an activity theory framework, we could say that engagement (i.e., the co-constructed socialization and enactment of successful college going by students and institutions) is at once the desired immediate outcome of SGSEs and a dialectical indicator of the larger process of college going, wherein the success of the individual and the collective are mutually entwined (Roth & Lee, 2007).

Research Question Revisited and Expanded

In light of the nuances of activity theory and the proposition that student engagement is a sociocultural framework by nature, the research question can be broken out in yet more specific ways, namely, What is the relationship between student engagement and the design of student success programs in terms of their (a) curricular elements (“tools” or “artifacts” in the words of activity theory), (b) the programs’ rules and social conventions (in the form of attendance duration and intensity [i.e., dosage], expectations for credit), (c) the community context within and for which the activity takes place, and (d) the division of labor (for instance, whether individual or group based or supported through auxiliary instruction)?

Method

To conserve space and to maintain the focus on the conceptual purposes of this study, the details of the data sources, the process for merging student-level and SGSE-level data, and the operationalization of variables are contained in Appendix A in the online supplemental material accompanying this article. In particular, Table S3 reports descriptive statistics for the student-level characteristics and Table S4 the SGSE-level characteristics.

Analytical Approach

Based primarily on the structure of the data, and anticipated variation between SGSE designs, this study employed hierarchical linear modeling (HLM), thus simultaneously modeling individual- and program-level effects (Gelman & Hill, 2007; Raudenbush & Bryk, 2002). I utilized SAS (v9.4) PROC MIXED following procedures recommended by Singer (1998). In accordance with recommendations for multilevel models, categorical variables were effect coded and continuous variables were grand-mean centered (Enders & Tofighi, 2007; Hox, 2010; Raudenbush & Bryk, 2002). To correct for student-level missing data, which may become a concern as I selectively—rather than randomly—reduced a large data set, I imputed 12 data sets to derive the statistical inferences reported below.

The analyses were done in three phases following recommended approaches (Raudenbush & Bryk, 2002): an unconditional null model, a model of student-level individual effects (within model), and a full (between) model with the addition of Level 2 effects. Although this approach of separating student factors and contextual factors is in accordance with most multilevel educational research studies, activity theory would suggest that individual- and contextual-level elements cannot be so easily separated because they jointly compose the activity system. Nonetheless, given that there is abundant literature on individual-level factors related to engagement and the current study’s variables of interest occur at the second level of analysis, this traditional approach is still appropriate for the task at hand, allowing for a special focus on the programmatic structure of engagement.

Limitations

The cross-sectional nature of the data used in this study presents conceptual and analytical limitations because the historical development of engagement over the course of SGSEs remains unaccounted for, and any observations about causal relationships rely on theory. This issue is alleviated somewhat in that I do not construe engagement as an outcome outside of the activity system. Rather, engagement measures are ways to gauge to what degree the objects (purposes) of the activity system are being enacted. Longitudinal data are needed to understand the relationship of program features with more distal student outcomes, whether or not mediated via engagement (Pike, Smart, & Ethington, 2012). A related limitation is that the use of activity theory does not realize its full potential to understand the interactions among activity system elements in order to uncover and ameliorate inherent tensions that inhibit successful outcomes. The theory does provide for the conceptualization of multiple programs as SGSEs and the selection of pertinent variables. But the study stops short of problematizing the longitudinal, sociocultural interactions among elements. Alternately, the use of the theory in this way for multivariate quantitative analysis is arguably an analytical delimitation that nonetheless illustrates the adaptability of an activity systems framework for different epistemological applications. Here and elsewhere I have argued (Hatch, 2016) how activity systems analysis is a useful framework to elicit complementary empirical evidence from different research paradigms. Separately, I have illustrated through case study analysis (Hatch et al., in press) the very historical, sociocultural interactions among student success course features that this study is not able to address but support its findings that beyond curricular elements, what may have the greatest impact on students is the chance to rehearse and reflect on the college-going experience in a supportive environment.

This study, like all survey research and similar engagement research, is attenuated by concerns of reliability and validity of self-reported behaviors, warranting caution in interpreting the results. Self-reported data in institutional research have been shown to be reasonable, given the trade-offs of broad-scale coverage for in-depth detail, when surveys are grounded in the empirical research literature, the information requested in known to respondents, the respondents believe the questions merit a thoughtful response, and the data collection process mitigates threats to validity, such as from social desirability bias (Gonyea, 2005; Pike, 2011). These conditions are met through the rigorous survey administration practices of CCCSE (McCormick et al., 2013). Still, the issue of self-reporting is a limitation of the data-matching process because it depends on student and institutional respondents alike correctly identifying program labels and, for institutions, a reliable accounting of program features. The resultant approximate match, although conservatively executed, certainly led to a loss of information. A more definite match and corroborated program descriptions would be preferable. Despite these limitations, the risk of errors in the results may be tolerable as a first step toward addressing the methodological gap this study responds to, as long as the findings are cautiously considered. A related caution in interpreting the results is that engagement benchmarks were designed not as psychometric scales but as a heuristic measure for engendering conversations about related practices by practitioners (Pike, 2013). Any relationships among modeled variables point to possible lines of inquiry to be investigated in more detail. Last, although the sample of colleges is diverse, it may not be representative. Participants come from self-selected colleges presumably with the motivation and/or resources to commit to implementing these special programs or at least to perform the institutional self-reflection needed to respond to a survey about them.

Results

Of the programmatic elements of SGSEs included in this study, three were significantly related with engagement measures: co-curricular and community activities, the number of credit hours awarded, and—in the case of the academic challenge benchmark—the inclusion of college success skills in the curriculum. Conversely, integrated academic planning and support was negatively associated with academic challenge. In some cases, I note findings within the p level cutoff of .10 if not for the sample size, due to the conditional data-matching process, to note relationships that may warrant further investigation.

Null Model: Difference in Engagement Across SGSEs

The unconditional, or null, model is analogous to a one-way ANOVA, in which the intercept varies across programs, thus revealing the variance in engagement within and among SGSEs. Table 1 presents the variance components for each outcome measure. Although relatively small, the variance between SGSEs is not trivial. Even relatively small intraclass variance can have important implications, and any estimation of standard errors of those situational effects is desirable given the research purpose and data source (Denson & Chang, 2009; Umbach & Kuh, 2006). Within and between variance establishes a baseline for comparison with subsequent models.

Table 1

Variance Components of Dependent Variable in Null Model

Variable	Active and collaborative learning	Student effort	Academic challenge
SGSE-level variables
Intercept	48.35	52.51	48.91
Variance components
Variance between SGSEs (intercept)	42.08	38.54	13.53
Variance within SGSEs (residual)	607.53	585.64	574.46
Proportion variance between (ICC)	0.06	0.06	0.02
Model fit
–2 log likelihood	20467.87	20386.92	20316.54

Note. SGSE = structured group socialization experience; ICC = intraclass correlation.

Within Model: Student-Level Variables

Table 2 presents results for both the within model and full between model for each of the three engagement measures. In HLM, it is possible to calculate standardized beta coefficients for fixed effects only (Hox, 2010), which are reported in parentheses for significant effects. For the within model, the intercept was allowed to vary across SGSEs as in the null model. Given the large proportion of variance within programs compared to across programs, combined with no conceptual justification to suspect site-specific regression slopes for predictors, I did not model any random components for covariates. Still, between 10% and 12% of the explainable variation within SGSEs is accounted for by the student-level predictors. This degree of residual variance at the individual level does not change in any important ways from the null model to the within model (nor to the full model, below). As Hox (2010) explains, this is at it should be because class-level variables cannot predict individual-level variation. In this case of the relationship between program design and engagement, the amount of variance explained by the student-level predictors at the program level is relatively small, thus reflecting the fact that student-level predictors are distributed almost equally across all SGSEs in the sample.

Table 2

Model Estimates for Relation of Engagement and Program-Level Curricular Features, Controlling for Student-Level Characteristics

	Active and collaborative learning		Student effort		Academic challenge
	Within	Between	Within	Between	Within	Between
Variable	B (β)	B (β)	B (β)	B (β)	B (β)	B (β)
SGSE-level variables
Intercept	47.57** (152.60)	48.47** (154.57)	51.01** (162.34)	51.00** (160.64)	47.50** (139.96)	48.15** (144.84)
Curricular tools
College success skills		0.01		0.05		0.68* (0.37)
Academic planning and student services		−0.21		−0.10		−0.92† (–0.94)
Rules and conventions
Duration (weekdays)		0.01		−0.04		−0.02
Intensity (hours per day)		0.37		−0.15		−0.21
Credit hours awarded		1.32* (1.57)		1.94** (2.29)		0.98† (0.98)
Community
Co-curricular and community activities		1.78* (3.00)		2.26* (3.85)		0.76
Contextualized and coordinated learning		−0.50		−0.65		−0.44
Division of labor
Assigned group work		3.29		−0.42		−0.15
Auxiliary instruction		−1.69		−1.34		−1.04
Variance components
Variance between SGSEs (intercept)	23.53**	10.78*	25.74**	11.64†	4.39	n/a
Variance within SGSEs (residual)	541.96**	543.70**	516.89**	518.60**	516.44**	515.16**
Proportion variance explained between	44.1%	74.4%	33.2%	69.8%	67.5%	n/a
Proportion variance explained within	10.8%	10.5%	11.7%	11.4%	10.1%	10.3%
Model fit
–2 log likelihood	20206.55	20194.11	20108.42	20094.68	20069.49	20049.44
Δ–2LL (compared to previous model)	261.32**	12.44	278.5**	13.74†	247.05**	20.05*
ΔFree parameters (df)	23	9	23	9	23	9

Note. All models control for enrollment intensity; enrolled in developmental course work; gender; hours spent working for pay, caring for dependents, and commuting; source of financial support; self-identified racial-ethnic background; international student or foreign national; highest academic credential; parents’ education level; age; whether has children at home; whether married; and how many SGSEs participated in. SGSE = structured group socialization experience.

†

p < .10. *p < .05. **p < .01.

Full Model: Student-Level and SGSE-Level Variables

At least three aspects of the results reported in Table 2 provide information about the nature and extent of the relationship between program variables and engagement: the change in model fit with the addition of Level 2 predictor variables, the change in the proportion of variance between SGSEs, and—naturally—the regression coefficients themselves. Overall, the findings reveal some limited evidence of the relationship of specific program structural elements with these three selected types of engagement.

Model fit

I used log likelihood ratio tests as part of the model-building process to check whether additional blocks of variables significantly improved model fit. The step from the null model to the within model for all three outcomes marked a notable improvement in the fit of the model to the data. For the step from the within to the full model, the model fit improved for the student effort and the academic challenge benchmarks (respectively, Δ–2LL = 13.74, df = 9, p < .10; and Δ–2LL = 20.05, df = 9, p < .05). However, in the case of active and collaborative learning, the addition of SGSE variables did not significantly improve model fit (Δ–2LL = 12.44, df = 9, p > .10), thus mitigating somewhat the saliency of the relationship between program design and this type of engagement.

Change in proportion of variance

The addition of program-level descriptors in the full model contributed a great deal to explaining the proportion of explainable variance between SGSEs. For active and collaborative learning, the proportion of variance explained between SGSEs went from 44.1% in the within model to 74.4% in the full model. For the student effort benchmark, the proportion of variance explained between went from 33.2% to 69.8% in the full model. And for the academic challenge outcome, whereas 67.5% of the explainable variance between SGSEs was accounted for in the within model alone, with the step to the full model there was not enough variation in the data to attribute any variation to the random intercept at all (as noted through warnings in the SAS output). Thus, the full model for academic challenge required a flat regression model, despite the fact that the fit of the model still improved. In other words, by including program-level descriptors, all of the significant variation between SGSEs on this engagement indicator was parceled out. The proportion of variance explained matters because it provides compelling evidence that the sociocultural structures of SGSEs are closely related to engagement in ways that merit the exploration of the coefficients of course features.

Variable coefficients

Regarding specific coefficients of program variables that were significant, results showed that co-curricular and community activities was positively and significantly related to two of the three engagement outcomes: For each additional type of element in this group, there was a 1.78-point increase in active and collaborative learning (p <.05) and a 2.26-point increase in student effort (p <.05). The standardized beta coefficients of 3.00 and 3.85, respectively, were the largest among SGSE-level predictors, underscoring their salience. That this effect is one of the most prominent is noteworthy because this curricular feature was relatively uncommon: 55% of SGSEs had factor scores of 0 on this measure, indicating most implemented no instances of any kind of service project, service learning, or participation in campus activities. Conversely, the implementation of program elements of academic planning and student support (ranging 0 to 5 different items), features receiving the greatest attention in defining and planning student success courses, was marginally associated with a 0.92-point decrease in academic challenge (p < .10).

The variable that was significantly and positively related to all engagement measures was the number of credit hours that a program afforded. Each additional credit hour was associated with a 1.78-point increase in active and collaborative learning (p < .05), a 1.94-point increase in student effort (p < .05), and a 0.98-point increase in academic challenge (p < .10).

Discussion and Implications

This study addresses the call of previous research regarding engagement and student success programs to investigate the variability in engagement across institutions that has remained mostly unexplained by considering the relationship of the structure of student success programs that are designed to purposively affect the college-going experience (e.g., Keup & Barefoot, 2005; Pike et al., 2011). The current study departed from previous research in this area by modeling variation in engagement due to program features instead of dichotomous program participation. Prior research showed that participation results in higher engagement (Inkelas & Weisman, 2003; Pike et al., 2011; Zhao & Kuh, 2004) but called for more detailed information about program features in order to account for the majority of variance between contexts left unexplained (Pike et al., 2011). Results here do indeed provide evidence that accounting for program features explains most or all of the cross-contextual variance in engagement for students who participated in these programs, thus addressing the question raised by Pike and colleagues (2011) about the explanatory power of program-level variables in relation to engagement.

Furthermore, results complement evidence that different kinds of engagement are related to student success program participation in nuanced ways. For instance, previous research has found that participation in learning communities, compared to nonparticipation, is most strongly associated with faculty interaction and peer collaboration but less markedly so with academic effort and supportive environments (Inkelas, Daver, Vogt, & Leonard, 2007; Inkelas & Weisman, 2003; Pike et al., 2011; Zhao & Kuh, 2004). The current study unpacks these associations further by showing that, for instance, the level of academic challenge that students experience does not differ according to cross-contextual factors of student success programs and appears to be related to program features in the same way regardless of the context. But in the case of active and collaborative learning, two findings reveal a limitation of this measure of engagement to describe the impact of SGSEs. First, the addition of program features did not significantly improve model fit. Second, as opposed to the other two engagement outcome measures, even after controlling for student characteristics and program features, there was still cross-site variation left unexplained. This may suggest relatively more room for intentional design and implementation. Alternately, the unexplained variance may be due to several of the items in this scale asking about instructor-driven activities—including class presentations, group work, and community projects—which depend not necessarily on students enacting learned metacognitive skills but rather on the prerogative of yet other instructors too. As such, this illustrates the limitations of using engagement measures as simple outcomes of interventions, given multiple interdependent systems. Thus, findings agree with the observation that engagement is not a unitary construct, more engagement is not always necessarily better, and types of engagement may be just as important as levels of engagement in relation to college experiences (Hu, 2011; Pike, 2013; Pike et al., 2011; Pike & Kuh, 2005).

Rules, Conventions, and Community

Of the programmatic features operationalized in the study through activity theory, it was the structuring rules of the program (specifically, credit hours) and the community setting (service and/or campus activities) that were most notably related to engagement measures. The positive relationship between credit hours awarded with all three measures of student engagement underscores the salience of this feature in many practitioners’ descriptions of programs (such as those published for years by the National Resource Center for the First-Year Experience; e.g., Young & Hopp, 2014) but has received very little attention in the research literature, despite evidence of the prominent role of credit hours in student motivation (Ward & Commander, 2011). These findings suggest that the distinction of credit-bearing status warrants relatively more attention in student success program research.

Curricular Features

The curricular “tools” of the programs, in the form of college success skills and academic planning, which receive a great deal of attention in practice and in the literature, were only marginally related to engagement, and only in terms of one of the three kinds of engagement. These results are a counterpoint, or at least a complement, to the literature that emphasizes the critical role of study skills, time management, and related competencies in college curricula (Allan & Clarke, 2007; Duggan & Williams, 2010; Engstrom & Tinto, 2008; Robbins et al., 2004; Struthers, Perry, & Menec, 2000).

In particular, the negative relationship of academic planning and student services with academic challenge is a counterintuitive finding that does not so readily compare to existing literature. One would normally expect that the inclusion of advising and student services in student success programs would translate to higher engagement. One possible explanation is that, with these activities integrated into the student success programs, students are less reliant on resources outside of the classroom. Indeed, research shows that many students who are underprepared for college—precisely the students for whom these programs are often meant—tend to utilize these services to lesser degrees than their peers (Robbins, Allen, et al., 2009). Because intrusive student support does not necessarily correspond to utilization, the negative coefficient could be an artifact of static utilization despite the deployment of relatively more resources, after parceling out variation due to other factors. If so, the saliency of this relationship relative to other covariates is noteworthy. Here, implications for practice are not forthcoming. Further research is needed to corroborate and clarify the phenomenon.

Conclusion

The results confirm previous research that shows limited impact of participation in student success programs on student engagement (Robbins, Oh, et al., 2009). But the results go beyond previous research by showing details of how, indeed, particular program features may have more or less relative impact and so promise to be valuable to community college practitioners who are tasked with designing and scaling up practices to reach as many students as possible but with limited resources to include all possible features. Foremost, the findings underscore that structural elements are likely more critical to fostering engagement than the skills-focused curricular questions that dominate the organizing rationale for these courses (Hatch, 2016; Robbins et al., 2004). In particular, if designers of student success programs aim to provide students the opportunity to enact and rehearse the often unspoken metacognitive behaviors and skills (Gildersleeve, 2010) that foster subsequent achievement, persistence, and completion, the findings here show that connections with the community/campus and academic credit afforded are fundamental. This speaks to programmatic debates and decisions whether such programs are integral or peripheral to the rest of the college experience, and best led by academic affairs or student services, or both (Nesheim et al., 2007; Song, Price, & Dodrill, 2016 ).

Conversely, the degree to which programs emphasize college success skills, although naturally related with the degree of academic challenge students perceive, seems to have little relation with students’ active and collaborative learning or effort exerted. This knowledge promises to be valuable to community college practitioners who are tasked with designing and scaling up practices to reach as many students as possible. Nonetheless, the results are just a first look at these types of relationships, warranting further research beyond what was feasible with the available data, even if arguably more complete and broader in scope than in most published program impact research. In particular, this study readily illustrates the limitations of single engagement measures to fully account for the interdependence of skills and knowledge taught in one course as enacted in the context of yet other college courses. A robust multisite activity systems analysis, for example, conducted with in-depth case studies over time, would promise to reveal how the SGSE activity system interacts with other salient activity systems, using the networked activity systems analysis proposed by Engeström (2010).

Beyond the immediate results and implications presented here regarding engagement outcomes, this study illustrates the general affordances of activity systems analysis in realizing some of the methodological improvements that have been called for in the student success program research literature (Bailey & Alfonso, 2005; Pike et al., 2011). Indeed, this framework and method show at least one way to address the dual methodological challenges of conceptualizing multiple instances of programs and operationalizing descriptors of their structure in comparable ways (Hatch, 2016). Because of the adaptability of activity theory to various levels of analysis, and to traditions of both qualitative and quantitative research methods (Atteberry & Bryk, 2011; Plewis & Mason, 2005), related research promises to be conceptually compatible across studies, potentially allowing for multiple complementary perspectives on the perennially difficult and neglected black box of college environments (Astin & Antonio, 2012). Whether subsequent research—limited mostly by the costs of gathering detailed, qualitatively rich, and longitudinal data regarding students, programs, and institutions—might corroborate or refute these findings, further inquiry promises to improve understanding of the central question of not just whether programs are effective but how and why.

Footnotes

Acknowledgements

This research utilized data originally gathered as part of the initiative “Identifying and Promoting High-Impact Educational Practices in Community Colleges,” conducted by the Center for Community College Student Engagement (CCCSE). This initiative was supported in part by generous funding from the Bill & Melinda Gates Foundation and the Lumina Foundation. However, this study was conducted independently and the findings strictly reflect the opinion of the author alone and not necessarily those of CCCSE or its directors, the Bill & Melinda Gates Foundation, or the Lumina Foundation.

1.

First-Year Experience is a registered trademark of the National Resource Center for the First-Year Experience and Students in Transition.

2.

The wording of the items in the National Survey of Student Engagement and the Community College Survey of Student Engagement instruments reflect this joint student–institutional conceptualization. For instance, the sometimes controversial “self-reported gains” items (Pike, Kuh, & Gonyea, 2007; Porter, 2013) do not, as opposed to how they are referred to in the shorthand, ask students to report their gains in knowledge, skills, and personal development. Rather, the items ask respondents to evaluate to what degree their experience at the college has contributed to gains in those areas—and therefore, by logical extension, regardless of actual gains. The wording of the items precisely gets at the construct of how students are reciprocally engaged in, and engaged by, their college experience.

ORCID iD

D. K. Hatch

Author

DERYL K. HATCH is an assistant professor at the University of Nebraska-Lincoln. His research focuses on the environments within community colleges and broad-access institutions that may foster or hinder equitable access, quality, and success in higher education, particularly for individuals from marginalized, traditionally underserved, and underrepresented groups.

References

Allan

Clarke

(2007). Nurturing supportive learning environments in higher education through the teaching of study skills: To embed or not to embed? International Journal of Teaching and Learning in Higher Education, 19(1), 64–76.

Astin

A. W.

(1984). Student involvement: A developmental theory for higher education. Journal of College Student Development, 25(4), 297–308.

Astin

A. W.

Antonio

A. L.

(2012). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education (2nd ed.). Lanham, MD: Rowman & Littlefield.

Atteberry

Bryk

A. S.

(2011). Analyzing teacher participation in literacy coaching activities. The Elementary School Journal, 112(2), 356–382.

Bailey

Alfonso

(2005). Paths to persistence: An analysis of research on program effectiveness at community colleges. Indianapolis, IN: Lumina Foundation for Education.

Brownell

J. E.

Swaner

L. E.

(2009). High-impact practices: Applying the learning outcomes literature to the development of successful campus programs. Peer Review, 11(2), 26–30.

Calcagno

J. C.

Crosta

Bailey

Jenkins

(2007). Stepping stones to a degree: The impact of enrollment pathways and milestones on community college student outcomes. Research in Higher Education, 48(7), 775–801. doi:10.1007/s11162-007-9053-8

Center for Community College Student Engagement. (2012). A matter of degrees: Promising practices for community college student success (a first look). Austin: University of Texas at Austin, Community College Leadership Program.

Crisp

Taggart

(2013). Community college student success programs: A synthesis, critique, and research agenda. Community College Journal of Research and Practice, 37(2), 114–130.

10.

Denson

Chang

M. J.

(2009). Racial diversity matters: The impact of diversity-related student engagement and institutional context. American Educational Research Journal, 46(2), 322–353.

11.

Duggan

M. H.

Williams

M. R.

(2010). Community college student success courses: The student perspective. Community College Journal of Research and Practice, 35(1/2), 121–134.

12.

Enders

C. K.

Tofighi

(2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121–138.

13.

Engeström

(1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki, Finland: Orienta-Konsultit.

14.

Engeström

(2000). Activity theory as a framework for analyzing and redesigning work. Ergonomics, 43(7), 960–974.

15.

Engeström

(2010). Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of Education and Work, 14(1), 133–156.

16.

Engstrom

C. M.

Tinto

(2008). Learning better together: The impact of learning communities on the persistence of low-income students. Opportunity Matters, 1, 5–21.

17.

Gelman

Hill

(2007). Data analysis using regression and multilevel/hierarchical models. New York, NY: Cambridge University Press.

18.

Gildersleeve

R. E.

(2010). Fracturing opportunity: Mexican migrant students and college-going literacy (2nd ed.). New York, NY: Peter Lang.

19.

Gonyea

R. M.

(2005). Self-reported data in institutional research: Review and recommendations. New Directions for Institutional Research, 2005(127), 73–89. https://doi.org/10.1002/ir.156

20.

Harper

S. R.

Quaye

S. J.

(2014). Making engagement equitable for students in U.S. higher education. In Harper

S. R.

Quaye

S. J.

(Eds.), Student engagement in higher education (2nd ed., pp. 1–14). New York, NY: Routledge.

21.

Hatch

D. K.

(2016). A brief history and a framework for understanding commonalities and differences of community college student success programs. New Directions for Community Colleges, 2016(175), 19–31. doi:10.1002/cc.20209

22.

Hatch

D. K.

Bohlig

E. M.

(2016). An empirical typology of the latent programmatic structure of promising practices at community colleges. Research in Higher Education, 57(1), 72–98.

23.

Hatch

D. K.

Garcia

C. E.

(2017). Academic advising and the persistence intentions of community college students in their first weeks in college. Review of Higher Education, 40(3), 353–390.

24.

Hatch

D. K.

Mardock-Uman

Garcia

C. E.

Johnson

(in press). Best laid plans: How community college student success courses work. Community College Review.

25.

Hox

J. J.

(2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.

26.

Humphreys

(2012). What’s wrong with the completion agenda, and what we can do about it. Liberal Education, 98(1). Retrieved from http://www.aacu.org/liberaleducation/le-wi12/humphreys.cfm

27.

(2011). Reconsidering the relationship between student engagement and persistence in college. Innovative Higher Education, 36(2), 97–106.

28.

Inkelas

K. K.

Daver

Z. E.

Vogt

K. E.

Leonard

J. B.

(2007). Living–learning programs and first-generation college students’ academic and social transition to college. Research in Higher Education, 48(4), 403–434.

29.

Inkelas

K. K.

Weisman

J. L.

(2003). Different by design: An examination of student outcomes among participants in three types of living-learning programs. Journal of College Student Development, 44(3), 335–368.

30.

Karp

(2011). Toward a new understanding of non-academic student support: Four mechanisms encouraging positive student outcomes in the community college (CCRC Working Paper No. 28). New York, NY: Community College Research Center, Teachers College, Columbia University.

31.

Karp

M. M.

(2016). A holistic conception of nonacademic support: How four mechanisms combine to encourage positive student outcomes in the community college. New Directions for Community Colleges, 2016(175), 33–44. doi:10.1002/cc.20210

32.

Karp

M. M.

Raufman

Efthimiou

Ritze

(2016). Revising a College 101 course for sustained impact: Early outcomes. Community College Journal of Research and Practice. Advance online publication. doi:10.1080/10668926.2016.1152929

33.

Keup

J. R.

Barefoot

B. O.

(2005). Learning how to be a successful student: Exploring the impact of first-year seminars on student outcomes. Journal of the First-Year Experience & Students in Transition, 17(1), 11–47.

34.

Kuh

G. D.

(2008). High-impact educational practices: What they are, who has access to them, and why they matter. Washington, DC: Association of American Colleges and Universities.

35.

Kuh

G. D.

Kinzie

Schuh

J. H.

Whitt

E. J.

(2005). Assessing conditions to enhance educational effectiveness: The inventory for student engagement and success (1st ed.). San Francisco, CA: Jossey-Bass.

36.

Kulik

C.-L. C.

Kulik

J. A.

Shwalb

B. J.

(1983). College programs for high-risk and disadvantaged students: A meta-analysis of findings. Review of Educational Research, 53(3), 397–414.

37.

Lee

J. M.

Rawls

Edwards

Menson

(2011). The college completion agenda: 2011 progress report. New York, NY: College Board.

38.

Leont’ev

A. N.

(1978). Activity, consciousness, and personality. Englewood Cliffs, NJ: Prentice Hall.

39.

Lester

(2014). The completion agenda: The unintended consequences for equity in community colleges. In Paulsen

M. B.

(Ed.), Higher education: Handbook of theory and research (Vol. 29, pp. 423–466). Dordrecht, Netherlands: Springer.

40.

Mayhew

M. J.

Rockenbach

A. B.

Bowman

N. A.

Seifert

T. A.

Wolniak

G. C.

Pascarella

E. T.

Terenzini

P. T.

(2016). How college affects students: 21st century evidence that higher education works. San Francisco, CA: Jossey-Bass.

41.

McCormick

A. C.

Kinzie

Gonyea

R. M.

(2013). Student engagement: Bridging research and practice to improve the quality of undergraduate education. In Paulsen

M. B.

(Ed.), Higher education: Handbook of theory and research (Vol. 28, pp. 47–92). Dordrecht, Netherlands: Springer.

42.

McCormick

A. C.

McClenney

K. M.

(2012). Will these trees ever bear fruit? A response to the special issue on student engagement. Review of Higher Education, 35(2), 307–333.

43.

Nesheim

B. E.

Guentzel

M. J.

Kellogg

A. H.

McDonald

W. M.

Wells

C. A.

Whitt

E. J.

(2007). Outcomes for students of student affairs–academic affairs partnership programs. Journal of College Student Development, 48(4), 435–454. doi:10.1353/csd.2007.0041

44.

Pike

G. R.

(2011). Using college students’ self-reported learning outcomes in scholarly research. New Directions for Institutional Research, 2011(150), 41–58. https://doi.org/10.1002/ir.388

45.

Pike

G. R.

(2013). NSSE benchmarks and institutional outcomes: A note on the importance of considering the intended uses of a measure in validity studies. Research in Higher Education, 54(2), 149–170.

46.

Pike

G. R.

Hansen

M. J.

Lin

C.-H.

(2011). Using instrumental variables to account for selection effects in research on first-year programs. Research in Higher Education, 52(2), 194–214. doi:10.1007/s11162-010-9188-x

47.

Pike

G. R.

Kuh

G. D.

(2005). A typology of student engagement for American colleges and universities. Research in Higher Education, 46(2), 185–209.

48.

Pike

G. R.

Kuh

G. D.

Gonyea

R. M.

(2007). Evaluating the rationale for affirmative action in college admissions: Direct and indirect relationships between campus diversity and gains in understanding diverse groups. Journal of College Student Development, 48(2), 166–182. https://doi.org/10.1353/csd.2007.0018

49.

Pike

G. R.

Kuh

G. D.

McCormick

A. C.

(2011). An investigation of the contingent relationships between learning community participation and student engagement. Research in Higher Education, 52(3), 300–322.

50.

Pike

G. R.

Smart

J. C.

Ethington

C. A.

(2012). The mediating effects of student engagement on the relationships between academic disciplines and learning outcomes: An extension of Holland’s theory. Research in Higher Education, 53(5), 550–575. doi:10.1007/s11162-011-9239-y

51.

Plewis

Mason

(2005). What works and why: Combining quantitative and qualitative approaches in large-scale evaluations. International Journal of Social Research Methodology, 8(3), 185–194.

52.

Porter

S. R.

(2013). Self-reported learning gains: A theory and test of college student survey response. Research in Higher Education, 54(2), 201–226.

53.

Porter

S. R.

Swing

R. L.

(2006). Understanding how first-year seminars affect persistence. Research in Higher Education, 47(1), 89–109.

54.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.

55.

Reid

K. M.

Reynolds

R. E.

Perkins-Auman

P. G.

(2014). College first-year seminars: What are we doing, what should we be doing? Journal of College Student Retention: Research, Theory and Practice, 16(1), 73–93.

56.

Rendón

L. I.

Muñoz

S. M.

(2011). Revisiting validation theory: Theoretical foundations, applications, and extensions. Enrollment Management Journal, 5(2), 12–33.

57.

Robbins

S. B.

Allen

Casillas

Akamigbo

Saltonstall

Campbell

. . . Gore

(2009). Associations of resource and service utilization, risk level, and college outcomes. Research in Higher Education, 50(1), 101–118.

58.

Robbins

S. B.

Lauver

Davis

Langley

Carlstrom

(2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130(2), 261–288.

59.

Robbins

S. B.

I.-S.

Button

(2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94(5), 1163–1184. doi:10.1037/a0015738

60.

Roth

W.-M.

Lee

Y.-J.

(2007). “Vygotsky’s neglected legacy”: Cultural-historical activity theory. Review of Educational Research, 77(2), 186–232.

61.

Rutschow

E. M. Z.

Cullinan

Welbeck

(2012). Keeping students on course: An impact study of a student success course at Guilford Technical Community College. New York, NY: MDRC.

62.

Sandelowski

Voils

C. I.

Knafl

(2009). On quantitizing. Journal of Mixed Methods Research, 3(3), 208–222. https://doi.org/10.1177/1558689809334210

63.

Singer

J. D.

(1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 23(4), 323–355.

64.

Song

Price

Dodrill

(2016, April). Bridging academic and student affairs in the first-year classroom. Paper presented at the annual conference of the Higher Learning Commission, Chicago, IL.

65.

Struthers

C. W.

Perry

R. P.

Menec

V. H.

(2000). An examination of the relationship among academic stress, coping, motivation, and performance in college. Research in Higher Education, 41(5), 581–592.

66.

Tinto

(1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago, IL: University Of Chicago Press.

67.

Trowler

(2005). A sociology of teaching, learning and enhancement: Improving practices in higher education. Revista de Sociologia, 76, 13–32.

68.

Umbach

P. D.

Kuh

G. D.

(2006). Student experiences with diversity at liberal arts colleges: Another claim for distinctiveness. Journal of Higher Education, 77(1), 169–192.

69.

University of Chicago Consortium on School Research. (2017). About us. Retrieved from https://consortium.uchicago.edu/about

70.

Ward

Commander

N. E.

(2011). The power of student voices: An investigation of the enduring qualities of freshmen learning communities. Journal of College Student Retention, 13(1), 63–85.

71.

Weiss

M. J.

Bloom

H. S.

Brock

(2013). A conceptual framework for studying the sources of variation in program effects. New York, NY: MDRC.

72.

Weiss

M. J.

Mayer

A. K.

Cullinan

Ratledge

Sommo

Diamond

(2014). A random assignment evaluation of learning communities at Kingsborough Community College: Seven years later. New York, NY: MDRC.

73.

Wolf-Wendel

Ward

Kinzie

(2009). A tangled web of terms: The overlap and unique contribution of involvement, engagement, and integration to understanding college student success. Journal of College Student Development, 50(4), 407–428.

74.

Yamagata-Lynch

L. C.

(2010). Activity systems analysis methods: Understanding complex learning environments. New York, NY: Springer.

75.

Young

D. G.

Hopp

J. M.

(2014). 2012-2013 National survey of first-year seminars: Exploring high-impact practices in the first college year. Columbia: National Resource Center for the First-Year Experience and Students in Transition, University of South Carolina.

76.

Young

D. G.

Keup

J. R.

(2016). Using hybridization and specialization to enhance the First-Year Experience in community colleges: A national picture of high-impact practices in first-year seminars. New Directions for Community Colleges, 2016(175), 57–69. doi:10.1002/cc.20212

77.

Zhao

C.-M.

Kuh

G. D.

(2004). Adding value: Learning communities and student engagement. Research in Higher Education, 45(2), 115–138.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.12 MB

0.00 MB