Abstract
Study abroad is believed to be a transformative learning experience for students. However, the extent to which study abroad adds value beyond what is possible on campus needs to be demonstrated. In this paper, we document the learning outcomes assessment undertaken by a faculty-led study-abroad program at a large university in the U.S. Southeast. Specifically, we describe the development of a theory-based pedagogical model of global citizenship for short-term study abroad and efforts to document student learning associated with its constructs. The results of these efforts indicate that when student learning outcomes and study-abroad pedagogy are aligned with theory, and rigorously assessed, the opportunity to demonstrate learning is possible, and opportunities for instructional improvement present themselves.
Keywords
In this paper we document the learning outcomes assessment process of a faculty-led study-abroad program (at a large university in the U.S. Southeast), rooted in global citizenship education, to conceptualize and measure student learning outcomes. Specially, we describe the development of a theoretical model of global citizenship used to frame study-abroad pedagogy and efforts to assess student learning related to its constructs. We argue that when study-abroad pedagogy and metrics of student learning are aligned with a coherent theoretical framework, and learning outcomes rigorously assessed, educators are able to assay whether instruction has been efficacious. They are also able to clearly identify opportunities for pursuing quality improvement. That is to say, building a theory-driven evidence base about study-abroad learning outcomes enables principled program evaluation and improvement. We offer our experience as an example for scholars and educators seeking to expand the scientific assessment of student learning in study abroad and develop data-driven methods for instructional reform.
Global Citizenship as the Guiding Framework for Short-Term Study Abroad
Global citizenship has emerged as a dominant theoretical construct in the study-abroad literature (de Wit, 2009). Proponents of global citizenship education contend that study abroad is an opportunity not only for the internationalization of the university campus but for transformation in students’ basic personal values (Lutterman-Aguilar & Gingerich, 2002), growth in intercultural competencies (Koskinen & Tossavainen, 2004; Williams, 2005), and the development of a more nuanced understanding of one’s role in an interconnected global society (Paige, Fry, Stallman, Josic, & Jon, 2009; Reysen & Katzarska-Miller, 2013). Solutions to the world’s most pressing social, ecological, and economic challenges require a citizenry whose identities and sense of moral obligation transcend the geopolitical boundaries of the locality or the nation state (Dolby, 2004) and therefore possess positive attitudes toward mitigating the impacts of emerging socioeconomic processes (e.g., globalization) on cultural others.
Global citizens take responsibility for the impacts of their consumer decisions on the environment and fellow humans, and seek to mitigate injustice through social, economic, and institutional change (Dobson, 2003). Reysen and Katzarska-Miller (2013) describe global citizens as individuals that possess awareness and caring; promote cultural diversity, social justice, and sustainability; and have internalized a responsibility to act. Similarly, Oxfam’s (1997) Curriculum Guide for Global Citizenship proposes that a global citizen is someone who has an awareness of his or her role as a world citizen, respects diversity, understands the complexity of the global socio-political-cultural-environmental system, and participates in the global community seeking to end injustice.
Research in the international education literature has hypothesized that the knowledge and skills required for one to actively and responsibly participate in a global society are developed through experience with, and education on, global social, economic, and environmental issues (Lutterman-Aguilar & Gingerich, 2002). Study abroad can be one way to foster these globally relevant skills and attitudes (de Wit, 2009). Metrics linked to global citizenship are potentially important measures of learning outcomes of study abroad and may have transformative real-world consequences (Tarrant, 2010). Therefore, global citizenship as a topic of international education transcends the academic foci of individual study-abroad programs (e.g., language acquisition, ecology) or the nations and institutions where they originate (Association of American Colleges and Universities, 2007).
Although global citizenship has been a powerful construct to help populate the broad class of learning outcomes potentially associated with participation in study abroad, greater conceptual specificity is needed to link it to meaningful and measurable metrics of instruction and experience (I. Davies & Reid, 2005; L. Davies, 2006). Learning outcomes are highly contextual as different study-abroad programs are designed to convey different topics, and students participating in different programs have different experiences. Consequently, attempts to assess student learning must consider program content and the learning activities that students are engaging in when selecting measures for outcomes assessment.
Value-Belief-Norm (VBN) Model of Global Citizenship
Given that our programs at the University of Georgia have a focus on sustainable development, and that environmentalism is a dominant theme in global citizenship education (Dobson, 2003), we chose pro-environmental attitudes and behaviors as measures of learning outcomes of participation in our programs. Specifically, we drew on the VBN theory of environmentalism (Stern, 2000; Stern, Dietz, Abel, Gugnano, & Kalof, 1999) as a basis to conceptualize our pedagogy. VBN theory is rooted in a cognitive hierarchy of human values (Schwartz, 1992; Stern, Kalof, Dietz, & Guagnano, 1995). The theory contends that altruistic behaviors indicative of global citizenship (voluntary natural resource conservation, active political participation, etc.) are a function of self-transcendent personal values as well as a mediating chain of more specific worldviews, attitudes, and internalized normative beliefs regarding one’s behavior in the social arena (Stern, 2000; Whittaker, Vaske, & Mandfredo, 2006). Values, in this view, are basic cognitions that represent an idealized end state of being (e.g., a world free of war, injustice, and environmental degradation) and serve as a filter for the evaluation of social stimuli and the development of attitudes toward those stimuli (Schwartz, 1992). A self-transcendent value orientation, or a belief in altruism as a desired end state of social interaction, is the psychological foundation for more specific prosocial attitudes, beliefs, and behaviors that serve as instruments for realizing this idealized state (Stern et al., 1999).
In addition to values, the constructs—from the most distal to most proximate causes of pro-environmental behaviors—include environmental worldview, awareness of consequence, ascription of responsibility, and personal norms (Figure 1). These variables represent an acceptance of a worldview that positions humans as a part of nature (environmental worldview), an awareness that one’s actions may have a negative consequence on valued objects (awareness of consequence), a sense of responsibility to take action to mitigate those consequences (ascription of responsibility), and a moral obligation to act in a manner consistent with those values (personal norm). A personal norm is said to be activated (i.e., to influence behavior) when individuals possess a responsibility for their actions and an awareness that their actions influence valued objects (Schwartz, 1977). Last, we consider one’s citizen type (Tarrant et al., 2011; Westheimer & Kahne, 2004) to reflect the manner in which individuals engage in environmentally altruistic behaviors. In our view, self-transcendent values, and the environmental attitudes and behaviors depicted in the VBN model, embody the ideals of global citizenship (Dobson, 2003; Stoner, Tarrant, Perry, Wearing, & Lyons, 2014; Tarrant, 2010). Although our model draws on global citizenship broadly, it is based on tested theory known to explain environmental behaviors (Stern, 2000; van Riper & Kyle, 2014). Our programs are purposefully designed with the VBN model constructs in mind as demonstrable (and achievable) learning outcomes.

Value-belief-norm conceptual model of global citizenship.
Aligning Theory and Pedagogy
We designed instructional practices to influence the causal chain of VBNs and behaviors hypothesized by the model of global citizenship. Students in our programs, for instance, completed a holistic (course-length) digital story assignment that asked them to reflect on the meaning of “progress” and the relationship between progress and sustainability. This assignment, and the associated experiences on which it is based, is designed to challenge student beliefs about the responsibilities that they hold for their actions as members of a global society (e.g., the unaccounted costs of their consumer decisions). Assignments such as these are intended to yield growth in the ascription of responsibility and awareness of consequence constructs in the model. Other assignments and activities that occur throughout the program provide content and perspective for the digital story. These additional assignments are designed to target additional aspects of the chain of mediating variables in the VBN model.
For example, personal normative beliefs are targeted for change when the study-abroad programs carefully build social expectations for students to adhere to sustainable practices (e.g., recycling, water conservation, leave no trace, etc.). Developing a student microculture that sanctions unsustainable behavior helps students to internalize personal environmental norms and potentially results in long-term behavior change (Thøgersen, 2006, 2009). Additionally, students’ values are challenged by critical reflection activities that ask them to consider the idea of “sufficiency” as it relates to their patterns of work, leisure, and consumption. Exercises and discussions ask them to consider what they hold as important, how they anticipate realizing their goals, and the potentially negative social and environmental impacts associated with their desired ways of living.
Growth in model constructs reflects specific learning outcomes that have real-world consequences. For instance, the ultimate dependent variable in our model is an intention to engage in pro-environmental behavior. In the social psychology literature, intentions are considered the direct antecedents to behavior (Ajzen, 1991). Accordingly, higher behavioral intentions to engage in environmentally responsible behaviors may lead to a reduction in resources use and the negative externalities (e.g., climate change, biodiversity loss, deforestation) that stem from it. We argue that the environment, and sustainability moreover, is a critical component of global citizenship education, as many of the world’s most pressing environmental issues are driven by individual consumer and political decisions. It is the intent of our programs that when students graduate from the university and make social, political, and economic decisions, they will do so with a greater sense of civic responsibility and stronger intentions to act in a socially and ecologically conscious manner. Our model of global citizenship, therefore, reflects this instructional intent and these specific learning outcomes.
Assessing Student Learning Outcomes
In the section to follow, we present key findings from our efforts to assess student learning outcomes stemming from program philosophy and student participation. Our efforts to assess learning outcomes have fallen under two main phases of research. First, we endeavored to confirm the hypothesized associations among VBN model constructs with real data and determine if we were able to generate growth in model constructs through instruction. To do this, we administered a survey querying items measuring model constructs on the 1st day of our programs (pretest) and again with a posttest on the last day. We hypothesized that students would report higher mean values on model constructs at posttest versus pretest and that our hypothesized model would be a fit for the data. We found that this was in fact the case. Students participating in programs in 2008, 2009, and 2010 (N = 674) demonstrated statistically significant growth in several model constructs (Table 1). Specifically, behavioral intentions, personal norms, ascription of responsibility, environmental worldview, and biospheric values were higher at the end of the program than at the beginning. This pattern of increments across these principled variables provides support for the proposition that our programs have been successful in generating the learning outcomes that they were designed to generate.
Paired-Sample t Test Value-Belief-Norm (VBN) Model of Global Citizenship Variables at Pretest And Posttest
p < .10. **p < .05. ***p < .001.
Using structural equation modeling techniques, we were also able to demonstrate that the hypothesized VBN model was an adequate fit for the data, χ2 = 440.64, df = 30, p < .00 (root mean square error of approximation = 0.09; non-normed fit index = 0.94; comparative fit index = 0.97). Moreover, the strength of some model relationships and amount of variance explained in behavioral intentions (the final outcome) increased from pretest to posttest (Wynveen, Kyle, & Tarrant, 2011). These results were encouraging given the stated purpose of the study-abroad programs. The theory-based instructional model in fact transformed the very attitudes and behaviors that we associated with global citizenship.
The theoretical model underpinning the instructional framework provides a benchmark to understand instructional efficacy. Knowing that we failed to change student attitudes in a dimension that our instruction was designed to allows us to make specific pedagogical improvements to correct the issue. Our empirical results, for instance, demonstrated inconsistent findings with respect to the awareness of consequence construct. Therefore, we have begun revising instructional practices to emphasize this dimension of the model. Different study-abroad programs may find that a different theoretical model better suits their intended outcomes and academic content. However, we argue that having one will naturally lead to transformative and accountable teaching and learning.
Quasiexperimental Design for Program Evaluation
Although the Phase 1 empirical work in model development did include pretest and posttest data collection, it did not require comparison group testing. In order to demonstrate that growth in student learning outcomes is attributable to program participation, we needed to demonstrate greater increments in pre- to postparticipation outcomes for students in our programs, compared to learning stemming from other forms of instruction. Beginning in 2011, and continuing through the present, we administered surveys to other study-abroad programs as well as on-campus (non-study-abroad) courses to serve as comparison groups. True experiments (demanding random selection of respondents from the population into T x groups) are all but impossible in situations like study abroad, where students self-select participation. Thus, a limitation exists in interpreting results, as potential bias is endogenous in responses to measures of student learning (Shadish, Cook & Campbell, 2002). However, the results obtained from quasiexperiments can offer nuanced understanding of student learning.
We hypothesized that students completing our programs, relative to students in courses on campus or students participating in study-abroad programs with a different academic focus, would demonstrate greater growth in pro-environmental behavioral intentions, the ultimate outcome of interest in our model. A repeated-measures analysis of variance with time as the repeated measure was used. Pro-environmental behavioral intentions (Stern, 2000) were measured for five consecutive years (2011 to 2015) at pretest and posttest for students participating in our programs (n = 2,394), other study-abroad programs (n = 1,004), and courses on campus (n = 1,664). A variable, format, was generated to represent the three formats of instruction that students received (our programs, other study-abroad programs, and on campus), and the variable time represents measured values at pretest and posttest. Where a significant Format × Time interaction was observed, the simple effects of time on each level of format were determined via post hoc pairwise mean contrasts.
The two-way interaction of Format × Time (Table 2) indicates that growth in behavioral intentions (from pretest to posttest) is different across the three formats of instruction (F = 59.29, p < .00, df = 2; ηp2 = .04). Simple post hoc contrasts (pairwise Bonferroni-corrected mean comparisons) indicate that growth from pretest to posttest in pro-environmental behavioral intentions was highly significant for students in our programs (F = 356.74, p < .00, df = 1; ηp2 = .12). Either mean pretest-to-posttest increments in pro-environmental behavioral intentions were not statistically significant, or mean differences were very small for students in other study-abroad programs and courses on campus (Table 2). These disordinal effects are depicted graphically in Figure 2. Students in other study-abroad programs and courses on campus show a relatively flat line, indicating little or no difference in pro-environmental behavioral intentions from pretest to posttest, whereas students in our programs, as we hypothesized, show significant growth. These findings corroborate the simple pretest-posttest findings demonstrated earlier and lend additional empirical evidence to suggest that our theory-based pedagogy is able to influence student attitudes and behaviors.
Repeated-Measures ANOVA Results for Pro-Environmental Behavioral Intentions

Graph of Format × Time interaction on pro-environmental behavioral intentions.
Discussion
The results of our efforts to conceptualize and assess higher-order student learning outcomes have broader implications for research and practice in short-term study abroad. First, theory is an important part of instructional design (Hovland, 2010). Understanding what variables to target through instructional practices can yield the types of values transformation that scholars of global citizenship suggest are needed for the success of a future global society (Dobson, 2003) but to date have remain underconceptualized (Davies, 2006; Streitwieser & Light, 2010). We have demonstrated that our theory-based model adequately depicted the relationships between the psychological dimensions of students’ pro-environmental behaviors and that the instructional practices that we developed to target those outcomes were effective. The specific learning outcomes and instructional practices adopted by a given study program may vary, but as our results demonstrate, when they are aligned with theory, significant student learning is possible.
This assessment program demonstrates also that program design has an influence on student learning. Global citizenship is too large a concept to apply equally to the learning outcomes associated with participation in study abroad in general (Davies, 2006). Our results, for instance, do not show growth in pro-environmental behavioral intentions for students participating in other study-abroad programs. Although pro-environmental attitudes and behaviors are hypothesized to be outcomes of global citizenship education (Dobson, 2003), students will evince growth in them only if they are receiving instruction and having experiences that are intended to influence them. Conversely, a study-abroad program targeted narrowly on sustainability and global citizenship, like ours, might not move the needle on other skills, such as world language proficiency.
As study-abroad programs adopt assessment measures targeted to their instructional goals, they must also adopt pretest–posttest–comparison group designs if they are to produce compelling evidence of growth in learning outcomes. Quasiexperiments remain rare in the study-abroad literature (Sutton & Rubin, 2004); however, they are necessary to demonstrate the value of study abroad as an opportunity for student learning. Assessing student learning outcomes in a rigorous manner requires the collection of data from a wide swath of the university student body for purposes of comparison with study abroad. Consequently, data collection requires the buy-in and participation of many faculty and university administrators.
Last, a number of limitations and areas for future research are worth noting. Although assessing student attitudes in the short term can shed light on instructional effectiveness, the long-term impacts of participation have not been well explored using quasiexperimental designs. In the context of our programs, for instance, it remains to be known if the gains in pro-environmental behavioral intentions that our students demonstrate have staying power or if they attenuate after the conclusion of the program. Assessing higher-order learning outcomes in a longitudinal design 1, 5, or even 10 years poststudy is an important area for future research in global citizenship education (Davies, 2006). Although the quasiexperimental methodology that we have adopted is an improvement over pretest-posttest-only designs, it is not without criticism. For instance, the courses that we are able to recruit for participation are a convenience sample, not a random sample of university courses on campus or abroad. Additionally, there is a potential for bias in students’ self-selection for participation in study abroad. Students who enroll in study abroad may possess predispositions that influence growth in measures of interest. In a quasiexperimental study of student learning in Brazil, Melguizo and Wainer (2016), for instance, address some of issues of selection bias by employing a propensity score matching estimator to account for differences in pretest scores.
Conclusion
Conceptualizing the relevant metrics, and measuring the learning outcomes of short-term study abroad is an ongoing and important challenge. For study abroad to complete the transition from a marginal experience, reserved for economically elite students, to a mainstream educational experience that all students have the opportunity to partake in requires unequivocal evidence that it is worth the investment. As of right now, our data, and the data of others, support such meaningful value added. However, fostering higher-order learning outcomes in short-term study abroad requires more than a philosophy of “just do it.” Simply traveling overseas and participating in a study-abroad program will not necessarily generate desired learning outcomes. First, those outcomes need to be clearly known and identified—beyond the disciplinary (i.e., knowledge) objectives often listed in the syllabus. Second, the program needs to be driven by a conceptual framework that explains and predicts how those outcomes arise. Ultimately, transforming students into global citizens requires a pedagogy that is attentive to instructional design and rooted in theory.
Footnotes
Authors
ADAM C. LANDON is a postdoctoral researcher in the Warnell School of Forestry and Natural Resources at the University of Georgia. His research is related to the human dimensions of natural resources and student learning outcomes of study abroad.
MICHAEL A. TARRANT is a Josiah Meigs Distinguished Teaching Professor in the Warnell School of Forestry and Natural Resources at the University of Georgia and Director of Global Programs in Sustainability. His research is related to sustainable development and student learning outcomes of study abroad.
DONALD L. RUBIN is professor emeritus in the Departments of Communication Studies and Language and Literacy Education at the University of Georgia. His research is related to student learning outcomes of study abroad.
LEE STONER is an assistant professor in the Department of Exercise and Sport Science at the University of North Carolina Chapel Hill. His research is related to cardiac and public health and student learning outcomes of study abroad.
