Abstract
Improving higher education student learning plays a critical role in advancing human capital. Assessment lies at the heart of such improvement, underlining the need for research and innovation in this field. This article reviews an AERA Open Special Topic published to advance assessment research and hence the success of higher education. It sketches key contexts, surveys selected articles, and stands back to frame future steps ahead.
This Special Topic of AERA Open was produced to advance the growing field of research into the assessment of student learning outcomes. Over the past decade, innovative contributions have been made via international initiatives, like the Organisation for Economic Co-operation and Development’s Assessment of Higher Education Learning Outcomes; national initiatives, like the Valid Assessment in Learning in Undergraduate Education work in the United States coordinated by Association of American Colleges and Universities and like SABER in Colombia; commercial work promulgated by the testing firms, like Education Testing Service and American College Testing; and a host of single and cross-institutional developments. Such large-scale innovation complements substantial work unfolding across thousands of institutions and people to enhance the more everyday forms of assessment. The field has spread globally, spanning all regions, dozens of countries, thousands of institutions, and millions of people (Coates, 2016).
The eight articles compiled in this AERA Open Special Topic contribute a diverse suite of insights and prospects. They flowed from an international call for contributions broadcast through several international platforms and networks. The published peer-reviewed articles were distilled from much a larger set. They have been prepared by authors from many disciplinary and cultural backgrounds.
Two articles focus on the challenges associated with creating nation- or systemwide assessment systems. Martin and colleague present a case study that reflects on development of the field in Australia. It discusses insights from a review of institutional websites and a survey of leaders regarding learning outcomes identified by institutions. It proposes a nationwide system and explores the challenges of implementation. Taking a more micro perspective, Swarat and colleagues describe the process of involving faculty from different disciplines in the identification of learning outcomes and describe tests and instruments to measure these outcomes. They conclude that successful assessment must be relevant to disciplinary nuances.
Another group of articles deals with the development, validation, and application of assessment instruments. Zlatkin-Troitschanskaia and colleagues describe the German KoKoHs initiative—one of the most comprehensive and ambitious projects yet delivered in this field. Barclay McKeown and Ercikan evaluate one of the most widely used measures of student engagement in higher education, the United States National Survey of Student Engagement. They examine whether students’ self-reported learning outcomes can be used to compare effective educational practices across majors and to make high-stakes decisions about the effectiveness of programs. They conclude that it is imperative to consider the multilevel nature of survey data in subsequent interpretations. Finally, Domingue and colleagues test the validity of formative assessment instruments. Based on item response theory analysis, they unpack the validity, differential functioning, and hence relevance of these assessments.
A third group of articles probes links between class-level pedagogical changes and learning outcomes. Wiggins and colleagues designed learning activities in a biology course that used either a constructive or an active framework. They gave students in a single classroom a pre- and a posttest and also used in-class observations to measure student-to-student interactions. They conclude that students in active classrooms demonstrate significantly improved learning outcomes compared to students in constructive classrooms. They argue that even though the learning was subtle, there is promise that seemingly small learning gains might propagate throughout a course or departmental curriculum. Landon and colleagues measured whether study-abroad experiences improved learning for students at a large university. They describe development of a faculty-led, short-term study-abroad program shaped by a model of global citizenship. The evaluation focused on three different modes of delivering content. They gave pre- and posttests, and tracked students across 5 years. The results suggest that students who participate in specially designed programs manifest better knowledge outcomes.
The final article in this AERA Open Special Topic, by Seifert and colleagues, summarizes 10 recommendations and challenges derived from a recent comprehensive synthesis of the college impact literature. Some of the recommendations and challenges noted include the need to provide working definitions of constructs, experiences, environments, outcomes, and student learning outcomes. They also advocate for the use of effect sizes instead of marginal effects in quasiexperimental evaluations as a way to create a common metric of measuring effects that can be compared across studies. The authors provide a number of research design recommendations to the field, with the goal of improving methodological rigor and research impact.
Although this AERA Open Special Topic sought to extend scientific research regarding student learning outcomes assessment rather than to contrive a formulated suite of articles, several interpretative themes emerge from reading the contributions as a whole. First, it can be helpful to understand each study in terms of the level at which it is pitched—individual, institutional, field, or national. The studies also focus on different areas of knowledge—specific disciplines, broader fields, or more generalized content. The studies are nuanced by the ways in which they are financed, for instance, from public sources, by institutions, by foundations, or by governments. The assessments examined have been developed in different ways—by firms, universities, or governments. They vary in scope, concentrating on articulating, building, implementing, analysing, or reporting learning outcomes assessments. Different kinds of governance are manifest for the assessments studied. The researchers themselves vary from being engaged directly as participants or evaluators to being situated externally in a more objectively framed scientific role.
A pulsing thread underpinning each article is the need to sustain and grow research and development in this field. There are evident and enduring reasons for this, such as the value in advancing assessment in the spirit of continuous improvements; strategic institutional rationales for finding innovative ways to assess student learning; enormous value for institutions, faculty, students, and governments in finding cheaper ways to assess learning; quality concerns prompting changes in assessment; the need to produce better data on outcomes to prove economic and social returns from education; and limitations or failure of other initiatives. Fundamentally, what unites the articles is a shared interest in establishing meaningful, valid, and efficient assessments for students, faculty, institutions, and broader communities.
As with any reasonably young field, like this one, there is an urgent need for much more research and development. Wide-ranging studies are required, from analysis of how to work with people with vested interests in entrenched approaches, to boosting the supply of relevant professional capability, to the related spasmodic training of academics, to new forms of governance and leadership, to tackling privatized notions of academic autonomy, to progressing techniques to address inherent security and confidentiality constraints, to understanding the increasing commercial considerations and value of assessment, to tackling prioritization and alignment issues within institutions. Education is complex and benefits from diverse and sustained analysis. This AERA Open Special Topic seeks to make a modest and formative contribution to the field.
Footnotes
Authors
TATIANA MELGUIZO is an associate professor in the University of Southern California Rossier School of Education. She works in the field of economics of higher education.
HAMISH COATES is a professor of higher education at the University of Melbourne. His work focuses on the quality and productivity if tertiary education, institutions, and systems.