What Are College Students Really Learning?

Abstract

Two books, Academically Adrift and Degrees of Inequality, are compared and reviewed as in-roads to understanding the college experience. The books both offer an investigation of the question “what are college students really learning?”

Academically Adrift: Limited Learning on College Campuses By Richard Arum and Josipa Roksa University of Chicago Press, 2011 272 pages

Degrees of Inequality: Culture, Class, and Gender in American Higher Education By Ann L. Mullen Johns Hopkins University Press, 2010 264 pages

Sociologists Richard Arum and Josipa Roksa have taken on the daunting question: What, if anything, do students learn in college? They should be credited with their willingness to ask such a tough question about a sector of the American educational system that promises a lot—consistent with its huge price tag—but is not assessed in any meaningful way. Under the umbrella of the Social Science Research Council, Arum and Roksa have assembled a dataset that includes relevant individual and institutional variables as well as baseline (first semester) and follow-up (fourth semester) college learning assessments. Their analysis of this dataset is the central focus of Academically Adrift: Limited Learning on College Campuses. Their most cited finding—that 45 percent of students “did not demonstrate any significant improvement in learning” during the first two years of college—raises serious questions about the scale of educational expenditure in the United States.

Arum and Roksa fear we’re not creating the engaged citizens upon which democracy depends.

This book has received tremendous media attention, and Arum and Roksa—through a variety of essays in popular publications—have attempted to reach the public directly. While the usual suspects (the Chronicle of Higher Education and Inside Higher Education) highlighted the findings, so too did such outlets as the New York Times, Wall Street Journal, Huffington Post, Salon, and National Public Radio. Louis Menand even ran a long piece in The New Yorker. Many of these reports hail the objectivity of the authors and their willingness to take a clear-eyed look at the industry that butters their bread. Personally, though, the criticisms that I would launch at “my” industry fall much more along the lines of those articulated by Ann Mullen in Degrees of Inequality: Culture, Class, and Gender in American Higher Education.

Mullen’s approach (as opposed to that of Arum and Roksa) facilitates the exploration of differences among institutions of higher education and among the students who attend college—especially with respect to their backgrounds and goals. Her argument and data show the tremendous role that higher education plays in helping to reproduce inequality in the U.S., even while ostensibly offering equal opportunity. Mullen’s deft presentation and analysis of the voices of students from Yale and Southern Connecticut State University demonstrate how institutions can be just two miles but many worlds apart. Indeed, Mullen shows that while students from different ends of the class structure attend college, their goals and experiences vary widely. Colleges may be quite successful in fulfilling the needs and aspirations of their students, but provide varying social and educational experiences. Mullen’s study could very well be seen as a confirming empirical test of economists Samuel Bowles and Herbert Gintis’s correspondence principle: that the social relations of education mirror the social relations of one’s eventual occupation. As a result, schooling helps reproduce class stratification.

But what about Arum and Roksa’s question: How do we study student learning in college? Granted, there are many tough decisions and compromises required in order to reach this goal. There are resource constraints—of time and money—that require the analyst to make sub-optimal choices on many fronts. So, what choices did Arum and Roksa make?

Arum and Roksa’s Determinants of College Learning (DCL) dataset includes data on over 2,300 traditional college-age students from 24 four-year institutions of “varying size, selectivity, and missions.” The authors note that their sample is geographically and institutionally diverse, including liberal arts colleges, large research universities, historically black colleges and universities, and Hispanic-serving institutions.

But most critical to any learning assessment are the tools of evaluation. For the DCL, Arum and Roksa chose to use the Collegiate Learning Assessment (CLA), a tool that includes a “performance task and two analytical writing tasks.” Avoiding multiple-choice questions as well as tools that focus on specific content knowledge, Arum and Roksa claim that the CLA, measured during the first and fourth semesters of college, will reveal student improvement in critical thinking, analytical reasoning, and problem solving. To control for variation among student attributes, Arum and Roksa designed a survey that tapped information about background (race/ethnicity, class, gender, high school type) and college experiences (how students studied, whether they volunteered). In addition, they “adequately accounted for a full set of non-school factors” and conclude that their study “yields quite illuminating results on the nature and character of collegiate experiences and variation in student learning that can significantly increase our understanding of the phenomenon.”

Arum and Roksa suggest four “important lessons” from their research: first, undergraduate learning is given short shrift in the undergraduate experience; second, there are few gains in student performance; third, there is persistent or even growing inequality in individual student learning; and fourth, there is variation between and within institutions in levels of student learning associated with different collegiate experiences. The authors fear that, with such low levels of learning and without adequate reform, we are not creating the engaged citizens on which our democracy depends.

Yale and Southern Connecticut State University are just two miles away from one another, but many worlds apart.

The question for this engaged citizen and sociologist, though, is whether the authors have proven their case.

From my point of view, Arum and Roksa make two fundamental mistakes. The first has to do with reifying the results of the CLA. The authors understand that the CLA measures a particular kind of learning but, very quickly, this understanding morphs into “American higher education is characterized by limited or no learning for a large proportion of students.” Lost in this rhetorical gloss is all the content and other learning that in many fields—especially those that lead most directly to jobs or professional training—are seen as the most critical outcome of higher education. Indeed, if students go to college for vocational preparation and are hired upon graduation, they could very well be considered failures by CLA and Arum and Roksa’s standards. The authors’ second mistake is to ignore the radically stratified nature of America’s higher education system. Mullen’s entire book is devoted to understanding just how different two four-year institutions can be; Arum and Roksa’s book considers each institution’s characteristics simply as additional attributes of each individual respondent, leaving out the social context of individual learning, which is so critical to grasping what higher education yields.

The problem is that there are massive differences among the institutions in Arum and Roksa’s study, many of which are revealed by their analysis, but many more of which are left uncovered. Some of their most interesting findings—having to do, for instance, with particularly poor performance on the CLA by business and education majors—aren’t even relevant to some of their institutions since many colleges (especially the most selective, which come out pretty well on many measures) do not offer these majors. Large differences in class sizes across institutions (which could have been tapped in their survey) are ignored, and we never find out whether class sizes are related, for example, to reading and writing assignments (which, by the way, are gleaned only by student report, not by an examination of syllabi). Institutions vary as well in the depth of their commitment to athletics; oddly enough for an exploration of this nature, Arum and Roksa fail to examine the impact of varsity athletic participation on learning outcomes. In fact, we never understand any of the key institutionally-related variables in their social context. Perhaps the most important factors that produce learning, according to Arum and Roksa, are high faculty expectations, taking classes that require more than 20 pages of writing per semester and 40 pages of reading per week, and studying alone rather than studying with peers. The authors recognize that they are hard-pressed for good, qualitative data, and they use other studies liberally in order to provide a bit of the flavor of daily life at college. Time and again, the authors emphasize the importance of “institutional differences,” yet the focus remains on the individual student as a collection of attributes. We learn very little about how these differences play out for given students in specific college contexts, though. In other words, we need to know about the culture of the institution.

To a certain extent, the authors’ quantitative analysis takes into account the relationships among these independent variables. But, because each respondent was seen as a collection of attributes—including which institution s/he attended—we never understand how the various attributes are related to one another.

Ultimately, the authors had to choose how to interpret their findings. They admit that, due to sample size limitations, they “cannot provide a detailed account of what students at each institution look like and what institutions are doing to facilitate their learning.” So, instead, we get a hodgepodge of findings that are impossible to fully comprehend. In line with “a picture being worth a thousand words,” the authors’ “overview of the conceptual framework” provides a graphic representation of their misguided attempt to understand college learning: they conceive of “college experiences” and “institutions attended” as two separate “factors after college entry.” And the authors seem to understand just how misguided it is since most of their suggestions for reform involve organizations effecting change. Yet how can we craft such strategies in the absence of a holistic understanding of how an institution’s students, structure, and culture intertwine?

Mullen’s analysis shows that we must pursue this path if we are to reach a deep understanding of the challenges and possibilities of student intellectual development. The result of Arum and Roksa’s choices concerning measurement, sample design, and analytic strategy is a book that made a lot of headlines and provided succor to conservative critics of higher education but ultimately fails to explain the how and why of our supposed academic waywardness.

With employers continuing to pay a premium for a college education and the president imploring all young people to attend a postsecondary institution, the demand for and cost of higher education will likely remain high. It seems clear, as Mullen suggests, that college will continue as one piece of a larger class transmission belt. But what actually happens in college—what and how much is learned, the amount and content of social and cultural capital accumulated, the nature of the values imbibed—remains a key question on our research agenda.