Classifying Institutions to Guide Mission-Relevant Performance Assessment

Abstract

The shift from a post-industrial to a knowledge economy has placed higher education at the center of attention among policy makers and the public at large. The increase in attention, along with the increasing diversity in the funding sources of higher education institutions (HEIs) has prompted calls for greater accountability. Efforts to assess the effectiveness of HEIs include a variety of accountability frameworks and review processes. These efforts produce competing pressures for both institutional differentiation and convergence: differentiation to efficiently accommodate broader participation and meet a wide array of societal needs; and convergence to compete for international prestige and recognition as demarcated by research university rankings. This paper examines the extent to which distinguishing between types of HEIs via taxonomies or classification systems has expanded or can contribute to expanding the basis upon which HEI performance is evaluated in order to meet the full range of societal expectations for higher education in the 21st Century.

The call for assessing the performance outcomes of higher education institutions (HEIs) is a relatively recent phenomenon that emerges from a set of interrelated societal trends including globalization, internationalization, consumerism, the rise of the knowledge-based economy, neo-liberalism, and the “new public management.” One impact of these changes is a normative shift from viewing higher education as a public good to a private good, accompanied by a shift in funding and subsidization. Internationally, students are expected to shoulder an increasing share of the cost, given the benefits that accrue to them as a result of obtaining a college degree. Accordingly, national, provincial and state governments have been shifting their allocation schemes away from institutions and toward the student, although the extent of this shift varies according to national and regional history and current context. Nowhere is this more obvious than in the United States, where higher education costs have spiraled upward as state allocations to public universities have contracted and the federal government has come to play an increasing role in funding higher education through grants and subsidized loans awarded directly to students and through research funds awarded to academic staff.

The impacts of these trends on higher education have been cast as both negative and positive. Critics have pointed to the dangers of this transformation, such as the commodification of education and the loss of public interest (Marginson, 2016), as well as increased attention to international research reputation and associated rankings that rely upon limited metrics of questionable validity and which favor some areas of scholarship over others (Hazelkorn, 2015; Shin, Toutkoushian, & Tiechler, 2011; Ioniddis, et al., 2007). On the positive side, economists note that these changes promote attention to efficient use of resources, institutional autonomy, and competition (Bonaccorsi & Daraio, 2007). Others have noted that this more “market-sensitive” approach results in better responsiveness to societal needs (Deiaco, Holmén, & McKelvey, 2009) and that costs are appropriately born by those who benefit most directly (Johnstone, 2004).

Regardless of the benefits or drawbacks of these changes, policy makers and the public are paying increased attention to the effectiveness of higher education as an increasingly larger percentage of the populace will need to participate to acquire the knowledge and skills needed for the majority of jobs and careers in a knowledge-based, global economy (Carnevale & Desrochers, 2001; Florida, 2006) This has resulted in a set of opposing pressures on HEIs that de Sousa Santos (2009) argues arises from a basic value conflict: production of high culture and exemplary scientific and humanistic knowledge for the training of elites versus the production of average cultural standards and instrumental knowledge required for training a qualified labor force to remain globally competitive.

The way in which higher education systems respond to a set of largely common challenges is shaped by their political, geographical, and historical contexts. Generally, however, this response entails two types of institutional diversification: vertical—that is, by prestige, reputation, or status; or horizontal—that is, by differences in missions or functions. The remainder of this paper explores the demands that higher education institutions face internationally to demonstrate effectiveness to various constituencies, the role of institutional classification in distinguishing among the types of institutions according to mission (i.e., horizontal) differentiation, and the extent to which current or emerging classification schemes can provide a basis for developing more nuanced approaches to higher education performance assessment that accommodates varying functions and purposes of higher education.

1 Demands for Higher Education Institution Accountability

Demands have increased globally for HEIs to demonstrate quality while educating more diverse students and contributing to the economic and social development of their regions and nations. These demands reveal the increasing centrality of higher education to the goals of individuals and the state. In most countries these demands are negotiated by the state, but in the US and increasingly in other countries, the diversification of revenue sources has prompted acute interest among a more diverse array of financial and political sponsors with differing interests.

Trow (1996) distinguished between internal and external accountability. This analysis focuses specifically on external accountability, which Trow defines as,

… the obligation of colleges and universities to their supporters, and ultimately to society at large, to provide assurance that they are pursuing their missions faithfully, that they are using their resources honestly and responsibly, and that they are meeting legitimate expectations

pp. 315-16

Trow additionally distinguishes between legal/financial accountability and academic accountability, the latter of which focuses specifically on the use of resources to realize institutional mission, that is, “… to further teaching, learning and public service, and to what effect” (Trow, 1996, p. 316). Thus external academic accountability is closely related to the interests and expectations of those who provide resources for HEIs to pursue their mission.

1.1 HEI Funding Sources and Accountability in the United States

The case of the United States is instructive given the wide variety of stakeholders from which HEIs draw financial support; a situation that is becoming more common in other countries as well. Despite its unique near equal mix of public and private sector institutions, HEI accountability is driven primarily by the federal government. Public sector institutions, which receive substantial general operating funds from state governments, must also respond to state-level accountability regimens. However, the federal government has taken on an increasing role in funding HEIs, both public and private, through its support of students through government-provided financial grants and subsidized loans, as well as its longstanding role as the primary funder of academic research. An analysis by the Pew Charitable Trusts (2015) estimated that in academic year 2013, the total investments of the federal and combined state governments were nearly equal with the federal share slightly over, and the state share slightly below US$75 billion. However, this does not include federal government’s role as direct student loan provider. Although loans are generally paid back with interest the estimated costs to the government are notable, at about US$15 billion a year. The near even distribution cited above also does include large tax credit programs that have been estimated to cost the federal government over $30 billion a year in recent years (Pew Charitable Trusts, 2015).

A large component of Federal accountability revolves around maintaining “Title IV” eligibility, which allows the institution to distribute federal aid (grants and loans) to students. One specific Title IV eligibility requirement is annual reporting to the National Center for Education Statistics an extensive range of data about student enrollments, persistence and graduation rates, staffing, and finance through the Integrated Postsecondary Education Data System (IPEDS). Another component of Title IV eligibility requires institutions to be accredited by a regional, national or specialized accrediting agency that is recognized for that purpose by the U.S. Department of Education.

Accreditation in the United States is the primary vehicle for academic quality assessment. Using a largely qualitative auditing process, accreditation is overseen by agencies that are constituted from within the academy. Because it is a highly contextualized and nuanced process compared to government accountability schemes, accrediting agents (agency staff and site visit teams of academic peers) can consider each institution’s performance in relation to its mission, regional context, and the types of students served by the institution.

Research and development funding in the United States is administered by a variety of governmental and non-governmental agencies, business and industry, non-profit organizations, and the institutions themselves. Figure 1 displays the proportion of funding by source for fiscal year 2016. Given the large proportion that originates federally, it is not surprising that federal rules and regulations dominate accountability for research funds. To fulfill the accountability requirements required for administering federal research funds, HEIs must develop a substantial administrative infrastructure, based on overall federal requirements, as well as specific funding agency requirements and general accounting standards. Although only a small proportion of U.S. degree-granting postsecondary institutions qualify as research universities 1 (334 or just over 7%), over 1,000 U.S. HEIs (more than 20%) have received federal funding over the past 10 years.

Figure 1

Higher education research expenditures (in $billions) by Funding Source, FY 2016

Source: National Science Foundation Higher Education Research and Development Survey, FY 2016, Table 3—total, by source of funds, R&D field, and survey population.

In addition to funding for educational operations and research and development, U.S. HEIs raise significant funds through charitable fundraising. As of Fiscal Year 2015, the total value of endowments of U.S. HEIs was just under $550 billion (U.S. Department of Education, National Center for Education Statistics, 2016). This type of institutional wealth is concentrated among institutions, with 120 accounting for $406 billion (74% of the total). It is also notable that this elite group includes 31 public universities. Indeed, the difference in revenue sources between public and private universities in the United States are not as distinct as one might expect. Although private universities depend more on revenue from tuition fees, a good portion of that source comes from the federal aid and student loan programs that students attending private institution can access. Perhaps most ironically, private-for-profit HEIs obtain the majority of their funds from governmental student aid. In fiscal year 2015, over one half (54.5%) of private-for-profit HEIs obtained at least 70% of their total revenues through federal student aid (Kelchen, 2017).

The purpose of describing the wide array of funding sources of U.S. HEIs and noting the differing distribution of sources among different types of institutions, is to underscore the complexity of accountability. As in most countries, higher education accountability is primarily negotiated by governments, both state and federal. Because U.S. HEIs have considerable autonomy in managing those revenues, significant accountability is required to ensure that these funds are used appropriately and toward the desired outcomes of the sector. Those outcomes include the benefits that accrue to students who attend and graduate, as well as the widely varied purposes of those who fund research and development, including the government, NGOs and non-profit foundations that seek HEI expertise to pursue their varied agendas.

The notable success of U.S. HEIs in meeting such a broad range of local, regional, state, national and international demands for its services has been related to the high level of differentiation across the sector. Currently, over two-thirds (69.7%) of U.S. secondary school completers attend college directly after completing high school (U.S. Department of Labor, Bureau of Labor Statististics, 2016). Accommodating this wide range of ability requires a range of institutional types, which the largely market-driven system provides. In the next section we consider how this differentiation is characterized and consider the alignment between accountability systems and the varying missions of this differentiated system.

2 Institutional Classification and Differentiation

Given their complexity as organizations and the variety of forms HEIs take, classification systems provide a useful platform for distinguishing among types of institutions. McCormick and Borden (2017) note that such classifications provide a shorthand for quickly characterizing HEIs, are useful tools for resource allocation and system differentiation among policy-makers, help students and academic staff understand the types of institutions with which they can engage, and can be used by researchers for sampling and analysis.

Historically, government agencies have established relatively simple typologies, such as the “sector” distinction in the United States, which distinguishes by both control (public, private-non-profit, private-for-profit) and level (confers no higher than a two-year, associate’s degree, or confers a bachelor’s degree of higher). Similarly, Germany distinguishes four types of institutions (universities; technical schools and colleges; colleges of art and music; and universities of applied science) and Australia between two types (universities v. technical and further education, or TAFE). Other typologies have emerged based on government development initiatives, like Project 211 and 985 institutions in China, or historical affiliations, like Ivy League institutions in the United States and the Russell Group in the United Kingdom.

Unlike these various typologies, classification systems “denote a formal system that differentiates institutions according to explicitly stated rules or procedures that typically assign all members of a defined population of institutions to designated categories” (McCormick & Borden, 2017, p. 2). The Carnegie Classification of Institutions of Higher Education (CCIHE), first published in 1973, is perhaps the most well-known classification system. Since its inception, the CCIHE has been employed within state legislation funding schemes, as part of eligibility criteria for membership in various higher education-related academic and professional associations, and even serves as the basis upon which universities are grouped within the most popular college ranking in the United States: The U.S. News & World Report Best Colleges.

The original, and what is now called “basic” Carnegie Classification employs two level of distinctions. It first groups institutions by whether they offer degrees across a comprehensive range of disciplines or only within a singular or very narrow range of disciplines. Special focus institutions are further sub-categorized by their disciplinary focus, for example, free-standing schools of medicine, engineering, culinary arts, divinity, etc. Comprehensive institutions are first categorized by highest degree level at which they confer a threshold number of degrees: associate’s, baccalaureate, master’s or doctoral. The sub-distinctions within each category vary by category. Associate’s colleges are further distinguished according to the population served (traditional, non-traditional or mixed) and the mix of degree programs (career/technical, transfer, or mixed). Baccalaureate institutions are separated into two sub-categories, based on whether they confer degrees primarily in arts and sciences disciplines (arts and science focus) or a broader range of fields, including professional disciplines (diverse fields). Master’s level institutions are distinguished by size, according to the number of degrees conferred annually (small, 50-99; medium 100-199; or large 200+). Doctoral Universities are divided into three groups depending on research activity, as measured through both research expenditures, and the breadth and quantity of doctorates conferred. The distribution of U.S. HEIs by the categories of the most recently released CCIHE can be found through the “Classification Summary Tables” link on the CCIHE Downloads page, http://carnegieclassifications.iu.edu/downloads.php.

In 2005, the CCIHE was expanded to include 5 other classification schemes. These included classifications according to undergraduate instructional program profile, graduate instructional program profile, enrollment profile, undergraduate profile, and size and setting. The added classifications have been used for research and internal benchmarking purposes, but none have entered the vernacular like the original, basic classification.

Several other HEI classification systems have been developed and used in the United States but primarily for the specific needs of a policy agency. For example, the American Association of University Professors (AAUP) and the Southern Regional Education Board (SREB) employ classifications similar in construction to the CCIHE, albeit more simplified. In addition, several researchers, noting limitations of the CCIHE, have proposed other classifications, based on the consumer-market choices (Zemsky, Shaman, & Iannozzi, 1997), perceptions of HEI presidents (Brint, Riddle, & Hanneman, 2006) and clustering methodologies (Ruef & Nag, 2015). These alternative configurations and approaches have not seen widespread use as has the CCIHE.

One of the enduring aspects of the CCIHE is that it is based on data sets collected by two federal agencies: The Integrated Postsecondary Education Data System (IPEDS) collected by National Center for Education Statistics for the U.S. Department of Education; and two surveys of research activity administered by the National Center for Science and Engineering Statistics of the National Science Foundation. As noted earlier, HEIs in the United States must submit IPEDS data in order to maintain Title IV status and thereby distribute federal aid to its students. Similarly, responding to the NSF surveys is a component of eligibility for access to certain federal research funds. As such, these date are comprehensive and have high integrity.

Within the European Union, the U-Map project (http://www.u-map.eu/) was undertaken, in part to establish a more comprehensive data collection on European HEIs. U-Map employs 25 indicators of activity across six dimensions: Teaching and learning profile, student profile, research involvement, investment in knowledge exchange, international orientation and regional engagement. The data used by U-Map are collected directly from institutions, with no stake for its consistency and reliability. Rather than cluster institutions into categories, U-Map is used to generate a profile of each institution, using a circular radar chart as shown in Figure 2. While informative for purposes of differentiating among institutions, the U-Map approach to differentiation does not provide a set of categories that enable the types of distinctions required for policy differentiation. It is possible that such categories could be developed, but the primary point of U-Map is that it compares institutions across several continua, rather than placing them into discrete categories.

Figure 2

U-Map institutional profile chart

Source: (van Vugt, et al., 2010)

Classification systems are subject to criticism for a number of reasons. They rely on the use of available and often proxy, rather than direct measures of institutional characteristics. The more esoteric and important aspects of HEI missions, such as developing student character and intellect, are not readily measurable and therefore subject to possible devaluing. Classifications are not intended to be neutral but rather justified on the basis of intended use. However, once developed, “classifications can be appropriated for purposes unrelated to those that informed their design, leading to sometimes perverse consequences” (McCormick & Borden, 2017, p. 8).

Classifications such as the CCIHE are intended to differentiate among institutions based on mission and function. Teichler (1996) described this type of differentiation as horizontal, as compared to vertical differentiation, which is based on prestige, reputation and status. While this distinction is useful in the abstract, it becomes more difficult to disentangle in reality as exemplified by the CCIHE, which is intended as a horizontal form of differentiation, however, the subgrouping of doctoral universities by level of research activity has become a basis for prestige seeking among institutions that desire to obtain “R1” status, the shorthand term used to refer to the “top” research category.

To the degree that true horizontal differentiation can be achieved, one can potentially differentiate among institutions according to the pursuit of mission objectives that are separate but equal. For example, this occurs within the CCIHE in comparing the top research universities (R1) to the “baccalaureate-arts & science” category, which includes such highly regarded institutions as Williams, Amherst, Bowdoin, and Swarthmore Colleges. Moreover, although they generally have lower prestige, Community Colleges (Associate’s level) in the United States are highly valued for their access and affordability and so are (more or less) evaluated on a different basis than the elite universities and colleges.

We next consider whether it is possible to extend this idea further so as to develop classifications that promote horizontally differentiated distinctions as the basis for performance evaluation that is more mission-appropriate.

3 Assessing the Performance of Differentiated Higher Education Institutions

The increase in accountability demands for the higher education sector is associated with a shift in interest among stakeholders from inputs and programs to outputs and outcomes. Olssen and Peters (2005) note that this shift replaces intellectual inquiry and debate with a “… stress on performativity, as evidenced by the emergence of an emphasis on measured outputs: on strategic planning, performance indicators, quality assurance measures and academic audits” (Olssen & Peters, 2005, p. 313). On a more positive note, Borden, Coates, and Bringle (2018) point to the broader and more diverse range of outcomes that can provide a basis for differentiated accountability. They review these outcome measures as related to research performance, educational affordability and cost, graduate labor market outcomes, student engagement and learning outcomes, and community engagement.

The last of these outcomes, community engagement, serves as the basis for a “voluntary classification” released initially as a companion to the CCIHE in 2006 and, since 2010 renewed on a five-year basis. The Carnegie Classification for Community Engagement (CCCE, https://www.brown.edu/swearer/carnegie), enables HEIs to demonstrate collaborations with their larger communities “for the mutually beneficial exchange of knowledge and resources in a context of partnership and reciprocity” (Borden, Coates, & Bringle, 2018, p. 196). In contrast to the CCIHE, which includes every U.S. degree-granting institution within the IPEDS data collection, institutions voluntarily participate in the CCCE, submitting extensive documentation that is reviewed by a national panel to determine if the institution qualifies for recognition. Relatedly, in contrast to the 4,665 institutions included in the 2015 update of the CCIHE, currently 361 institutions have been recognized with the CCCE.

A similar type of designation is awarded in the United States to recognize HEIs for excellence in student learning outcomes assessment, which “recognizes institutions for their efforts in intentional integration of campus-level learning outcomes assessment” (http://www.learningoutcomesassessment.org/eiadesignation.html). In the two years of its existence, the National Institute for Learning Outcomes Assessment has awarded its top “sustained excellence” designation to five HEIs and its “excellence award” to 10 HEIs including one from outside the United States.

Another example of these voluntary designations is the Quality Assurance QA Commons, which is seeking to develop new forms of quality assurance in the United States that “… respond to the changing landscape of higher and postsecondary education and will serve the needs of learners, employers, and the larger society” (https://theqacommons.org/). Their pilot Essential Employability Qualities (EEQ) attempts to provide evidence to students and prospective employers “that a program of study provides quality preparation for employability” (https://theqacommons.org/the-eeq-pilot/).

Outside the United States, several countries have established research excellence frameworks that parallel the assessment of research performance contained within popular research rankings, but incorporate more substantial measures and documented activity beyond superficial metrics. In addition, the UK’s Teaching Excellence and Student Outcomes Framework (TEF) provides a similar process for institutions to seek recognition for their excellence in undergraduate teaching (https://www.officeforstudents.org.uk/advice-and-guidance/teaching/). Previously, the UK and several other countries, like Australia, focused more directly on employability outcomes of graduates, which has helped create greater transparency for employment outcomes but has also revealed that differences in institutional selectivity (student inputs), and the close relationship between disciplines and employment outcomes have made it difficult to connect these assessments directly to institutional performance.

The voluntary designations thus reviewed provide a basis for recognizing areas of excellence beyond traditional accountability measures. Institutions self-select to participate in these designations as expressions of mission-related quality in engaging with their communities, assessing the quality of student learning outcomes, and educating students who will be well-prepared for jobs and careers in the 21st century knowledge economy. Although most of the examples thus reviewed are managed by non-governmental agencies, their usefulness to broader accountability has been recognized by several U.S. states. Moreover, government sponsored instances, like the UK’s TEF demonstrate that this type of distinction can be incorporated into national accountability. However, these systems do not represent a more comprehensive approach to differentiated accountability as related to encompassing classification schemes.

The CCIHE has been used extensively by state higher education systems in the United States, as a basis for funding schemes that explicitly take into account the differences in function that underlie the CCIHE. Doctoral universities are often provided targeted funds for research investment. Associate’s colleges often receive special funds for the remediation programs and intensive support required for the large proportions of underrepresented and lower income students that they serve. Recent trends in performance funding have both stimulated and obstructed efforts to recognize differentiated missions. On the positive side, several states have incorporated measures specifically related to the enrollment and completion of under-represented populations that are more often served by associate’s colleges as well as the large array of public, Carnegie master’s level institutions that often serve larger proportions of place-bound, low-income and under-represented students.

Although not specifically designed as a basis for differentiated performance assessment the manner in which the CCIHE reflects differences in the types of students served and other goals related to social equity and mobility suggests that, if specifically designed to differentiate as related to higher education performance goals, a taxonomy or classification may contribute to a more nuanced approach to metric-based HEI performance assessment. As noted by Borden and McCormick (2018, in press), differentiation for performance assessment can reveal existing bias in the metrics now relied upon for accountability. However, its use adds complexity to the process by moving away from applying all measures to all institutions, to a more contingent logic that applies various sets of metrics to the various types of institutions in accordance with their mission and purposes. Borden and McCormick also note that the limitations to developing such a nuanced system may be related to the relatively early stage of development for metric-based performance assessment rather than to the possibilities for developing a classification system that differentiates among institutions for the purposes of performance assessment.

4 Developing Mission-Related Performance Measures

Although not specifically designed for the purposes of modern-day higher education institutional performance assessment, the CCIHE provides a basis for considering the types of measures that can be considered for assessment as related to horizontally differentiated missions and functions. As a starting point, Table 1 arrays several prospective performance measures, and the CCIHE types of institutions to which they are most relevant.

Table 1

Performance measures for the various Carnegie classification categories

The performance measures and relevant types of institutions arrayed in Table 1 are not intended to be either comprehensive or perfectly matched, primarily because the Basic Carnegie Classification was not designed specifically for the purposes of differential performance assessment. It is instructive to note, however, that it is not possible to isolate measures for one type of institutions that cannot be considered for other types of institutions. For example, quality of undergraduate learning may be more mission-central to the relatively smaller, teaching-focused Baccalaureate Colleges and field-specific Special Focus institutions, but other types of colleges and universities may claim and provide evidence of such quality. Similarly, some Research Universities, especially those in the public sector, may be competitively measured according to measures of access and affordability. The point of this exercise is to demonstrate that it is possible to differentiate to at least some extent using the Carnegie Basic Classification, the types of performance measures that are expected to be more central to the missions of different types of institutions. To the extent that policy makers and governments can adopt measures that reflect the broadening array of outcomes to which higher education institutions are expected to contribute, then it is possible that a broader array of higher education institutions, and not just “world class research universities” can be more highly valued for their contributions to society.