Sage Journals: Discover world-class research

Abstract

Performance-based funding (PBF) policies with research incentives have grown in popularity over the years despite little understanding regarding whether they actually work. This study leverages a novel national data set to examine the impact of PBF research incentives on the research expenditures and total state appropriations among public 4-year institutions, with a particular focus on minority-serving institutions. We find that PBF research incentives had no impact on the total amount or relative share of total research expenditures or the total amount of state appropriations allocated to treated institutions, regardless of minority-serving institution status. PBF policies that allowed institutions to self-select or opt into including research incentives as part of their PBF formula had a positive impact on the relative share of expenditures allocated to research and total state appropriations.

Keywords

accountability educational policy finance higher education minorities quasi-experimental analysis

Introduction

Public 4-year universities in the United States, which are subsidized by government appropriations, typically have institutional missions centered on a combination of research, teaching, and service (Rhoten & Calhoun, 2011). The research function of higher education is critically important to not only institutional prestige but also economic development (e.g., Eid, 2012; Guisan, 2005; Jongbloed et al., 2008; Volkwein & Sweitzer, 2006). State policy makers look to colleges and universities to foster research activities as a way to improve innovation and economic development within their individual states, and a growing number of states have begun to increase their financial commitment to efforts designed to expand the research capacity of their public colleges and universities (Toutkoushian & Paulsen, 2016).

Performance-based funding (PBF), which has grown in popularity and is currently used by two thirds of states, ties a portion of a public college or university’s level of state appropriations to institutional outcomes (Ortagus et al., 2020). The metrics states use to evaluate institutional performance most often include student outcomes, such as progression toward a degree and degree production, but an increasing share of PBF systems are focusing specifically on a given institution’s research activities (Rosinger et al., 2020). Research metrics for PBF-adopting states have varied over the years but often include institutions’ research expenditures from externally funded grants and broad measures of research and development (R&D) expenditures. Slightly fewer than half (19) of the 41 states that have adopted PBF over time have included research-oriented metrics within their PBF formulas (authors’ calculations). Research incentives in PBF systems encourage institutions to support research activities by tying state appropriations to research-oriented expenditures and outcomes in alignment with the institutional missions of public research universities (Burke, 1998; Miller, 2016; Snyder & Boelscher, 2018). But PBF can increase the tension between policy makers’ desire to hold institutions accountable and the financial realities of already underresourced colleges or universities (Boland, 2020; Hillman & Corral, 2018).

Due to the unequal funding distribution of public higher education, minority-serving institutions (MSIs), in particular, often receive insufficient resources to maximize their research capabilities and are left with limited financial flexibility when compared with predominantly White institutions (Boland & Gasman, 2014; Cunningham et al., 2014). Similarly, less-resourced institutions, such as MSIs, may not receive enough support from the state to substantially alter their expenditures in ways that would boost their research capacity (Leslie et al., 2012; Ryan, 2004). PBF policies typically lead to funding systems in which already-advantaged institution types receive a disproportionate share of funding and underfunded institutions, such as MSIs, are asked to continually do more with less (Hagood, 2019; Hillman & Corral, 2018; Li et al., 2018; Ortagus et al., 2020) and have a higher share of funding at stake in PBF systems (T. Jones et al., 2017). For MSIs, the unique mission of these institution types related to serving targeted student populations is often overlooked in PBF metrics (Gasman et al., 2017).

While a large body of previous research has focused on the intended and unintended consequences of PBF on students’ academic outcomes (Hillman et al., 2014; Ortagus et al., 2020; Umbricht et al., 2017), PBF policies incentivizing research activities have a substantive impact on institutional behavior but have yet to be studied in the academic literature. To examine the impact of PBF with research incentives on the behaviors of public 4-year colleges and universities with a focus on MSIs, this study is guided by the following research questions:

Research Question 1: To what extent do PBF research incentives influence the level of research expenditures at public 4-year institutions?

Research Question 2: To what extent do PBF research incentives influence the total state appropriations received by public 4-year institutions?

Research Question 3: Does the influence of PBF policies with research incentives vary according to an institution’s MSI status?

Literature Review

Research incentives featured less prominently than student-oriented measures of institutional performance in early PBF policies during the 1990s (now known as PBF 1.0), emphasizing states’ strategic investment in degree completion over institutional prestige indicators (Burke 1998; Dougherty & Natow, 2015; Toutkoushian & Danielson, 2002). However, PBF policies have linked a portion of state appropriations to research outcomes for certain institutions since the 1990s. Roughly two thirds of states that operated PBF 1.0 systems adopted a research metric for at least one institution (Burke & Serban, 1998; Dougherty & Natow, 2015), and states have incorporated measures of research activity into more recent PBF policies (known as PBF 2.0) at a similar rate (Rabovsky, 2012). As one example, every university in Florida’s State University System is able to choose whether or not to include research expenditures as one of its 10 performance metrics (Cornelius & Cavanaugh, 2016; Snyder & Fox, 2016). PBF policies in other states, such as Kansas, Maine, and Montana, have tied state appropriations to research activity for a small subset of 4-year institutions within each state (Boelscher & Snyder, 2019).

Research Expenditures at Public 4-Year Institutions

The level of institutional expenditures on research activities plays a pivotal role in the extent to which a college or university is able to increase its ranking or institutional prestige (Morse & Brooks, 2021; Volkwein & Sweitzer, 2006), research productivity (Dundar & Lewis, 1998; Eid, 2012), and other important outcomes related to institutional efficiency and effectiveness (Powell et al., 2012; Robst, 2001). For example, R&D expenditures in higher education in European countries were positively related to innovation (Pegkas et al., 2019). Guisan (2005) pointed out that research expenditures at universities in the United States greatly contribute to regional development and solidify a comparative advantage relative to the majority of European regions and countries.

American colleges and universities spent nearly $84 billion on R&D in Fiscal Year 2019, with more than half of all expenditures coming from federal funds (Gibbons, 2021). Most of this funding goes to support university-based research projects and facilities, but indirect cost recovery from federal grants can be an important source of general operating revenue for the largest research universities (Ledford, 2014). The level of a given institution’s reliance on research expenditures is largely dependent on its available revenue sources and stability in funding (Leslie et al., 2012). Historically, private 4-year universities have higher levels of research expenditures than public colleges and universities (Blasdell et al., 1993). Moreover, private universities have accelerated their spending on research activities and have experienced corresponding advantages in institutional prestige over the years (Lau & Rosen, 2016), including when it comes to recruiting faculty (Alexander, 2001; Rippner & Toutkoushian, 2015).

Due to the ways in which the prestige structure in American higher education emphasizes research resources (Brewer et al., 2004; Stocum, 2013), MSIs can face pressure to demonstrate traditional measures of research resources as institutional quality indicators (W. A. Jones, 2013; O’Meara, 2007). As a growing number of MSIs seek to maximize their prestige (Contreras et al., 2008; Doran, 2015), one approach is to increase their research capacity and potentially obtain a larger share of state funding. However, MSIs have been underfunded in ways that may restrict their ability to expand their research expenditures and build their research capacity (Cunningham et al., 2014; Gasman & Commodore, 2014). Prior research attributes these funding disparities to unequal state funding systems, including PBF (Hillman & Corral, 2018; T. Jones et al., 2017; Li et al., 2018) and insufficient support from federal R&D funds (Boland & Gasman, 2014; Matthews, 2011; National Center for Science and Engineering Statistics, 2021).

Due to the institutional mission of MSIs focused on empowering racially marginalized students before implementing prestige-seeking behaviors, MSI faculty often carry larger teaching and advising loads and receive lower levels of research support when compared with non-MSI faculty (Clark et al., 2016). With an increasing number of MSIs seeking to emphasize institutional prestige and become research-intensive institutions (Contreras et al., 2008; Doran, 2015; O’Meara, 2007), it is critical to understand the potential impact of state-level policies on research expenditures for MSIs. While previous research has centered around the impact of PBF on student success at MSIs (Boland, 2020; Hu, 2019), little is known regarding the extent to which PBF policies with research-oriented metrics may alter institutional expenditure patterns at MSIs or the extent to which these policies impact state appropriations these institutions receive.

Incentivizing Research Activities With Performance-Based Funding

To incentivize research activities, many European countries (e.g., Belgium, Italy, Norway, Sweden) have adopted PBF with varying provisions related specifically to research activities. In general, previous research has found that PBF adoption within European countries is positively related to research productivity for colleges and universities (Aagaard et al., 2015; Cattaneo et al., 2016; Checchi et al., 2019; Sīle & Vanderstraeten, 2018; Vanecek, 2014). However, the impact of PBF on research productivity can vary greatly depending on the academic discipline (Engels et al., 2012) and selectivity of the institution (Abramo et al., 2011), which can exacerbate already-existing inequities among colleges and universities (Mateos-González & Boliver, 2019). Because the funding mechanisms and accountability systems for higher education in European countries are fundamentally different from the ones in the United States (e.g., Cretan & Gherghina, 2015), it is unclear if PBF with research incentives functions similarly to influence institutional behaviors in the U.S. context.

In the United States, state governments currently provide funding for about 5% of all university R&D expenditures (Gibbons, 2021), and this money is generally separate from capital appropriations used to build new facilities. However, interest in tying state funding to research funding and productivity has grown over time. Indiana’s research incentive was the first metric used to measure performance in the state’s PBF program (Umbricht et al., 2017), whereas other PBF states have incorporated research-oriented metrics into existing systems. PBF policies incentivize research activity for universities by tying state funding to R&D metrics relating to the academic, economic, and research-related missions of institutions. While a common measure of research activity is externally generated research funding (e.g., Arizona Joint Legislative Budget Committee, 2012), PBF policies also incentivize nonmonetary research outcomes, such as increasing entrepreneurship and commercialization (Kansas Board of Regents, 2019), technology transfer, and licensing and patents (Board of Governors of the State University System of Florida, 2019; Board of Trustees of State Institutions of Higher Learning, 2013). PBF systems in South Dakota and Tennessee have linked research funding to educational opportunities for students (South Dakota Board of Regents, 2000; Tennessee Higher Education Commission, 2015), while research metrics in Kansas are meant to encourage R&D activity at universities that improve employment outcomes for students and support state economic priorities (Kansas Board of Regents, 2019). The amount of institutional funding tied to research metrics has ranged from roughly 1% of state funding in Mississippi (Board of Trustees of State Institutions of Higher Learning, 2013) to nearly 20% of performance funding in New Mexico (New Mexico Higher Education Department, 2018). Prior studies have examined the relationship between PBF adoption and university research activity and expenditures, despite their common drawback of not actually identifying PBF programs with research incentives.

Early PBF programs in Florida and South Carolina coincided with increases in externally funded research activity (Shin & Milton, 2004). Using spline linear modeling to explain variation in research funding at 4-year colleges and universities from 1997 to 2007, Shin (2010) determined that institutional characteristics, rather than PBF policies, contribute to institutional differences in revenue growth from federal research grants and contracts. Kelchen and Stedrak (2016) reported that PBF adoption was not related to expenditures for research for all 4-year institutions; however, PBF policies were associated with decreases in annual research spending of less than 1% for research universities specifically (Rabovsky, 2012). Spending of gift, grant, and contract revenues by research universities in PBF states was also associated with minimal decreases in state appropriations (Rabovsky, 2012).

State policy makers’ desires to measure efficiency and productivity can create tension with measures of institutional performance given the complexity of varying institutional missions (T. Jones, 2016). On the adoption of any performance-oriented programs, including PBF and other performance budgeting programs, research funding has been found to increase at flagship universities specifically, while nonflagship universities experienced decreases in both research funding and publication activity (Payne & Roberts, 2010). Forty-one states have adopted PBF over time (Ortagus et al., 2020), but 22 of those PBF-adopting states have never included research-oriented metrics in their PBF formula. Previous literature related to PBF adoption and research-related priorities offers mixed findings due in part to the misalignment between the policy lever and outcomes being examined (i.e., the metrics of PBF systems vary across states and may not incentivize research-related outcomes). Although prior work has identified numerous PBF studies with strong difference-in-differences designs to support causal inference (Ortagus et al., 2020), recent developments in econometrics literature have shown that any difference-in-differences study with time-varying policy adoption is subject to a host of methodological threats that were not addressed in previous PBF literature (e.g., Goodman-Bacon, 2021; Sun & Abraham, 2020). In this study, we offer the first evidence to date related to the direct impact of PBF research incentives and the subsequent research activities and level of state appropriations among PBF-adopting institutions.

Theoretical Framework

The theoretical framework of this study is guided by principal agent theory, which suggests that the principal (state government) pays the agent (public college or university) to carry out an objective (Jensen & Meckling, 1976; Spence & Zeckhauser, 1971). In the case of PBF policies that center research-oriented metrics, the objective relates to investing in (and ultimately producing) research in ways that can improve the prestige of the institution and a given state’s economic environment. Importantly, the logic of any principal–agent model rests on the assumption that the outcomes of the agent (e.g., investments in research activities) must be observable and measurable by both the principal and agent.

For external resource providers, such as the state government, the extent to which they can influence a public institution’s behavior depends on whether the resource being provided is deemed critical and not obtained easily by another funding source (Emerson, 1962; Harnisch, 2011). In addition, Rabovsky (2012) has reported that shifts in how states allocate resources will likely lead to the adoption of new strategies by colleges and universities seeking to enhance their performance according to the prescribed funding formula. Under a resource dependence perspective, any public colleges or universities that rely heavily on state appropriations may alter their institutional behaviors in response to changes in their state’s funding criteria, such as the introduction of a PBF metric incentivizing research activities.

PBF policies are typically created to directly tie at least a portion of public institutions’ state funding to their academic outcomes, with a particular focus on the intended outcomes of student retention and degree completion (Ortagus et al., 2020). The logical rationale of principal–agent theory coupled with a resource dependence perspective (e.g., Pfeffer & Salancik, 1978) suggests that public research universities, which rely heavily on state funding allocations, are likely to respond to the implementation of PBF in ways that will increase their ability to garner state funding by leveraging resources to improve their performance on the particular metrics states incentivize. In the case of research-oriented performance metrics, institutions may respond by reorganizing their activities in search of external resources and thereby increasing their research output and expenditures. Prior research, for example, has found that colleges in states with PBF metrics that incentivized science, technology, engineering, and mathematics (STEM) fields led to increases in STEM degree production (Li, 2020).

While PBF with STEM incentives has been effective at increasing degree production in those fields, a large body of research indicates PBF has not consistently led to improvements in overall degree production (Ortagus et al., 2020). Although principal–agent and resource dependence theories offer an explanation of how institutions may respond to state incentives, performance management research has shown that performance reform efforts frequently fail to meet their stated objectives (Radin, 2006; Thompson, 1999). This literature has noted that complex structures, organizations, and politics can limit reform efforts’ ability to improve outcomes, and in practice, can lead to unintended consequences (Radin, 2006). Prior research on PBF has noted that underresourced institutional types, such as MSIs, face capacity and financial constraints when it comes to reorganizing activities to align with PBF incentives (Hillman et al., 2014). The historic underfunding of MSIs may limit their capacity to increase research infrastructure and meet their PBF goals (Boland & Gasman, 2014; Cunningham et al., 2014; Leslie et al., 2012; Ryan, 2004), putting them at a further disadvantage when it comes to state funding linked to R&D activities. In addition, these institutions often receive a larger share of institutional revenues from state appropriations (Kelchen et al., 2020), making them simultaneously more likely to respond to state incentives but also more vulnerable if they are unable to improve on particular metrics.

Prior work has reported that traditional PBF systems disadvantage underresourced institutional types, particularly MSIs, while the same policy can considerably benefit more advantaged institutional types (Hagood, 2019; Hillman & Corral, 2018; T. Jones et al., 2017). The introduction of additional metrics in PBF programs that incentivize research and prestige-seeking behaviors may exacerbate already-existing inequities facing MSIs in PBF-adopting states. Given these dynamics, this study explores what happens after PBF policies with research incentives are introduced, focusing specifically on the institutional responses of public research universities and MSIs within these PBF systems.

Methods

Data and Sample

In this study, we use institution-level data from Integrated Postsecondary Education Data System (IPEDS), and state-level data from the Council of State Governments and the National Association of State Budget Officers. We identified every public university in the United States subject to a PBF program that includes research incentives by systematically analyzing more than 2,000 state budget or policy documents between 1997 and 2020 (Kelchen et al., 2019; Ortagus et al., 2021). These documents include state appropriation bills, state budgets, policy documents, audit reports, financial statements, higher education commission or coordinating board reports, personal communication with higher education policy makers, and other firsthand sources that provide information for the years of operation, amounts of funding at stake, sectors and institutions affected, and performance metrics of PBF policies. In addition, we leveraged the Wayback Machine when older budget or policy documents were no longer available online. These data collection procedures allowed our research team to obtain accurate information pertaining to PBF policy adoption, whether the PBF policy was actually funded, the timing of the PBF policy, and the specific PBF policy metrics. The analysis window for this study is 2002 to 2018, which is when data on research expenditures in IPEDS aligns with the years from the PBF data set.

To select institutions with comparable mission and research capacity, we restricted our sample to public 4-year universities that were classified as doctoral research universities or master’s colleges and universities based on the 2000 Carnegie classification. We included master’s institutions that annually conferred more than 40 master’s degrees because many master’s institutions are eligible to include PBF research incentives in PBF-adopting states (e.g., Tennessee and Florida). We excluded institutions that did not offer any undergraduate programs or closed between 2002 and 2018. To examine the influence of PBF with research incentives on institutional research expenditures, we excluded institutions in states that either adopted or abandoned PBF with research incentives in the first 2 years (2001–2002, 2002–2003) or the past 2 years (2016–2017, 2017–2018) as a way to ensure at least 2 years of pre- and posttreatment observations, respectively (Wooldridge, 2002). The final panel data set consisted of 17 years of observations from 374 public universities (n = 6,333), with data for some years missing for 3% of institutions.

Finally, we created an MSI indicator and created an MSI subgroup (n = 1,882) and a non–minority-serving institution (NMSI) subgroup with 263 institutions (n = 4,450) after identifying an institution’s MSI eligibility prior to 2004 defined by the Office of Postsecondary Education (2021). The 111 MSI institutions consist of 25 historically Black colleges and universities or any colleges or universities eligible to be primarily Black institutions, 49 Hispanic-serving institutions, 65 colleges or universities eligible to be Asian American and Native American Pacific Islander-Serving Institutions or Alaska Native and Native Hawaiian-serving institutions, and seven Native American-Serving Nontribal Institutions.¹ Due to the composition and size of the MSIs included in the current study, we caution the readers to interpret our findings as the average treatment effects on treated MSIs that may not be statistically generalizable to all MSIs in the United States.

Variables

The dependent variables for this study are (1) the amount of research expenditures transformed using a natural logarithm, (2) the relative share of research expenditures relative to total expenditures, (3) the annual change in the total amount of research expenditures, (4) the amount of state appropriations transformed using a natural logarithm, and (5) the annual change of total amount of state appropriations. The treatment variable is a binary indicator of an approved PBF policy that includes research incentives, with the treatment turning on or off at the institution level. By approved PBF policy, we mean a policy through which funds could be allocated based in part on institutional performance existed in state legislation or, if a state higher education agency allocated state dollars to institutions, existed in board documents. The treatment variable is coded as 1 for institutions subject to PBF policies with research incentives, and it is coded as 0 if PBF is not in place or the PBF policy does not have research incentives for the institution. Between 2002 and 2018, 64 institutions across 13 states were subject to approved PBF policies that included research incentives (see Figure 1 and Supplemental Appendix A, available in the online version of this article).

Figure 1.

Performance-based funding (PBF) research incentives in place by state for eligible institutions.

Because the binary treatment variable only captures adoption and potentially masks the complexity of PBF policies, exploratory studies need to identify how PBF policies with varying designs affect financial outcomes of institutions, particularly those that are already underresourced (Bell et al., 2018; Ortagus et al., 2020). Thus, we used variations of the treatment indicator in additional model specifications to capture (1) if the PBF research incentives were actually funded and (2) if PBF-adopting institutions were given the option to include research incentives as part of their PBF formula (i.e., the research-oriented metric was not mandated). Among institution-year observations subject to PBF research incentives, 83.9% of the observations were actually funded and coded as 1 for the first alternative specification. Institutions with no PBF research incentive or a PBF research incentive that was not actually funded were coded as 0. The treatment variable for the second alternative specification is coded in a categorical manner, indicating whether colleges and universities are granted autonomy to choose a research-oriented PBF metric. PBF-adopting institutions that were able to opt in for research metrics (coded as 2 in this additional specification) represent 41.8% of all institution-year observations subject to PBF policies with research incentives. Institutions subject to PBF policies that mandated the use of research incentives were coded as 1. Institutions with no PBF research incentives were coded as 0.

In the outcome model examining research expenditures, we controlled for both state- and institution-level covariates that could affect the level of research expenditures or state appropriations. We included two state-level covariates: state legislative control indicating if the same party held both the legislative chamber and the governorship and the percentage of state annual appropriations allocated to higher education (Kelchen & Stedrak, 2016; McLendon et al., 2009). Institution-level variables included institutional characteristics of enrollment size and MSI status (Cunningham et al., 2014; Rabovsky, 2012), percentage of applicants admitted as a proxy for selectivity (Kim, 2018), revenues (e.g., revenue from tuition and fees per full-time equivalent [FTE] student, revenue from state appropriations per FTE student, revenue from federal, state, local/private contract and grant per FTE student, respectively; Leslie et al., 2012), and instructional cost per FTE as a proxy for competing institutional expenses (McClure & Titus, 2018). To examine the impact of PBF policies with research incentives on total state appropriations, we removed state appropriations as a control variable but instead controlled for the presence of any PBF policy for the 4-year sector (including PBF systems that did not include research incentives; Hagood, 2019). All dollar values were adjusted for inflation and online Supplemental Appendix B lists all variables and their sources used in our analyses.

Empirical Strategy

To estimate the average treatment effect of PBF research incentives on treated institutions’ institutional research expenditures and state appropriations (our first two research questions), we used a generalized difference-in-differences (GDiD) model, which allows the treatment to turn on or off for the individual institution between 2002 and 2018 (Angrist & Pischke, 2009). Specifically, the GDiD estimator ( $δ_{1}$ ) was used to compare the difference in the outcomes between treated and untreated units after the adoption of PBF research incentives and then subtract the difference in outcomes before the adoption of PBF research incentives. Formally, we used ordinary least squares regression in the model, holding covariates constant:

y_{i j} = β_{0} + δ_{1} t r e a t m e n t_{i j} + c_{i} + h_{j} + Z_{i j} + δ_{i j} + ε_{i j}

where y_ij represents the outcome variables at institution i in year j. $β_{0}$ is an institution-specific intercept. Treatment is an indicator of the adoption of PBF research incentives (and its alternative specifications) for institution i in year j. $δ_{1}$ is the coefficient of interest. c_i represents the time-invariant institution-level fixed effect, and h_j represents the year fixed effect. By incorporating institution and year fixed effects, the model controls for potential institution-specific effects over time as well as any time effects that were common across institutions in each year (Allison, 2009). Z_ij is a vector of state- and institution-level covariates described in the previous section. We also included institution-specific linear time trends ( $δ_{i j}$ ) by interacting institution fixed effects with a continuous time trend (Furquim et al., 2020). To correct for heteroscedasticity and serially correlated error terms in panel data (Bertrand et al., 2004), we estimated robust standard errors in each model by clustering at the institution level.

Prior research on PBF policies has shown that PBF-adopting institutions may need to take a year or two to alter their behavior in response to policy changes (e.g., Gándara, 2019; Li & Ortagus, 2019). In practice, changes in research expenditures in particular can be lagged due to the application and award process of major grants (e.g., at least 10 months for National Science Foundation grants). To account for potential delays in institutions’ response to PBF research incentives, we estimated additional specifications modeling a 1-year lag and a 2-year lag to capture both the lagged effect under treatment and the persistent effect despite treatment suspension. Additionally, to examine whether institutional responses to PBF research incentives or level of state appropriations institutions receive differ based on an institution’s MSI status (our third research question), we estimated the equation identified above for subsamples comprised solely of MSIs and NMSIs, respectively.

To better account for the impact of PBF research incentives on institutional research expenditures and state appropriations, we selected multiple comparison groups of untreated institutions to construct counterfactual situations of institutional responses in the absence of PBF research incentives (Meyer, 1995). The first comparison group was restricted to 198 public 4-year universities that were not subject to PBF research incentives in adjacent or neighboring untreated states (Cook et al., 2008). The second comparison group was a national sample of 294 public 4-year universities that were not subject to PBF research incentives in all untreated states.

For the third comparison group, we accounted for differences across institutions by using inverse propensity score weighting (IPSW). The logic of IPSW requires researchers to leverage pretreatment covariates in order to improve balance and reduce bias by creating statistically comparable institutions in the comparison group (Guo & Fraser, 2015; Ho et al., 2007). Specifically, we estimated a logit model predicting a college’s probability of being subject to PBF research incentives conditional on all institution-level covariates and other characteristics related to the institution’s likelihood of being treated (i.e., location, affiliated hospital, medical degree conferring status; Birdsall, 2019; McClure & Titus, 2018). After removing seven institutions with extreme propensity scores (Austin & Stuart, 2015), we used the inverse of the propensity score to weight each institution based on its likelihood of adopting PBF research incentives then applied to all descriptive and regression analyses to estimate the average treatment effect of PBF research incentives on treated institutions for the third comparison group. Table 1 provides a descriptive summary of the variables for the treatment and comparison groups.

Table 1

Descriptive Summary of the Variables

Covariates	Treated	Neighboring group	National group	IPSW treated	IPSW comparison
State-level covariates
State control
Republican, %	52.67	40.99	29.89	52.85	33.40
Democratic, %	6.43	21.16	24.65	5.88	23.23
Divided/others, %	40.90	37.85	45.46	41.27	43.37
Percentage of state annual appropriation to HE	0.107 (0.001)	0.127 (0.001)	0.114 (0.001)	0.106 (0.001)	0.117 (0.001)
PBF in place, %	58.36	27.10	25.04	59.01	24.80
Institution-level covariates
Institutional size, %
Under 1,000	0.00	0.00	0.08	0.00	0.00
1,000–4,999	2.21	10.69	11.18	0.66	2.27
5,000–9,999	20.04	24.93	29.36	20.68	20.91
10,000–19,999	32.08	29.49	30.03	33.11	30.25
Above 20,000	45.68	34.90	29.34	45.54	46.57
Location, %
City	65.99	57.97	52.25	66.51	63.38
Suburb	12.68	13.34	19.78	13.09	15.59
Town	18.01	25.28	24.63	16.98	18.52
Rural	3.31	3.40	3.34	3.42%	2.51%
Institution has hospital, %	5.97	8.69	8.29	6.17	9.29
Institution grants a medical degree, %	33.00	18.39	16.97	32.45	30.25
MSI, %	13.05	27.13	24.69	12.52	9.90
Percentage of applicants admitted	0.733 (0.005)	0.713 (0.003)	0.688 (0.003)	0.741 (0.005)	0.704 (0.005)
Tuition and fee revenue per FTE	9158.43 (152.69)	8685.90 (78.88)	8784.25 (65.90)	9208.88 (156.74)	10368.70 (144.35)
Federal contract and grant revenue per FTE	5635.88 (253.89)	4046.79 (109.69)	3809.26 (86.39)	4832.71 (170.16)	6106.85 (312.93)
State contract and grant revenue per FTE	752.40 (27.53)	837.27 (20.49)	1001.63 (20.14)	687.55 (23.53)	964.64 (25.40)
Local/private contract and grant revenue per FTE	1904.58 (101.84)	1012.56 (34.11)	1069.83 (40.37)	1486.44 (67.25)	1574.47 (66.72)
State appropriations per FTE	10161.80 (164.28)	9157.31 (88.07)	9708.35 (121.70)	9643.04 (134.53)	10201.81 (242.69)
Instructional cost per FTE	10445.61 (166.22)	10083.77 (112.27)	10329.40 (108.03)	10271.06 (166.51)	11516.24 (263.24)
Number of institutions	64	198	294	64	290
Number of observations	1,073	3,309	4,772	1,039	4,870

Note. Standard error in parentheses. PBF = performance-based funding; IPSW = inverse propensity score weighting; HE = higher education; MSI = minority=serving institution; FTE = full-time equivalent.

We applied the same procedure to the MSI and NMSI subsamples to create multiple comparison groups. Table 2 and Figure 2 present the balance for the unweighted and weighted groups, indicating that we met the common support assumption for the full sample and NMSI subsample. However, due to the small number of MSIs in the base year (n = 110), we did not meet the common support assumption using the IPSW approach. To create a comparison group of MSIs for the treated MSIs with PBF research incentives, we also used a coarsened exact matching (CEM) approach (see Hillman et al. [2014] and Hu et al. [2020] for examples of prior quasi-experimental work employing a CEM approach). Different from IPSW, CEM matches institutions based on select characteristics to improve balance for each variable in isolation rather than using one propensity score based on a set of covariates. In other words, CEM allows comparisons between treated and untreated observations for each variable separately without reducing balance in other covariates, and this approach is particularly appropriate given the small sample size of MSIs (King & Nielsen, 2019; Wells et al., 2013). Table 3 shows improved covariate balance after implementing the CEM procedure (Blackwell et al., 2009), including covariates based on its associated p value recommended by Rosenbaum (2002) and Rosenbaum and Rubin (1984).

Table 2

Standardized Differences of the Unweighted and Weighted Sample

Variable	Full sample		NMSI sample
Variable	Preweighting	Postweighting	Preweighting	Postweighting
Institutional size	0.486	−0.034	0.430	−0.105
Location	−0.176	0.035	−0.193	0.025
Institution has hospital	0.005	−0.025	−0.060	−0.097
Institution grants a medical degree	−0.264	0.078	−0.176	0.041
Minority-serving institution	−0.408	0.068	—
Percentage of applicants admitted	0.076	0.018	−0.016	−0.040
Tuition and fee revenue per FTE	0.322	−0.051	0.115	−0.124
Federal contract and grant revenue per FTE	0.301	0.138	0.367	−0.015
State contract and grant revenue per FTE	−0.052	−0.073	0.014	−0.014
Local/private contract and grant revenue per FTE	0.412	−0.007	0.434	−0.016
State appropriations per FTE	0.156	−0.063	0.255	−0.057
Instructional cost per FTE	0.245	−0.138	0.240	−0.131

Note. NMSI = non–minority-serving institutions; FTE = full-time equivalent.

Figure 2.

Estimated propensity scores pre- and postweighting.

Table 3

Imbalance Measurement of the Continuous Variables Before and After CEM Procedure for MSIs

Variable	L1	Mean	Min	25%	50%	75%	Max
Pre-CEM
Tuition and fees revenue per FTE	0.229	2516.9	1835.3	728.2	−371.6	2763.0	8068.4
Federal contract and grant revenue per FTE	0.529	4880.9	2126.1	1216.8	4899.3	5952.8	7752.7
Post-CEM
Tuition and fees revenue per FTE	0.128	836.4	−460.9	−586.6	−235.9	−522.2	8068.4
Federal contract and grant revenue per FTE	0.156	−2232.1	13.9	−1660.1	−4817.6	−7190.2	7752.7

Note. MSI = minority-serving institutions; CEM = coarsened exact matching; FTE = full-time equivalent.

Robustness Checks

We used several approaches to check the robustness of our analyses. First, we included multiple comparison groups with multiple pre- and posttreatment periods (typically examining a 1- or 2-year lead and lag) to examine if the results are consistent (Furquim et al., 2020; Meyer, 1995). The lead model specifications serve as a falsification test that the treatment was largely not statistically significantly associated with the outcomes prior to the adoption of PBF research metrics for each sample. We also ran the lag model specifications for institutions that had only one adoption which stayed in effect in 2020 to address the potential difference between the lagged effect under treatment from the persistent effect despite treatment suspension. The consistent pattern across model specifications suggested that the average treatment effects of PBF research incentives on treated institutions were robust regardless of comparison samples. Additionally, we ran alternative model specifications without the institution-specific linear time trends due to the risk of overcontrolling for unit-specific trends, which can greatly reduce the power needed to detect statistical significance (Furquim et al., 2020). The results of model specifications without the institution-specific linear time trends are largely consistent with the findings of our preferred models with IPSW or CEM adjustments.

We tested the treatment effect on the treated by capturing whether the PBF research incentive was approved, whether the PBF research incentive was funded, and whether institutions have the option to opt in to including the research incentives as part of their PBF system. In additional model specifications, we controlled for the proportion of PBF funding relative to total state appropriations as a continuous variable. Specifically, nearly three quarters of treated institution-year observations (n = 799) have less than 5% of state appropriations tied to performance metrics, with the majority of treated institutions with a higher PBF dosage clustered in the state of Tennessee (n = 153). The results controlling for PBF dosage are highly consistent with the results in our preferred model specifications with IPSW or CEM adjustments.

Limitations

This study was subject to several limitations. First, according to Jaquette and Parra (2016), IPEDS finance data have inconsistencies for institutions in a parent–child reporting relationship. We excluded institutions as child records, which represents less than 1% of the sample, because their finance data were reported with the parent institutions. In all analyses, we used data disaggregated by the U.S. Department of Education’s Office of Postsecondary Education ID. We also ran model specifications including institutions in a parent–child relationship. Across alternative outcome model specifications, our findings remain highly consistent, indicating that the point estimates were not sensitive to the parent–child data limitation. IPEDS data have other inherited limitations as the survey items have changed over time, and we chose to restrict our panel data starting from 2002 to balance data consistency and sample size (Aliyeva et al., 2018; Delta Cost Project, 2011; Jaquette & Parra, 2016).

In addition, treatment with time-varying adoption between 2002 and 2018 can bias the estimator and the inclusion of institution-specific trends may not be sufficient to address potential biases (Goodman-Bacon, 2021). Unfortunately, the Goodman-Bacon decomposition test only applies to staggered treatment adoption designs in which an increasing number of institutions become treated over time. The bacondecomp command is not applicable to treatments that turn on and off with a decreased number of treated cases over time, as is the case with PBF policies with research incentives that are occasionally abandoned and readopted, so we cannot examine which comparisons are weighted the most in the aggregated treatment effect. We followed recommendations by Sun and Abraham (2020) to conduct two-way fixed-effects (TWFE) event study analyses and estimated the average treatment effects of PBF research incentives for institutions that had only one adoption between 2002 and 2018. Coupled with IPSW and CEM models, this approach allows for variation in treatment timing across multiple periods and the parallel trends assumption to hold conditional on covariates. This approach can be extremely flexible when determining control units to consider, and the treatment effect estimates do not suffer from biases associated with TWFE regressions with time-varying treatment adoption. The results for the average treatment effects on treated institutions are largely consistent with the results in our main analyses for the full sample and subgroups (see online Supplemental Appendices C and D).

Finally, the sample size in subgroup analyses is relatively small, which can lead to Type II errors (i.e., when one accepts a null hypothesis that is actually false). For example, in the subgroup of MSIs, given the existing sample size and significance level at 0.05, the statistical power of the treatment coefficient needs to be 1 to reach an effect size of 0.1 in Cohen’s f. Because it is possible that the true effect size of PBF research metrics can be smaller than 0.1 (i.e., the treatment representing less than 0.012% of variances explained, which may not be substantively meaningful from the policy perspective), the current model with existing sample size and power may not detect statistically significant results of the true effect size. Given this limitation, we urge caution in the interpretation of null results. In other words, the small number of MSIs in the sample can lead to estimation bias such that the true effect of the treatment is not detected by the current study. The specific types of MSIs included in the study should not be generalized to all MSIs as a monolith; rather, our findings should represent the average treatment effect on the treated MSIs as indicated in online Supplemental Appendix A.

Results

Descriptive Results

Table 4 presents descriptive differences in the outcome variables between the treated group and comparison groups. Before weighting, treated institutions had larger research expenditures in total ($113.9 million) relative to comparable institutions in neighboring states ($76.6 million) and comparable institutions in the national group ($67 million). Accounting for the likelihood of adopting PBF research incentives, treated institutions had lower research expenditures ($106.4 million) than comparable institutions ($117.7 million) after weighting. The relative share of total institutional expenditures allocated to research was higher for treated institutions when compared with institutions in any comparison group before and after weighting. Similarly, regardless of weighting, the average amount of state appropriations for treated institutions was consistently higher than the average amount for institutions in any of the comparison groups.

Table 4

Descriptive Summary of Dependent Variable

Outcome	Treated	Neighboring group	National group	IPSW treated	IPSW comparison
Total amount of research expenditure (in $1,000)	113,910 (5,217)	76,590 (2,800)	67,012 (2,168)	106,446 (4,985)	117,666 (5,027)
Relative share of research expenditure	0.117 (0.003)	0.070 (0.001)	0.067 (0.001)	0.109 (0.003)	0.097 (0.002)
Total amount of state appropriations (in $1,000)	165,852 (4,571)	118,881 (2,115)	112,539 (1,737)	159,325 (4,186)	154,896 (4,716)

Note. Standard error in parentheses. IPSW = inverse propensity score weighting.

GDiD Results

We report results for the full sample and MSI and NMSI subsamples, controlling for state- and institution-level covariates, institution-level and year fixed effects, and institution-specific linear time trends. Importantly, models using subsamples according to MSI status do not allow for comparisons between MSIs and NMSIs but do allow for an estimation of the treatment effect on treated institutions within the subsample of interest. Table 5 presents the treatment effect of adopted PBF research incentives on research expenditures for the treated institutions, and Table 6 presents the treatment effect of funded PBF research incentives on research expenditures for the treated institutions.

Table 5

Coefficients of PBF Research Incentives on Research Expenditure

Variable	No lag			1-year lag			2-year lag
Variable	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group
Panel A: Full sample
Total amount of research expenditure (logged)	−0.004 (0.042)	−0.002 (0.042)	−0.010 (0.038)	0.023 (0.040)	0.019 (0.039)	0.018 (0.035)	0.057 (0.034)	0.054 (0.034)	0.058 (0.033)
Relative share of research expenditure	−0.003 (0.003)	−0.002 (0.003)	0.000 (0.002)	−0.001 (0.002)	−0.001 (0.003)	0.001 (0.002)	−0.001 (0.002)	0.000 (0.002)	0.002 (0.002)
Annual difference of research expenditure (in $1,000)	348.631 (1682.140)	384.015 (1724.917)	508.149 (1742.511)	3387.422* (1514.915)	3315.645* (1554.360)	3109.669* (1576.600)	2,388.217 (1501.066)	2,535.716 (1580.339)	2,078.260 (1511.245)
Panel B: MSI subsample
Total amount of research expenditure (logged)	0.111	0.148	0.140 (0.105)	0.115 (0.071)	0.122 (0.072)	0.098 (0.075)	0.064 (0.075)	0.057 (0.078)	−0.052 (0.059)
Total amount of research expenditure (logged)	(0.086)	(0.085)	0.140 (0.105)	0.115 (0.071)	0.122 (0.072)	0.098 (0.075)	0.064 (0.075)	0.057 (0.078)	−0.052 (0.059)
Relative share of research expenditure	0.001 (0.006)	0.001 (0.006)	0.002 (0.006)	−0.003 (0.007)	−0.002 (0.007)	0.000 (0.007)	−0.007 (0.006)	−0.006 (0.006)	−0.005 (0.006)
Annual difference of research expenditure (in $1,000)	−3533.756 (7892.887)	−2836.813 (7477.328)	47.247 (7898.544)	9306.756 (7974.897)	10193.269 (7403.404)	11909.024 (7727.110)	9871.209 (9771.574)	9456.546 (9591.058)	8972.271 (13186.013)
Panel C: NMSI subsample
Total amount of research expenditure (logged)	−0.025 (0.046)	−0.026 (0.045)	−0.029 (0.043)	0.011 (0.044)	0.009 (0.043)	0.011 (0.040)	0.055 (0.037)	0.062 (0.037)	0.066 (0.038)
Relative share of research expenditure	−0.004 (0.003)	−0.003 (0.003)	−0.003 (0.003)	−0.002 (0.003)	−0.001 (0.003)	−0.001 (0.003)	0.000 (0.002)	0.001 (0.003)	0.001 (0.003)
Annual difference of research expenditure (in $1,000)	475.105 (1640.116)	866.082 (1694.361)	699.961 (1589.152)	1918.293 (1227.594)	2002.036 (1267.870)	1628.478 (1250.429)	1351.984 (1012.439)	1608.819 (1011.008)	889.489 (1066.047)

Note. Standard error in parentheses. All model specifications controlled for institution and year fixed effects, and included institution-specific linear time trends. PBF = performance-based funding; IPSW = inverse propensity score weighting; CEM = coarsened exact matching; MSI = minority-serving institutions; NMSI = non–minority-serving institutions.

p < .05.

Table 6

Coefficients of PBF Research Incentives (Funded) on Research Expenditure

Variable	No lag			1-year lag			2-year lag
Variable	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group
Panel A: Full sample
Total amount of research expenditure (logged)	−0.014 (0.038)	−0.013 (0.038)	−0.020 (0.034)	0.023 (0.040)	0.022 (0.039)	0.019 (0.035)	0.031 (0.032)	0.029 (0.032)	0.037 (0.032)
Relative share of research expenditure	−0.002 (0.003)	−0.002 (0.003)	0.001 (0.002)	−0.001 (0.002)	0.000 (0.002)	0.002 (0.002)	0.000 (0.002)	0.000 (0.002)	0.002 (0.001)
Annual difference of research expenditure (in $1,000)	−236.767 (1494.992)	−231.584 (1535.297)	−242.394 (1557.410)	3115.971* (1372.532)	2991.765* (1401.875)	2891.533 (1482.476)	1796.465 (1365.565)	1943.717 (1419.020)	1197.222 (1409.948)
Panel B: MSI Subsample
Total amount of research expenditure (logged)	0.064 (0.058)	0.097 (0.063)	0.083 (0.057)	0.197 (0.114)	0.202 (0.102)	0.190 (0.122)	−0.027 (0.125)	−0.034 (0.134)	−0.123 (0.116)
Relative share of research expenditure	0.004 (0.004)	0.004 (0.004)	0.006 (0.004)	0.000 (0.005)	0.001 (0.005)	0.004 (0.005)	−0.003 (0.004)	−0.003 (0.004)	−0.002 (0.004)
Annual difference of research expenditure (in $1,000)	−3203.189 (7171.551)	−2529.864 (6847.081)	635.659 (6933.784)	8131.562 (7467.571)	8807.782 (7105.513)	10229.666 (7508.397)	7993.961 (9170.638)	7763.277 (9015.887)	6671.435 (12590.480)
Panel C: NMSI Subsample
Total amount of research expenditure (logged)	−0.031 (0.042)	−0.031 (0.042)	−0.033 (0.040)	−0.002 (0.040)	−0.001 (0.039)	−0.001 (0.037)	0.041 (0.032)	0.047 (0.032)	0.051 (0.033)
Relative share of research expenditure	−0.003 (0.003)	−0.003 (0.003)	−0.002 (0.003)	−0.001 (0.002)	−0.001 (0.002)	0.000 (0.002)	0.000 (0.002)	0.001 (0.002)	0.001 (0.002)
Annual difference of research expenditure (in $1,000)	−278.388 (1449.003)	−1.789 (1497.544)	−64.548 (1439.126)	2342.887* (1136.710)	2247.462 (1175.977)	1725.866 (1150.020)	1059.395 (980.976)	1293.695 (951.303)	643.556 (1034.622)

p < .05.

Consistently, the results indicate that PBF research incentives, funded or not, are largely not significantly related to the total amount of research expenditures, the relative share of total expenditures allocated to research, or the annual change in total research expenditures for treated institutions in the full sample or MSI and NMSI subsamples. The only exception was the significantly positive relationship between treatment and the annual change in research expenditures in the 1-year lag model for the full sample. When compared with institutions in the comparison groups, the adoption of PBF research metrics is associated with a $3.11 million to $ 3.39 million increase in research expenditures, respectively (p < .05), and the funding of PBF research metrics is associated with a $2.99 million and $3.12 million increase in research expenditure, respectively (p < .05). This increase on the funding of PBF metrics seems to be driven by the annual change of research expenditures for NMSIs that treated NMSIs experienced a $2.34 million increase in research expenditure relative to untreated NMSIs in neighboring states in the 1-year lag model (p < .05). The TWFE event study analysis results are largely consistent with the GDiD findings that the adoption of PBF research metrics is not associated with institutions’ research expenditures for the treated institutions (see online Supplemental Appendix C and Figure 3). However, because of the relatively small sample size of treated MSIs, the null findings can be due to potential Type II errors such that the true treatment effect cannot be detected by this exploratory study.

Figure 3.

Event study of PBF research incentives for CEM/IPSW Group.

Additionally, our analysis of the total amount of state appropriations typically indicates that, funded or not, PBF research incentives are not statistically related to the total amount of state appropriations distributed to treated institutions relative to those institutions without PBF research incentives in the full sample or MSI and NMSI subsamples (Tables 7 and 8). Consistently, the TWFE event study results reveal no differences in the total amount of state appropriations between treated and untreated institutions on adopting PBF research metrics (see online Supplemental Appendix D and Figure 3). Only in one model specification, the TWFE event study results provide suggestive evidence that 2 years after the adoption of PBF research metrics, treated institutions experienced a 0.6% increase in the total amount of state appropriations received when compared with untreated institutions in the national sample (p < .05).² This increase appears to be driven by treated NMSIs, which experienced a 0.7% to 0.74% increase in the total amount of state appropriations received when compared with untreated NMSIs in neighboring states and the national sample 2 years after the adoption of PBF research metrics (p < .05). Similarly, though the GDiD results indicate a significant relationship between PBF research metrics and the annual change in state appropriations distributed to treated institutions for several groups, the TWFE event study results are either not statistically significant or indicate a potential violation of the parallel trends assumption. The preexisting differences in pretrends can bias the estimates, and readers should interpret the findings on the relationship between PBF research metrics and the annual change in state appropriations with caution.

Table 7

Coefficients of PBF Research Incentives on State Appropriations

Variable	No lag			1-Year lag			2-Year lag
Variable	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group
Panel A: Full sample
Total amount of state appropriations (logged)	0.038 (0.028)	0.027 (0.025)	0.014 (0.032)	0.047 (0.035)	0.043 (0.030)	0.007 (0.041)	0.062 (0.035)	0.062 (0.032)	0.011 (0.043)
Annual difference of state appropriations (in $1,000)	8694.928*** (2564.434)	8542.313*** (2448.610)	8602.261** (2752.997)	2999.219 (1566.900)	3838.987* (1591.310)	1139.868 (1782.704)	4586.885* (1945.712)	4455.706* (1942.326)	3711.580 (1950.741)
Panel B: MSI Subsample
Total amount of state appropriations (logged)	−0.049 (0.045)	−0.005 (0.033)	0.012 (0.037)	−0.045 (0.042)	0.044 (0.041)	0.022 (0.050)	−0.005 (0.044)	0.095 (0.059)	0.095 (0.070)
Annual difference of state appropriations (in $1,000)	1605.243 (6365.583)	8659.941 (6236.080)	13269.674 (11052.689)	1714.514 (5101.338)	4534.150 (4812.717)	5341.074 (6553.783)	9409.194 (7157.012)	10283.354 (5932.838)	14975.764* (5518.212)
Panel C: NMSI sample
Total amount of state appropriations (logged)	0.050 (0.033)	0.037 (0.029)	0.015 (0.034)	0.048 (0.043)	0.046 (0.037)	0.012 (0.043)	0.050 (0.042)	0.054 (0.038)	0.014 (0.045)
Annual difference of state appropriations (in $1,000)	8217.476** (2994.503)	8262.085** (2814.103)	8817.229** (3315.957)	2359.317 (1643.978)	3461.976* (1692.305)	2960.756 (1829.556)	3766.228 (2018.646)	3483.027 (2024.103)	3462.855 (2179.952)

Note. Standard error in parentheses. All model specifications controlled for general PBF policy in place/funded, institution and year fixed effects, and included institution-specific linear time trends. PBF = performance-based funding; IPSW = inverse propensity score weighting; CEM = coarsened exact matching; MSI = minority-serving institutions; NMSI = non–minority-serving institutions.

p < .05. **p < .01. ***p < .001.

Table 8

Coefficients of PBF Research Incentives (Funded) on State Appropriations

Variable	No lag			1-year lag			2-year lag
Variable	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group
Panel A: Full sample
Total amount of state appropriations (logged)	0.023 (0.021)	0.016 (0.018)	0.003 (0.022)	0.030 (0.025)	0.028 (0.022)	0.000 (0.027)	0.042 (0.025)	0.043 (0.023)	0.005 (0.027)
Annual difference of state appropriations (in $1,000)	6754.376** (2444.690)	6687.383** (2361.043)	6003.813* (2618.211)	3047.848* (1519.560)	3715.526* (1560.200)	1834.168 (1723.155)	3523.522* (1454.282)	3593.177* (1410.967)	2786.631 (1541.677)
Panel B: MSI subsample
Total amount of state appropriations (logged)	−0.035 (0.034)	−0.001 (0.026)	0.009 (0.023)	−0.033 (0.032)	0.035 (0.034)	0.019 (0.035)	0.004 (0.033)	0.075 (0.044)	0.059 (0.045)
Annual difference of state appropriations (in $1,000)	2024.030 (5414.461)	7204.452 (5638.678)	7456.048 (7880.190)	536.177 (4376.614)	2713.043 (4227.893)	985.004 (4593.076)	5590.044 (5374.546)	6401.949 (4337.452)	5462.673 (4586.449)
Panel C: NMSI sample
Total amount of state appropriations (logged)	0.031 (0.025)	0.022 (0.022)	0.003 (0.024)	0.031 (0.031)	0.029 (0.027)	0.003 (0.028)	0.032 (0.030)	0.036 (0.027)	0.007 (0.029)
Annual difference of state appropriations (in $1,000)	6320.034* (2856.786)	6380.861* (2709.815)	6369.653* (3093.607)	2969.788 (1661.538)	3793.626* (1699.828)	3474.008 (1840.379)	3289.939* (1577.767)	3182.077* (1523.927)	3003.366 (1792.809)

p < .05. **p < .01.

Table 9 indicates the treatment effect of PBF policies with research metrics on treated institutions while accounting for specific design features of the PBF system. The results show that the option to self-select PBF research metrics as part of a PBF system is positively related to the relative share of research expenditures as well as total state appropriations for treated institutions. In other words, institutions that were able to opt into research-oriented metrics as part of their PBF system experienced a 0.8% to 0.9% increase in the relative share of research expenditures when compared with untreated institutions in the neighboring states or national samples (p < .05). The adoption of mandatory PBF research incentives, however, is associated with a decrease between 0.4% and 0.8% in treated institutions’ relative share of research expenditures when compared with untreated institutions (p < .05). The GDiD analyses also provided suggestive evidence that PBF-adopting institutions given the option to self-select PBF research incentives are positively associated with the total amount of state appropriations by 8.9% to 11.3% (p < .05) and an annual change of total amount of state appropriations between $6.2 million and $14.7 million (p < .05) relative to institutions without PBF research incentives.

Table 9

Coefficients of PBF Research Incentive Types on Research Expenditure and State Appropriations

Variable		No lag			1-year lag			2-year lag
Variable		Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group	Neighboring group	National group	IPSW/CEM group
Total amount of research expenditure (logged)	Mandatory	−0.053 (0.043)	−0.045 (0.042)	−0.049 (0.039)	−0.033 (0.041)	−0.034 (0.039)	−0.026 (0.034)	0.037 (0.038)	0.032 (0.037)	0.045 (0.034)
Total amount of research expenditure (logged)	Self-selected	0.940 (0.086)	0.082 (0.087)	0.073 (0.117)	0.139 (0.080)	0.129 (0.080)	0.106 (0.102)	0.106 (0.068)	0.107 (0.068)	0.060 (0.079)
Relative share of research expenditure	Mandatory	−0.008* (0.004)	−0.008 (0.004)	−0.005* (0.002)	−0.007* (0.003)	−0.007* (0.003)	−0.004* (0.002)	−0.004 (0.003)	−0.004 (0.003)	−0.001 (0.002)
Relative share of research expenditure	Self-selected	0.008* (0.004)	0.008* (0.004)	0.007 (0.005)	0.009* (0.004)	0.009* (0.004)	0.010 (0.005)	0.009* (0.004)	0.009* (0.004)	0.008 (0.005)
Annual difference of research expenditure (in $1,000)	Mandatory	−2322.934 (1,980.530)	−2289.225 (2,028.267)	−1520.263(1,797.107)	1517.619 (1,293.561)	1300.764 (1,339.329)	1374.516 (1,222.523)	3278.657 (1,909.999)	3164.940 (2,047.113)	3467.552 (1,916.662)
Annual difference of research expenditure (in $1,000)	Self-selected	4685.457 (2,591.615)	4969.240 (2,621.521)	2462.867 (2,103.023)	6155.331 (3,693.966)	6776.057 (3,659.557)	8317.083 (5,280.432)	−251.948 (2,225.881)	651.833 (2,191.931)	−1917.546 (1,986.365)
Total amount of state appropriations (logged)	Mandatory	−0.001 (0.033)	−0.011 (0.029)	−0.042 (0.047)	−0.009 (0.041)	−0.012 (0.036)	−0.033 (0.059)	0.031 (0.040)	0.028 (0.037)	−0.008 (0.058)
Total amount of state appropriations (logged)	Self-selected	0.101** (0.033)	0.089** (0.032)	0.086* (0.040)	0.105** (0.035)	0.108** (0.034)	0.075 (0.041)	0.091** (0.034)	0.103** (0.031)	0.074 (0.039)
Annual difference of state appropriations (in $1,000)	Mandatory	4566.116 (2,857.130)	4317.992 (2,734.155)	5093.328* (2,174.689)	1103.261 (2,163.597)	1801.988 (2,163.910)	2476.314 (1,717.546)	6893.64** (2,427.735)	6305.125* (2,447.477)	4728.046 (2,531.510)
Annual difference of state appropriations (in $1,000)	Self-selected	14703.31*** (3,874.394)	14746.51*** (3,794.192)	13460.86** (4,058.593)	4838.262 (2,859.352)	6187.381* (2,936.067)	2883.140 (3,939.792)	687.732 (3,184.166)	1433.681 (3,157.299)	5846.029 (5,068.950)

p < .05. **p < .01. ***p < .001.

Discussion

This national analysis of the impact of PBF research incentives provides a complex picture of what happens after a PBF-adopting state introduces research-oriented metrics, such as externally funded grants or institutional R&D expenditures. Contrary to previous research (e.g., Rabovsky, 2012), we found that PBF policies with research incentives—regardless of whether they were merely approved or actually funded—were unrelated to the total amount of research expenditures or the relative share of total expenditures allocated to research for treated institutions. This particular finding is typically robust across MSI and NMSI subsamples and event study results in alignment with recent advances in econometrics literature (e.g., Sun & Abraham, 2020).

In addition, we found that PBF research incentives are largely not related to the total amount of state appropriations allocated to treated institutions in the full sample and subsamples of MSIs or NMSIs. Although the GDiD results show some targeted increases in the annual change in research expenditures and total state appropriations for treated institutions within the subgroups of NMSIs, event study results largely indicate that no subsample of interest has statistically significant increases in research expenditures or total state appropriations, indicating that positive effects in our GDiD models and should be interpreted with caution.

One area in which there was a positive relationship between PBF research incentives and institutional research expenditures was when institutions could self-select or opt into including research incentives in their performance agreement with the state. In this case, there was an increase in the relative share of total expenditures that institutions allocated to research. On the contrary, when the research-oriented metric was mandated, institutions under the PBF research metrics policy tended to decrease their relative share of total expenditures that they allocated to research. We also found that institutions that were able to select their own research metrics saw gains in the annual change of the total amount of state appropriations as well as total state appropriations. Universities that were able to self-select or opt into being evaluated based on research performance may have done so either because they wanted to place a higher priority on research than other metrics or they knew that the research performance goals were more easily attainable for their institution than other goals. Universities that were required to use PBF research metrics may choose to prioritize other more attainable performance goals, given that their level of research expenditures can be heavily dependent on their available revenue sources and stability in funding (Leslie et al., 2012).

Although our findings do not align with prior work by Rabovsky (2012), they do align with the overwhelming majority of PBF studies suggesting that PBF policies are typically not related to their outcome of interest, such as research expenditures (Kelchen & Stedrak, 2016) and retention or degree completion (Ortagus et al., 2020). Additional work in an international context has focused to a further extent on research outcomes, suggesting that PBF adoption within European countries has a positive impact on research productivity for PBF-adopting colleges and universities (Aagaard et al., 2015; Cattaneo et al., 2016; Checchi et al., 2019; Sīle & Vanderstraeten, 2018; Vanecek, 2014). However, the funding mechanisms and accountability systems for colleges and universities in European countries are fundamentally different than those in the United States (Cretan & Gherghina, 2015), which may explain the distinctions in the present study focused on the United States and prior studies focused on international contexts.

Implications for Future Research and State Policy

These findings raise questions for future research about the effects of required versus optional metrics in PBF systems, as self-selection may occur in unintended ways when institutions are given the option to choose their own metrics. If institutions select evaluation metrics on which they are likely to excel, they may improve in these particular areas but not necessarily in other areas in which state policy makers intend. For instance, future research might examine whether institutions that select a specific set of metrics see gains in those metrics while not showing similar gains (or even showing losses) in other metrics they did not select. Prior work indicates PBF policies in some states have become something akin to a “choose your own adventure” approach to state funding with institutions selecting one or more metrics on which they will be evaluated (Rosinger et al., 2020). This study demonstrates that such an approach may lead to gains, at least in research expenditures, in the specific metrics selected but also raises questions for future research about whether improvements will happen along other metrics as well.

In the case of research metrics, research universities that are positioned to compete for federal R&D funding may stand to gain the most under PBF systems that allow institutions to select research metrics as a component of their PBF funding formula. While a growing number of universities, including some MSIs (Contreras et al., 2008; Doran, 2015; O’Meara, 2007), are engaging in striving activities that position themselves to win in competitions for research dollars, some institutions are better positioned to leverage their existing financial advantages to garner additional resources in the form of external research support (Cantwell & Taylor, 2015; Rosinger et al., 2016; Slaughter & Rhoades, 2004). Research universities with stronger political representation and external interest groups may advocate for PBF with research metrics in the policy design process (Ness et al., 2015; Tandberg, 2010). Under PBF systems with research incentives, these already-advantaged institutions are also better positioned to see gains in state resources given that a portion of appropriations is linked to research expenditures and institutional efforts to secure external R&D funds. Similarly, those same institutions may see gains in federal R&D funds as a result of their efforts, leaving a select group of well-resourced institutions that stand to potentially gain both state and federal funds under these systems.

Less-resourced institutions might reorganize activities in pursuit of external funding and, in doing so, could see gains in state support, but such efforts are not likely to yield the same returns in the form of external R&D support. After all, competition for external resources creates an environment in which some institutions are positioned to win while others are not, leading to widening inequities in resources across institutions (Cantwell & Taylor, 2015; Slaughter & Rhoades, 2004). In other words, PBF systems with research incentives offer a potential mechanism through which institutions with greater resources and prestige, which often go hand in hand, are able to have built-in advantages to pursue prestige-seeking behaviors such that their relative advantage begets further advantages (Taylor & Cantwell, 2019).

Under this scenario, PBF systems with research incentives as part of their PBF formula may widen existing funding disparities between higher education institutions, leading to yet another potential unintended consequence of PBF policies. Returning to our theoretical framework, our findings are consistent with principal-agent and resource dependence theories—that agents will respond to incentives—but also highlights how the impacts of PBF are mediated by complex structures, organizations, and politics with some institutions better positioned to leverage their greater resources to obtain more resources.

PBF systems with research incentives that institutions can opt into could serve to level the playing field for colleges and universities to be able to compete for both state funds and external research funds if lower-resourced institutions are able to reallocate resources toward prioritizing R&D activity. Since we did not find differential impacts on institutional research expenditures or state appropriations, it is possible that any institution that is able to reallocate funds toward research activities may see gains in state funding under such a policy. If so, PBF systems with research incentives that institutions can opt into could serve as one way to build research capacity of institutions within a state and thereby better position any institution to compete for both state and federal dollars.

However, findings from our study also highlight the limitations of PBF research incentives as an effective strategy for aligning institutional behavior with state workforce and economic development goals. Absent a PBF policy that allows institutions to select the metrics on which they are evaluated, we found no evidence of an impact of PBF research incentives on institutional research expenditures or state appropriations. PBF research incentives without the “choose your own adventure” option may not be an effective lever when it comes to leveraging state and institutional resources to compete for and secure external R&D funds. The limits of PBF in altering institutional behavior to improve research activities are hardly surprising given decades of empirical work demonstrating the limits of PBF policies in the higher education sector when it comes to other incentivized metrics (Ortagus et al., 2020).

Conclusion

A growing number of PBF policies have included research-oriented metrics—such as institutional research expenditures from externally funded grants and broad measures of R&D expenditures—as a way to expand the research capacity of their public colleges and universities. Although such a development can lead to improvements in innovation and economic development throughout the state, it also has the potential to exacerbate financial inequities facing underresourced institutions, such as MSIs, that may lack the financial flexibility required to build research capacity. In this study, we typically found that PBF research metrics were not related to the total amount or relative share of total research expenditures; however, PBF research incentives are positively related to the annual change in research expenditures, which captures the difference in research expenditures from the prior year. Limited and suggestive evidence of positive effects of PBF research incentives on treated institutions’ research expenditures or state appropriations appears to be driven by NMSIs, calling into question the potential for disparate impacts of PBF research incentives according to MSI status. Future research can focus on potential funding inequities among specific types of institutions (e.g., MSIs, less research-intensive, rural) to build on the current study and attempt to provide statistically generalizable findings.

In addition, exploratory analyses revealed that the specific design features of the PBF system may be an important consideration for policy makers seeking to better understand the implications of incorporating research-oriented metrics into their state’s PBF system. We showed that mandating PBF research incentives did not increase research expenditures or state appropriations, but providing the option to self-select into including research-oriented metrics had some targeted positive effects on the annual change and total amount of state appropriations for PBF-adopting institutions. Future PBF should consider the implications—for equity and effectiveness—of specific PBF policy designs to identify what works within a state policy that appears to be a firmly entrenched aspect of higher education finance throughout the United States.

Supplemental Material

sj-docx-1-ero-10.1177_23328584211071109 – Supplemental material for Disparate Impacts of Performance Funding Research Incentives on Research Expenditures and State Appropriations

Supplemental material, sj-docx-1-ero-10.1177_23328584211071109 for Disparate Impacts of Performance Funding Research Incentives on Research Expenditures and State Appropriations by Xiaodan Hu, Justin C. Ortagus, Nicholas Voorhees, Kelly Rosinger and Robert Kelchen in AERA Open

Footnotes

ORCID iDs

Xiaodan Hu

Justin C. Ortagus

Robert Kelchen

Open Practices

The data and analysis files for this article can be found at

Notes

Authors

XIAODAN HU is an assistant professor of higher education and student affairs at Northern Illinois University. Her work has focused on the impact of state and institutional initiatives on postsecondary success of historically marginalized students, with particular attention to the community college sector.

JUSTIN C. ORTAGUS is an associate professor of higher education administration and policy and the director of the Institute of Higher Education at the University of Florida. His research examines how online education, community colleges, and various state policies affect the opportunities and outcomes of underserved students.

NICHOLAS VOORHEES is a doctoral candidate in the Higher Education Administration and Policy program at the University of Florida. His research interests include postbaccalaureate student outcomes and the impact of state policies in higher education.

KELLY ROSINGER is an assistant professor in the Department of Education Policy Studies and a research associate with the Center for the Study of Higher Education at Pennsylvania State University. Her research focuses on the barriers students face going to and through college and the impact of policies and interventions designed to improve college access and success.

ROBERT KELCHEN is a professor of higher education and the head of the Department of Education Leadership, Management and Policy at the University of Tennessee, Knoxville. His research interests include higher education finance, accountability policies and practices, and student financial aid.

References

Aagaard

Bloch

Schneider

J. W.

(2015). Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator. Research Evaluation, 24(2), 106–117. https://doi.org/10.1093/reseval/rvv00

Abramo

Cicero

D’Angelo

C. A.

(2011). The dangers of performance-based research funding in non-competitive higher education systems. Scientometrics, 87(3), 641–654. https://doi.org/10.1007/s11192-011-0355-4

Alexander

F. K.

(2001). The silent crisis: The relative fiscal capacity of public universities to compete for faculty. Review of Higher Education, 24(2), 113–129. https://doi.org/10.1353/rhe.2000.0027

Aliyeva

Cody

C. A.

Low

(2018). The history and origins of survey items for the Integrated Postsecondary Education Data System (2016-17 update). U.S. Department of Education.

Allison

P. D.

(2009). Fixed effects regression models. Sage. https://doi.org/10.4135/9781412993869

Angrist

J. D.

Pischke

(2009). Most harmless econometrics: An empiricist’s companion. Princeton University Press. https://doi.org/10.1515/9781400829828

Arizona Joint Legislative Budget Committee. (2012). FY 2013 appropriations report. https://www.azleg.gov/jlbc/13AR/apprpttoc.pdf

Austin

P. C.

Stuart

E. A.

(2015). Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661–3679. https://doi.org/10.1002/sim.6607

Bell

Fryar

A. H.

Hillman

(2018). When intuition misfires: A meta-analysis of research on performance-based funding in higher education. In Hazelkorn

Coates

McCormick

A. C.

(Eds.), Research handbook on quality, performance and accountability in higher education (pp. 108–124). Edward Elgar.

10.

Bertrand

Duflo

Mullainathan

(2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249–275. https://doi.org/10.1162/003355304772839588

11.

Birdsall

(2019). Policy adoption, innovation, and performance management: The case of performance-funding policies in state postsecondary education. State and Local Government Review, 51(1), 34–45. https://doi.org/10.1177/0160323X19861425

12.

Blackwell

Iacus

King

Porro

(2009). CEM: Coarsened exact matching in Stata. Stata Journal, 9(4), 524–546. https://doi.org/10.1177/1536867X0900900402

13.

Blasdell

S. W.

McPherson

M. S.

Schapiro

M. O.

(1993). Trends in revenues and expenditures in U.S. higher education: Where does the money come from? Where does it go? In McPherson

M. S.

Schapiro

M. O.

Winston

G. C.

(Eds.), Paying the piper: Productivity, incentives, and financing in U.S. higher Education (pp. 15–34). University of Michigan Press.

14.

Board of Governors of the State University System of Florida. (2019). 2019 Performance-based funding model: Final metrics score sheet.

15.

Board of Trustees of State Institutions of Higher Learning. (2013). Performance allocation model.

16.

Boelscher

Snyder

(2019). Driving better outcomes: Fiscal year 2019 state status and typology update. HCM Strategists. https://hcmstrategists.com/resources/driving-better-outcomes-2019/

17.

Boland

W. C.

(2020). Performance funding and historically black colleges and universities: An assessment of financial incentives and baccalaureate degree production. Educational Policy, 34(4), 644–673. https://doi.org/10.1177/0895904818802118

18.

Boland

W. C.

Gasman

(2014). America’s public HBCUs: A four state comparison of institutional capacity and state funding priorities. Penn Center for Minority Serving Institutions. http://repository.upenn.edu/gse_pubs/340

19.

Brewer

D. J.

Gates

S. M.

Goldman

C. A.

(2004). In pursuit of prestige: Strategy and competition in U. S. higher education. RAND. https://www.rand.org/content/dam/rand/pubs/drafts/2005/DRU2541.pdf

20.

Burke

J. C.

(1998). Performance funding indicators: Concerns, values, and models for state colleges and universities. New Directions for Institutional Research, 1998(97), 49–60. https://doi.org/10.1002/ir.9704

21.

Burke

J. C.

Serban

A. M.

(1998). State synopses of performance funding programs. New Directions for Institutional Research, 97, 25–48. https://www.academia.edu/11021818/State_synopses_of_performance_funding_programs

22.

Cantwell

Taylor

B. J.

(2015). Rise of the science and engineering postdoctorate and the restructuring of academic research. Journal of Higher Education, 86(5), 667–696. https://doi.org/10.1353/jhe.2015.0028

23.

Cattaneo

Meoli

Signori

(2016). Performance-based funding and university research productivity: The moderating effect of university legitimacy. Journal of Technology Transfer, 41(1), 85–104. https://doi.org/10.1007/s10961-014-9379-2

24.

Checchi

Malgarini

Sarlo

(2019). Do performance-based research funding systems affect research production and impact? Higher Education Quarterly, 73(1), 45–69. https://doi.org/10.1111/hequ.12185

25.

Clark

Allen

Hodge

S. R.

Murata

(2016). One small step, a giant leap: D-PETE within minority serving institutions. Quest, 68(4), 406–419. https://doi.org/10.1080/00336297.2016.1233121

26.

Contreras

F. E.

Malcom

L. E.

Bensimon

E. M.

(2008). Hispanic-serving institutions: Closeted identity and the production of equitable outcomes for Latino/a students. In Gasman

Baez

Turner

C. S. V.

(Eds.), Understanding minority-serving institutions (pp. 71–90). State University of New York Press.

27.

Cook

T. D.

Shadish

W. R.

Wong

V. C.

(2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750. https://doi.org/10.1002/pam.20375

28.

Cornelius

L. M.

Cavanaugh

T. W.

(2016). Grading the metrics: Performance-based funding in the Florida State University System. Journal of Education Finance, 42(2), 153–187.

29.

Cretan

G. C.

Gherghina

(2015). Funding higher education in a few EU countries: Implications for competition and competitiveness in higher education. Journal of Knowledge Management, Economics and Information Technology, 1(1), 1–22. http://www.scientificpapers.org/wp-content/files/1504_Cretan_Gherghina-Funding_Higher_Education_in_a_few_EU_Countries.pdf

30.

Cunningham

Park

Engle

(2014). Minority-serving institutions: Doing more with less. Institute for Higher Education Policy. http://hdl.handle.net/10919/83120

31.

Delta Cost Project. (2011). Delta Cost Project documentation of IPEDS database and related products. Delta Project on Postsecondary Education Costs, Productivity, and Accountability. https://nces.ed.gov/ipeds/deltacostproject/download/DCP_History_Documentation.pdf

32.

Doran

E. E.

(2015). Negotiating access and tier one aspirations: The historical evolution of a striving Hispanic-serving institution. Journal of Hispanic Higher Education, 14(4), 343–354. https://doi.org/10.1177/1538192715570638

33.

Dougherty

K. J.

Natow

R. S.

(2015). The politics of performance funding for higher education: Origins, discontinuations, and transformations. Johns Hopkins University Press.

34.

Dundar

Lewis

D. R.

(1998). Determinants of research productivity in higher education. Research in Higher Education, 39(6), 607–631. https://doi.org/10.1023/A:1018705823763

35.

Eid

(2012). Higher education R&D and productivity growth: An empirical study on high-income OECD countries. Education Economics, 20(1), 53–68. https://doi.org/10.1080/09645291003726855

36.

Emerson

R. M.

(1962). Power-dependence relations. American Sociological Review, 27(1), 31–41.

37.

Engels

T. C.

Ossenblok

T. L.

Spruyt

E. H.

(2012). Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics, 93(2), 373–390. https://doi.org/10.1007/s11192-012-0680-2

38.

Furquim

Corral

Hillman

(2020). A primer for interpreting and designing difference-in-differences studies in higher education research. In Perna

L. W.

(Ed.), Higher education: Handbook of theory and research (Vol. 35, pp. 1–58). Springer.

39.

Gándara

(2019). Does evidence matter? An analysis of evidence use in performance-funding policy design. Review of Higher Education, 42(3), 991–1022. https://doi.org/10.1353/rhe.2019.0027

40.

Gasman

Commodore

(2014). The state of research on historically Black colleges and universities. Journal for Multicultural Education, 8(2), 89–111. https://doi.org/10.1108/JME-01-2014-0004

41.

Gasman

Nguyen

T. H.

Samayoa

A. C.

Corral

(2017). Minority serving institutions: A data-driven student landscape in the outcomes-based funding universe. Berkeley Review of Education, 7(1), 5–24. https://doi.org/10.5070/B87125036

42.

Gibbons

M. T.

(2021). Universities report 5.7% growth in R&D spending in FY 2019, reaching $84 billion. National Center for Science and Engineering Statistics. https://ncses.nsf.gov/pubs/nsf21313

43.

Goodman-Bacon

(2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014

44.

Guisan

M. C.

(2005). Universities and research expenditure in Europe and the USA, 1993-2003: An analysis of countries and regions. Regional and Sectoral Economic Studies, 5(2), 35–46. https://www.usc.es/economet/reviews/eers523.pdf

45.

Guo

Fraser

(2015). Propensity score analysis. Sage.

46.

Hagood

L. P.

(2019). The financial benefits and burdens of performance funding in higher education. Educational Evaluation and Policy Analysis, 41(2), 189–213. https://doi.org/10.3102/0162373719837318

47.

Halvorsen

Palmquist

(1980). The interpretation of dummy variables in semilogarithmic equations. American Economic Review, 70(3), 474–475. https://fvela.files.wordpress.com/2010/11/dummyinterpretation.pdf

48.

Harnisch

T. L.

(2011). Performance-based funding: A re-emerging strategy in public higher education financing. American Association of State Colleges and Universities.

49.

Hillman

Corral

(2018). The equity implications of paying for performance in higher education. American Behavioral Scientist, 61(14), 1757–1772. https://doi.org/10.1177/0002764217744834

50.

Hillman

N. W.

Tandberg

D. A.

Gross

J. P.

(2014). Performance funding in higher education: Do financial incentives impact college completions? Journal of Higher Education, 85(6), 826–857. https://doi.org/10.1353/jhe.2014.0031

51.

D. E.

Imai

King

Stuart

E. A.

(2007). Matching as nonparametric pre-processing for reducing model dependence in parametric causal inference. Political Analysis, 15(3), 199–236. https://doi.org/10.1093/pan/mpl013

52.

(2019). Efficiency for whom? Varying impact of performance-based funding on community colleges in Louisiana. Community College Review, 47(4), 323–359. https://doi.org/10.1177/0091552119864409

53.

Fernandez

Gandara

(2020). Are donations bigger in Texas? Analyzing the impact of a policy to match donations to Texas’ emerging research universities. American Educational Research Journal, 58(4), 850–882. https://doi.org/10.3102/0002831220968947

54.

Jaquette

Parra

(2016). The problem with the Delta Cost Project database. Research in Higher Education, 57(5), 630–651. https://doi.org/10.1007/s11162-015-9399-2

55.

Jensen

M. C.

Meckling

W. H.

(1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics, 3(4), 305–360. https://doi.org/10.1016/0304-405X(76)90026-X

56.

Jones

(2016). A historical mission in the accountability era: A public HBCU and state performance funding. Educational Policy, 30(7), 999–1041. https://doi.org/10.1177/0895904815586852

57.

Jones

Elliott

K. C.

Owens

L. R.

Assalone

A. E.

Gándara

(2017). Outcomes based funding and race in higher education: Can equity be bought? Palgrave Macmillan. https://doi.org/10.1007/978-3-319-49436-4

58.

Jones

W. A.

(2013). Prestige among Black colleges: Examining the predictors of HBCU peer academic reputation. Journal of African American Studies, 17(2), 129–141. http://doi.org/10.1007/s12111-012-9231-4

59.

Jongbloed

Enders

Salerno

(2008). Higher education and its communities: Interconnections, interdependencies and a research agenda. Higher Education, 56(3), 303–324. http://doi.org/10.1007/s10734-008-9128-2

60.

Kansas Board of Regents. (2019). Foresight 2020: Annual progress report.

61.

Kelchen

Ortagus

Baker

Rosinger

(2020). Trends in state funding for public higher education. InformEd States. https://static1.squarespace.com/static/5d9f9fae6a122515ee074363/t/5f4d2ae0e8f13c2a2dabd9ea/1598892770478/IS_Brief_TrendsinStateFunding_Aug2020.pdf

62.

Kelchen

Rosinger

K. O.

Ortagus

J. C.

(2019). How to create and use state-level policy data sets in education research. AERA Open, 5(3). https://doi.org/10.1177/2332858419873619

63.

Kelchen

Stedrak

L. J.

(2016). Does performance-based funding affect colleges’ financial priorities? Journal of Education Finance, 41(3), 302–321. https://doi.org/10.1353/jef.2016.0006

64.

Kennedy

(1981). Estimation with correctly interpreted dummy variables in semilogarithmic equations. American Economic Review, 71(4), 801.

65.

Kim

(2018). The functions and dysfunctions of college rankings: An analysis of institutional expenditure. Research in Higher Education, 59(1), 54–87. https://doi.org/10.1007/s11162-017-9455-1

66.

King

Nielsen

(2019). Why propensity scores should not be used for matching. Political Analysis, 27(4), 435–454. https://doi.org/10.1017/pan.2019.11

67.

Lau

Rosen

H. S.

(2016). Are universities becoming more unequal? Review of Higher Education, 39(4), 479–514. http://doi.org/10.1353/rhe.2016.0023

68.

Ledford

(2014, November 19). Indirect costs: Keeping the lights on. Nature. https://www.nature.com/news/indirect-costs-keeping-the-lights-on-1.16376

69.

Leslie

L. L.

Slaughter

Taylor

B. J.

Zhang

(2012). How do revenue variations affect expenditures within U.S. research universities? Research in Higher Education, 53(6), 614–639. https://doi.org/10.1007/s11162-011-9248-x

70.

A. Y.

(2020). Performance funding policy impacts on STEM degree attainment. Educational Policy, 34(2), 312–349. https://doi.org/10.1177/0895904818755455

71.

A. Y.

Gándara

Assalone

(2018). Equity or disparity: Do performance funding policies disadvantage 2-year minority-serving institutions? Community College Review, 46(3), 288–315. https://doi.org/10.1177/0091552118778776

72.

A. Y.

Ortagus

J. C.

(2019). Raising the stakes: Impacts of the Complete College Tennessee Act on underserved student enrollment and sub-baccalaureate credentials. Review of Higher Education, 43(1), 295–333. https://doi.org/10.1353/rhe.2019.0097

73.

Mateos-González

J. L.

Boliver

(2019). Performance-based university funding and the drive towards “institutional meritocracy” in Italy. British Journal of Sociology of Education, 40(2), 145–158. https://doi.org/10.1080/01425692.2018.1497947

74.

Matthews

C. M.

(2011). Federal research and development funding at historically black colleges and universities. Congressional Research Service. https://digital.library.unt.edu/ark:/67531/metadc795348/

75.

McClure

K. R.

Titus

(2018). Spending up the ranks? The relationship between striving for prestige and administrative expenditure at U.S. public research universities. Journal of Higher Education, 89(6), 961–987.

76.

McLendon

M. K.

Hearn

J. C.

Mokher

C. G.

(2009). Partisans, professionals, and power: The role of political factors in state higher education funding. Journal of Higher Education, 80(6), 686–713. https://doi.org/10.1080/00221546.2009.11779040

77.

Meyer

B. D.

(1995). Natural and quasi-experiments in economics. Journal of Business & Economic Statistics, 13(2), 151–161. https://doi.org/10.1080/07350015.1995.10524589

78.

Miller

(2016). Higher education outcomes-based funding models and academic quality. Lumina Foundation. https://www.luminafoundation.org/files/resources/ensuring-quality-1.pdf

79.

Morse

Brooks

(2021, September 12). How U.S. News calculated the 2021 best colleges rankings. U.S. News. https://www.usnews.com/education/best-colleges/articles/how-us-news-calculated-the-rankings

80.

National Center for Science and Engineering Statistics. (2021). Higher Education Research and Development: Fiscal year 2019. https://www.nsf.gov/statistics/srvyherd/#tabs-2

81.

Ness

E. C.

Tandberg

D. A.

McLendon

M. K.

(2015). Interest groups and state policy for higher education: New conceptual understandings and future research directions. In Paulsen

M. B.

(Ed.), Higher education: Handbook of theory and research (pp. 151–186). Springer.

82.

New Mexico Higher Education Department. (2018). New Mexico higher education I&G funding formula [Data set]. https://hed.state.nm.us/uploads/documents/New_Mexico_HED_Funding_Formula_-_2020_Technical_Guide_for_FY21_Budget_Cycle.pdf

83.

O’Meara

(2007). Striving for what? Exploring the pursuit of prestige. In Smart

J. C.

(Ed.), Higher education: Handbook of theory and research (Vol. 22, pp. 121–179). Springer. https://doi.org/10.1007/978-1-4020-5666-6_3

84.

Office of Postsecondary Education. (2021). Eligibility designations and applications for waiver of eligibility requirements. https://www2.ed.gov/about/offices/list/ope/idues/eligibility.html#tips

85.

Ortagus

J. C.

Kelchen

Rosinger

Voorhees

(2020). Performance-based funding in American higher education: A systematic synthesis of the intended and unintended consequences. Educational Evaluation and Policy Analysis, 42(4), 520–550. https://doi.org/10.3102/01623737209

86.

Ortagus

J. C.

Rosinger

Kelchen

(2021). InformEd States performance-based funding policies dataset. InformEd States. https://informedstates.org/pbf-funding-policies-dataset-download

87.

Payne

Roberts

(2010). Government oversight of public universities: Are centralized performance schemes related to increased quantity or quality? Review of Economics and Statistics, 92(1), 207–2012. https://doi.org/10.1162/rest.2009.10157

88.

Pegkas

Staikouras

Tsamadias

(2019). Does research and development expenditure impact innovation? Evidence from the European Union countries. Journal of Policy Modeling, 41(5), 1005–1025. https://doi.org/10.1016/j.jpolmod.2019.07.001

89.

Pfeffer

Salancik

G. R.

(1978). The external control of organizations: A resource dependence perspective. Harper & Row.

90.

Powell

B. A.

Gilleland

D. S.

Pearson

L. C.

(2012). Expenditures, efficiency, and effectiveness in U.S. undergraduate higher education: A national benchmark model. Journal of Higher Education, 83(1), 102–127. https://doi.org/10.1353/jhe.2012.0005

91.

Rabovsky

T. M.

(2012). Accountability in higher education: Exploring impacts on state budgets and institutional spending patterns. Journal of Public Administration Research and Theory, 22(4), 675–700. https://doi.org/10.1093/jopart/mur069

92.

Radin

(2006). Challenging the performance movement: Accountability, complexity, and democratic values. Georgetown University Press.

93.

Rhoten

Calhoun

(2011). Knowledge matters: The public mission of the research university. Columbia University Press.

94.

Rippner

J. A.

Toutkoushian

R. K.

(2015). The “big bang” in public and private faculty salaries. Journal of Education Finance, 41(2), 103–123. https://www.jstor.org/stable/44162893

95.

Robst

(2001). Cost efficiency in public higher education institutions. Journal of Higher Education, 72(6), 730–750. https://doi.org/10.2307/2672901

96.

Rosenbaum

P. R.

(2002). Observational studies (2nd ed.). Springer. https://doi.org/10.1007/978-1-4757-3692-2

97.

Rosenbaum

P. R.

Rubin

D. B.

(1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387), 516–524. https://doi.org/10.1080/01621459.1984.10478078

98.

Rosinger

Ortagus

Kelchen

Cassell

Voorhees

(2020). The landscape of performance-based funding in 2020. https://informedstates.org/policy-briefs

99.

Rosinger

Taylor

Slaughter

(2016). The crème de la crème: Stratification and accumulative advantage within US private research universities. In Slaughter

Taylor

(Eds.), Higher education, stratification, and workforce development (pp. 81–101). Springer. https://doi.org/10.1007/978-3-319-21512-9_5

100.

Ryan

J. F.

(2004). The relationship between institutional expenditures and degree attainment at baccalaureate colleges. Research in Higher Education, 45(2), 97–114. https://doi.org/10.1023/B:RIHE.0000015691.02545.61

101.

Shin

J. C.

(2010). Impacts of performance-based accountability on institutional performance in the U.S. Higher Education, 60(1), 47–68. https://doi.org/10.1007/s10734-009-9285-y

102.

Shin

J. C.

Milton

(2004). The effects of performance budgeting and funding programs on graduation rate in public four-year colleges and universities. Education Policy Analysis Archives, 12, 22. https://doi.org/10.14507/epaa.v12n22.2004

103.

Sīle

Vanderstraeten

(2019). Measuring changes in publication patterns in a context of performance-based research funding systems: The case of educational research in the University of Gothenburg (2005–2014). Scientometrics, 118(1), 71–91. https://doi.org/10.1007/s11192-018-2963-8

104.

Slaughter

S. A.

Rhoades

(2004). Academic capitalism and the new economy: Markets, state, and higher education. Johns Hopkins University Press.

105.

Snyder

Boelscher

(2018). Driving better outcomes: Fiscal year 2018 state status and typology update. HCM Strategists. https://hcmstrategists.com/resources/driving-better-outcomes-fiscal-year-2018-state-status-typology-update/

106.

Snyder

Fox

(2016). Driving better outcomes: Fiscal year 2016 state status and typology update. HCM Strategists. https://hcmstrategists.com/resources/driving-better-outcomes-fiscal-year-2016-state-status-and-typology-update/

107.

South Dakota Board of Regents. (2000). Factbook: Fiscal year 2001. https://www.sdbor.edu/mediapubs/factbook/Documents/FY01factbook.pdf

108.

Spence

Zeckhauser

(1971). Insurance, information, and individual action. American Economic Review, 61(2), 380–387.

109.

Stocum

D. L.

(2013). Killing public higher education: The arms race for research prestige. Academic Press.

110.

Sun

Abraham

(2020). Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics, 225(2), 175–199. https://doi.org/10.1016/j.jeconom.2020.09.006

111.

Tandberg

D. A.

(2010). Interest groups and governmental institutions: The politics of state funding of public higher education. Educational Policy, 24(5), 735–778. https://doi.org/10.1177/0895904809339163

112.

Taylor

B. J.

Cantwell

(2019). Unequal higher education: Wealth, status, and student opportunity. Rutgers University Press. https://doi.org/10.36019/9780813593531

113.

Tennessee Higher Education Commission. (2015). Quality assurance funding: 2015-20 cycle standards.

114.

Thompson

J. R.

(1999). Devising administrative reform that works: The example of the reinvention lab program. Public Administration Review, 59(4), 283–293.

115.

Toutkoushian

Danielson

(2002). Using performance indicators to evaluate decentralized budgeting systems and institutional performance. In Priest

D. M.

Becker

Hossler

St. John

E. P.

(Eds.), Incentive-based budgeting systems in public universities (pp. 205–226). Elgar.

116.

Toutkoushian

R. K.

Paulsen

M. B.

(2016). Economics of higher education. Springer. https://doi.org/10.1007/978-94-017-7506-9

117.

Umbricht

M. R.

Fernandez

Ortagus

J. C.

(2017). An examination of the (un)intended consequences of performance funding in higher education. Educational Policy, 31(5), 643–673. https://doi.org/10.1177/0895904815614398

118.

Vanecek

(2014). The effect of performance-based research funding on output of R&D results in the Czech Republic. Scientometrics, 98(1), 657–681. https://doi.org/10.1007/s11192-013-1061-1

119.

Volkwein

J. F.

Sweitzer

K. V.

(2006). Institutional prestige and reputation among research universities and liberal arts colleges. Research in Higher Education, 47(2), 129–148. https://doi.org/10.1007/s11162-005-8883-5

120.

Wells

A. R.

Hamar

Bradley

Gandy

W. M.

Harrison

P. L.

Sidney

J. A.

Coberley

C. R.

Rula

E. Y.

Pope

J. E.

(2013). Exploring robust methods for evaluating treatment and comparison groups in chronic care management programs. Population Health Management, 16(1), 35–45. https://doi.org/10.1089/pop.2011.0104

121.

Wooldridge

J. M.

(2002). Econometric analysis of cross section and panel data. MIT Press. https://jrvargas.files.wordpress.com/2011/01/wooldridge_j-_2002_econometric_analysis_of_cross_section_and_panel_data.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB