How Accurately Can Short-Term Outcomes Approximate Long-Term Outcomes? Examining the Predictive Power of Early Momentum Metrics for Community College Credential Completion Using Machine Learning

Abstract

Objective: This study examines how accurately a small set of short-term academic indicators can approximate long-term outcomes of community college students so that decision-makers can take informed actions based on those indicators to evaluate the current progress of large-scale reform efforts on long-term outcomes, which in practice will not be observed until several years later. Method: Using transcript-level data of approximately 50,000 students at over 30 institutions in two states, I compare the out-of-sample predictive power of the early momentum metrics (EMMs), 13 short-term academic indicators suggested in the literature, to that of more complex, Machine Learning (ML)-based models that employ 497 predictors. Results: This study found that EMMs accurately predict credential completion for 75% to 77% of students in an out-of-sample dataset, with a predictive power largely comparable to that of ML-based models. This study also found similar results among the gender and race/ethnicity groups. However, the predictive power for certificate completion is lower than that for associate and bachelor’s degrees by 5 percentage points, implying that this set of EMMs are likely to be less relevant to certificate completion. Contribution: This study validates EMMs as informative predictors of credential completion, confirming that decision makers can use them to understand the probable long-term impact of current reforms on credential outcomes. However, room for continued research and refinement of EMMs remains, especially for certificate.