A theory-driven approach to evaluation (TDE) emphasizes the development and empirical testing of conceptual models to understand the processes and mechanisms through which programs achieve their intended goals. However, most reported applications of TDE are limited to large-scale experimental/quasi-experimental program evaluation designs. Very few (limited) examples of the relevance of TDE to nonexperimental program evaluation designs exist in literature. Using the method of structural equation modeling to analyze data from the Interns for Indiana (IfI) program, this study demonstrates how evaluation practitioners can test logical and sequential relationships among tiers of outcomes of nonexperimental programs, especially programs with limited datasets. The study also describes how the empirical feedback can be used to understand program dynamics and improve program implementation and evaluation.
ArbuckleJ. L. (2005). Amos 6.0 user’s guide. Chicago, IL: SPSS.
2.
BeranT. N.ViolatoC. (2010). Structural equation modeling in medical research: A primer. BMC Research Notes, 3, 267, 1-10. doi:10.1186/1756-0500-3-267.
3.
BiermanK. L.NixR. L.GreenbergM. T.BlairC.DomitrovichC. E. (2008). Executive functions and school readiness intervention: impact, moderation, and mediation in the Head Start REDI program. Developmental Psychopathology, 20, 821-843.
4.
BirckmayerJ.WeissC. H. (2000). Theory-based evaluation in practice: What do we learn?. Evaluation Review, 24, 407-431.
5.
BledsoeK. L.GrahamJ. A. (2005). The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation, 26, 302-319.
6.
ChenH. T. (1990). Theory-driven evaluations. Newbury-Park, CA: Sage.
7.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155-159.
8.
CorynC. L. S.NoakesL. A.SchröterD. C.WestineC. D. (n.d.). An empirical review of theory-driven evaluation practice from 1990-2009. Advance online publication. doi: 10.1177/1098214010389321
9.
DonaldsonS. I. (2001). Mediator and moderator analysis in program development. In SussmanS. (Ed.), Handbook of program development for health behavior research and practice (pp.470-500). Thousand Oakes, CA: Sage.
10.
DonaldsonS. I. (2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum.
11.
DonaldsonS. I.GoolerL. E. (2003). Theory-driven evaluation of the work and health initiative: A focus on winning new jobs. American Journal of Evaluation, 23, 341-346.
12.
FairchildA. J.McQuillinS. D. (2010). Evaluating mediation and moderation effects in school psychology: A presentation of methods and review of current practice. Journal of School Psychology, 48, 53-84.
13.
HansenW. B.McNealR. B.Jr. (1996). The law of maximum expected potential effect: Constraints placed on program effectiveness by mediator relationships. Health Education Research, 11, 501-507.
14.
HassanS. B.BanC.HugginsL. (2003). Explainning the “brain drain” from older industrial cities: The Pittsburgh region. Economic Development Quarterly, 17, 132-147.
15.
HektnerJ. M. (1995). When moving up implies moving out: Rural adolescents' conflict in the transition to adulthood. Journal of Research in Rural Education, 11, 3-14.
16.
HenneseyM.GreenbergJ. (1999). Bringing it all together: Modeling intervention processes using structural equation modeling. American Journal of Evaluation, 20, 471-480.
17.
KlineR. B. (2005). Principles and practice of structural equation modeling. (2nd ed.) New York, NY: Guilford Press.
18.
KolbA. Y.KolbD. A. (2005). Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of Management Learning and Education, 4, 193-212.
19.
KolbD. A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall.
20.
LewinK. (1951). Field theory in social sciences. New York, NY: Harper & Row.
21.
LivingstonE. H. (2004). The mean and standard deviation: What does it all mean?. Journal of Surgical Research, 119, 117-123.
22.
MarkM. M. (1990). From program theory to tests of program theory. New Directions for Program Evaluation, 47, 37-51.
23.
PreacherK. J.HayesA. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in simple and multiple mediator models. Behavior Research Methods, 40, 879-891.
24.
RosasS. R. (2005). Concept mapping as a technique for program theory development: An illustration using family support programs. American Journal of Evaluation, 26, 389-401.
25.
SidaniS.SechrestL. (1999). Putting program theory into operation. American Journal of Evaluation, 20, 227-238.
26.
SmithN. L. (1990). Using path analysis to develop and evaluate program theory and impact. New Directions for Evaluation, 47, 53-57.
27.
SobelM. E. (1982). Asymptotic confidence intervals for indirect effects in structural equations models . In LeinhartS. (Ed.), Sociological methodology (pp.290-312). San Francisco, CA: Jossey-Bass.
28.
SobelM. E. (1986). Some new results on indirect effects and their standard errors in covariance structure models . In TumaN. (Ed.), Sociological methodology (pp.159-186). Washington, DC: American Sociological Association.
29.
WeissC. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community based initiatives for children and families . In ConnellJ. P.KubischA. C.SchorrL. B.WeissC. H. (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp.65-92). Washington, DC: The Aspen Institute.
30.
WhiteH. (2009). Theory-based impact evaluation: Principle and practice. Working paper of the International Initiative for Impact Evaluation. Retrieved March 16, 2010, fromhttp://www.3ieimpact.org/page.php?pg=papers/48.pdf
