In a recent issue of the American Journal of Evaluation, Mark (2001) challenges evaluators to “derive more cumulative knowledge from and about evaluation” (p. 471). The aim of the current study was to address this challenge by illustrating a three-phase approach for program theory building and testing. This approach is proposed as a compliment to the traditional theory-driven outcomes evaluation as a means of deriving and cumulating evaluation knowledge. Hence, this study was intended as an example of how to cumulate knowledge about evaluation practice itself. It was also intended to illustrate how knowledge can be derived from evaluation by replicating models of program theory across multiple contexts. The study concludes with a discussion of methodological issues related to developing and testing models of program theory.
Get full access to this article
View all access options for this article.
References
1.
Alper, S., Tjosvold, D., & Law, K. S. (1998). Interdependence and controversy in group decision making: Antecedents to effective self-managing teams. Organizational Behavior and Human Decision Processes, 74, 33-52.
2.
Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411-423.
3.
Byrne, B. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum.
4.
Campbell, D. J., & Gingrich, K. F. (1986). The interactive effects of task complexity and participation on task performance: A field experiment. Organizational Behavior and Human Decision Processes, 38, 162-180.
5.
Decision-making. Educational Evaluation and Policy Analysis, 16(3), 302-319.
6.
Chen, H., & Rossi, P. H. (1992). Using theory to improve program and policy evaluations. New York: Greenwood Press.
7.
Deutsch, M. (1980). Over fifty years of conflict research. In L. Festinger (Ed.), Four decades of social psychology. New York, NY: Oxford University Press.
8.
Erez, M., & Arad, R. (1986). Participative goal-setting: Social, motivational, and cognitive factors. Journal of Applied Psychology, 4, 591-597.
9.
Jöreskog, K. G., & Sörbom, D. (1989). LISREL7: A guide to the program and applications (2nd ed.). Chicago: SPSS Inc.
10.
Latham, G. P., & Saari, L. M. (1979). Importance of supportive relationships in goal setting. Journal of Applied Psychology, 64, 151-156.
11.
Latham, G. P., Winters, D. C., & Locke, E. A. (1994). Cognitive and motivational effects of participation: A mediator study. Journal of Organizational Behavior, 15, 49-63.
12.
Locke, E. A., & Latham, G. P. (1990). A theory of goal setting and task performance. Englewood Cliff, NJ: Prentice Hall.
13.
Locke, E. A., & Schweiger, D. M. (1979). Participation in decision-making: One last look. Research in Organizational Behavior, 1, 265-339.
14.
Mark, M. (2001). Evaluation’s future: Furor, futile, or fertile?The American Journal of Evaluation, 22, 457-479.
15.
Marsh, H. W. (1994). Confirmatory factor analysis models of factorial invariance: A multifaceted approach. Structural Equation Modeling, 1, 5-34.
16.
Miller, K. I., & Monge, P. R. (1986). Participation, satisfaction, and productivity: A meta-analytic review. Academy of Management Journal, 29, 727-753.
17.
Pawson, R., & Tilley, N. (2001). Realistic evaluation bloodlines. The American Journal of Evaluation, 22, 317-324.
18.
Rahim, M. A., & Magner, N. R. (1995). Confirmatory factor analysis of the styles of handling interpersonal conflict: First order model and its invariance across groups. Journal of Applied Psychology, 80, 122-132.
19.
Reyes, P. (1990). Organizational commitment of teachers. In P. Reyes (Ed.), Teachers and their workplace: Commitment, performance and productivity (pp. 38-47). Newbury Park, CA: Sage.
20.
Schneider, B., & Bryk, A. S. (2000). Studying trust in school reform in Chicago. Paper presented at the 2000 Meeting of the American Educational Research Association, New Orleans, LA.
21.
Schumacker, R. E., & Lomax, R. G. (1996). A beginner’s guide to structural equation modeling. Hillsdale, NJ: Lawrence Erlbaum.
22.
Senge, P. A. (1990). The fifth discipline: The art and practice of the learning organization. New York: Doubleday.
23.
Smith, M. F. (2001). Evaluation: Preview of the future #2. The American Journal of Evaluation, 22, 281-300.
24.
Tjosvold, D. (1985). Dynamics of participation: An experimental investigation. Group and Organization Studies, 10, 260-277.
25.
Tjosvold, D. (1986). Organizational test of goal linkage theory. Human Relations, 39, 517-528.
26.
Tjosvold, D. (1987). Participation: A close look at its dynamics. Journal of Management, 13, 731-750.
27.
Tjosvold, D., Andrews, I. R., & Jones, H. (1983). Cooperative and competitive relationships between leaders and subordinates. Human Relations, 36, 1111-1124.
28.
Tjosvold, D., & Deemer, D. K. (1980). Effects of controversy within a cooperative or competitive context on organizational decision making. Journal of Applied Psychology, 65, 590-595.
29.
Tjosvold, D., & Johnson, D. W. (1978). Controversy within a cooperative or competitive context and cognitive perspective taking. Contemporary Educational Psychology, 3, 376-386.
30.
Turnbull, B. (1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131-140.
31.
Wagner, J. A., & Gooding, R. Z. (1987a). Effects of societal trends on participation research. Administrative Science Quarterly, 32, 241-262.
32.
Wagner, J. A., & Gooding, R. Z. (1987b). Shared influence and organizational behavior: A meta-analysis of situational variables expected to moderate participation-outcome relationships. Academy of Management Journal, 30, 524-541.
33.
Wohlstetter, P., Smyer, R., & Mohrman, S. A. (1994). New boundaries for school-based management. Educational Evaluation and Policy Analysis, 16, 268-286.
