This paper discusses methods for reconstructing theories underlying programs and policies. It describes three approaches. One is empirical-analytical in nature and focuses on interviews, documents and argumentational analysis. The second has strategic assessment, group dynamics, and dialogue as its core. The third has cognitive and organizational psychology as its foundation. For each of the three approaches, case-illustrations are given. These approaches can help to make the process of reconstructing underlying program theories more open for scrutiny. This is important because mis-reconstruction of policy and program theories is dangerous. All three approaches have a number of weaknesses to be remedied. The paper discusses these shortcomings and presents some suggestions for overcoming the limitations are presented.
Get full access to this article
View all access options for this article.
References
1.
Andrews, M. (2002). A theory-based approach to evaluating budget reforms. International Public Management Journal, 5, 135-154.
2.
Argyris, C., & Schön, D. A. (1978). Organizational learning: A theory of action perspective. Reading, MA: Addison-Wesley.
3.
Bell, P. (1997). Using argument representations to make thinking visible for individuals and groups. In R. Hall, N. Miyake, & N. Enyedy (Eds.), Proceedings of CSCL ’97: The Second International Conference on Computer Support for Collaborative Learning (pp. 10-19). Toronto: University of Toronto Press.
4.
Bickman, L. (2000). Summing up program theory. In P. J. Rogers, T. A. Hacsi, A. Petrosino, & T. A. Huebner (Eds.), Program theory in evaluation: Challenges and opportunities. New directions in evaluation (Vol. 87, pp. 103-113). San Francisco, CA: Jossey-Bass.
5.
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. In P. J. Rogers, T. A. Hacsi, A. Petrosino, & T. A. Huebner (Eds.), Program theory in evaluation: Challenges and opportunities. New directions in evaluation (Vol. 87, pp. 27-35). San Francisco: Jossey-Bass.
6.
Daniels, K., De Chernatony, L., & Johnson, G. (1995). Validating a method for mapping managers’ mental models of competition. Human Relations, 48, 971-975.
7.
Davidson, E. J. (2000). Assserting causality in theory-based evaluation. In P. J. Rogers, T. A. Hacsi, A. Petrosino, & T. A. Huebner (Eds.), Program theory in evaluation: Challenges and opportunities. New directions in evaluation (Vol. 87, pp. 17-27). San Francisco, CA: Jossey-Bass.
8.
Eden, C., & Spender, J.-C. (Eds.). (1998). Managerial and organizational cognition. Theory, methods, and research. London: Sage.
9.
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
10.
Hoogerwerf, A. (1987). Beleid berust op veronderstellingen: De beleidstheorie. Acta Politica, 19, 493-531.
11.
Jackson, M. C. (1989). Assumptional analysis. Systems Practice, 14, 11-28.
12.
Leeuw, F. L. (1991). Policy theories, knowledge utilization, and evaluation. Knowledge and Policy, 4, 73-92.
13.
Leeuw, F. L., & van Gils, G. H. C. (1999). Evaluating anti-corruption initiatives. Underlying logic and mid-term impact of a World Bank Program. Evaluation, 5, 194-219.
14.
Manheim, K. (1935/1967). Man and society in an age of reconstruction: Studies in modern social structure. New York: Hardcourt, Brace & World.
15.
Mason, I., & Mitroff, I. (1981). Challenging strategic planning assumptions. New York: Wiley.
16.
Midgley, G. (1996). Evaluating services for people with disabilities: A critical systems perspective. Evaluation, 2, 67-85.
17.
Pawson, R. (2002). Does Megan’s Law work? A theory-driven systematic review. ESCR UK Centre for Evidence Based Policy and Practice Working Paper 8, Queen Mary University of London.
18.
Rogers, P. J., Hacsi, T. A., Petrosino, A., & Huebner, T. A. (Eds.). (2000a). Program theory evaluation: Practice, promise and problems. New directions in evaluation (Vol. 87, pp. 5-17). San Francisco, CA: Jossey-Bass.
19.
Rogers, P. J., Hacsi, T. A., Petrosino, A., & Huebner, T. A. (Eds.). (2000b). Program theory in evaluation: Challenges and opportunities. New directions in evaluation (Vol. 87). San Francisco, CA: Jossey-Bass.
20.
Rossi, P., Freeman, H., & Lipsey, M. (1999). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.
21.
Schwandt, T. A. (2001). A postscript on thinking about dialogue. Evaluation, 7, 264-277.
22.
Schwenk, C. (1984). Devil’s advocate in managerial decision-making. Journal of Management Studies, 21, 153-168.
23.
Shaw, I. (2000). Evaluating public progammes: Contexts and issues. Aldershot: Ashgate.
24.
Stern, E. (2001). Editorial foreword to the special issue of evaluation: Dialogue in evaluation. Evaluation, 7, 1-3.
25.
Suthers, D., & Weiner, A. (1995). Groupware for developing critical discussion skills. http://www-cscl95.indiana.edu/cscl95/suthers.html.
26.
Suthers, D. D., Toth, E. E., & Weiner, A. (1997). An integrated approach to implementing collaborative inquiry in the classroom. In R. Hall, N. Miyake, & N. Enyedy (Eds.), Proceedings of CSCL ’97: The Second International Conference on Computer Support for Collaborative Learning (pp. 272-279). Toronto: University of Toronto Press.
27.
Swanborn, P. G. (1999). Evalueren. Amsterdam: Boom Press.
28.
Tilley, N. (1999). Evaluation and evidence-(mis)led policy. Evaluation Journal of Australasia, 11, 48-64.
29.
Toulmin, S. (1958). The uses of argument. Cambridge: Cambridge University Press.
30.
Trochim, W. (1989). Concept mapping. Soft science or hard art?Evaluation and Program Planning, 12, 87-110.
31.
Van der Heijden, K., & Eden, C. (1998). The theory and praxis of reflective learning in strategy making. In C. Eden & J.-C. Spender (Eds.), Managerial and organizational cognition: Theory, methods and research (pp. 58-76). London: Sage.
32.
Vedung, E. (1998). Public policy and program evaluation. New Brunswick & London: Transaction.
33.
Weick, K. E., & Bougon, M. G. (1986). Organizations as cognitive maps. Charting ways to success and failure. In H. P. Sims, et al. (Eds.), The thinking organization: Dynamics of organizational and social cognition. San Francisco, CA: Jossey-Bass.
