In the field of program evaluation, one of the first things evaluators often do to evaluate a program is to identify the “theory of change” behind the program and create a “logic model” to articulate why the program is expected to work. This allows evaluators to make an assessment about the expectations and feasibility of the program, as well as assess program implementation. In this article, the author uses the theory of change approach to evaluation and creates a logic model for the “accountability movement” or the tendency for funders to require nonprofit organizations to conduct evaluations and report performance information. The model is tested with empirical data, finding that the theory behind the accountability movement suffers from four design and implementation flaws well-known to evaluators: (a) unstandardized treatment, (b) confusion among targets, (c) coverage, and (d) intensity issues. The article concludes by offering insights and recommendations for improving accountability practices.
Alaimo, S.P. ( 2008). Nonprofits and evaluation: Managing expectations from the leader’s perspective. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and evaluation: New directions for program evaluation (Vol. 119, pp. 73-92). San Francisco: Jossey-Bass.
2.
Anheier, H. ( 2005). Nonprofit organizations: Theory, management, policy. New York: Routledge.
3.
Axelrod, N.R. ( 2005). Board leadership and development. In R. D. Herman & Associates (Eds.), The Jossey-Bass handbook of nonprofit leadership and management (2nd ed., pp. 131-152). San Francisco, CA: Jossey-Bass.
4.
Bamberger, M., Rugh, J., Church, M., & Fort, L. ( 2004). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation , 25(1), 5-37.
5.
Boulmetis, J., & Dutwin, P. ( 2005). The ABCs of evaluation: Timeless techniques for program and project managers (2nd ed.). San Francisco, CA: Jossey-Bass.
6.
Carlton, D.W. ( 1989). The theory and the facts of how markets clear: Is industrial organization valuable for understanding macroeconomics? In R. Schmalensee & R. D. Willig (Eds.), Handbook of industrial organization (Vol. 1, pp. 909-946). New York: Elsevier Science.
7.
Carman, J.G. ( 2007). Evaluation practice among community-based organizations: Research into the reality. American Journal of Evaluation , 28(1), 60-75.
8.
Carman, J.G. ( 2008). Nonprofits, funders, and evaluation: Accountability in action. American Review of Public Administration. Retrieved August 6, 2008, from http://arp.sagepub.com/pap.dtl
9.
Carman, J.G., & Fredericks, K.A. (2008). Nonprofits and evaluation: Empirical evidence from the field. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and evaluation: New directions for program evaluation (Vol. 119, pp. 51-72). San Francisco: Jossey-Bass.
10.
Carman, J.G., & Millesen, J.L. (2005). Nonprofit program evaluation: Organizational challenges and resource needs. Journal of Volunteer Administration, 23(3), 36-43.
11.
Chen, H. ( 2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage.
12.
Cohen, R. ( 2007). Conflicts of interest: Mischief, thou art afoot. Nonprofit Quarterly, 14(1), 54-60.
13.
Council on Foundations. (2002). Board briefing: Type of funding. Retrieved October 20, 2007 , from http://www.cof.org/files/Documents/Governing_Boards/BBProjectOperating.PDF
14.
Cutt, J., & Murray, V. ( 2000). Accountability and effectiveness evaluation in non-profit organizations. New York: Routledge .
15.
David, T. ( 2006, spring). Evaluation and foundations: Can we have an honest conversation?The Nonprofit Quarterly, 13(1), 90-93.
16.
Davis, J.H., Schoorman, F.D., & Donaldson, L. ( 1997). Toward a stewardship theory of management. Academy of Management Review, 1(22), 20-47.
17.
De Hoog, R.H., & Salamon, L.M. ( 2002). Purchase of service contracting. In L. M. Salamon (Ed.), The tools of government: A guide to the new governance (pp. 319-339). New York: Oxford University Press.
18.
DiMaggio, P.J., & Powell, W.W. ( 1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48, 147-160.
19.
Ebrahim, A. ( 2005). Accountability myopia: Losing sight of organizational learning. Nonprofit and Voluntary Sector Quarterly, 34(1), 56-87.
20.
Ferris, J.M. ( 1993). The double edged sword of social service contracting: Public accountability vs. nonprofit autonomy. In J. S. Ott (2001), The nature of the nonprofit sector (pp. 391-398). Boulder, CO: Westview Press.
21.
Ferris, J.M., & Graddy, E. ( 1994). Organizational choices for public service supply. Journal of Law, Economics and Organization, 11(1), 126-141.
22.
Gammal, D.L., Simard, C., Hwang, H., & Powell, W.W. ( 2005). Managing through challenges: A profile of San Francisco Bay Area nonprofits. Stanford, CA: Stanford Graduate School of Business, Stanford Project on the Evolution of Nonprofits.
23.
Gibelman, M., & Gelman, S.R. ( 2004). A loss of credibility: Patterns of wrongdoing among nongovernmental organizations. Voluntas, 15(4), 255-281.
24.
Grantmakers for Effective Organizations. ( 2004). Funding effectiveness: Lessons in building nonprofit capacity. San Francisco: Jossey-Bass .
25.
Gray, S. T., & Associates. (Eds.). (1998). Evaluation with power: Developing organizational effectiveness, empowerment and excellence. San Francisco: Jossey-Bass.
26.
Grobman, G.M. ( 2007). An introduction to the nonprofit sector: A practical approach for the 21st century (2nd ed.). Harrisburg, PA: White Hat Communications.
27.
Harlan, S.L., & Saidel, J.R. ( 1994). Board members’ influence on the government-nonprofit relationship. Nonprofit Management and Leadership, 5(2), 173-196.
Hatry, H.P., Wholey, J.S., & Newcomer, K.E. (2004). Other issues and trends in evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 670-684). San Francisco : Jossey-Bass.
30.
Hendricks, M. ( 2000, February). Outcomes measurement in the nonprofit sector: Recent developments, incentives, and challenges. Paper presented at the meeting of the American Society for Public Administration , Washington, DC.
31.
Hoefer, R. ( 2000). Accountability in action? Program evaluation in nonprofit human service agencies. Nonprofit Management and Leadership , 11(2), 167-177.
32.
Hoole, E., & Patterson, T. ( 2008). Voices from the field: Evaluation as part of a learning culture. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and evaluation: New directions for program evaluation (Vol. 119, pp. 93-114). San Francisco: Jossey-Bass.
33.
Howard, C. ( 2002). Tax expenditures. In L. M. Salamon (Ed.), The tools of government: A guide to the new governance (pp. 410-444). New York: Oxford University Press.
34.
International Network on Strategic Philanthropy. ( 2005). Theory of change tool manual. Retrieved October 30, 2007, from http://wingsweb.org/download/Theory_of_Change_Tool_Manual_Full_Version.pdf
35.
Kee, J.E. ( 2004). Cost-effectivness and cost-benefit analysis. In Wholey, J. S.,Hatry, H. P. & Newcomer, K. E. (Eds.). Handbook of practical program evaluation. (2nd Ed.) (pp. 506-542). San Francisco, CA: Jossey-Bass.
36.
Kettl, D.F. ( 1993). Sharing power: Public governance and private markets . Washington, DC: Brookings Institution .
37.
Kopczynski, M.E., & Pritchard, K. ( 2004). The use of evaluation by nonprofit organizations. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 649-669). San Francisco: Jossey-Bass.
38.
Light, P. ( 2000). Making nonprofits work: A report on the tides of nonprofit management reform. Washington, DC: The Aspen Institute and the Brookings Institution Press.
39.
Love, A. ( 2004). Implementation evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 63-97). San Francisco: Jossey-Bass.
40.
Love, A. ( 2006). Harnessing the power of technology for evaluation. Nonprofit Quarterly, 13(1), 86-89.
41.
Martin, L.J. & Kettner, P.M. ( 1996). Measuring the performance of human service programs. (Sage Human Service Guides, Volume 71). Thousand Oaks, CA: SAGE Publications.
42.
McLaughlin, J.A., & Jordan, G.B. ( 2004). Using logic models. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 7-32). San Francisco: Jossey-Bass.
43.
Meyer, J.W., & Rowan, B. ( 1977). Institutionalized organizations: Formal structure as myth and ceremony. American Journal of Sociology, 83(2), 340-363.
44.
Meyer, J.W., & Scott, W.R. ( 1983). Organizational environments: Ritual and rationality. Beverly Hills, CA: Sage.
45.
Middleton, M. ( 1987). Nonprofit boards of directors: Beyond the governance function. In W. W. Powell (Ed.), The nonprofit sector: A research handbook (pp. 141-153). New Haven, CT: Yale University Press.
46.
Miller, J.L., & Faerman, S.R. ( 2003). Making good board choices: A competing values approach . Nonprofit Quarterly, 10(1), 48-53.
47.
Mirabella, R.M. ( 2007). University-based educational programs in nonprofit management and philanthropic studies: A 10-year review and projections of future trends . Nonprofit and Voluntary Sector Quarterly, 36(4), 11S-27S.
48.
Moe, T.M. ( 1984). The new economics of organization. American Journal of Political Science, 28(4), 739-777.
49.
Mohr, L.B. ( 1995). Impact analysis for program evaluation (2nd ed.). Thousand Oaks, CA: Sage.
50.
Mott, A. ( 2006). Hand in hand: Evaluation and organizational development . Nonprofit Quarterly, 13(1), 42-47.
51.
Mullen, E. J., & Magnabosco, J. L. (Eds.). (1997). Outcomes measurement in the human services: Crosscutting issues and methods. Washington, DC: National Association of Social Workers .
52.
Murphy, D.M., & Mitchell, R. ( 2007). Building evaluation capacity in North Carolina’s nonprofit sector: A survey report. Raleigh, NC: Institute for Nonprofits.
53.
Murray, V. ( 2005). Evaluating the effectiveness of nonprofit organizations . In R. D. Herman & Associates (Eds.), The Jossey-Bass handbook of nonprofit leadership and management (2nd ed., pp. 345-370). San Francisco: Jossey-Bass.
54.
Office of Management and Budget. (2004). Program evaluation: What constitutes strong evidence of program effectiveness? Retrieved December 9, 2007, from http://www.whitehouse.gov/omb/part/2004_program_eval.pdf
55.
Osborne, D., & Plastrick, P. ( 1998). Banishing bureaucracy: The five strategies for reinventing government. New YorkPenguinPutnam .
56.
Philliber, S. ( 1998). The virtue of specificity in theory of change evaluation . In K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell (Eds.), New approaches to evaluating community initiatives, Vol. 2: Theory, measurement and analysis (pp. 87-100). Queenstown, MD: The Aspen Institute, Roundtable on Comprehensive Community Initiatives for Children and Families.
57.
Rampersad, H.K. ( 2008). The way to a high-performance culture with the Total Performance Scorecard. Strategic Change, 1(1-2), 43-55.
58.
Rittenhouse, T., Campbell, J., & Daltro, M. ( 2002). Dressed-down research terms: A glossary for non-researchers . Retrieved August 5, 2008, from http://www.cstprogram.org/PCS&T/Research%20Glossary/Dressed_Down_Glossary.pdf
59.
Roberts, A.R., & Yeager, K.R. ( 2004). Evidence-based practice manual: Research and outcome measures in health and human services. New York: Oxford University Press.
60.
Rossi, P.H., & Freeman, H.E. ( 1993). Evaluation: A systematic approach (5th ed.). Newbury Park, CA: Sage .
61.
Salamon, L.M. ( 2005). The changing context of American nonprofit management . In R. D. Herman & Associates (Eds.), The Jossey-Bass handbook of nonprofit leadership and management (2nd ed., pp. 81-101). San Francisco: Jossey-Bass.
62.
Snibbe, A.C. ( 2006). Drowning in data. Stanford Social Innovation Review. Retrieved October 30, 2007, from http://www.ssireview.org/images/articles/2006FA_feature_snibbe.pdf
63.
Steuerle, C.E., & Twombly, E.C. ( 2002). Vouchers. In L. M. Salamon (Ed.), The tools of government: A guide to the new governance (pp. 445-465). New York: Oxford University Press.
64.
Swanson, D.A., & McKibben, J.N. (1998). On teaching statistics to non-specialists: A course aimed at increasing both learning and retention . Paper presented at the International Association for Statistical Education conference in Singapore. Retrieved August 6, 2008, from http://www.stat.auckland.ac.nz/~iase/publications/2/Topic2g.pdf
65.
Thomas, C.T. ( 2005). Outcome assessment and program evaluation. In R. D. Herman & Associates (Eds.), The Jossey-Bass handbook of nonprofit leadership and management (2nd ed., pp. 391-418). San Francisco: Jossey-Bass.
66.
Tuan, M.T. ( 2004). Cultivating a culture of measurement. In Grantmakers for Effective Organizations (Ed.), Funding effectiveness: Lessons in building nonprofit capacity (pp. 105-142). San Francisco: Jossey-Bass.
67.
United Way of America. (1996). Measuring program outcomes: A practical approach._Alexandria, VA: Author.
68.
U.S. Government Accountability Office. (2005 , October). Program evaluation: OMB’s PART reviews increased agencies’ attention to improving evidence of program results. Retrieved July 19, 2008, from www.gao.gov/cgi-bin/getrpt?GAO-06-67
69.
Van Slyke, D.M. ( 2007). Agents or stewards: Using theory to understand the government-nonprofit social service contracting relationship. Journal of Public Administration Research and Theory, 17, 157-187.
70.
W. K. Kellogg Foundation. (n.d.). Evaluation handbook. Available from Collateral Management Company, 1255 Hill Brady Road, Battle Creek, MI 49015, item number 1203.
71.
W. K. Kellogg Foundation. (2004). Logic model development guide. Retrieved September 15, 2007, from http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf
72.
Walker, G., & Grossman, J. ( 1999). Philanthropy and outcomes. In C. T. Clotfelter & T. Erlich (Eds.), Philanthropy and the nonprofit sector in a changing America (pp. 449-460). Bloomington: Indiana University Press.
73.
Weiss, C.H. ( 1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families . In J. P. Connell, A. C. Kubish, L. B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives, Vol. 1: Concepts, methods, and contexts (pp. 65-92). Queenstown, MD: The Aspen Institute, Roundtable on Comprehensive Community Initiatives for Children and Families.
74.
Werner, A. ( 2004). A guide to implementation research. Washington, DC: Urban Institute Press.
75.
Wong, H.K. ( 2007). Social services: A case study of Hong Kong consumerism and quality standards. International Journal of Public Administration , 30, 441-458.
