Although scholars have proposed many steps to increase evaluation use, there has been little comparative empirical study of whether researchers follow these recommendations and whether specific steps are associated with greater utilization. This study of state legislative evaluators finds that those that regularly meet with stakeholders and provide readily actionable products were considered by senior legislative staff to have more impact, as were larger offices. Legislative evaluators working in auditing organizations were viewed less favorably than those working in other units; this appears to be related to their adherence to Government Auditing Standards that prescribe organizational independence. Evaluators following these standards had less stakeholder engagement than did those working in other legislative units that adhered to research standards that stress meeting stakeholder needs. Environmental factors such as changes in party control may also play a role in how the work of evaluators is valued and used in the legislative process.
Alkin, M., Christie, C., & Rose, M. ( 2006). Communicating evaluation. In I. Shaw, J. Green, & M. Mark (Eds.), Sage handbook of evaluation-policies, programs, and practices (pp. 384-403). London, England: Sage.
2.
American Evaluation Association. (2004). Guiding principles for evaluators. Fairhaven, MAAuthor .
3.
Berry, F., Turcotte, J., & Latham, S. ( 2002). Program evaluation in state legislatures: professional services delivered in a complex, competitive environment. In R. Mohan, D. Bernstein , & M. Whitsett (Eds.), New directions for evaluation (No. 95, pp. 73-88). San Francisco: Jossey-Bass.
4.
Chelimsky, E. ( 2009). A clash of cultures-improving the ‘‘fit’’ between evaluative independence and the political requirements of a democratic society. American Journal of Evaluation, 29, 400-415.
5.
Cousins, J.B. ( 2003). Utilization effects of participatory evaluation. In T. Kelligan, & D. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 245-66). Dordrecht, the Netherlands: Kluwer Academic Press.
6.
Cousins, J.B. & Shulha, L. ( 2006). A comparative analysis of evaluation utilization and its cognate fields of inquiry: Current issues and trends. In I. F. Shaw, J. C. Greene, & M. Mark (Eds.), The sage handbook of evaluation: Policies, programs and practice (pp. 266-291). Thousand Oaks, CA: Sage.
7.
Haddock, R.E. ( 1998). State legislative program evaluation: An assessment of recent claims of direct utilization in the states. Dissertation Abstracts International: Section A: The Humanities and Social Sciences, 60, 881(UMI 0021400).
8.
Hird, J. ( 2005). Power, knowledge and politics. Washington, DC: Georgetown University Press.
9.
Jonas, K. ( 1999). Against the whim: State legislatures’ use of program evaluation. In K. Jonas (Ed.), Legislative program evaluation: Utilization-driven research for decision makers (pp. 3-22). San Francisco, CA: Jossey-Bass.
10.
Klein, P. ( 1999). New institutional economics. In B. Bouckeart, & G. DeGeest (Eds.), Encyclopedia of law and economics (pp. 456-489). Northampton, MA: Edward Elgar.
11.
Mark, M. & Henry, G. ( 2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293-314.
12.
Mark, M. & Henry, G. ( 2004). The mechanisms and outcomes of evaluation use. Evaluation, 10, 35-57.
13.
Moran, R. & Stutzman, M. ( 1999). The impact of term limits on legislative program evaluation . In K. Jonas (Ed.), Legislative program evaluation: Utilization-driven research for decision makers . New Directions for Evaluation (vol. 81, pp. 95-105). San Francisco: Jossey-Bass.
14.
Mooney, C. ( 1991). Information sources in state legislative decision making . Legislative Studies Quarterly, 16, 445-455.
15.
National Conference of State Legislatures. ( 2004). Ensuring the public trust 2004: Program policy evaluation’s role in serving state legislatures. Denver, CO: NCSL.
16.
National Conference of State Legislatures. ( 2009a). Legislative staff count. Denver, CO: NCSL.
17.
National Conference of State Legislatures. ( 2009b). Full and part-time legislatures. Denver, CO: NCSL.
18.
National Conference of State Legislatures. ( 2009c). Party competition in state legislatures. Denver, CO: NCSL.
19.
North, D. ( 1990). Institutions, institutional change and economic performance . Cambridge, England: Cambridge University Press.
20.
Patton, M. ( 2008). Utilization-focused evaluation. (4th ed.). Los Angeles, CA: Sage.
21.
Peck, L. & Gorzalski, L. ( 2009). An evaluation use framework and empirical assessment . Journal of Multidisciplinary Evaluation, 6, 139-156.
22.
Preskill, H. & Torres, R. ( 1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage.
23.
Radin, B. ( 2000). Beyond Machiavelli-policy analysis comes of age. Washington, DC: Georgetown University Press.
24.
Sabatier, P., & Jenkins-Smith, H. (1999). The advocacy coalition framework: An assessment. In P. Sabatier (Ed.), Theories of the Policy Process. Boulder, Co: Westview Press.
25.
Schneider, M., Scholz, J., Lubell, M., Mindruta, D., & Edwardsen, M. ( 2003). Building consensual institutions: Networks and the national estuary program. American Journal of Political Science, 47, 143-158.
26.
Shulock, N. ( 1999). The paradox of policy analysis: If it is not used, why do we produce so much of it? Journal of Policy Analysis and Management, 18, 226-244.
27.
Snow, C. ( 1961). Science and government. Cambridge, MA: Harvard University Press.
28.
The Joint Committee on Standards for Educational Evaluation , J. Sanders , Chair. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.
29.
U.S. Government Accountability Office. (2007 ). Government auditing standards (2007 Revision). Washington, DC: USGAO.
30.
VanLandingham, G.R. ( 2006). A voice crying in the wilderness: Legislative oversight agencies efforts to achieve utilization. In Mohan & Sullivan (Eds.), Promoting the use of government evaluations in policymaking. New Directions for Evaluation (vol. 112). Danvers, MA: Jossey-Bass.
31.
Weiss, C. ( 1973). Where politics and evaluation research meet. Evaluation Practice, 14, 93-106. (Original work published 1973).
32.
Weiss, C. ( 1977). Introduction. In C. Weiss (Ed.), Using social research in public policy making (pp. 1-22). Lexington, MA: D.C. Heath.
33.
Weiss, C. ( 1989). Congressional committees as users of analysis. Journal of Policy Analysis and Management, 8, 411-431.
34.
Whiteman, D. ( 1995). Communication in congress: Members, staff, and the search for information. Lawrence, KS: University Press of Kansas.
35.
Wholey, J., Scanlon, J., Duffy, H., Fukumortu, J., & Vogt, L. ( 1970). Federal evaluation policy: Analyzing the effects of public programs. Washington, DC: Urban Institute.
36.
Zajano, N., & Lochtefeld, S. (1999). The nature of knowledge and language in the legislative arena. In K. Jonas (Ed.), Legislative program evaluation: Utilization-driven research for decision makers (pp. 3-22). San Francisco: Jossey-Bass.
