Sage Journals: Discover world-class research

Abstract

A simulation study was conducted in an attempt to examine how evaluators modify their evaluation design in response to differing stakeholder groups. In this study, evaluators were provided with a fictitious description of a school-based program. They were then asked to design an evaluation of the program. After the evaluation design decisions were made, evaluators were presented with feedback from three differing stakeholder groups (i.e., decision maker, implementer, recipient) either endorsing or rejecting the evaluation design. Evaluators were then given the opportunity to modify (or not modify) their original design in response to stakeholder feedback. The findings revealed that the more political power or influence stakeholder groups held over evaluation logistical factors (i.e., funding, data access), the more evaluators were willing to modify their design choices to accommodate perceived stakeholder concerns. These design modifications were typically implemented to ensure data access, reduce stakeholder resistance, and increase stakeholder buy-in.

Keywords

evaluation simulation stakeholder involvement responsiveness experimental design

Get full access to this article

View all access options for this article.

References

Alkin, M.C. , Daillak, R. , & White, B. ( 1979). Using evaluations. Beverly Hills, CA: SAGE.

Boruch, R.F. ( 1997). Randomized experiments for planning and evaluation: a practical guide. Thousand Oaks, CA: SAGE.

Braskamp, L.A. ( 1980). Credibility of a local education program evaluation report: author source and client characteristics. American Educational Research Journal, 15, 441-450.

Brown, R.D. , Braskamp, L.A. , & Newman, D.L. ( 1978). Evaluator credibility and acceptance as a function of report styles: Do jargon and data make a difference? Evaluation Q, 2, 331-341.

Brown, R.D. , Braskamp, L.A. , & Newman, D.L. ( 1982a). Studying evaluation utilization through simulations . Evaluation Review, 6, 114-126.

Brown, R.D. , & Newman, D.L. ( 1982b). An investigation of the effects of different data presentation formats and order of arguments in a simulated adversary evaluation. Education Evaluation and Policy Analysis, 4, 197-204.

Brown, R.D. , Newman, D.L. , & Rivers, L.S. ( 1985). Does the Superintendent’s opinion affect School Boards’ evaluation information needs? An Empirical Investigation. Urban Education, 20, 204-221.

Cannon, H.M. , & Burns, A.C. ( 1999). A framework for assessing the competencies reflected in simulation performance. Developments in Business Simulation & Experiential Exercises, 26, 40-44.

Chelimsky, E. ( 1987). The politics of program evaluation. Society, 25, 24-32.

10.

Christie, C. ( 2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. New Directions in Evaluation , 97, 7-37.

11.

Cronbach, L.J. , Ambron, S.R. , Dornbusch, S.M. , Hess, R.D. , Hornik, R.C. , Phillips, D.C. , et al. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.

12.

Eastmond, N. ( 1998). When funders want to compromise your design. American Journal of Evaluation, 19, 392-395.

13.

Fetterman, D.M. ( 2001). Foundations of empowerment evaluation. Thousand Oaks, CA: SAGE.

14.

Granville, A. ( 1977). Where do school decisions really come from? Sources of influence and the influence of evaluation. Unpublished doctoral dissertation, University of California, Los Angeles, California, United States.

15.

Greene, J.C. ( 1990). Technical quality vs. responsiveness in evaluation practice . Evaluation and Program Planning, 13, 267-274.

16.

Joint Committee on Standards for Educational Evaluation. (1994 ). The Program Evaluation Standards (2nd ed.). Thousand Oaks, CA: SAGE .

17.

Kirk, R.E. ( 1995). Experimental design: Procedures for behavioral sciences (3rd ed.). Belmont, CA: Wads-worth Publishing.

18.

Mathison, S. ( 1991). Role conflicts for internal evaluators. Evaluation and Program Planning, 14, 173-179.

19.

Patton, M.Q. ( 1997). Utilization-focused evaluation: the new century text (3rd ed.). Thousand Oaks, CA : SAGE.

20.

Patton, M.Q. , Patricia, G. , Guthrie, K. , Brennan, N. , French, B. , & Blyth, D. ( 1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using Social Research in Public Policy Making (pp. 141-164). Lexington, MA: D. C. Health.

21.

O’Neil, H.F. , Allred, K. , & Dennis, R.A. ( 1997). Validation of a computer simulation for assessment of interpersonal skills. In H. F. O’Neil Jr (Ed.), Workforce readiness: Competencies and assessment (pp. 229-254). Mahwah, NJ: Lawrence Erlbaum Associates.

22.

Scriven, M. ( 1967). The methodology of evaluation. In R. W. Tyler , R. W. Gagne , & M. Scriven (Eds.), The American Educational Research Association Monograph Series on Curriculum Evaluation, Vol.1: Perspectives of Curriculum Evaluation (pp.39-83). Rand McNally.

23.

Smith, H.W. , Winer, J.L. , & George, C.E. ( 1983). The relative efficacy of simulation experiments. Journal of Vocational Behavior, 22, 96-104.

24.

Ward, E. ( 1954). The theory of decision making. Psychological Bulletin, 51, 38.

25.

Weiss, C.H. ( 1978). Improving the linkage between social research and public policy. In L. E. Lynn (Ed.), Knowledge and policy: The uncertain connection (pp. 23-81). Washington, DC: National Academy of Sciences.

Evaluator Responsiveness to Stakeholders

Abstract

Keywords

Get full access to this article

References