Stakeholder Participation for the Purpose of Helping Ensure Evaluation Validity: Bridging the Gap Between Collaborative and Non-collaborative Evaluations

Abstract

Collaborative evaluations, in which program stakeholders participate extensively, typically are conducted for the primary purpose of enhancing the use of evaluation findings, and non-collaborative evaluations, in which stakeholders do not participate extensively, typically are conducted for the primary purpose of generating valid findings. This article shows how the gap can be bridged between these two types of evaluations. The article synthesizes, and elaborates on, a small body of recent research that showed that considerable interaction with stakeholders during the evaluation of small education programs helped achieve the raison d’etre of non-collaborative evaluations-that is, to enhance validity. It is also shown that theories and methods of stakeholder participation for the purpose of enhancing validity, which by and large have been ignored in the collaborative evaluation literature, can improve the quality of collaborative studies.

Get full access to this article

View all access options for this article.

References

Bacharach, S. B. , & Lawler, E. J. (1980). Power and politics in organizations. San Francisco: Jossey-Bass.

Brandon, P. R. (in press). Involving program stakeholders in reviews of evaluators’ recommendations for program revisions. Evaluation and Program Planning.

Brandon, P. R. , and Higa, T. A. F. (1998, April). Setting standards to use when judging program performance in stakeholder-assisted evaluations of small educational programs. Paper presented at the meeting of the American Educational Research Association, San Diego, CA.

Brandon, P. R. , Lindberg, M. A. , & Wang, Z. (1993). Involving program beneficiaries in the early stages of evaluation: Issues of consequential validity and influence. Educational Evaluation and Policy Analysis, 15, 420-428.

Brandon, P. R. , Newton, B. J. , & Harman, J. W. (1993). Enhancing validity through beneficiaries’ equitable involvement in identifying and prioritizing homeless children’s educational problems. Evaluation and Program Planning, 16.287-293.

Bryk, A. (1983). (Ed.), Stakeholder-based evaluation: New directions for program evaluation 17. San Francisco: Jossey-Bass.

Carey, M. A. , & Smith, M. W. (1992). Enhancement of validity through qualitative approaches: Incorporating the patient’s perspectives. Evaluation and the Health Professions, 15(4), 107-114.

Conrad, K. J. , & Conrad, K. M. (1994). Reassessing validity threats in experiments: Focus on construct validity. In K. J. Conrad (Ed.), Critically evaluating the role of experiments: New Directions for Program Evaluation 63 (pp. 5-25). San Francisco: Jossey-Bass.

Cook, T. D. (1991). Clarifying the warrant for generalized causal inferences in quasi-experimentation. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century. Ninetieth yearbook of the National Society for the Study of Education, Purt II (pp. 115-144). Chicago: National Society for the Study of Education.

10.

Cousins, J. B. , Donohue, J. J. , & Bloom, G. A. (1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices and consequences. Evaluation Practice, 17, 207-226.

11.

Cousins, J. B. , Donohue, J. J. , & Bloom, G. A. (1997, November). Dimensions of form of collaborative evaluation us predictors of utilization. Paper presented at the annual meeting of the American Evaluation Association, San Diego, CA.

12.

Cousins, J. B. , & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397-418.

13.

Cousins, J. B. , & Earl, L. M. (Eds.). (1995). Participatory evaluation in education: Studies in evaluation use and organizational learning. Washington, D.C.: Falmer.

14.

Cousins, J. B. , & Whitmore, E. (1997, May). Framing participatory evaluation. Paper presented at the annual meeting of the Canadian Evaluation Society, Ottawa.

15.

Dawson, J. A. , & D’Amico, J. J. (1985). Involving program staff in evaluation studies: A strategy for increasing information use and enriching the data base. Evaluation Review, 9, 173-188.

16.

Deutsch, S. J. , & Malmborg, C. J. (1986). A study on the consistency of stakeholder preferences for different types of information in evaluating police services. Evaluation and Program Planning, 9, 13-24.

17.

Donmoyer, R. (1990). Curriculum evaluation and the negotiation of meaning. Language Arts, 67, 274-286.

18.

Edwards, E. L. , Guttentag, M. , & Snapper, K. (1975). A decision-theoretic approach to evaluation research. In E. L. Struening & M. Guttentag (Eds.), Handbook of evaluation research (pp. 139-181). Beverly Hills, CA: Sage.

19.

Glaser, R. , & Chi., M. T. H. (1988). Introduction: What is it to be an expert? In M. T. H. Chi , R. Glaser , & M. J. Fatr (Eds.), The nature of expertise (pp. xv-xxiix). Hillsdale, NJ: Lawrence Erlbaum.

20.

Greene, J. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10, 379-394.

21.

Greene, J. C. (1990). Technical quality versus user responsiveness in evaluation practice. Evaluation and Program Planning, 13, 267-274.

22.

Henry, G. T. , Dickey, K. A. , & Areson, J. C. (1991). Stakeholder participation in educational performance monitoring systems. Educational Evaluation and Policy Analysis, 13, 177-188.

23.

Hoffman, R. R. (1994). (Ed.), The psychology of expertise: Cognitive research and empirical AI. New York: Springer-Verlag.

24.

House, E. R. (1990). Trends in evaluation. Educational Researcher, 19(3), 24-28.

25.

Huberman, M , & Cox, P. (1990). Evaluation utilization: Building links between action and reflection. Studies in Educational Evaluation, 16, 157-179.

26.

Joint Committee on Standards for Educational Evaluation . (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.

27.

Lawrence, J. E. S. , & Cook, T. J. (1982). Designing useful evaluations: The stakeholder survey. Evaluation and Program Planning, 5, 327-336.

28.

Mercurio, J. A. (1979.) Community involvement in cooperative decision making: Some lessons learned. Educational Evaluation and Policy Analysis, I(6), 37-46.

29.

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.) (pp. 13-103). New York: American Council on Education/Macmillan.

30.

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

31.

Mulder, M. , & Wilke, H. (1970). Participation and power equalization. Organizational Behavior and Human Performance, 5, 430-448.

32.

O’Neill, T. (1995). Implementation frailties of Cuba and Lincoln’s fourth generation evaluation theory. Studies in Educational Evaluation, 21, 5-21.

33.

Owston, R. D. (1986). Establishing validity in evaluation: The Canadian Indian school evaluations. Evaluation and Program Planning, 9, 319-323.

34.

Perry, P. D. , & Backus, C. A. (1995). A different perspective on empowerment in evaluation: Benefits and risks to the evaluation process. Evaluation Practice, 16, 37-46.

35.

Rogers, W. T. (1995). The treatment of measurement issues in the revised Program Evaluation Standards. Journal of Experimental Education, 61, 13-28.

36.

Striven, M. (1997). Empowerment evaluation examined. Evaluation Practice, 18, 165-175.

37.

Shanteau, J. (1988). Psychological characteristics and strategies of expert decision makers. Acta Psychologica, 68, 203-215.

38.

Shepard, L. A. (1993). Evaluating test validity. In L. Darling-Hammond (Ed.), Review of research in education (Vol. 19, pp. 405-450). Washington, DC: American Educational Research Association.

39.

Smith, N. L. (1983). Citizen involvement in evaluation: Empirical studies. Studies in Educational Evaluation, 9, 105-117.

40.

Smith, N. L. (1995). The influence of societal games on the methodology of evaluative inquiry. In D. M. Fournier (Ed.), New Directions for Evaluation 44 (pp. 5-14). San Francisco: Jossey-Bass.

41.

Smith, N. L. (1997, November). An investigative framework for characterizing evaluation practice. In N. L. Smith (Chair), Examining Evaluation Practice. Symposium conducted at the meeting of the American Evaluation Association, San Diego, CA.

42.

Tjosvold, D. (1987). Participation: A close look at its dynamics. Journal of Management, 13, 739-750.

43.

Torres, R. T. , Preskill, H. S. , & Piontek, M. E. (1996). Evaluation strategies for communication and reporting: Enhancing learning in organizations. Thousand Oaks, CA: Sage.

44.

Tovar, M. (1989). Representing multiple perspectives: Collaborative-democratic evaluation in distance education. The American Journal of Distance Education, 3(2), 44-56.

45.

Trochim, W. M. K. , & Linton, R. (1986). Conceptualization for planning and evaluation. Evaluation and Program Planning, 9, 289-308.

46.

Wood, C. J. (1989). Challenging the assumptions underlying the use of participatory decision-making strategies. Small Group Behavior, 20, 428-448.