“You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation

Abstract

Practical-participatory evaluation is often considered an effective approach to enhancing evaluation use. Despite this generally positive connotation, potential negative effects of participatory evaluation have also been discussed in the literature but never empirically investigated before. Drawing upon the social–psychological procedural justice literature, we assumed that stakeholders’ perceptions about the justice of an evaluation might explain possible negative effects of participatory evaluation on evaluation use. Negative effects of participatory evaluation on instrumental and conceptual use were explored in two experiments (Study 1: N = 85; Study 2: N = 78). The results show that negative effects of participation on evaluation use are mediated by stakeholders’ justice perceptions. Implications of the present research include considering stakeholders’ socioemotional needs, negotiation, and transparency when conducting participatory evaluation.

Keywords

participatory evaluation negative effects procedural justice experimental approach

Get full access to this article

View all access options for this article.

References

Aguinis

Bradley

K. J.

(2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17, 351–371. doi:10.1177/1094428114547952

Ajzen

(1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211. doi:10.1016/0749-5978(91)90020-T

Alkin

M. C.

King

J. A.

(2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 76, 1–17. doi:10.1177/1098214017717015

Alkin

M. C.

Taut

S. M.

(2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1–12. doi:10.1016/S0191-491X(03)90001-0

Blader

S. L.

Tyler

T. R.

(2015). Relational models of procedural justice. In Cropanzano

R. S.

Ambrose

M. L.

(Eds.), The Oxford handbook of justice in the workplace (pp. 351–370). New York, NY: Oxford University Press.

Borenstein

(2009). Effect sizes for continuous data. In Cooper

H. M.

Hedges

L. V.

Valentine

J. C.

(Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–237). New York, NY: Russell Sage Foundation.

Brandon

P. R.

(1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337. doi:10.1177/109821409801900305

Brandon

P. R.

Fukunaga

L. L.

(2014). The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation, 35, 26–44. doi:10.1177/1098214013503699

Brockner

Fishman

A. Y.

Reb

Goldman

Spiegel

Garden

(2007). Procedural fairness, outcome favorability, and judgments of an authority’s responsibility. Journal of Applied Psychology, 92, 1657–1671. doi:10.1037/0021-9010.92.6.1657

10.

Cohen

R. L.

(1985). Procedural justice and participation. Human Relations, 38, 643–663. doi:10.1177/001872678503800703

11.

Cohen-Charash

Mueller

J. S.

(2007). Does perceived unfairness exacerbate or mitigate interpersonal counterproductive work behaviors related to envy? Journal of Applied Psychology, 92, 666–680. doi:10.1037/0021-9010.92.3.666

12.

Cousins

J. B.

(1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3–27. doi:10.1016/0191-491x(96)00001-6

13.

Cousins

J. B.

Chouinard

J. A.

(2012). Participatory evaluation up close: An integration of research-based knowledge. Greenwich, England: Information Age.

14.

Cousins

J. B.

Donohue

J. J.

Bloom

G. A.

(1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices and consequences. American Journal of Evaluation, 17, 207–226. doi:10.1177/109821409601700302

15.

Cousins

J. B.

Whitmore

(1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23. doi:10.1002/ev.1114

16.

Cropanzano

R. S.

Rupp

D. E.

Thornton

M. A.

Shao

(2016). Organizational justice and organizational citizenship behavior. In Podsakoff

P. M.

Mackenzie

S. B.

Podsakoff

N. P.

(Eds.), The Oxford handbook of organizational citizenship behavior. Oxford, England: Oxford University Press.

17.

Cullen

A. E.

Coryn

C. L. S.

Rugh

(2011). The politics and consequences of including stakeholders in international development evaluation. American Journal of Evaluation, 32, 345–361. doi:10.1177/1098214010396076

18.

Daigneault

P. M.

(2014). Taking stock of four decades of quantitative research on stakeholder participation and evaluation use: A systematic map. Evaluation and Program Planning, 45, 171–181. doi:10.1016/j.evalprogplan.2014.04.003

19.

de Cremer

Tyler

T. R.

(2005). Am I respected or not? Inclusion and reputation as issues in group membership. Social Justice Research, 18, 121–153. doi:10.1007/s11211-005-7366-3

20.

Deutsch

(1949). A theory of cooperation and competition. Human Relations, 2, 129–152. doi:10.1177/001872674900200204

21.

Fishbein

Triandis

H. C.

Kanfer

F. H.

Becker

Middlestadt

S. E.

Eichler

(2001). Factors influencing behavior and behavior change. In Baum

A. S.

Revenson

T. A.

Singer

J. E.

(Eds.), Handbook of health psychology (pp. 3–17). Mahwah, NJ: Lawrence Erlbaum.

22.

Fleischer

D. N.

Christie

C. A.

(2009). Evaluation use: Results from a survey of U.S. American evaluation association members. American Journal of Evaluation, 30, 158–175. doi:10.1177/1098214008331009

23.

Gastic

Irby

D. J.

Zdanis

(2008). When stakeholders rebel: Lessons from a safe schools program. Studies in Educational Evaluation, 34, 208–211. doi:10.1016/j.stueduc.2008.10.004

24.

Goodwin

Ross

(1992). Consumer responses to service failures: Influence of procedural and interactional fairness perceptions. Journal of Business Research, 25, 149–163. doi:10.1016/0148-2963(92)90014-3

25.

Greenberg

Folger

R. G.

(1983). Procedural justice, participation, and the fair process effect in groups and organizations. In Paulus

P. B.

(Ed.), Springer series in social psychology: Basic group processes (pp. 235–256). New York, NY: Springer.

26.

Greene

J. C.

(1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10, 379–394. doi:10.1016/0149-7189(87)90010-3

27.

Greene

J. C.

(1988). Communication of results and utilization on participatory program evaluation. Evaluation and Program Planning, 11, 341–351. doi:10.1016/0149-7189(88)90047-x

28.

Gregory

(2000). Problematizing participation: A critical review of approaches to participation in evaluation theory. Evaluation, 6, 179–199. doi:10.1177/13563890022209208

29.

Groves

Guijt

(2015). Positioning Participation on the Power Spectrum. Retrieved from http://www.betterevaluation.org/en/blog/positioning_participation_on_the_power_spectrum

30.

Hayes

A. F.

(2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY: Guilford Press.

31.

Hayes

A. F.

Preacher

K. J.

(2014). Statistical mediation analysis with a multicategorical independent variable. The British Journal of Mathematical and Statistical Psychology, 67, 451–470. doi:10.1111/bmsp.12028

32.

Henry

G. T.

Mark

M. M.

(2003). Beyond Use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293–314. doi:10.1177/109821400302400302

33.

Jacob

Ouvrard

Bélanger

J. F.

(2011). Participatory evaluation and process use within a social aid organization for at-risk families and youth. Evaluation and Program Planning, 34, 113–123. doi:10.1016/j.evalprogplan.2010.08.002

34.

Johnson

D. W.

Greenseid

L. O.

Toal

S. A.

King

J. A.

Lawrenz

Volkov

(2009). Research on evaluation use. A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30, 377–410. doi:10.1177/1098214009341660

35.

Johnson

D. W.

Johnson

R. T.

(1989). Cooperation and competition: Theory and research. Edina, MN: Interaction Book.

36.

Johnson

D. W.

Johnson

R. T.

Stevahn

(2011). Social interdependence and program evaluation. In Mark

M. M.

Donaldson

S. I.

Campbell

(Eds.), Social psychology and evaluation (pp. 288–317). New York, NY: Guilford Press.

37.

King

J. A.

Stevahn

(2013). Interactive evaluation practice: Mastering the interpersonal dynamics of program evaluation. Los Angeles, CA: Sage.

38.

Kirkhart

K. E.

(2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 2000, 5–23. doi:10.1002/ev.1188

39.

Lakens

(2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 1–12. doi:10.3389/fpsyg.2013.00863

40.

Lennie

(2005). An evaluation capacity building process for sustainable community IT initiatives. Empowering and disempowering impacts. Evaluation, 11, 390–414. doi:10.1177/1356389005059382

41.

Leviton

L. C.

Hughes

E. F.

(1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review, 5, 525–548. doi:10.1177/0193841X8100500405

42.

Lind

E. A.

Kanfer

Earley

P. C.

(1990). Voice, control, and procedural justice: Instrumental and noninstrumental concerns in fairness judgments. Journal of Personality and Social Psychology, 59, 952–959. doi:10.1037/0022-3514.59.5.952

43.

Lind

E. A.

Tyler

T. R.

(1988). The social psychology of procedural justice. New York, NY: Plenum Press.

44.

Mathie

Greene

J. C.

(1997). Stakeholder participation in evaluation: How important is diversity? Evaluation and Program Planning, 20, 279–285. doi:10.1016/S0149-7189(97)00006-2

45.

Mark

M. M.

Henry

G. T.

(2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35–57. doi:10.1177/1356389004042326

46.

Mark

M. M.

Shotland

R. L.

(1985). Stakeholder-based evaluation and value judgments. Evaluation Review, 9, 605–626. doi:10.1177/0193841X8500900504

47.

McFarlin

D. B.

Sweeney

P. D.

(1992). Distributive and procedural justice as predictors of satisfaction with personal and organizational outcomes. Academy of Management Journal, 35, 626–637. doi:10.2307/256489

48.

Mertens

D. M.

(2009). Transformative research and evaluation. New York, NY: Guilford Press.

49.

Moorman

R. H.

Byrne

Z. S.

(2005). How does justice affect organizational citizenship behavior? In Greenberg

Colquitt

J. A.

(Eds.), Handbook of organizational justice (pp. 355–380). Mahwah, NJ: Lawrence Erlbaum Associates.

50.

Oelerich

(2013). Evaluation Schulsozialarbeit im Rahmen des Bildungs- und Teilhabepakets (BuT) in Wuppertal [School social work evaluation in the context of the education and participation package in Wuppertal]. Retrieved from http://www.sozpaed.uni-wuppertal.de/fileadmin/sozpaed/Abschlussbericht_Evaluation_Schulsozialarbeit_BuT_Wuppertal-Oelerich-9.2013.pdf

51.

Patton

M. Q.

(2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

52.

Poth

C. A.

Shulha

(2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34, 218–223. doi:10.1016/j.stueduc.2008.10.006

53.

Potter

P. W.

(2006a). Procedural justice and voice effects. Journal of Organizational Culture, Communications and Conflict, 10, 33–62.

54.

Potter

P. W.

(2006b). Procedural justice, voice effects and sham: Examining the decision maker from a research context perspective. Journal of Organizational Culture, Communications and Conflict, 10, 61–76.

55.

Preskill

Caracelli

(1997). Current and developing conceptions of use: Evaluation use TIG survey results. American Journal of Evaluation, 18, 209–225. doi:10.1016/s0886-1633(97)90028-3

56.

Rebien

C. C.

(1996). Participatory evaluation of development assistance: Dealing with power and facilitative learning. Evaluation, 2, 151–171. doi:10.1177/135638909600200203

57.

Rebolloso

Fernández-Ramírez

Cantón

(2005). The influence of evaluation on changing management systems in educational institutions. Evaluation, 11, 463–479. doi:10.1177/1356389005060263

58.

Rodríguez-Campos

(2005). Collaborative evaluations: A step-by-step model for the evaluator. Tamarac, FL: Llumina Press.

59.

Roseland

Lawrenz

Thao

(2015). The relationship between involvement in and use of evaluation in multi-site evaluations. Evaluation and Program Planning, 48, 75–82. doi:10.1016/j.evalprogplan.2014.10.003

60.

Rucker

D. D.

Preacher

K. J.

Tormala

Z. L.

Petty

R. E.

(2011). Mediation analysis in social psychology: Current practices and new recommendations. Social and Personality Psychology Compass, 5, 359–371. doi:10.1111/j.1751-9004.2011.00355.x

61.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

62.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

63.

Shulha

Whitmore

Cousins

J. B.

Gilbert

Al Hudib

(2015). Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation, 37, 193–217. doi:10.1177/1098214015615230

64.

Speck

(2006). Qualität und Evaluation in der Schulsozialarbeit: Konzepte, Rahmenbedingungen und Wirkungen [Quality and evaluation in school social work: Concepts, general conditions and impacts] . Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.

65.

Sullins

C. D.

(2003). Adapting the empowerment evaluation model: A mental health drop-in centre case example. American Journal of Evaluation, 24, 387–398. doi:10.1177/109821400302400307

66.

Taut

(2007). Methodological and conceptional challenges in studying evaluation process use. Canadian Journal of Program Evaluation, 22, 1–19.

67.

Taut

(2008). What have we learned about stakeholder involvement in program evaluation? Studies in Educational Evaluation, 34, 224–230. doi:10.1016/j.stueduc.2008.10.007

68.

Taut

Brauns

(2003). Resistance to evaluation: A psychological perspective. Evaluation, 9, 247–264. doi:10.1177/13563890030093002

69.

Turnbull

(1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131–140. doi:10.1016/s0149-7189(99)00012-9

70.

Tyler

T. R.

Blader

S. L.

(2003). The group engagement model: Procedural justice, social identity, and cooperative behavior. Personality and Social Psychology Review, 7, 349–361. doi:10.1207/S15327957PSPR0704_07

71.

van den Bos

Lind

E. A.

Vermunt

Wilke

H. A. M.

(1997). How do I judge my outcome when I do not know the outcome of others? The psychology of the fair process effect. Journal of Personality and Social Psychology, 72, 1034–1046. doi:10.1037/0022-3514.72.5.1034

72.

van den Bos

Wilke

H. A. M.

Vermunt

Lind

E. A.

(1998). Evaluating outcomes by means of the fair process effect: Evidence for different processes in fairness and satisfaction judgements. Journal of Personality and Social Psychology, 74, 1493–1503. doi:10.1037//0022-3514.74.6.1493

73.

Weiss

C. H.

(1986). The stakeholder approach to evaluation: Origins and promise. In House

E. R.

(Ed.), New directions for educational evaluation (pp. 145–157). Abingdon, England: Routledge Falmer.

74.

Whitmore

(1998). “We need to rebuild this house”: The role of empowerment in evaluation of a Mexican Farmers’ cooperative. In Jackson

E. T.

Kassam

(Eds.), Knowledge shared: Participatory evaluation in development cooperation (pp. 217–230). West Hartford, CT: Kumarian Press.

75.

Zhao

Lynch

J. G.

Jr Chen

(2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37, 197–206. doi:10.1086/651257

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB