Randomized Experiments and Reality of Public and Nonprofit Organizations: Understanding and Bridging the Gap

Abstract

This article aims to contribute to the methodological advancement of experimental public administration, a nascent and promising literature stream. The article discusses the assumptions behind the theory of experimentation and the consequences of their violation; the main types of experimental designs (i.e., lab experiments, artefactual, framed, and natural field experiments); discrete choice experiments, neglected so far in our discipline for no good reason; the computation of optimal sample sizes; and the procedures for dealing with noncompliance in field experiments. The article concludes by providing tips to help public administration scholars bridge the gap between randomized trials and reality.

Keywords

randomized control trials experiments experimental designs external validity discrete choice experiment optimal sample size noncompliance

Get full access to this article

View all access options for this article.

References

Andersen

S. C.

Hjortskov

(2016). Cognitive biases in performance evaluations. Journal of Public Administration Research and Theory, 26, 647-662. doi:10.1093/jopart/muv036

Anderson

D. M.

Edwards

B. C.

(2015). Unfulfilled promise: Laboratory experiments in public management research. Public Management Review, 17, 1518-1542.

Angrist

J. D.

Pischke

J. S.

(2014). Mastering metrics: The path from cause to effect. Princeton, NJ :Princeton University Press.

Baekgaard

Baethge

Blom-Hansen

Dunlop

C. A.

Esteve

Jakobsen

. . . Wolf

P. J.

(2015). Conducting experiments in public management research: A practical guide. International Public Management Journal, 18, 323-342.

Bandiera

Barankay

Rasul

(2005). Social preferences and the response to incentives: Evidence from personnel data. The Quarterly Journal of Economics, 120, 917-962.

Belle

(2013). Experimental evidence on the relationship between public service motivation and job performance. Public Administration Review, 73, 143-153.

Belle

(2014). Leading to make a difference: A field experiment on the performance effects of transformational leadership, perceived social impact, and public service motivation. Journal of Public Administration Research and Theory, 24, 109-136.

Belle

(2015). Performance-related pay and the crowding out of motivation in the public sector: A randomized field experiment. Public Administration Review, 75, 230-241.

Belle

Cantarelli

(2015). Monetary incentives, motivation, and job effort in the public sector: An experimental study with Italian government executives. Review of Public Personnel Administration, 35, 99-123.

10.

Benz

Meier

(2008). Do people behave in experiments as in the field? Evidence from donations. Experimental Economics, 11, 268-281.

11.

Blom-Hansen

Morton

Serritzlew

(2015). Experiments in public management research. International Public Management Journal, 18, 151-170.

12.

Bouwman

Grimmelikhuijsen

(2016). Experimental public administration from 1992 to 2014: A systematic literature review and ways forward. International Journal of Public Sector Management, 29, 110-131.

13.

Braun

F. G.

(2010). Determinants of households’ space heating type: A discrete choice analysis for German households. Energy Policy, 38, 5493-5503.

14.

Breustedt

Müller-Scheeßel

Latacz-Lohmann

(2008). Forecasting the adoption of GM oilseed rape: Evidence from a discrete choice experiment in Germany. Journal of Agricultural Economics, 59, 237-256.

15.

Brewer

G. A.

Brewer

G. A.

(2011). Parsing public/private differences in work motivation and performance: An experimental study. Journal of Public Administration Research and Theory, 21(Suppl. 3), i347-i362.

16.

Dal Bó

Finan

Rossi

M. A

. (2013). Strengthening state capabilities: The role of financial incentives in the call to public service. The Quarterly Journal of Economics, 128, 1169-1218.

17.

de Bekker-Grob

E. W.

Ryan

Gerard

. (2012). Discrete choice experiments in health economics: A review of the literature. Health Economics, 21, 145-172.

18.

Doty

R. L.

Silverthorne

(1975). Influence of menstrual cycle on volunteering behaviour. Nature, 254, 139-140.

19.

Fehr

List

J. A.

(2004). The hidden costs and returns of incentives—Trust and trustworthiness among CEOs. Journal of the European Economic Association, 2, 743-771.

20.

Fryer

R. G.

Jr. Levitt

S. D.

List

Sadoff

(2012). Enhancing the efficacy of teacher incentives through loss aversion: A field experiment (No. w18237). Cambridge, MA: National Bureau of Economic Research.

21.

George

Desmidt

Nielsen

P. A.

Baekgaard

(2016). Rational planning and politicians’ preferences for spending and reform: Replication and extension of a survey experiment. Public Management Review, 19, 1251-1271.

22.

Gerber

A. S.

Green

D. P.

(2012). Field experiments: Design, analysis, and interpretation. New York City, NY: W. W. Norton.

23.

Grant

A. M.

(2008). The significance of task significance: Job performance effects, relational mechanisms, and boundary conditions. Journal of Applied Psychology, 93, 108-124.

24.

Grant

A. M.

Wall

T. D.

(2009). The neglected science and art of quasi-experimentation: Why-to, when-to, and how-to advice for organizational researchers. Organizational Research Methods, 12, 653-686.

25.

Grimmelikhuijsen

S. G.

Jilke

Olsen

A. L.

Tummers

L. G.

(2016). Behavioral public administration: Combining insights from public administration and psychology. Public Administration Review, 77, 45-56. doi:10.1111/puar.12609

26.

Grimmelikhuijsen

Klijn

(2015). The effects of judicial transparency on public trust: Evidence from a field experiment. Public Administration, 93, 995-1011.

27.

Guthrie

R. V.

(2004). Even the rat was white: A historical view of psychology. Upper Saddle River, NJ. Pearson Education.

28.

Hackbarth

Madlener

(2013). Consumer preferences for alternative fuel vehicles: A discrete choice analysis. Transportation Research Part D: Transport and Environment, 25, 5-17.

29.

Harrison

G. W.

List

J. A.

(2004). Field Experiments. Journal of Economic Literature, 42, 1009-1055.

30.

Hatry

H. P.

Winnie

R. E.

Fisk

D. M.

(1981). Practical program evaluation for state and local governments. Washington, DC: Urban Institute Press

31.

Henshel

R. L.

(1980). The purposes of laboratory experimentation and the virtues of deliberate artificiality. Journal of Experimental Social Psychology, 16, 466-478.

32.

Holt

C. A.

(2007). Markets, games, & strategic behavior: Recipes for interactive learning. Boston, MA:Pearson Addison Wesley.

33.

Imbens

G. W.

Angrist

J. D.

(1994). Identification and estimation of local average treatment effects. Econometrica, 62, 467-475.

34.

James

Jilke

Van Ryzin

(2017). Experiments in public administration research: Challenges and opportunities. Cambridge, UK: Cambridge University Press.

35.

Jilke

S. G.

Petrovsky

Meuleman

James

(2016). Measurement equivalence in replications of experiments: When and why it matters and guidance on how to determine equivalence. Public Management Review, 19, 1293-1310.

36.

Jilke

Van

Walle

Kim

(2016). Generating Usable Knowledge through an Experimental Approach to Public Administration. Public Administration Review, 76, 69-72.

37.

Jilke

Van Ryzin

G. G.

Van

Walle

(2015). Responses to decline in marketized public services: An experimental evaluation of choice overload. Journal of Public Administration Research and Theory, 26, 421-432.

38.

Kramer

M. S.

Shapiro

S. H.

(1984). Scientific challenges in the application of randomized trials. The Journal of the American Medical Association, 252, 2739-2745.

39.

Levitt

S. D.

List

J. A.

(2007). What do laboratory experiments measuring social preferences reveal about the real world? The Journal of Economic Perspectives, 21, 153-174.

40.

List

J. A.

(2007). Field experiments: A bridge between lab and naturally occurring data. The BE Journal of Economic Analysis & Policy, 5(2), 1-47.

41.

List

J. A.

(2011). Why economists should conduct field experiments and 14 tips for pulling one off. The Journal of Economic Perspectives, 25, 3-15.

42.

List

J. A.

Sadoff

Wagner

(2011). So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design. Experimental Economics, 14, 439-457.

43.

Margetts

H. Z.

(2011). Experiments for public management research. Public Management Review, 13, 189-208.

44.

Newton

(1687). Principia mathematica ( Motte

Ed.). London, England. (Original work published 1729)

45.

Olsen

A. L.

(2015). Citizen (dis) satisfaction: An experimental equivalence framing study. Public Administration Review, 75, 469-478.

46.

Orne

M. T.

(1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776-783.

47.

Paier

Scherngell

(2011). Determinants of collaboration in European R&D networks: Empirical evidence from a discrete choice model. Industry and Innovation, 18, 89-104.

48.

Perry

J. L.

(2012). How can we improve our science to generate more usable knowledge for public professionals? Public Administration Review, 72, 479-482.

49.

Roethlisberger

F. J.

Dickson

W. J.

(1939). Management and the worker. Cambridge, MA: Harvard University Press.

50.

Rosenberg

M. J.

(1969). The conditions and consequences of evaluation apprehension. In Rosenthal

Rosnow

R. L.

(Eds.), Artifact in behavioral research (pp. 280-351). Oxford, UK.

51.

Ryan

Kolstad

Rockers

Dolea

(2012). How to conduct a discrete choice experiment for health workforce recruitment and retention in remote and rural areas: A user guide with case studies. World Health Organization, CapacityPlus, & World Bank. Geneca, Switzerland.

52.

Samuelson

P. A.

Nordhaus

W. D.

(1985). Economics (12th ed.). New York, NY: McGraw-Hill.

53.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. United Kingdom. Wadsworth Cengage Learning.

54.

Street

D. J.

Burgess

Louviere

J. J.

(2005). Quick and easy choice sets: Constructing optimal and nearly optimal stated choice experiments. International Journal of Research Marketing, 22, 459-470.

55.

Ziegler

(2012). Individual characteristics and stated preferences for alternative energy sources and propulsion technologies in vehicles: A discrete choice analysis for Germany. Transportation Research Part A: Policy and Practice, 46, 1372-1385.