Sage Journals: Discover world-class research

Abstract

To begin statistical analysis, Bayesians quantify their confidence in modeling hypotheses with priors. A prior describes the probability of a certain modeling hypothesis apart from the data. Bayesians should be able to defend their choice of prior to a skeptical audience. Collaboration between evaluators and stakeholders could make their choices more defensible. This article describes how evaluators and stakeholders could combine their expertise to select rigorous priors for analysis. The article first introduces Bayesian testing, then situates it within a collaborative framework, and finally illustrates the method with a real example.

Keywords

Bayesian statistics collaborative evaluation Bayes’s factor posterior odds ratios

Get full access to this article

View all access options for this article.

References

Algozzine

Morsbach Sweeney

Choi

Horner

Sailor

McCart

… Lane

(2014). SWIFT fidelity implementation tool: Development and preliminary technical adequacy. Lawrence: University of Kansas, SWIFT Center.

Azzam

(2010). Evaluator responsiveness to stakeholders. American Journal of Evaluation, 31, 45–65.

Berger

J. O.

(1985). Statistical decision theory and Bayesian analysis (2nd ed.). New York, NY: Springer.

Bolstad

W. M.

(2007). Introduction to Bayesian statistics. Hoboken, NJ: John Wiley.

Brandon

P. R.

(1989). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337.

Christensen

Johnson

Branscum

Hanson

T. E.

(2011). Bayesian ideas and data analysis. New York, NY: CRC Press.

Cousins

J. B.

Earl

L. M.

(1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397–418.

Fetterman

D. M.

(2001). The transformation of evaluation into a collaboration: A vision of evaluation in the 21st century. American Journal of Evaluation, 22, 381–385.

Fetterman

D. M.

Rodriguez-Campos

Wandersman

O’Sullivan

(2014). Collaborative, participatory and empowerment evaluation: Building a strong conceptual foundation for stakeholder involvement approaches to evaluation. (A response to Cousins, Whitmore and Shulha, 2013) [Letters to the Editor]. American Journal of Evaluation, 35, 144–148.

10.

Gelman

Shalizi

C. R.

(2013). Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology, 66, 8–38.

11.

Gill

Walker

L. D.

(2005). Elicited priors for Bayesian model specifications in political science research. Journal of Politics, 67, 841–872.

12.

Hacking

(2001). An introduction to probability and inductive logic. New York, NY: Cambridge University Press.

13.

Hawthorne

(1994). On the nature of Bayesian convergence. Philosophy of Science, 1, 241–249.

14.

Hicks

(2015). What you know counts: How to elicit priors from experts to improve quantitative analysis with qualitative analysis (Doctoral dissertation). Retrieved from http://scholarcommons.usf.edu/etd/5493

15.

Hoff

P. D.

(2009). A first course in Bayesian statistical methods. New York, NY: Springer.

16.

House

E. R.

Howe

K. R.

(1998). The issue of advocacy in evaluations. American Journal of Evaluation, 19, 233–236.

17.

Howson

Urbach

(2006). Scientific reasoning: The Bayesian approach (3rd ed.). Chicago, IL: Open Court.

18.

Jaynes

E. T.

(1968). Prior probabilities. IEEE Transactions of System Science and Cybernetics, 4, 227–241.

19.

Jeffreys

(1946, 9). An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 186, 453–461. doi:10.1098/rspa.1946.0056

20.

Kass

R. E.

Raftery

A. E.

(1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.

21.

Kass

R. E.

Wasserman

(1996). The selection of prior distributions by formal rules. Journal of American Statistical Association, 91, 1343–1370.

22.

Keynes

(1923). A track on monetary reform. Toronto, Canada: MacMillan.

23.

Kruschke

J. K.

(2011). Doing Bayesian data analysis: A tutorial with R and BUGS. New York, NY: Elsevier.

24.

McCart

A. B.

Sailor

W. S.

Bezdek

J. M.

Satter

A. L.

(2014). A framework for inclusive educational delivery systems. Inclusion, 2, 252–264. doi:10.1352/2326-6988-2.4.252

25.

Orr

S. T.

(2010). Exploring stakeholder values and interests in evaluation. American Journal of Evaluation, 31, 557–569.

26.

Patton

M. Q.

(1978). Utilization-focused evaluation. Thousand Oaks, CA: Sage.

27.

Pollard

W. E.

(1986). Bayesian statistics for evaluation research. Thousand Oaks, CA: Sage.

28.

Rodríguez-Campos

(2012). Stakeholder involvement in evaluation: Three decades of the American journal of evaluation. Journal of Multidisciplinary Evaluation, 8, 57–59.

29.

Rodríguez-Campos

Rincones-Gomez

(2012). Collaborative evaluations: Step-by-step (2nd ed.). Stanford, CA: Stanford University Press.

30.

Rouder

J. N.

Speckman

P. L.

Sun

Morey

R. D.

Iverson

(2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychological Bulletin Review, 16, 225–237.

31.

Smith

M. F.

(1999). Participatory evaluation: Not working or not tested? American Journal of Evaluation, 20, 295–308.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB