When evaluations are broadly disseminated, the public can use them to support a program or to advocate for change.
Methods:
To explore how evaluations are perceived and used by the public, individuals in a sample of 425 people in the United States were recruited through an online crowdsourcing service called Mechanical Turk (www.mturk.com). Participants were randomly assigned to receive different versions of a press release describing a summative evaluation of a program. Each condition contained a unique combination of methods (e.g., randomized controlled design) and findings (positive or negative) to describe the evaluation and its findings. Participants in each condition responded to questions about their trust in the content of the evaluation findings and their attitudes toward the program.
Results:
Results indicated that the type of evaluation methods and the direction of the findings both influenced the credibility of the findings and that the credibility of the findings moderated the relationship between the direction of the evaluation findings and attitudes toward the evaluated program. Additional evaluation factors to explore in future research with the public are recommended.
AzzamT.JacobsonM. R. (2013). Finding a comparison group: Is online crowdsourcing a viable option?American Journal of Evaluation, 34, 372–384. doi:10.1177/1098214013490223
2.
AzzamT.LevineB. (2015). Politics in evaluation: Politically responsive evaluation in high stakes environments. Evaluation and Program Planning, 53, 44–56. doi:10.1016/j.evalprogplan.2015.07.002
3.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N. (2011). The viability of crowdsourcing for survey research. Behavioral Research Methods, 43, 800–813. doi:10.3758/s13428-011-0081-0
4.
BoardmanA. G.ArguellesM. E.VaughnS.HughesM. T.KlingnerJ. (2005). Special education teachers’ views of research-based practices. The Journal of Special Education, 39, 168–180. doi:10.1177/00224669050390030401
5.
BrandonP. R.FukunagaL. L. (2014). The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation, 35, 26–44. doi:10.1177/1098214013503699
6.
BrandonP. R.SinghJ. M. (2009). The strength of the methodological warrants for the findings of research on program evaluation use. American Journal of Evaluation, 30, 123–157. doi:10.1177/1098214009334507
7.
BraskampL. A.BrownR. D.NewmanD. L. (1982). Studying evaluation utilization through simulations. Evaluation Review, 6, 114–126.
8.
BridgesF.AppelL.GrossklagsJ. (2012). Young adults’ online participation behaviors: An exploratory study of Web 2.0 use for political engagement. Information Polity, 17, 163–176. doi:10.3233/IP-2012-0271
9.
BuhrmesterM.KwangT.GoslingS. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980
10.
BushawW. J.CalderonV. J. (2014a). Try it again, Uncle Sam: The 46th Annual PDK/Gallup Poll of the public’s attitudes toward the public schools. Phi Delta Kappan, 96, 8–20. doi:10.1177/0031721714547856
11.
BushawW. J.CalderonV. J. (2014b). Americans put teacher quality on center stage: The 46th annual PDK/Gallup poll of the public’s attitudes toward the public schools: Part II. Phi Delta Kappan, 96, 49–59. doi:10.1177/0031721714553411
12.
CarsonR.T. (2012). Contingent valuation: A practical alternative when prices aren’t available. Journal of Economic Perspectives, 26, 27–42. doi:10.1257/jep.26.4.27
13.
ChristieC.A. (2007). Reported influence of evaluation data on decision makers’ actions: An empirical examination. American Journal of Evaluation, 28, 8–25.doi:10.1177/1098214006298065
14.
CousinsJ. B.ChouinardJ. A. (2012). Participatory evaluation up close: An integration of research-based knowledge. Charlotte, NC: Information Age.
15.
CousinsJ. B.LeithwoodK. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56, 331–364.
16.
CranoW. D.BrewerM. D. (2002). Principles and methods of social research (2nd ed.). New York, NY: Taylor and Francis.
17.
CreswellJ. W.Plano ClarkV. L. (2011). Designing and conducting mixed method research. Thousand Oaks, CA: Sage.
18.
DagenaisC.LysenkoL.AbramiP. C.BernardR. M.RamdeJ.JanoszM. (2012). Use of research-based information by school practitioners and determinants of use: A review of empirical research. Evidence & Policy, 8, 285–309.doi:/10.1332/174426412X654031
19.
DaigneaultP. M. (2014). Taking stock of four decades of quantitative research on stakeholder participation and evaluation use: A systematic map. Evaluation and Program Planning, 45, 171–181. doi:10.1016/j.evalprogplan.2014.04.003
20.
DaigneaultP. M.JacobS. (2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30, 330–348. doi:10.1177/1098214009340580
21.
DattaL. (2007). Looking at the evidence: What variations in practice might indicate. New Directions for Evaluation, 113, 35–54. doi:10.1002/ev.214
22.
FinniganK. S.DalyA. J.CheJ. (2013). Systemwide reform in districts under pressure: The role of social networks in defining, acquiring, using, and diffusing research evidence. Journal of Educational Administration, 51, 476–497. doi:10.1108/09578231311325668
23.
FlemingM. A. (2011). Applying social psychology to increase evaluation use. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 212–243). New York, NY: The Guilford Press.
24.
GuskeyT. R. (2007). Multiple sources of evidence: An analysis of stakeholders’ perceptions of various indicators of student learning. Educational Measurement: Issues and Practice, 26, 19–27.
25.
HenryG. T. (2009). When getting it right matters: The case for high-quality policy and program impact evaluations. In DonaldsonS. I.ChristieC. A.MarkM. M. (Eds.), What counts as credible evidence in applied research and evaluation practice? (pp. 32–50). Thousand Oaks, CA: Sage.
26.
HenryG. T.MarkM. M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293–314.
27.
HilligossB.RiehS. (2008). Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, 44, 1467–1484. doi:10.1016/j.ipm.2007.10.001
28.
HitchmanS. C.DriezenP.LogelC.HammondD.FongG. T. (2014). Changes ineffectiveness of cigarette health warnings over time in Canada and the United States, 2002-2011. Nicotine & Tobacco Research, 16, 536–543. doi:10.1093/ntr/ntt196
29.
HonigM. I.CoburnC. (2008). Evidence-based decision making in school district central offices: Toward a policy and research agenda. Educational Policy, 22, 578–608. doi:10.1177/0895904807307067
30.
House. (1980). Evaluating with validity. Beverly Hills, CA: Sage.
31.
HurteauM.HouleS.MongiatS. (2009). How legitimate and justified are judgments in program evaluation?Evaluation, 15, 307–319. doi:10.1177/1356389009105883
32.
JabbarH.GoelP.LondeL. A.DebrayE.LubienskiC. (2014). How policymakers define “evidence”: The politics of research use in New Orleans. Policy Futures in Education, 12, 1013–1027.
33.
JohnsonK.GreenseidL. O.ToalS. A.KingJ. A.LawrenzF.VolkovB. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30, 377–410. doi:10.1177/1098214009341660
34.
KennedyM. M. (1999). A test of some common contentions about educational research. American Educational Research Journal, 36, 511–541. doi:10.3102/00028312036003511
35.
LittleT. D.CardN. A.BovairdJ. A.PreacherK. J.CrandallC. S. (2007). Structural equation modeling of mediation and moderation with contextual factors. In LittleT. D.BovairdJ. A.CardN. A. (Eds.), Modeling Contextual Effects in Longitudinal Studies (pp. 207–230). doi:10.1198/tech.2006.s369
36.
LoboscoA. F.NewmanD. L. (1992). Stakeholder information needs: Implications for evaluation practice and policy development in early childhood special education. Evaluation Review, 16, 443–463.
37.
MarkM. M. (2015). Credible and actionable evidence: A framework, overview, and suggestions for future practice and research. In DonaldsonS. I.ChristieC. A.MarkM. M. (Eds.), Credible and actionable evidence: The foundation for rigorous and influential evaluations (pp. 275–302). Thousand Oaks, CA: Sage.
38.
MarkM. M.HenryG. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35–57. doi:10.1177/1356389004042326
39.
MetzgerM. J.FlanaginA. J.MeddersR. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60, 413–439. doi:10.1111/j.1460-2466.2010.01488.x
40.
MillerR. L. (2015). How people judge the credibility of information: Lessons for cognitive and information sciences. In DonaldsonS. I.ChristieC. A.MarkM. M. (Eds.), Credible and actionable evidence: The foundation for rigorous and influential evaluations (pp. 39–62). Thousand Oaks, CA: Sage.
41.
NelsonS. R.LefflerJ. C.HansenB. A. (2009). Towards a research agenda for understanding and improving the use of research evidence. Portland, OR: Northwest Regional Educational Laboratory. Retrieved fromhttp://educationnorthwest.org/webfm_send/311
42.
NewmanD. L.BrownR. D.BraskampL. A. (1980). Communication theory and the utilization of evaluation. New Directions for Program Evaluation, 5, 29–35.
PeerE.VosgerauJ.AcquistiA. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. doi:10.3758/s13428-013-0434-y
45.
PetrosinoA.BirkelandS.HacsiT. A.Murphy-GrahamE.WeissC. H. (2006). US state government and DARE: The story in four states. Evidence & Policy, 2, 291–319.
46.
PreacherK. J.RuckerD. D.HayesA. F. (2007). Addressing moderated mediation hypotheses: Theory, methods, and prescriptions. Multivariate Behavioral Research, 42, 185–227. doi:10.1080/00273170701341316
47.
RobertsC. (2010). Correlation among variables in message and messenger credibility scales. American Behavioral Scientist, 54, 43–56. doi:10.1177/0002764210376310
ScottJ.LubienskiC.DeBrayE.JabbarH. (2014). The intermediary function in evidence production, promotion, and utilization: The case of educational incentives. In FinniganK. S.DalyA. J. (Eds.), Using research evidence in education: From the schoolhouse door to Capitol Hill (pp. 69–92). New York, NY: Springer.
50.
ShadishW. R. (2011). The truth about validity. New Directions for Evaluation, 130, 107–117. doi:10.1002/ev
51.
SimonsD. J.ChabrisC. F. (2012). Common (mis)beliefs about memory: A replication and comparison of telephone and Mechanical Turk survey methods. PLoS One, 7, 1–5. doi:10.1371/journal.pone.0051876
52.
ThayerC. E.FineA. H. (2001). Evaluation and outcome measurement in the non-profit sector: Stakeholder participation. Evaluation and Program Planning, 24, 103–108.
53.
TsengV. (2012). The uses of research in policy and practice. Social Policy Report, 26, 1–23.
WeinbergJ.FreeseJ.McElhattanD. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310. doi:10.15195/v1.a19
58.
WeissC. H.BucuvalasM. J. (1980). Truth tests and utility tests: Decision-makers’ frames of reference for social science research. American Sociological Review, 45, 302–313.
59.
WeissC. H.Murphy-GrahamE.PetrosinoA.GandhiA. G. (2008). The fairy godmother—and her warts: Making the dream of evidence-based policy come true. American Journal of Evaluation, 29, 29–47. doi:10.1177/1098214007313742
