Material Incentives and Other Potential Factors Associated With Response Rates to Internet Surveys of American Evaluation Association Members: Findings From a Randomized Experiment
Restricted accessNoteFirst published online June, 2020
Material Incentives and Other Potential Factors Associated With Response Rates to Internet Surveys of American Evaluation Association Members: Findings From a Randomized Experiment
Internet surveys of American Evaluation Association (AEA) members are a common method for studying evaluation practice. Response rates obtained from Internet surveys of AEA members are, however, frequently very small. To investigate whether or not material incentives increase response rates to Internet surveys of AEA members, a between-subjects three-treatment and one control randomized experiment in which a randomly selected sample of AEA members were randomly assigned to a no-incentive control condition, lottery condition, token incentive condition, or philanthropic donation incentive condition was utilized. The overall response rate to the survey was 39.66% and the response rates for each of the four conditions were control = 36.24%, lottery = 44.39%, token incentive = 43.28%, and philanthropic donation = 34.67%, respectively. The cost-effectiveness of each of the four conditions also was examined, demonstrating that the lottery was the most cost-effective. Other factors potentially influencing response or nonresponse decisions also are discussed.
ArmstrongJ. S. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39, 111–116.
2.
AzzamT.SzanyiM. (2011). Designing evaluations: A study examining preferred evaluation designs. Studies in Educational Evaluation, 37, 134–143.
3.
BosnjakM.TutenT. L. (2003). Prepaid and promised incentives in Web surveys: An experiment. Social Science Computer Review, 21, 208–217.
4.
BoulianneS. (2008). Incentives. In LavrakasP. J. (Ed.), Encyclopedia of survey research methods (Vol. 1, pp. 326–331). Thousand Oaks, CA: Sage.
5.
ChurchA. H. (1993). Estimating the effect of incentives on mail survey rates: A meta-analysis. Public Opinion Quarterly, 57, 62–79.
6.
ConverseP. D.WolfeE.HuangX.OswaldF. (2008). Response rates for mixed-mode surveys using mail and e-mail/web. American Journal of Evaluation, 29, 99–107.
7.
CookC.HeathF.ThompsonR. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60, 821–836.
8.
CorynC. L. S.OzekiS.WilsonL. N.Greenman IIG. D.SchröterD. C.HobsonK. A.…VoA. T. (2016). Does research on evaluation matter? Findings from a survey of American evaluation association members and prominent evaluation theorists and scholars. American Journal of Evaluation, 37, 159–173.
9.
CorynC. L. S.WilsonL. N.WestineC. D.HobsonK. A.OzekiS.FiekowskyE. L.…SchröterD. C. (2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38, 329–347.
10.
DillmanD. A.SmythJ. D.ChristianL. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, NJ: Wiley.
11.
DeutskensE.de RuyterK.WetzelsM.OosterveldP. (2004). Response rate and response quality of Internet-based surveys: An experimental study. Marketing Letters, 15, 21–36.
12.
FanW.YanZ. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26, 132–139.
13.
FleischerD. N.ChristieC. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30, 158–175.
14.
FoxR. J.CraskM. R.KimJ. (1988). Mail survey response rate—A meta-analysis of selected techniques for inducing response. Public Opinion Quarterly, 52, 467–491.
15.
GalportN.AzzamT. (2017). Evaluator training needs and competencies: A gap analysis. American Journal of Evaluation, 38, 80–100.
16.
GalportM.GalportN. (2015). Methodological trends in research on evaluation. New Directions for Evaluation, 148, 17–29.
17.
GöritzA. S. (2006). Incentives in Web studies: Methodological issues and a review. International Journal of Internet Science, 1, 58–70.
18.
GreenlawC.Brown-WeltyS. (2009). A comparison of web-based and paper-based survey methods: Testing assumptions of survey mode and response cost. Evaluation Review, 33, 464–480.
19.
GugiuP. C.CorynC. L. S.ApplegateE. B. (2010). Structure and measurement properties of the patient assessment of chronic illness care instrument. Journal of Evaluation in Clinical Practice, 16, 509–516.
20.
HenryG. T.MarkM. M. (2003). Toward an agenda for research on evaluation. New Directions for Evaluation, 97.
21.
LevyP. S.LemeshowS. (2013). Sampling of populations: Methods and applications (4th ed.). Hoboken, NJ: Wiley.
22.
LewisN. R.HarrisonG. M.Ah SamA. F.BrandonP. R. (2015). Evaluators’ perspectives on research on evaluation. New Directions for Evaluation, 148, 89–102.
23.
MarkM. M. (2007). Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In SmithN. L.BrandonP. R. (Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guilford.
24.
OzekiS. (2016). Three empirical investigations into the logic of evaluation and valuing practices (Unpublished doctoral dissertation).Western Michigan University, Kalamazoo.
25.
PreskillH.CaracelliV. (1997). Current and developing conceptions of use: Evaluation Use TIG survey results. American Journal of Evaluation, 18, 209–225.
26.
RogelbergS. G. (2006). Understanding nonresponse and facilitating response to organizational surveys. In KrautA. I. (Ed.), Getting action from organizational surveys: New concepts, methods, and applications (pp. 312–325). San Francisco, CA: Jossey-Bass.
27.
SeidlingM. B. (2015). Evaluator certification and credentialing revisited: A survey of American Evaluation Association members in the United States. New Directions for Evaluation, 145, 87–102.
28.
SeifertT. (2008). Leverage-salience theory. In LavrakasP. J. (Ed.), Encyclopedia of survey research methods (Vol. 1, pp. 423–425). Thousand Oaks, CA: Sage.
29.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
30.
SheehanK.McMillanS. (1999). Response variation in e-mail surveys: An exploration. Journal of Advertising Research, 39, 45–54.
31.
SingerE.Van HoewykJ.GeblerN.RaghunathanT.McGonagleK. (1999). The effect of incentives on response rates in interviewer-mediated surveys. Journal of Official Statistics, 15, 217–230.
32.
SzanyiM.AzzamT.GalenM. (2013). Research on evaluation: A needs assessment. Canadian Journal of Program Evaluation, 27, 39–64.
33.
YuJ.CooperH. (1983). A quantitative review of research design effects on response rates to questionnaires. Journal of Marketing Research, 20, 36–44.
