With the exponential increase in Web access, program evaluators need to understand the methodological benefits and barriers of using the Web to collect survey data from program participants. In this experimental study, the authors examined whether a Web survey can be as effective as the more established mail survey on three measures of survey effectiveness: response rate, question completion rate, and the lack of evaluative bias. Community- and university-based educators (n = 274) attending a 2-day program were randomly assigned to receive a Web or mail survey evaluating the program. Among those participants successfully solicited by e-mail, Web survey participants were more likely to respond (95%) than mail survey participants (79%). Web survey participants completed similarly high numbers of quantitative questions as mail survey participants, provided longer and more substantive responses to qualitative questions, and did not demonstrate evidence of evaluative bias. These results suggest that program evaluators could expand their use of Web surveys among computer users.
Archer, T. M. (2003). Web-based surveys. Journal of Extension, 41(4), 1-5.
2.
Bachmann, D., Elfrink, J., & Vazzana, G. (1996). Tracking the progress of e-mail vs. snail-mail. Marketing Research, 8(2), 30-35.
3.
Booth-Kewley, S., Rosenfeld, P., & Edwards, J. E. (1993). Computer-administered surveys in organizational settings: Alternatives, advantages, and applications. In P. Rosenfeld, J. E. Edwards, & M. D. Thomas (Eds.), Improving organizational surveys: New directions, methods, and applications (pp. 73-101). Newbury Park, CA: Sage.
4.
Cobanoglu, C., Warde, B., & Moreo, P. J. (2001). A comparison of mail, fax and Web-based survey methods. International Journal of Market Research, 43(4), 441-452.
5.
Couper, M. P., Blair, J., & Triplett, T. (1999). A comparison of mail and e-mail for a survey of employees in U. S. statistical agencies. Journal of Official Statistics, 15(1), 39-56.
6.
Couper, M. P., Tourangeau, R., Conrad, F., & Crawford, S. D. (2004). What they see is what we get: Response options for Web surveys. Social Science Computer Review, 22(1), 111-127.
7.
Couper, M. P., Traugott, M. W., & Lamias, M. J. (2001). Web survey design & administration. Public Opinion Quarterly, 65, 230-253.
8.
Crawford, S. D., Couper, M. P., & Lamias, J. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19(2), 146-162.
9.
Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: John Wiley.
10.
Dillman, D. A., Eltinge, J. L., Groves, R. M., & Little, R. J. A. (2002). Survey nonresponse in design, data collection and analysis. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 3-26). New York: John Wiley.
11.
Goodfellow, M., Kiernan, N. E., Ahern, F., & Smyer, M. A. (1988). Response bias using two-stage data collection: A study of elderly participants in a program. Evaluation Review, 12(6), 638-654.
12.
Hosseini, J. C., & Armacost, R. L. (1993). Gathering sensitive data in organizations. In P. Rosenfeld, J. E. Edwards, & M. D. Thomas (Eds.), Improving organizational surveys: New directions, methods, and application (pp. 29-57). Newbury Park, CA: Sage.
13.
Horrigan, J. B., & Rainie, L. (2002). Getting serious online. Available at http://www.pewinternet.org/pdfs/PIP_Getting_Serious_Online3ng.pdf
14.
Hulley, S. B., Cummings, S. R., Browner, W. S., Grady, D., Hearst, N., & Newman, T. B. (2001). Designing clinical research. Philadelphia: Lippincott Williams & Wilkins.
15.
Kiernan, N. E. (1999). [Pilot study: Using the Web for program evaluation]. Unpublished data.
16.
McCabe, S. E., Boyd, C., Couper, M. P., Crawford, S., & d’Arcy, H. (2002). Mode effects for collecting alcohol & other drug use data: Web & U. S. mail. Journal of Studies on Alcohol, 63(6), 755-761.
17.
Mehta, R., & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of the Market Research Society, 37, 429-439.
18.
Morrel-Samuels, P. (2003). Web surveys’ hidden hazards: Companies replacing paper surveys with Web-based versions. Harvard Business Review, 81(7), 16-17.
19.
Nielsen, J. (2000). Designing Web usability. Indianapolis, IN: New Riders.
20.
Parker, L. (1992, July). Collecting data the e-mail way. Training and Development, pp. 52-54.
21.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.
22.
Pest Watch. (2004). Home page. Retrieved October 22, 2004, from http://www.pestwatch.psu.edu
23.
Porter S. R., & Whitcomb, J. E. (2003). The impact of contact type on Web survey response rates. Public Opinion Quarterly, 67(4), 579-588.
24.
The Rodale Institute Farm. (2004). 2004 field day and workshop survey. Retrieved October 22, 2004, from http://www.newfarm.org/surveys/ifafs2004a.htm
25.
Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage.
26.
Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62, 378-397.
27.
Singer, E., Mathiowetz, N. A., & Couper, M. P. (1993). The impact of privacy and confidentiality concerns on survey participation. Public Opinion Quarterly, 57, 465-482.
28.
Truell, A. D., Bartlett, J. E., & Alexander, M. W. (2002). Response rate, speed, & completeness: A comparison of Internet-based and mail surveys. Behavior Research Methods, Instruments, & Computers, 34(1), 50-54.
29.
Vehovar, V., Batagelj, Z., Manfreda, K. L., & Zaletel, M. (2002). Nonresponse in Web surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 229-242). New York: John Wiley.
30.
Watt, J. H. (1999). Internet systems for evaluation research. In G. Gay & T. L. Bennington (Eds.), Information technologies in evaluation: Social, moral, epistemological and practical implications (pp. 23-43). San Francisco: Jossey-Bass.
31.
Wright, D. L., Aquilino, W. S., & Supple, A. J. (1998). A comparison of computer-assisted and paper-and-pencil self-administered questionnaires in a survey on smoking, alcohol, and drug use. Public Opinion Quarterly, 62, 331-353.
