Firm-level data from executives are necessary for testing many theories in organizational science. To date, researchers who have used survey methods to get data from executives have relied on response-rate techniques that were validated in general public, customer-, or employee-level populations. In a factorial field experiment, we manipulated the four most strongly supported of these techniques: incentives (gifts), advance notice, follow-up, and personalization. We also examined firm size as a potential moderator of their effects. In addition, we tested whether these response-rate enhancements affected the psychometric quality (acquiescence, self-presentation bias) of the executive survey data. Despite high statistical power(n = 1,200, 1-β≥.95), none of the conventional response-rate enhancement techniques were effective. Small influences on response acquiescence were found but these were confined to questions that arguably had the strongest demand cues. We conclude that distinctly different techniques may be needed to generate interest and involvement among executives targeted by firm-level mail surveys, and we provide some suggestions.
Get full access to this article
View all access options for this article.
References
1.
Aguinis, H.1995. Statistical power problems with moderated multiple regression in management research. Journal of Management, 21: 1141-1158.
2.
Amason, A. C.1996. Distinguishing the effects of functional and dysfunctional conflict on strategic decision making: Resolving a paradox for top management teams. Academy of Management Journal, 39: 123-148.
3.
Andrews, K. R.1971. The concept of corporate strategy. Homewood, IL: Dow-Jones-Irwin.
4.
Baruch, Y.1999. Response rate in academic studies—a comparative analysis. Human Relations, 52(4): 421-438.
5.
Bean, V. L., & Medewitz, J. N.1988. Mail questionnaires: Some methodological considerations. Journal of Data Collection, 28(1): 50-52.
6.
Blau, P. M., & Schoenherr, R. A.1971. The structure of organizations. New York: Basic Books.
7.
Boyd, B. K., Dess, G. G., & Rasheed, M. A.1993. Divergence between archival and perceptual measures of environment: Causes and consequences. Academy of Management Review, 18: 204-226.
8.
Brennan, M., & Hoek, J.1992. The behavior of respondents, nonrespondents, and refusers across mail surveys. Public Opinion Quarterly, 56: 530-535.
9.
Brown, C. M.1999. Information seeking behavior of scientists in the electronic information age: Astronomers, chemists, mathematicians, and physicists. Journal of the American Society for Information Science, 50(10): 929-943.
10.
Browne, M. W., & Cudeck, R.1993. Alternative ways of assessing model fit. In Bollen, K. A., Long, J. S. (Eds.), Testing structural equation models. Newbury Park, CA: Sage Publications.
11.
Brush, C. G.1992. Marketplace information scanning activities of new manufacturing ventures. Journal of Small Business Management, 30(4): 41-53.
12.
Busenitz, L. W., & Barney, J. B.1997. Differences between entrepreneurs and managers in large organizations: Biases and heuristics in strategic decision-making. Journal of Business Venturing, 12(1): 9-30.
13.
Bush, A. J., Bush, V., & Harris, S.1998. Advertiser perception of the Internet as a marketing communications tool. Journal of Advertising Research, 38(2): 17-27.
14.
Chen, M. J., Farh, J. L., & MacMillan, I. C.1993. An exploration of expertness of outside informants. Academy of Management Journal, 36(6): 1614-1632.
15.
Child, J.1997. Strategic choice in the analysis of action, structure, organizations and environment: Retrospect and prospect. Organizational Studies, 18(1): 43-76.
16.
Chow, G.1960. Test of equality between sets of coefficients in two linear regressions. Econometrica, 28: 591-604.
17.
Church, A. H.1993. Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57: 62-79.
18.
Cohen, J.1988. Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lauwrence Erlbaum.
19.
D’Aveni, R. A.1995. Hypercompetitive rivalries. New York: Free Press.
20.
Delery, J. E., & Doty, D. H.1996. Modes of theorizing in strategic human resource management: Tests of universalistic, contingency, and configurational performance predictions. Academy of Management Journal, 39(4): 802-835.
21.
Dess, G. G., Newport, S., & Rasheed, A.1993. Configurations in strategic management research: Key issues and suggestions. Journal of Management, 19: 775-795.
22.
Dillman, D. A.1978. Mail and telephone surveys: The total design method. New York: Wiley.
23.
Eisenhardt, K. M.1989. Making fast strategic decisions in high-velocity environments. Academy of Management Journal, 32: 543-576.
24.
Fabrigar, L. R., MacCallum, R. C., Wegener, D. T., & Strahan, E. J.1999. Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3): 272-299.
25.
Farmer, J., Richardson, A., & Lawton, S.1999. Improving access to information for nursing staff in remote areas: The potential of the Internet and other networked information resources. International Journal of Information Management, 19(1): 49-62.
26.
Fox, R. J., Crask, M. R., & Kim, J.1988. Mail survey response rates: A meta-analysis of selected techniques for inducing responses. Public Opinion Quarterly, 52: 467-491.
27.
Gendall, P., Hoek, J., & Brennan, M.1998. The tea bag experiment: More evidence on incentives in mail surveys. Journal of the Market Research Society, 40(4): 347-351.
28.
Haggett, S., & Mitchell, V. W.1994. Effect of industrial prenotification on response rate, speed, quality, bias, and cost. Industrial Marketing Management, 23(2): 101-110.
29.
Haleblian, J., & Finkelstein, S.1987. Top management team size, CEO dominance, and firm performance: The moderating role of environmental turbulence and discretion. Academy of Management Journal, 36: 844-863.
30.
Harrigan, K. R.1983. Research methodologies or contingency approaches to business strategy. Academy of Management Review, 8: 398-405.
31.
Hambrick, D. C., Gelekanycz, M. A., & Frederickson, J. W.1993. Top executive commitment to status quo: Some tests of its determinants. Strategic Management Journal, 14(6): 401-418.
32.
Hambrick, D. C., & Mason, P. A.1984. Upper echelons: The organization as a reflection of its top managers. Academy of Management Review, 9(2): 193-206.
33.
Harrison, D. A.2001. Structure and timing in limited range dependent variables: Regression models for predicting if and when. In Schmitt, N., Drasgow, F. (Eds.), Frontiers of industrial and organizational psychology: Advances in measurement and data analysis. San Francisco, CA: Jossey-Bass.
34.
Heberlein, T. A., & Baumgartner, R.1978. Factors affecting response rates to mailed questionnaires: A quantitative analysis of published literature. American Sociological Review, 43: 447-462.
35.
Hitt, M. A., Hoskisson, R. E., & Kim, H.1997. International diversification: Effects on innovation and firm performance in product-diversified firms. Academy of Management Journal, 40(4): 767-798.
36.
Hopkins, K. D., & Gullickson, A. R.1993. Response rates in survey research: A meta-analysis of the effects of monetary gratuities. Journal of Experimental Education, 61: 52-62.
37.
Hosmer, D. U., & Lemeshaw, S.1989. Applied logistic regression. New York: Wiley.
38.
Hu, L., Bentler, P. M., & Kano, Y.1992. Can test statistics in covariance structure analysis be trusted?Psychological Bulletin, 112: 351-362.
39.
Humphreys, L. G., & Montanelli, R. G., Jr.1975. An investigation of the parallel analysis criterion for determining the number of common factors. Multivariate Behavioral Research, 10: 193-205.
40.
Huselid, M. A.1995. The impact of human resource management practices on turnover, productivity, and corporate financial performance. Academy of Management Journal, 38: 635-672.
41.
Huselid, M. A., Jackson, S. E., & Schuler, R. S.1997. Technical and strategic human resource management effectiveness as determinants of firm performance. Academy of Management Journal, 40(1): 171-188.
42.
James, J. M., & Bolstein, R.1990. The effect of monetary incentives and follow-up mailings on response rate and response quality in mail surveys. Public Opinion Quarterly, 54(3): 346-361.
43.
Jarillo, C. J.1989. Entrepreneurship and growth: The strategic use of external resources. Journal of Business Venturing, 4: 133-147.
44.
Koch, M. J., & McGrath, R. G.1996. Improving labor productivity: Human resource management policies do matter. Strategic Management Journal, 17(5): 335-354.
45.
Kumar, K., Subramanian, R., & Yauger, C.1998. Examining the market orientation-performance relationship: A context-specific study. Journal of Management, 24(2): 201-233.
46.
Kumar, N., Stern, L. W., & Anderson, J. C.1993. Conducting interorganizational research using key informants. Academy of Management Journal, 36(6): 1633-1651.
47.
Lane, P. J., & Lubatkin, M.1998. Relative absorptive capacity and interorganizational learning. Strategic Management Journal, 19(5): 461-488.
48.
McArdle, J. J.1996. Current directions in structural factor analysis. Current Directions in Psychological Science, 4: 11-18.
49.
Markides, C. C., & Williamson, P. J.1996. Corporate diversification and organizational structure: A resource-based view. Academy of Management Journal, 39(2): 340-367.
50.
Martin, W. S., Duncan, W. J., Powers, T. L., & Sawyer, J. C.1989. Costs and benefits of selected response inducement techniques in mail survey research. Journal of Business Research, 19: 67-79.
51.
Melone, N. P.1994. Reasoning in the executive suite: The influence of role/experience-based expertise on decision processes of corporate executives. Organization Science, 5: 438-455.
52.
Miller, C. C., Burke, L. M., & Glick, W. H.1998. Cognitive diversity among upper-echelon executives: Implications for strategic decision processes. Strategic Management Journal, 19(1): 39-58.
53.
Miller, D., & Droge, C.1986. Psychological and traditional determinants of structure. Administrative Science Quarterly, 31(4): 539-560.
54.
Miller, D., & Shamsie, J.1996. The resource-based view of the firm in two environments: The Hollywood film studios from 1936 to 1965. Academy of Management Journal, 40: 534-559.
55.
Milliken, F. J., Martin, L. L., & Morgan, H.1998. Explaining organizational responsiveness to work-family issues The role of human resource executives as issue interpreters. Academy of Management Journal, 41(5): 580-592.
56.
Mintzberg, H.1973. The nature of managerial work. London: Prentice-Hall.
57.
Norburn, D.1989. The chief executive: A breed apart. Strategic Management Journal, 10: 1-15.
58.
Patton, M. Q.1990. Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage Publication.
59.
Paxson, M. C.1995. Increasing survey response rates: Practical instructions from the total-design method. Cornell Hotel and Restaurant Administration Quarterly, 36(4): 66-73.
60.
Podsakoff, P. M., & Organ, D. W.1986. Self-reports in organizational research: Problems and prospects. Journal of Management, 12(4): 531-544.
61.
Priem, R. L., Lyons, D. W., & Dess, G. G.1999. Inherent limitations of demographic proxies in top management teams heterogeneity research. Journal of Management, 25(6): 935-953.
62.
Roth, P. L., & BeVier, C. A.1998. Response rates in HRM/OB survey research: Norms and correlates, 1990-1994. Journal of Management, 24(1): 97-117.
63.
Sawka, K.1996. Demystifying business intelligence. Management Review, 85(10): 47-51.
64.
Schaefer, D. R., & Dillman, D. A.1998. Development of a standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62(3): 378-397.
65.
Schlegelmilch, B. B., & Diamantopoulos, A.1991. Prenotification and mail survey response rates: A quantitative integration of the literature. Journal of the Market Research Society, 33(3): 243-255.
66.
Schwarz, N.1999. Self-reports: How the questions shape the answers. American Psychologist, 54(2): 93-105.
67.
Sexton, D. L., & Bowman, N.1985. The entrepreneur: A capable executive and more. Journal of Business Venturing, 1: 129-140.
68.
Simonin, B. L.1997. The importance of collaborative know-how: An empirical test of the learning organization. Academy of Management Journal, 40(5): 1150-1174.
69.
Singer, E., Groves, R. M., & Corning, A. D.1999. Differential incentives: Beliefs about practices, perceptions of equity, and effects on survey participation. Public Opinion Quarterly, 63(2): 251-260.
70.
Stimpert, J. L., & Duhaime, I. M.1997. In the eyes of the beholders: Conceptualizations of relatedness held by the managers of large diversified firms. Strategic Management Journal, 18(2): 111-125.
71.
Stubbart, C.1989. Managerial cognition: A missing link in strategic management research. Journal of Management Studies, 26: 325-347.
72.
Sudman, S., & Blair, E.1999. Sampling in the twenty-first century. Journal of the Academy of Marketing Science, 27(2): 269-277.
73.
Sutcliffe, K. M.1994. What executives notice: Accurate perceptions in top management teams. Academy of Management Journal, 37(5): 1360-1378.
74.
Sutton, R. J., & Zeits, L. L.1992. Multiple prior notifications, personalizations, and reminder surveys: Do they have an effect on survey response rates?Marketing Research: A Magazine of Management and Applications, 4(4): 14-21.
75.
Thomas, A. S., Litschert, R. J., & Ramaswamy, K.1991. The performance impact of strategy-manager co-alignment: An empirical examination. Strategic Management Journal, 12: 509-522.
Tosi, H. L., Jr., & Gomez-Mejia, L. R.1994. CEO compensation monitoring and firm performance. Academy of Management Journal, 37(4): 1002-1016.
78.
Tse, A. C. B.1998. Comparing the response rate, response speed and response quality of two methods of sending questionnaires: E-mail vs. mail. Journal of the Market Research Society, 40(4): 353-361.
79.
Tyler, B. B., & Steensma, H. K.1998. The effects of executives’ experiences and perceptions on their assessment of potential technological alliances. Strategic Management Journal, 19(10): 939-966.
80.
U.S. Small Business Administration, 1996. Office of Advocacy. http://www.sba.gov
81.
Venkatraman, N.1989. The concept of fit in strategy research: Toward verbal and statistical correspondence. Academy of Management Review, 14.: 423-444.
82.
Venkatraman, N., & Ramanujam, V.1987. Measurement of business economic performance: An examination of method convergence. Journal of Management, 13(1): 109-122.
83.
Woodward, J.1965. Industrial organizations: Theory and practice (2nd ed.). Oxford: Oxford University Press.
84.
Yammarino, F. J., Skinner, S. J., & Childers, T. L.1991. Understanding mail survey response behavior. Public Opinion Quarterly, 55: 613-629.
85.
Yu, J., & Cooper, H.1983. A quantitative review of research design efforts on response rates to questionnaires. Journal of Marketing Research, 20: 36-44.
