In this article, the authors provide an introduction to the use of a matched-groups design as a strategy for controlling nuisance variables. Building on the insightful comments of Campbell and Kenny and of Meehl, the authors utilized Monte Carlo simulations to highlight three major limitations of this control strategy (i.e., regression toward different means, systematic unmatching, and the generalizability of results) that they believe have received insufficient attention by researchers. Particular attention is given to how the effect of regression toward different means can result in researchers' drawing erroneous conclusions from their data. Recommendations for researchers considering the use of a matched-groups design are provided.
Austin, J.T., Scherbaum, C.A., & Mahlman, R.A. (2002). History of research methods in industrial and organizational psychology: Measurement, design, analysis. In S. G. Rogelberg (Eds.), Handbook of research methods in industrial and organizational psychology (pp. 3-33). Malden, MA: Blackwell.
2.
*Baker, A., Roach, G., Ferguson, S., & Dawson, D. (2003). The impact of different rosters on employee work and non-work time preferences. Time & Society, 12, 315-332.
3.
*Bartkus, B.R., Morris, S.A., & Seifert, B. (2002). Governance and corporate philanthropy. Business and Society, 41, 319-344.
4.
Becker, T.E. (2005). Potential problems in the statistical control of variables in organizational research. Organizational Research Methods, 8, 274-289.
5.
*Budhwar, P.S., & Sparrow, P.R. (2002). Strategic HRM through the cultural looking glass: Mapping the cognition of British and Indian managers. Organizational Studies, 23, 599-638.
6.
Campbell, D.T., & Kenny, D.A. (1999). A primer of regression artifacts. New York: Guilford.
7.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum .
8.
Cohen, J., Cohen, P., West, S.G., & Aiken, L.S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.
9.
*Dunn, P. (2004). The impact of insider power on fraudulent financial reporting. Journal of Management, 30, 397-412.
10.
*Farrell, K.A., & Whidbee, D.A. (2002). The impact of forced CEO turnover on committee structure. Journal of Managerial Issues, 14, 49-67.
11.
*Fosberg, R.H. (1999). Leadership structure and CEO compensation. American Business Review, 17, 50-57.
12.
*Fulmer, I.S., Gerhart, B., & Scott, K.S. (2003). Are the 100 best better? An empirical examination of the relationship between being a “great place to work” and firm performance. Personnel Psychology, 56, 965-993.
13.
Goodwin, C.J. (2003). Research in psychology. New York: John Wiley.
14.
*Hare, A.P., Koenigs, R.J., & Hare, S.E. (1997). Perceptions of observed and model values of male and female managers. Journal of Organizational Behavior, 18, 437-447.
Klein, R.B. (1998). Principles and practice of structural equation modeling. New York: Guilford.
17.
*Lam, S.S., Chen, X., & Schaubroeck, J. (2002). Participative decision making and employee performance in different cultures: The moderating effects of allocentrism/idiocentrism and efficacy. Academy of Management Journal, 45, 905-914.
18.
Little, R.J., An, H., Johanns, J., & Giordani, B. (2000). A comparison of subset selection and analysis of covariance for the adjustment of confounders. Psychological Methods, 5, 459- 476.
19.
*Lyness, K.S., & Thompson, D.T. (1997). Above the glass ceiling? A comparison of matched samples of female and male executives. Journal of Applied Psychology, 82, 359-375.
20.
*Lyness, K.S., & Thompson, D.T. (2000). Climbing the corporate ladder: Do female and male executives follow the same route. Journal of Applied Psychology, 85, 86-101.
21.
*MacDonald, S., Lothian, S., & Wells, S. (1997). Evaluation of an employee assistance program at a transportation company. Evaluation and Program Planning , 20, 495-505.
22.
Maxwell, S.E., & Delaney, H.D. (2004). Designing experiments and analyzing data. Mahwah, NJ: Lawrence Erlbaum.
23.
*McGurr, P.T., & DeVaney, S.A. (1998). Predicting business failure of retail firms: An analysis using mixed industry models. Journal of Business Research, 43, 169-177.
24.
Meehl, P.E. (1970). Nuisance variables and the ex post facto design . In M. Radner & S. Winokur (Eds.), Analyses of theories and methods of physics and psychology (pp. 373-402). Minneapolis : University of Minnesota Press.
25.
Meehl, P.E. (1971). High school yearbooks: A reply to Schwarz. Journal of Abnormal Psychology, 77, 143-148.
26.
Mervis, C.B., & Klein-Tasman, B.P. (2004). Methodological issues in group-matching designs. Journal of Autism and Developmental Disorders, 34, 7-17.
27.
*Mezias, J.M. (2002). Identifying liabilities of foreignness and strategies to minimize their effects: The case of labor lawsuit judgments in the United States. Strategic Management Journal, 23, 229-244.
28.
Mook, D.G. (2001). Psychological research: The ideas behind the methods . New York: Norton.
29.
Pedhazur, E.J. (1997). Multiple regression in behavioral research. Fort Worth, TX: Harcourt, Brace.
30.
*Ragins, B.R., & Scandura, T.A. (1997). The way we were: Gender and the termination of mentoring relationships. Journal of Applied Psychology, 82, 945-953.
31.
Rogelberg, S. G. (Ed.). (2002). Handbook of research methods in industrial and organizational psychology. Malden, MA: Blackwell.
32.
Rubin, D.B., & Thomas, N. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics, 52, 249-264.
33.
*Sarchione, C.D., Cuttler, M.J., Muchinsky, P.M., & Nelson-Gray, R.O. (1998). Prediction of dysfunctional job behaviors among law enforcement officers. Journal of Applied Psychology, 83, 904-912.
34.
*Savvas, M., El-Kot, G., & Sadler-Smith, E. (2001). Comparative study of cognitive styles in Egypt, Greece, Hong Kong, and the UK. International Journal of Training and Development, 5, 64-73.
35.
Schmidt, F., & Hunter, J. (2003). History, development, evolution, and impact of validity generalization and meta-analysis methods, 1975-2001. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 31-65). Mahwah, NJ: Lawrence Erlbaum.
36.
Schwab, D.B. (2005). Research methods for organizational studies. Mahwah, NJ: Lawrence Erlbaum.
37.
*Seifert, B., Morris, S.A., & Bartkus, B.R. (2003). Comparing big givers and small givers: Financial correlates of corporate philanthropy. Journal of Business Ethics, 45, 195-203.
38.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
39.
*Slaski, M., & Cartwright, S. (2003). Emotional intelligence training and its implications for stress, health, and performance. Stress and Health, 19, 233-239.
40.
Smith, H.L. (1997). Matching with multiple controls to estimate treatment effects in observational studies. Sociological Methodology , 27, 325-353.
41.
*Thomas, P.J., Edwards, J.E., Perry, Z.A., & David, K.M. (1998). Racial differences in male officer fitness reports . Military Psychology, 10, 127-143.
