Leadership scholars are faced with pressing methodological issues that challenge our current scholarship. In this paper, we discuss methodological concerns such as multilevel hypothesis testing, construct validity of group-level scales, sampling countries in cross-cultural research, self-report measurement — issues that are endemic to organizational leadership research. We point out the methodological challenges facing modern leadership researchers, and point to possible solutions that can provide opportunities for future progress in our field.
BachiochiP.D.WeinerS.P. (2002): Qualitative data collection and analysis. InRogelbergS.G. (Ed.): Handbook of Research Methods in Industrial and Organizational Psychology. (pp. 161–183). Malden, MA: Blackwell.
2.
BarrettG.V.BassB.M.. (1976): Cross-cultural issues in Industrial and Organizational Psychology. In: DunnetteM. D. (Ed.): Handbook of Industrial and Organizational Psychology. (pp. 1639–1686). Chicago, IL: Rand McNally.
3.
BassB.M. (1990): Bass & Stogdill's handbook of leadership: Theory, research, and managerial applications (3rd ed.). New York: Free Press.
4.
BlieseP.D.HalversonR. R.SchriesheimC.A. (2002): Benchmarking multilevel methods in leadership: The articles, the data, and the data set. In: Leadership Quarterly, 13, 3–14.
5.
BlieseP.D.HangesP.J. (In press): Being both too liberal and too conservative: The perils of treating grouped data as though it is independent. In: Organizational Research Methods.
6.
BrykA. S.RaudenbushS. W. (1987): Hierarchical linear models.Newbury Park, CA: SAGE Publications.
7.
CarlD.GuptaV.JavidanM. (2004): Power distance. (pp. 513–560). In: HouseR. J.HangesP. J.JavidanM.DorfmanP.W.GuptaV. (Eds.): Culture, leadership, and organizations: The GLOBE study of 62 societies.Thousand Oaks, CA: Sage Publications, Inc.
8.
CastroS.L. (2002): Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling. In: Leadership Quarterly, 13, 69–93.
9.
ChanD. (1998): Functional relations among constructs in the same content domain in different levels of analysis: A typology of composition models. In: Journal of Applied Psychology, 83, 234–246.
10.
ChongL. M. A.ThomasD. C. (1997): Leadership perceptions in cross-cultural context: Pakeha and Pacific islanders in New Zealand. In: Leadership Quarterly, 8(3), 275–293.
11.
CronbachL. J. (1976): Research on classrooms and schools: Formulation of questions, design, and analysis. Unpublished manuscript, Stanford University, California.
12.
DansereauF.AluttoJ.A.YammarinoF.J. (1984): Theory testing in organizational behavior: The variant approach.Englewood Cliffs, NJ: Prentice-Hall.
13.
DansereauF.YammarinoF.J. (1998a): Leadership: The Multiple-Level Approaches, Classical and New Wave. Samford, Connecticut: JAI Press, Inc.
14.
DansereauF.YammarinoF.J. (1998b): Leadership: The Multiple-Level Approaches, Contemporary and Alternative. Stamford, Connecticut: JAI Press, Inc.
DeMeuseK.P. (1986): A compendium of frequently used measures in Industrial/Organizational Psychology. In: The Industrial and Organizational Psychologist, 23, 53.
17.
Den HartogD.HouseR. J.HangesP. J.Ruiz-QuintanillaS. A.DorfmanP. W., Associates. (1999): Culture specific and cross culturally generalizable implicit leadership theories: Are attributes of charismatic/transformational leadership universally endorsed? In: Leadership Quarterly, 10(2), 219–256.
18.
DevineP. G. (2001): Implicit prejudice and stereotyping: How automatic are they? Introduction to the special section. In: Journal of Personality and Social Psychology, 81, 757–759.
19.
DorfmanP.W.HouseR.J. (2004): Cultural influences on organizational leadership. In: HouseR. J.HangesP.J.JavidanM.DorfmanP.W.GuptaV. (Eds.): Leadership, Culture, and Organizations: The GLOBE study of 62 societies. (pp. 51–73). Thousand Oaks, CA: SAGE.
20.
DorfmanP.W. (1996): International and cross-cultural leadership. In: PunnittJ.ShankerO. (Eds.): Handbook for International Management Research. (pp. 267–349). Cambridge, MA: Blackwell.
21.
DorfmanP.HangesP. J.BrodbeckF. C. (2004): Leadership and cultural variation: The identification of culturally endorsed leadership profiles. InHouseR. J.HangesP. J.JavidanM.DorfmanP.GuptaV. (Eds.), Leadership, culture, and organizations: The GLOBE study of 62 societies (pp. 667–718). Thousand Oaks, CA: Sage Publications, Inc.
22.
DuntonB. C.FazioR. H. (1997): An individual difference measure of motivation to control prejudiced reactions. In: Personality and Social Psychology Bulletin, 23, 316–326.
23.
DyerN.HangesP.J.HallR.A. (2004): Applying multilevel confirmatory factor analysis techniques to the study of leadership. Working paper.
24.
FazioR. H.OlsonM. A. (2003): Implicit measures in social cognition research: Their meaning and use. In: Annual Review of Psychology, 54, 297–327.
25.
FazioR. H.JacksonJ. R.DuntonB. C.WilliamsC. J. (1995): Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? In: Journal of Personality and Social Psychology, 69, 1013–1027.
26.
GreenwaldA. G.NosekB. A. (2001): Health of the Implicit Association Test at age 3. In: Zeitschrift fur Experimentelle Psychologie, 48, 85–93.
27.
GreenwaldA. G.McGheeD. E.SchwartzJ. L. K. (1998): Measuring individual differences in implicit cognition: The Implicit Association Test. In: Journal of Personality and Social Psychology, 74, 1464–1480.
28.
GuptaV.HangesP.J. (2004): Regional and climate clustering of social cultures. In: HouseR. J.HangesP.J.JavidanM.DorfmanP.W.GuptaV. (Eds.): Leadership, Culture, and Organizations: The GLOBE study of 62 societies. (pp. 178–218). Thousand Oaks, CA: SAGE.
29.
HallR.J.HangesP.J.DyerN.G. (in press): Two-level structural equation modeling with latent variables: An illustration of confirmatory factor analysis with applications to organizational research. In: Organizational Research Methods.
30.
HallR. J.MakineyJ. D.MarchioroC. A.TanJ. A.KleinK. J. (April 1999): Applying multilevel structural equation modeling techniques to the study of organizational behavior. Paper presented at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, GA.
31.
HangesP.J.DicksonM.W. (2004): The development and validation of the GLOBE culture and leadership scales. In: HouseR. J.HangesP.J.JavidanM.DorfmanP.W.GuptaV. (Eds.): Leadership, Culture, and Organizations: The GLOBE study of 62 societies. (pp. 122–151). Thousand Oaks, CA: SAGE.
32.
HangesP. J.DicksonM. W.SipeM. T. (2004): Rationale for GLOBE statistical analyses: Societal rankings and test of hypotheses. InHouseR. J.HangesP. J.JavidanM.DorfmanP.GuptaV. (Eds.): Leadership, culture, and organizations: The GLOBE study of 62 societies (pp. 219–234). Thousand Oaks, CA: Sage.
33.
HangesP.J.DorfmanP.W.ShteynbergGBatesA. (in press): Implications of culture and leadership information processing: Leadership style doesn't work a bit? Maybe you're a cultural misfit! In: MobleyW. H.WeldonE. (Eds): Advances in Global Leadership (Vol 4). NY: JAI Press.
34.
HangesP. J.LordR. G.DicksonM. W. (2000): An information processing perspective on leadership and culture: A case for connectionist architecture. In: Applied Psychology: An International Review, 49, 133–161.
35.
HangesP.J.LyonJ.DorfmanP.W., (In press): Managing an international research team: Lessons from project GLOBE. In: ShapiroD.Von GlinowM.A.ChengJ. (Eds.) Managing Multinational Teams: Cultural, Organizational, and National Influences. Elsevier Press.
36.
HarnqvistK. (1978): Primary mental abilities of collective and individual levels. In: Journal of Educational Psychology, 70, 706–716.
37.
HofmannD. A.GriffinM. A.GavinM. B. (2000): The application of hierarchical linear modeling to organizational research. InKleinK. J.KoslowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 467–511). San Francisco: Jossey-Bass.
38.
HofmannD. A. (1997): An overview of the logic and rationale of hiearchical linear models. In: Journal of Management, 23, 723–744.
39.
HouseR. J.HangesP. J.JavidanM.DorfmanP. W.GuptaV., GLOBE Associates. (2004): Culture, leadership, and organizations: The GLOBE study of 62 societies.Thousand Oaks, CA: Sage Publications, Inc.
40.
HouseR. J.HangesP. J.Ruiz-QuintanillaS. A.DorfmanP. W.JavidanM.DicksonM.GuptaV., 170 co-authors. (1999): Cultural influences on leadership and organizations: Project GLOBE. In: MobleyW. F.GessnerM. J.ArnoldV. (Eds.): Advances in global leadership (Vol. 1, pp. 171–233). Stamford, CT: JAI Press.
41.
JamesL. R.WilliamsL. J. (2000): The cross-level operator in regression, *ANCOVA, and contextual analysis. In: KleinK. J.KoslowskiS. W. J. (Eds.): Multilevel theory, research, and methods in orgainzations: Foundations, extensions, and new directions.San Francisco: Jossey-Bass.
42.
JavidanM.HouseR. J.DorfmanP. W. (2004): A nontechnical summary of GLOBE findings. In: HouseR. J.HangesP. J.JavidanM.DorfmanP.GuptaV. (Eds.), Leadership, culture, and organizations: The GLOBE study of 62 societies (pp. 29–48). Thousand Oaks, CA: Sage.
43.
KleinK.J.BlieseP.D.KozlowskiS.W.J.DansereauF.GavinM.B.GriffinM.A.HofmannD.A.JamesL.R.YammarinoF.J.BlighM.C. (2000): Multilevel analytic techniques: Commonalities, differences, and continuing questions. In: KleinK. J.KoslowskiS. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 512–556). San Francisco: Jossey-Bass.
44.
KleinK.J.KozlowskiS.W.J. (2000): From micro to meso: Critical steps in conceptualizing and conducting multilevel research. In: Organizational Research Methods, 3, 211–236.
LeslieJ.Van VelsorE. (1998): A cross-national comparison of effective leadership and teamwork: Toward a global workforce.Greensboro, NC: Center for Creative Leadership.
47.
LordR. G.BrownD. J. (2004): Leadership processes and follower self-identity.Mahwah, NJ: Lawrence Erlbaum.
48.
LordR. G.FotiR. J.DeVaderC. L. (1984): A test of leadership categorization theory: Internal structure, information processing, and leadership perceptions. Organizational Behavior and Human Performance, 34, 343–378.
49.
LordR. G.FotiR. J.PhillipsJ. S. (1982): A theory of leadership categorization. InHuntJ. G.SekaranU.SchriesheimC. A. (Eds.), Leadership: Beyond establishment views (pp. 104–121). Carbondale: Southern Illinois University Press.
50.
LordR.G.MaherK.J. (1991): Leadership and information processing: Linking perceptions and performance.Cambridge, MA: Unwin Hyman.
51.
MaczynskiJ.JagoA.ReberG.BoehnischW. (1994): Culture and leadership styles: A comparison of Polish, Austrian, and U.S. managers. In: Polish Psychological Bulletin, 25(4), 303–315.
52.
MitchellT. R. (1979): Organizational behavior. In: Annual Review of Psychology, 30, 243–281.
53.
MuthenB.O. (1990): Mean and covariance structure analysis of hierarchical data. Paper presented at the Psychometric Society, Princeton, New Jersey.
PhelpsE. A.O'ConnorK. J.CunninghamW. A.GatenbyJ. C.FunayamaE. S.GoreJ. C.BanajiM. R. (2000): Amygdala activation predicts performance on indirect measures of racial bias. In: Journal of Cognitive Neuroscience, 12, 729–738.
57.
PlantE. A.DevineP. G. (1998): Internal and external motivation to respond without prejudice. In: Journal of Personality and Social Psychology, 75, 811–832.
58.
RonenS.ShenkarO. (1985): Clustering countries on attitudinal dimensions: A review and synthesis. In: Academy of Management Review, 10, 435–454.
59.
RudmanL. A.KilianskiS. E. (2000): Implicit and Explicit attitudes toward female authority. In: Personality and Social Psychology Bulletin, 26, 1315–1328.
60.
StogdillR.M. (1974): Handbook of leadership (1st ed.). New York: Free Press.
61.
SuutariV. (1996): Leadership ideologies among European managers: A comparative survey in a multinational company. In: Scandinavian Journal of Management, 12(4), 389–409.
62.
WaldmanD.A.YammarinoF.J. (1990): A multiple level investigation of personnel ratings. In: Personnel Psychology, 43, 811–835.
63.
YammarinoF.J.BassB.M. (1991): Person and situation views of leadership: A multiple levels of analysis approach. In: Leadership Quarterly, 2, 121–139.
64.
YammarinoF.J.MarkhamS.E. (1992): On the application of within and between analysis: Are absence and affect really group-based phenomena? In: Journal of Applied Psychology, 77, 168–176.
65.
ZaccaroS.J.KlimoskiR.J. (2001): The nature of organizational leadership: An introduction. In: ZaccaroS.J.KlimoskiR.J. (Eds.): The nature of organizationanl leadership: Understanding the performance imperatives confronting today's leaders. (pp. 3–41). San Francisco, CA: Jossey-Bass.
66.
ZiegertJ. C.HangesP.J. (In press): Employment Discrimination: The Role of Implicit Attitudes, Motivation, and a Climate for Racial Bias. In: Journal of Applied Psychology.
