The value of multiple informant methodology for improving the validity in determining organizational properties has been increasingly recognized. However, the majority of empirical research still relies on single (key) informants. This is partly due to the lack of comprehensive methodological narratives and precise recommendations on the application of this important methodology. Therefore, the authors have developed a critical review and derived clear recommendations for the key challenges that researchers face in using multiple informants: (1) Which and how many informants should be considered? (2) How should the consensus among the informants be judged? (3) How are multiple responses combined into a single, organizational response to conduct further data analyses?
Anderson, James C.1987. ‘‘An Approach for Confirmatory Measurement and Structural Equation Modeling of Organizational Properties.’’Management Science33(4):525-41.
2.
Ashton, Alison H. and Robert H. Ashton.1985. ‘‘Aggregating Subjective Forecasts: Some Empirical Results.’’Management Science31(12):1499-1508.
3.
Bakir, Aysen, Gregory M. Rose, and Aviv Shoham.2005. ‘‘Consumption Communication and Parental Control of Children’s Television Viewing: A Multi-rater Approach.’’Journal of Marketing Theory and Practice13(2):47-58.
4.
Bartlett, Frederic C.1954. Remembering: A Study in Experimental and Social Psychology . 3rd ed. Cambridge , UK: Cambridge University Press.
5.
Bliese, Paul D.2000. ‘‘Within-group Agreement, Non-independence, and Reliability: Implications for Data Aggregation and Analysis.’’ Pp. 349-81 in Multilevel Theory, Research, and Methods in Organizations, edited by K. J. Klein and S. W. J. Kozlowski.San Francisco, CA : Jossey-Bass.
6.
Bowman, Cliff and Véronique Ambrosini.1997. ‘‘Using Single Respondents in Strategy Research.’’ British Journal of Management8(2):119-31.
7.
Bowman, Douglas and Das Narayandas.2004. ‘‘Linking Customer Management Effort to Customer Profitability in Business Markets.’’ Journal of Marketing Research41(4):433-47.
8.
Boyer, Kenneth K. and Rohit Verma.2000. ‘‘Multiple Raters in Survey-based Operations Management Research: A Review and Tutorial.’’Production and Operations Management9(2):128-40.
9.
Brown, Reagan D. and Neil M. A. Hauenstein.2005. ‘‘Interrater Agreement Reconsidered: An Alternative to the rwg Indices.’’Organizational Research Methods8(2):165-84.
10.
Burke, Michael J. and William P. Dunlap.2002. ‘‘Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide.’’Organizational Research Methods5(2):159-72.
11.
Burke, Michael J., Lisa M. Finkelstein, and Michelle S. Dusig .1999. ‘‘On Average Deviation Indices for Estimating Interrater Agreement.’’Organizational Research Methods2(1):49-68.
12.
Butler, Richard, Les Davies, Richard Pike, and John Sharp.1991. ‘‘Strategic Investment Decision-making: Complexities, Politics and Processes.’’Journal of Management Studies28(4):395-415.
13.
Campbell, Donald T.1955. ‘‘The Informant in Quantitative Research.’’American Journal of Sociology60(4):339-42.
14.
Chan, David.1998. ‘‘Functional Relations Among Constructs in the Same Context Domain at Different Levels of Analysis: A Typology of Composition Models.’’ Journal of Applied Psychology83(2):234-46.
15.
Chen, Ming-Jer, Jiingh-Lih Farh, and Ian C. Macmillan .1993. ‘‘An Exploration of the Expertness of Outside Informants.’’Academy of Management Journal36(6):1614-32.
16.
Cohen, Jacob.1960. ‘‘A Coefficient of Agreement for Nominal Scales .’’ Educational and Psychological Measurement20(1):37-46.
17.
Cohen, Jacob.1968. ‘‘Weighted Kappa: Nominal Scale Agreement With Provision for Scaled Disagreement or Partial Credit. ’’ Psychological Bulletin70:213-20.
18.
Dean, James W. Jr. and Mark P. Sharfman.1995. ‘‘Procedural Rationality in the Strategic Decision-making Process.’’Journal of Management Studies30(4):587-610.
19.
Deery, Stephen J. and Roderick D. Iverson.2005. ‘‘Labor-management Cooperation: Antecedents and Impact on Organizational Performance.’’Industrial and Labor Relations Review58(4):588-609.
20.
Dooley, Robert S., Gerald E. Fryxell, and William Q. Judge .2000. ‘‘Belaboring the Not-so-obvious: Consensus, Commitment, and Strategy Implementation Speed and Success.’’Journal of Management26(6):1237-57.
21.
Dunlap, William P., Michael J. Burke, and Kristin Smith-Crowe .2003. ‘‘Accurate Tests of Statistical Significance for rwg and Average Deviation Interrater Agreement Indexes.’’Journal of Applied Psychology88(2):356-62.
22.
Enticott, Gareth.2004. ‘‘Multiple Voices of Modernization: Some Methodological Implications.’’ Public Administration82(3):743-56.
23.
Ferrell, William R.1985. ‘‘Combining Individual Judgments.’’ Pp. 111-45 in Behavioral Decision Making, edited by G. Wright.New York: Plenum .
24.
Finn, R.H.1970. ‘‘A Note on Estimating the Reliability of Categorical Data.’’ Educational and Psychological Measurement30(1):71-6.
25.
Fleenor, John W., Julie B. Fleenor, and William F. Grossnickle .1996. ‘‘Interrater Reliability and Agreement of Performance Ratings: A Methodological Comparison.’’Journal of Business and Psychology10(3):367-80.
26.
George, Jennifer M.1990. ‘‘Personality, Affect, and Behavior in Groups.’’Journal of Applied Psychology75(2):107-16.
27.
Gigone, Daniel and Reid Hastie.1997. ‘‘Proper Analysis of the Accuracy of Group Judgments.’’ Psychological Bulletin121(1):149-67.
28.
Glick, William H., George P. Huber, C. Chet Miller, D. Harold Doty, and Kathleen M. Sutcliffe .1990. ‘‘Studying Changes in Organizational Design and Effectiveness: Retrospective Event Histories and Periodic Assessments.’’Organization Science1(3):293-312.
29.
Golden, Brian R.1992. ‘‘The Past is the Past-Or Is It? The Use of Retrospective Accounts as Indicators of Past Strategy.’’Academy of Management Journal35(4):848-60.
30.
Grayson, Kent and Roland Rust.2001. ‘‘Interrater Reliability.’’Journal of Consumer Psychology10(1&2):71-3.
31.
Grove, William M., David H. Zald, Boyd S. Lebow, Beth E. Snitz, and Chad Nelson.2000. ‘‘Clinical Versus Mechanical Prediction: A Meta-analysis.’’Psychological Assessment12(1):19-30.
32.
Hambrick, Donald C.1981. ‘‘Strategic Awareness Within Top Management Teams.’’Strategic Management Journal2(3):263-79.
33.
Hill, Gayle W.1982. ‘‘Group vs. Individual Performance: Are N+1 Heads Better than One?’’Psychological Bulletin91(3):517-39.
34.
Hoegl, Martin and Stephan M. Wagner.2005. ‘‘Buyer-supplier Collaboration in Product Development Projects.’’Journal of Management31(4):530-48.
35.
Hsu, Louis M. and Ronald Field.2003. ‘‘Interrater Agreement Measures: Comments on Kappan, Cohen’s Kappa, Scott’s π, and Aickin’s α.’’Understanding Statistics2(3):205-19.
36.
Huber, George P. and Daniel J. Power.1985. ‘‘Retrospective Reports of Strategic-level Managers: Guidelines for Increasing Their Accuracy.’’Strategic Management Journal6(2):171-80.
37.
James, Lawrence R.1982. ‘‘Aggregation Bias in Estimates of Perceptual Agreement.’’Journal of Applied Psychology67(2):219-29.
38.
James, Lawrence R.1988. ‘‘Organizational Climate: Another Look at a Potentially Important Construct.’’ Pp. 253-82 in Applications of Interactionist Psychology: Essays in Honor of Saul B. Sells, edited by S. G. Cole and R. G. Demaree.Hillsdale, NJ: Lawrence Erlbaum.
39.
James, Lawrence R., Robert G. Demaree, and Gerrit Wolf.1984. ‘‘Estimating Within-group Interrater Reliability With and Without Response Bias.’’Journal of Applied Psychology69(1):85-98.
40.
James, Lawrence R., Robert G. Demaree, and Gerrit Wolf.1993. ‘‘rWG: An Assessment of Within-group Interrater Agreement.’’Journal of Applied Psychology78(2):306-09.
41.
James, Lawrence R., Gerrit Wolf, and Robert G. Demaree .1981. Estimating Interrater Reliability in Incomplete Designs. Fort Worth, TX: Institute of Behavioral Research.
42.
Janis, Irving L.1972. Victims of Groupthink. Boston, MA: Houghton Mifflin.
43.
John, George and Torger Reve.1982. ‘‘The Reliability and Validity of Key Informant Data From Dyadic Relationships in Marketing Channels. ’’ Journal of Marketing Research19(4):517-24.
44.
Jones, Allen P., Lee A. Johnson, Marc C. Butler, and Deborah S. Main .1983. ‘‘Apples and Oranges: An Empirical Comparison of Commonly Used Indices of Interrater Agreement.’’Academy of Management Journal26(3):507-19.
45.
Kozlowski, Steve W. J. and Keith Hattrup.1992. ‘‘A Disagreement About Within-group Agreement: Disentangling Issues of Consistency Versus Consensus.’’Journal of Applied Psychology77(2):161-67.
46.
Kozlowski, Steve W. J. and Katherine J. Klein.2000. ‘‘A Multilevel Approach to Theory and Research in Organizations: Contextual, Temporal, and Emergent Processes.’’ Pp. 3-90 in Multilevel Theory, Research, and Methods in Organizations, edited by K. J. Klein and S. W. J. Kozlowski.San Francisco, CA: Jossey-Bass.
47.
Krannich, Richard S. and Craig R. Humphrey.1986. ‘‘Using Key Informant Data in Comparative Community Research.’’Sociological Methods & Research14(4):473-93.
48.
Kumar, Nirmalya, Louis W. Stern, and James C. Anderson .1993. ‘‘Conducting Inter-organizational Research Using Key Informants.’’Academy of Management Journal36(6):1633-51.
49.
Lance, Charles E., Marcus M. Butts, and Lawrence C. Michels .2006. ‘‘The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?’’Organizational Research Methods9(2):202-20.
50.
Lawlis, G. Frank and Elba Lu.1972. ‘‘Judgment of Counseling Process: Reliability, Agreement and Error.’’Psychological Bulletin78(1):17-20.
51.
LeBreton, James M., Jennifer R. D. Burgess, Robert B. Kaiser, E. Kate Atchley, and Lawrence R. James .2003. ‘‘The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?’’Organizational Research Methods6(1):80-128.
52.
LeBreton, James M., Lawrence R. James, and Micheal K. Lindell .2005. ‘‘Recent Issues Regarding rWG, r*WG, rWG(J), and r*WG(J).’’Organizational Research Methods8(1):128-38.
53.
LeBreton, James M. and Jenell L. Senter.2008. ‘‘Answers to 20 Questions About Interrater Reliability and Interrater Agreement.’’Organizational Research Methods11(4):815-52.
54.
Libby, Robert and Roger K. Blashfield.1978. ‘‘Performance of a Composite as a Function of the Number of Judges.’’Organizational Behavior and Human Performance21(2):121-29.
55.
Lindell, Michael K. and Christina J. Brandt.1999. ‘‘Assessing Interrater Agreement on the Job Relevance of a Test: A Comparison of the CVI, T, rWG(J), and r*WG(J) Indexes.’’Journal of Applied Psychology84(4):640-47.
56.
Lindell, Michael K., Christina J. Brandt, and David J. Whitney .1999. ‘‘A Revised Index of Agreement for Multi-item Ratings of a Single Target.’’Applied Psychological Measurement23(2):127-35.
57.
Lipscomb, Joseph, Giovanni Parmigiani, and Vic Hasselblad .1998. ‘‘Combining Expert Judgment by Hierarchical Modeling: An Application to Physician Staffing . ’’ Management Science44(2):149-61.
58.
Malhotra, Manoj K. and Varun Grover.1998. ‘‘An Assessment of Survey Research in POM: From Constructs to Theory.’’Journal of Operations Management16(4):407-25.
59.
McGraw, Kenneth O. and S.P. Wong.1996. ‘‘Forming Inferences About Some Intra-class Correlation Coefficients.’’Psychological Methods1(1):30-46.
60.
Mitchell, Sandra K.1979. ‘‘Interobserver Agreement, Reliability, and Generalizability of Data Collected in Observational Studies.’’Psychological Bulletin86(2):376-90.
61.
Mitroff, Ian I.1972. ‘‘The Myth of Objectivity or Why Science Needs a New Psychology of Science.’’Management Science18(10):B613-18.
62.
Money, R. Bruce.2004. ‘‘Word-of-mouth Promotion and Switching Behavior in Japanese and American Business-to-business Service Clients.’’Journal of Business Research57(3):297-305.
63.
Mood, Alexander M., Franklin A. Graybill, and Duane C. Boes .1974. Introduction to the Theory of Statistics. 3rd ed. New York : McGraw-Hill.
64.
Nutt, Paul C.1986. ‘‘Tactics of Implementation.’’Academy of Management Journal29(2):230-61.
65.
Pagell, Mark and Daniel R. Krause.2005. ‘‘Determining When Multiple Respondents Are Needed in Supply Chain Management Research: The Case of Purchasing and Operations.’’Academy of Management Proceedings, B1-B5.
66.
Phillips, Lynn W.1981. ‘‘Assessing Measurement Error in Key Informant Reports: A Methodological Note on Organizational Analysis in Marketing.’’Journal of Marketing Research18(4):395-415.
67.
Roberson, Quinetta M., Michael C. Sturman, and Tony L. Simons .2007. ‘‘Does the Measure of Dispersion Matter in Multilevel Research? A Comparison of the Relative Performance of Dispersion Indexes.’’Organizational Research Methods10(4):564-88.
68.
Salancik, Gerald R. and James R. Meindl.1984. ‘‘Corporate Attributions as Strategic Illusions of Management Control.’’Administrative Science Quarterly29(2):238-54.
69.
Schwab, Donald P. and Herbert G. Heneman III.1986. ‘‘Assessment of a Consensus-based Multiple Information Source Job Evaluation System.’’Journal of Applied Psychology71(2):354-56.
70.
Schwenk, Charles R.1985. ‘‘The Use of Participant Recollection in the Modeling of Organizational Decision Processes.’’Academy of Management Review10(3):496-503.
71.
Seidler, John.1974. ‘‘On Using Informants: A Technique for Collecting Quantitative Data and Controlling for Measurement Error in Organizational Analysis.’’ American Sociological Review39(6):816-31.
72.
Shrout, Patrick E. and Joseph L. Fleiss.1979. ‘‘Intraclass Correlations: Uses in Assessing Rater Reliability.’’Psychological Bulletin86(2):420-28.
73.
Silk, Alvin J. and Manohar U. Kalwani.1982. ‘‘Measuring Influence in Organizational Purchase Decisions.’’Journal of Marketing Research19(2):165-81.
74.
Snow, Charles C. and Lawrence G. Hrebiniak.1980. ‘‘Strategy, Distinctive Competence, and Organizational Performance.’’Administrative Science Quarterly25(2):317-36.
75.
Story, Victoria, Louise Hurdley, Gareth Smith, and James Saker.2001. ‘‘Methodological and Practical Implications of the Delphi Technique in Marketing Decision-making: A Re-assessment.’’The Marketing Review1(4):487-504.
76.
Subramaniam, Mohan and Mark A. Youndt.2005. ‘‘The Influence of Intellectual Capital on the Types of Innovative Capabilities.’’Academy of Management Journal48(3):450-63.
77.
van Bruggen, Gerrit H., Gary L. Lilien, and Manish Kacker.2002. ‘‘Informants in Organizational Marketing: Why Use Multiple Informants and How to Aggregate Responses.’’Journal of Marketing Research39(4):469-78.
