In an effort to establish and enhance the accuracy of key informant data, organizational survey studies are increasingly relying on triangulation techniques by including supplemental data sources that complement information acquired from key informants. Despite the growing popularity of triangulation, little guidance exists as to when and how it should be conducted. Addressing this gap, the authors develop hypotheses linking a comprehensive set of study characteristics at the construct, informant, organizational, and industry levels to key informant accuracy. Two studies test these hypotheses. The first study is a meta-analysis of triangulation applications. Using data from 127 studies published in six major marketing and management journals, the authors identify antecedents to key informant reliability. The second study, using eight multi-informant data sets, analyzes antecedents to key informant validity. The results from these studies inform survey researchers as to which conditions particularly call for the use of triangulation. The authors conclude by offering guidelines on when and how to employ triangulation techniques.
BaddeleyAlan (1986), Working Memory.Oxford: Clarendon.
2.
BagozziRichard P., and YiYoujae (1988), “On the Evaluation of Structural Equation Models,”Journal of the Academy of Marketing Science, 16 (1), 74–94.
3.
BagozziRichard P., and YiYoujae (1991), “Multitrait-Multimethod Matrices in Consumer Research,”Journal of Consumer Research, 17 (4), 426–39.
4.
BagozziRichard P., YiYoujae, and LynnW. Phillips (1991), “Assessing Construct Validity in Organizational Research,”Administrative Science Quarterly, 36 (3), 421–58.
5.
BarnettWilliam P. (1997), “The Dynamics of Competitive Intensity,”Administrative Science Quarterly, 42 (1), 128–60.
6.
BiggadikeE. Ralph (1979), Corporate Diversification: Entry, Strategy, and Performance.Boston: Harvard University Press.
7.
BliesePaul D. (2000), “Within-Group Agreement, Non-independence, and Reliability: Implications for Data Aggregation and Analysis,” in Multilevel Theory, Research, and Methods in Organizations, KatherineJ. Klein, and SteveW. J. Kozlowski, eds. San Francisco: Jossey-Bass.
8.
BorensteinMichael, LarryV. Hedges, JulianP. T. Higgins, and HannahR. Rothstein (2009), Introduction to Meta-Analysis.Chichester, UK: John Wiley & Sons.
9.
BrownReagan D., and NeilM.A. Hauenstein (2005), “Interrater Agreement Reconsidered: An Alternative to the rWG Indices,”Organizational Research Methods, 8 (2), 165–84.
10.
CampbellDonald T., and DonaldW. Fiske (1959), “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix,”Psychological Bulletin, 56 (2), 81–105.
11.
ChurchillGilbert A. (1979), “A Paradigm for Developing Better Measures of Marketing Constructs,”Journal of Marketing Research, 16 (February), 64–73.
12.
CohenWesley M., and DanielA. Levinthal (1990), “Absorptive Capacity: A New Perspective on Learning and Innovation,”Administrative Science Quarterly, 35 (1), 128–52.
13.
ConwayJames M., RobertA. Jako, and DeborahE. Goodman (1995), “A Meta-Analysis of Interrater and Internal Consistency Reliability of Selection Interviews,”Journal of Applied Psychology, 80 (5), 565–79.
14.
CoteJoseph A., and Ronald BuckleyM. (1987), “Estimating Trait, Method, and Error Variance: Generalizing Across 70 Construct Validation Studies,”Journal of Marketing Research, 24 (August), 315–18.
15.
CoteJoseph A., and Ronald BuckleyM. (1988), “Measurement Error and Theory Testing in Consumer Research: An Illustration of the Importance of Construct Validation,”Journal of Consumer Research, 14 (4), 579–82.
16.
DavisHarry L., SusanP. Douglas, and AlvinJ. Silk (1981), “Measure Unreliability: A Hidden Threat to Cross-National Marketing Research?”Journal of Marketing, 45 (Spring), 98–109.
17.
DenzinNorman K. (1978), The Research Act.New York: McGraw-Hill.
18.
DotyD. Harold, and WilliamH. Glick (1998), “Common Methods Bias: Does Common Methods Variance Really Bias Results?”Organizational Research Methods, 1 (4), 374–406.
19.
DunlapWilliam P., MichaelJ. Burke, and Smith-CroweKristin (2003), “Accurate Tests of Statistical Significance for rWG and Average Deviation Interrater Agreement Indexes,”Journal of Applied Psychology, 88 (2), 356–62.
20.
DuvalSue, and TweedieRichard (2000), “A Nonparametric 'Trim And Fill' Method of Accounting for Publication Bias in Meta-Analysis,”Journal of the American Statistical Association, 95 (449), 89–98.
21.
EidMichael (2000), “A Multitrait-Multimethod Model with Minimal Assumptions,”Psychometrika, 65 (2), 241–61.
22.
FazioRussell H., DavidM. Sanbonmatsu, MarthaC. Powell, and FrankR. Kardes (1986), “On the Automatic Activation of Attitudes,”Journal of Personality and Social Psychology, 50 (2), 229–38.
23.
FeldmanJack M., and JohnG. Lynch (1988), “Self-Generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior,”Journal of Applied Psychology, 73 (3), 421–35.
24.
FisherR.A. (1925), “Theory of Statistical Estimation,”Proceedings of the Cambridge Philosophical Society, 22 (1), 700–725.
25.
FrankeGeorg R., and ParkJeong-Eun (2006), “Salesperson Adaptive Selling Behavior and Customer Orientation: A Meta-Analysis,”Journal of Marketing Research, 43 (November), 693–702.
26.
GeyskensInge, KrishnanRekha, Jan-BenedictE.M. Steenkamp, and PauloV. Cunha (2009), “A Review and Evaluation of Meta-Analysis Practices in Management Research,”Journal of Management, 35 (2), 393–419.
27.
GillundGary, and RichardM. Shiffrin (1984), “A Retrieval Model for Both Recognition and Recall,”Psychological Review, 91 (1), 1–67.
28.
GlickWilliam H. (1985), “Conceptualizing and Measuring Organizational and Psychological Climate: Pitfalls in Multilevel Research,”Academy of Management Review, 10 (3), 601–616.
29.
GoldenBrian R. (1992), “The Past Is the Past—Or Is It? The Use of Retrospective Accounts as Indicators of Past Strategy,”Academy of Management Journal, 35 (4), 848–60.
30.
HambrickDonald C. (1981), “Strategic Awareness Within Top Management Teams,”Strategic Management Journal, 24 (3), 263–79.
31.
HarrisonDavid A., MaryE. McLaughlin, and TerryM. Coalter (1996), “Context, Cognition, and Common Method Variance: Psychometric and Verbal Protocol Evidence,”Organizational Behavior and Human Decision Processes, 68 (3), 246–61.
32.
HastieReid, and ParkBernadette (1986), “The Relationship Between Memory and Judgment Depends on Whether the Judgment Is Memory-Based or Online,”Psychological Review, 93 (3), 258–68.
33.
HomburgChristian, GrozdanovicMarko, and KlarmannMartin (2007), “Responsiveness to Customers and Competitors: The Role of Cognitive and Affective Organizational Systems,”Journal of Marketing, 71 (July), 18–38.
34.
HoustonMichael J., and SudmanSeymour (1975), “A Methodological Assessment of the Use of Key Informants,”Social Science Research, 4 (2), 151–64.
35.
HuberGeorge P., and DanielJ. Power (1985), “Retrospective Reports of Strategic-Level Managers: Guidelines for Increasing Their Accuracy,”Strategic Management Journal, 6 (2), 171–80.
36.
HunterJohn E., and FrankL. Schmidt (2004), Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, 2d ed.Thousand Oaks, CA: Sage Publications.
37.
JamesLawrence R., RobertG. Demaree, and WolfGerrit (1984), “Estimating Within-Group Interrater Reliability with and Without Response Bias,”Journal of Applied Psychology, 69 (1), 85–98.
38.
JamesLawrence R., RobertG. Demaree, and WolfGerrit (1993), “rWG: An Assessment of Within-Group Interrater Agreement,”Journal of Applied Psychology, 78 (2), 306–309.
39.
JapSandy D. (1999), “Pie-Expansion Efforts: Collaboration Processes in Buyer–Supplier Relationships,”Journal of Marketing Research, 36 (November), 461–75.
40.
JickTodd D. (1979), “Mixing Qualitative and Quantitative Methods: Triangulation in Action,”Administrative Science Quarterly, 24 (4), 602–611.
41.
JobeJared B., TourangeauRoger, and AlbertF. Smith (1993), “Contributions of Survey Research to the Understanding of Memory,”Applied Cognitive Psychology, 7 (7), 567–84.
42.
KozlowskiSteve W., and HattrupKeith (1992), “A Disagreement About Within-Group Agreement: Disentangling Issues of Consistency Versus Consensus,”Journal of Applied Psychology, 77 (2), 161–67.
43.
KumarAjith, and WilliamR. Dillon (1990), “On the Use of Confirmatory Measurement Models in the Analysis of Multiple-Informant Reports,”Journal of Marketing Research, 27 (February), 102–111.
44.
KumarAjith, and WilliamR. Dillon (1992), “An Integrative Look at the Use of Additive and Multiplicative Covariance Structure Models in the Analysis of MTMM Data,”Journal of Marketing Research, 29 (February), 51–64.
45.
KumarNirmalya, LouisW. Stern, and JamesC. Anderson (1993), “Conducting Interorganizational Research Using Key Informants,”Academy of Management Journal, 36 (6), 1633–51.
46.
LanceCharles E., DawsonBryan, BirkelbachDavid, and BrianJ. Hoffman (2010), “Method Effects, Measurement Error, and Substantive Conclusions,”Organizational Research Methods, 13 (3), 435–55.
47.
LanceCharles E., CarrieL. Noble, and StevenE. Scullen (2002), “A Critique of the Correlated Trait-Correlated Method and Correlated Uniqueness Models for Multitrait-Multimethod Data,”Psychological Methods, 7 (2), 228–44.
48.
LeeChang-Yang, and IshtiaqP. Mahmood (2009), “Inter-Industry Differences in Profitability: The Legacy of the Structure-Efficiency Debate Revisited,”Industrial and Corporate Change, 18 (3), 351–80.
49.
LipseyMark W., and DavidB. Wilson (2001), Practical Meta-Analysis.Thousand Oaks, CA: Sage Publications.
50.
MarchJames G., and RobertI. Sutton (1997), “Organizational Performance as a Dependent Variable,”Organization Science, 8 (6), 698–706.
51.
MarshHerbert W., BarbaraM. Byrne, and CravenRhonda (1992), “Overcoming Problems in Confirmatory Factor Analyses of MTMM Data: The Correlated Uniqueness Model and Factorial Invariance,”Multivariate Behavioral Research, 27 (4), 489–507.
52.
McGrawKenneth O., and WongS.P. (1996), “Forming Inferences About Some Intraclass Correlation Coefficients,”Psychological Methods, 1 (1), 30–46.
53.
NunnallyJum C. (1978), Psychometric Theory.New York: McGraw-Hill.
54.
PaulhusDelroy L. (1991), “Measurement and Control of Response Bias,” in Measures of Personality and Social Psychological Attitudes, JohnP. Robinson, PhillipR. Shaver, and LawrenceS. Wright, eds. San Diego: Academic Press.
55.
PhillipsLynne W. (1981), “Assessing Measurement Error in Key Informant Reports: A Methodological Note on Organizational Analysis in Marketing,”Journal of Marketing Research, 18 (November), 395–415.
56.
PillemerDavid B., LynnR. Goldsmith, AbigailT. Panter, and SheldonH. White (1988), “Very Long-Term Memories of the First Year in College,”Journal of Experimental Psychology: Learning, Memory, and Cognition, 14 (4), 709–715.
57.
PodsakoffPhilip M., ScottB. MacKenzie, LeeJeong-Yeon, and NathanP. Podsakoff (2003), “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies,”Journal of Applied Psychology, 88 (5), 879–903.
58.
PorterMichael E. (1980), Competitive Strategy: Techniques for Analyzing Industries and Competitors.New York: The Free Press.
59.
RindfleischAric, AlanJ. Malter, GanesanShankar, and MoormanChristine (2008), “Cross-Sectional Versus Longitudinal Survey Research: Concepts, Findings, and Guidelines,”Journal of Marketing Research, 45 (August), 261–79.
60.
RogelbergSteven G., and JeffreyM. Stanton (2007), “Introduction Understanding and Dealing with Organizational Survey Nonresponse,”Organizational Research Methods, 10 (2), 195–209.
61.
SeidlerJohn (1974), “On Using Informants: A Technique for Collecting Quantitative Data and Controlling for Measurement Error in Organizational Analysis,”American Sociological Review, 39 (12), 816–31.
62.
SilkAlvin J., and ManoharU. Kalwani (1982), “Measuring Influence in Organizational Purchase Decisions,”Journal of Marketing Research, 19 (May), 165–81.
63.
SlaterStanley F., and JohnC. Narver (1994), “Does Competitive Environment Moderate the Market Orientation–Performance Relationship?”Journal of Marketing, 58 (January), 46–55.
64.
SlovicPaul (1962), “Convergent Validation of Risk Taking Measures,”Journal of Abnormal and Social Psychology, 65 (1), 68–71.
65.
SpectorPaul E. (2006), “Method Variance in Organizational Research: Truth or Urban Legend?”Organizational Research Methods, 9 (2), 221–32.
66.
StarbuckWilliam H., and JohnM. Mezias (1996), “Opening Pandora's Box: Studying the Accuracy of Managers' Perceptions,”Journal of Organizational Behavior, 17 (2), 99–117.
67.
SudmanSeymour, BickartBarbara, BlairJohnny, and MenonGeeta (1994), “The Effects of Level of Participation on Reports of Behavior and Attitudes by Proxy Reporters,” in Autobiographical Memory and the Validity of Retrospective Reports, SchwarzNorbert, and SudmanSeymour, eds. New York: Springer.
68.
SutcliffeKathleen M. (1994), “What Executives Notice: Accurate Perceptions in Top Management Teams,”Academy of Management Journal, 37 (5), 1360–78.
69.
ThompsonSimon G., and StephenJ. Sharp (1999), “Explaining Heterogeneity in Meta-Analysis: A Comparison of Methods,”Statistics in Medicine, 18 (20), 2693–2708.
70.
TinsleyHoward E.A., and DavidJ. Weiss (1975), “Interrater Reliability and Agreement of Subjective Judgements,”Journal of Counseling Psychology, 22 (4), 358–74.
71.
TourangeauRoger, LanceJ. Rips, and RasinskiKenneth (2000), The Psychology of Survey Response.Cambridge, UK: Cambridge University Press.
72.
Van BruggenGerrit H., GaryL. Lilien, and KackerManish (2002), “Informants in Organizational Marketing Research: Why Use Multiple Informants and How to Aggregate Responses,”Journal of Marketing Research, 39 (November), 469–78.
73.
WebbEugene J., DonaldT. Campbell, RichardD. Schwartz, and SechrestLee (1966), Unobtrusive Measures: Nonreactive Research in the Social Sciences.Chicago: Rand McNally.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
