Metric conjoint studies are a popular research design in the entrepreneurship domain. For these studies, test-retest reliabilities of ρ > .70 or higher are an often-cited reliability criterion. Despite their widespread use, however, there is little rigorous analysis of whether test-retest reliability in metric conjoint studies relates to model efficacy. Informed by a systematic literature review, we conducted two Monte Carlo simulations to evaluate the effects of various determinants of test-retest reliability in conjoint experiments. We then illustrate a workflow for entrepreneurship researchers employing conjoint designs to better evaluate—and communicate—confidence in statistical models estimated from conjoint data.
AguinisH.GottfredsonR. K.CulpepperS. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39(6), 1490–1528.
2.
Aiman-SmithL.ScullenS. E.BarrS. H. (2002). Conducting studies of decision making in organizational contexts: A tutorial for policy-capturing and other regression-based techniques. Organizational Research Methods, 5(4), 388–414.
3.
AndersonB. S.SchuelerJ.BaumM.WalesW. J.GuptaV. K. (2022). The chicken or the egg? Causal inference in entrepreneurial orientation–performance research. Entrepreneurship Theory and Practice, 46(6), 1569–1596.
4.
AntonakisJ.BendahanS.JacquartP.LaliveR. (2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21(6), 1086–1120.
5.
BaronR. A. (2006). Opportunity recognition as pattern recognition: How entrepreneurs “connect the dots” to identify new business opportunities. The Academy of Management Perspectives, 20(1), 104–119.
6.
BaumgartnerT. A. (2000). Estimating the stability reliability of a score. Measurement in Physical Education and Exercise Science, 4(3), 175–178.
7.
BlieseP. D.ChanD.PloyhartR. E. (2007). Multilevel methods. Organizational Research Methods, 10(4), 551–563.
8.
BowlingN. A.GibsonA. M.HouptJ. W.BrowerC. K. (2021). Will the questions ever end? Person-level increases in careless responding during questionnaire completion. Organizational Research Methods, 24(4), 718–738.
9.
BusenitzL. W.BarneyJ. B. (1997). Differences between entrepreneurs and managers in large organizations: Biases and heuristics in strategic decision-making. Journal of Business Venturing, 12(1), 9–30.
10.
CarminesE.ZellerR. (1979). Reliability and validity assessment. SAGE Publications.
11.
CloggC. C.PetkovaE.HaritouA. (1995). Statistical methods for comparing regression coefficients between models. American Journal of Sociology, 100(5), 1261–1293.
12.
CronbachL. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12(1), 1–16.
13.
DeSimoneJ. A. (2015). New techniques for evaluating temporal consistency. Organizational Research Methods, 18(1), 133–152.
14.
GelmanA.HillJ. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
15.
GelmanA.SternH. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328–331.
16.
GoldfeldK.Wujciak-JensJ. (2020). simstudy: Illuminating research methods through data generation. Journal of Open Source Software, 5(54), 2763.
17.
GrahamM. E.CableD. M. (2001). Consideration of the incomplete block design for policy-capturing research. Organizational Research Methods, 4(1), 26–45.
18.
GreenP. E.SrinivasanV. (1978). Conjoint analysis in consumer research: issues and outlook. The Journal of Consumer Research, 5(2), 103.
19.
GrégoireD. A.BinderJ. K.RauchA. (2019). Navigating the validity tradeoffs of entrepreneurship research experiments: A systematic review and best-practice suggestions. Journal of Business Venturing, 34(2), 284–310.
GuttmanL. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10, 255–282.
22.
HensherD. A. (2006). How do respondents process stated choice experiments? Attribute consideration under varying information load. Journal of Applied Econometrics (Chichester, England), 21(6), 861–878.
23.
HessS.HensherD. A.DalyA. (2012). Not bored yet—revisiting respondent fatigue in stated choice experiments. Transportation Research Part A: Policy and Practice, 46(3), 626–644.
24.
HyslopD. R.ImbensG. W. (2001). Bias from classical and other forms of measurement error. Journal of Business & Economic Statistics, 19(4), 475–481.
25.
KamC. C. S.MeyerJ. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541.
26.
KarrenR. J.BarringerM. W. (2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5(4), 337–361.
27.
KooT. K.LiM. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.
28.
LanceC. E.ButtsM. M.MichelsL. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say?Organizational Research Methods, 9(2), 202–220.
29.
LeeB.-K.LeeW.-N. (2004). The effect of information overload on consumer choice quality in an online environment. Psychology & Marketing, 21(3), 159–183.
30.
LeighT. W.MacKayD. B.SummersJ. O. (1984). Reliability and validity of conjoint analysis and self-explicated weights: A comparison. Journal of Marketing Research, 21(4), 456–462.
31.
LohrkeF. T.HollowayB. B.WoolleyT. W. (2010). Conjoint analysis in entrepreneurship research: A review and research agenda. Organizational Research Methods, 13(1), 16–30.
32.
LurieN. H. (2004). Decision making in information-rich environments: The role of information structure. The Journal of Consumer Research, 30(4), 473–486.
33.
MaulaM.StamW. (2020). Enhancing rigor in quantitative entrepreneurship research. Entrepreneurship Theory and Practice, 44(6), 1059–1090.
34.
McCulloughJ.BestR. (1979). Conjoint measurement: Temporal stability and structural reliability. Journal of Marketing Research, 16(1), 26–31.
35.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
36.
MoserK. J.TumasjanA.WelpeI. M. (2017). Small but attractive: Dimensions of new venture employer attractiveness and the moderating role of applicants’ entrepreneurial behaviors. Journal of Business Venturing, 32(5), 588–610.
37.
NunnallyJ. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
ScherbaumC. A.FerreterJ. M. (2009). Estimating statistical power and required sample sizes for organizational research using multilevel modeling. Organizational Research Methods, 12(2), 347–367.
40.
SchmidtF. L.HunterJ. E. (1999). Theory testing and measurement error. Intelligence, 27(3), 183–198.
41.
SchwarzN.OysermanD. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. The American Journal of Evaluation, 22(2), 127–160.
42.
ShaneS.VenkataramanS. (2000). The promise of entrepreneurship as a field of research. Academy of Management Review, 25(1), 217–226.
43.
ShepherdD. A.ZacharakisA. (2018). Conjoint analysis: A window of opportunity for entrepreneurship research. In KatzJ. A.CorbettA. (Eds.), Reflections and extensions on key papers of the first twenty-five years of advances (pp. 149–183). Emerald Publishing Limited.
44.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
45.
SireciS. G.SukinT. (2013). Test validity. In GeisingerK. F.BrackenB. A.CarlsonJ. F.HansenJ.-I. C.KuncelN. R.ReiseS. P.RodriguezM. C. (Eds.), APA handbook of testing and assessment in psychology, Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 61–84). American Psychological Association. https://doi.org/10.1037/14047-004
46.
TayL.DrasgowF. (2012). Theoretical, statistical, and substantive issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15(3), 363–384.
47.
TourangeauR.LanceJ. R.RasinskiK. (2000). The psychology of survey response. Cambridge University Press.
48.
TyszkaT. (1998). Two pairs of conflicting motives in decision making. Organizational Behavior and Human Decision Processes, 74(3), 189–211.
49.
van RosmalenJ.van HerkH.GroenenP. J. F. (2010). Identifying response styles: A latent-class bilinear multinomial logit model. Journal of Marketing Research, 47(1), 157–172.
50.
WeirJ. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research/National Strength & Conditioning Association, 19(1), 231–240.
51.
WennbergK.AndersonB. S. (2020). Editorial: Enhancing the exploration and communication of quantitative entrepreneurship research. Journal of Business Venturing, 35(3), 105938.
52.
ZhuZ.TomassettiA. J.DalalR. S.SchraderS. W.LooK.SabatI. E.AlaybekB.ZhouY.JonesC.FyffeS. (2022). A test-retest reliability generalization meta-analysis of judgments via the policy-capturing technique. Organizational Research Methods, 25(3), 541–574.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
