Analyzing Policy Capturing Data Using Structural Equation Modeling for Within-Subject Experiments (SEMWISE)

Abstract

We present the SEMWISE (structural equation modeling for within-subject experiments) approach for analyzing policy capturing data. Policy capturing entails estimating the weights (or utilities) of experimentally manipulated attributes in predicting a response variable of interest (e.g., the effect of experimentally manipulated market-technology combination characteristics on perceived entrepreneurial opportunity). In the SEMWISE approach, a factor model is specified in which latent weight factors capture individually varying effects of experimentally manipulated attributes on the response variable. We describe the core SEMWISE model and propose several extensions (how to incorporate nonbinary attributes and interactions, model multiple indicators of the response variable, relate the latent weight factors to antecedents and/or consequences, and simultaneously investigate several populations of respondents). The primary advantage of the SEMWISE approach is that it facilitates the integration of individually varying policy capturing weights into a broader nomological network while accounting for measurement error. We illustrate the approach with two empirical examples, compare and contrast the SEMWISE approach with multilevel modeling (MLM), discuss how researchers can choose between SEMWISE and MLM, and provide implementation guidelines.

Keywords

structural equation modeling within-subject experiments multilevel data policy capturing conjoint analysis

Get full access to this article

View all access options for this article.

References

Aguinis

Bradley

K. J.

(2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17(4), 351–371. doi:10.1177/1094428114547952

Aiman-Smith

Bauer

T. N.

Cable

D. M.

(2001). Are you attracted? Do you intend to pursue? A recruiting policy-capturing study. Journal of Business and Psychology, 16(2), 219–237. doi:10.1023/A:1011157116322

Aiman-Smith

Scullen

S. E.

Barr

S. H.

(2002). Conducting studies of decision making in organizational contexts: A tutorial for policy-capturing and other regression-based techniques. Organizational Research Methods, 5(4), 388–414. doi:10.1177/109442802237117

Bollen

K. A.

(1989). Structural equations with latent variables. New York, NY: Wiley.

Caruso

E. M.

Rahnev

D. A.

Banaji

M. R.

(2009). Using conjoint analysis to detect discrimination: Revealing covert preferences from overt choices. Social Cognition, 27(1), 128–137. doi:10.1521/soco.2009.27.1.128

Catlin

J. R.

Wang

(2013). Recycling gone bad: When the option to recycle increases resource consumption. Journal of Consumer Psychology, 23(1), 122=127. doi:10.1016/j.jcps.2012.04.001

Cheung

G. W.

Rensvold

R. B.

(2001). The effects of model parsimony and sampling error on the fit of structural equation models. Organizational Research Methods, 4(3), 236–264. doi:10.1177/109442810143004

Connelly

B. L.

Ketchen

D. J.

Gangloff

K. A.

Shook

C. L.

(2016). Investor perceptions of CEO successor selection in the wake of integrity and competence failures: A policy capturing study. Strategic Management Journal, 37(10), 2135–2151. doi:10.1002/smj.2430

Cuddy

A. J.

Glick

Beninger

(2011). The dynamics of warmth and competence judgments, and their outcomes in organizations. Research in Organizational Behavior, 31, 73–98. doi:10.1016/j.riob.2011.10.004

10.

Curran

P. J.

(2003). Have multilevel models been structural equation models all along? Multivariate Behavioral Research, 38(4), 529–568. doi:10.1207/s15327906mbr3804_5

11.

Edwards

J. R.

(2001). Multidimensional constructs in organizational behavior research: An integrative analytical framework. Organizational Research Methods, 4(2), 144–192. doi:10.1177/109442810142004

12.

Feingold

(2009). Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychological Methods, 14(1), 45–53. doi:10.1037/a0014699

13.

Fink

Penton-Voak

(2002). Evolutionary psychology of facial attractiveness. Current Directions in Psychological Science, 11(5), 154–158. doi:10.1111/1467-8721.00190

14.

Fiske

S. T.

Cuddy

A. J.

Glick

(2002). A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology, 82(6), 878–902. doi:10.1037//0022-3514.82.6.878

15.

Gelman

Hill

(2007). Data analysis using regression and multilevelhierarchical models (Vol. 1). New York, NY: Cambridge University Press.

16.

Graham

M. E.

Cable

D. M.

(2001). Consideration of the incomplete block design for policy-capturing research. Organizational Research Methods, 4(1), 26–45. doi:10.1177/109442810141002

17.

Grégoire

D. A.

Shepherd

D. A.

(2012). Technology-market combinations and the identification of entrepreneurial opportunities: An investigation of the opportunity-individual nexus. Academy of Management Journal, 55(4), 753–785. doi:10.5465/amj.2011.0126

18.

Helm

J. L.

Castro-Schilo

Oravecz

(2017). Bayesian versus maximum likelihood estimation of multitrait-multimethod confirmatory factor models. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 17–30. doi:10.1080/10705511.2016.1236261

19.

Hinkin

T. R.

(1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121. doi:10.1177/109442819800100106

20.

Jones

D. A.

Willness

C. R.

(2013). Corporate social performance, organizational reputation, and recruitment. In Cable

D. M.

K. Y. T.

(Eds.), The Oxford handbook of recruitment (pp. 298–313). Oxford, UK: Oxford University Press.

21.

Karren

R. J.

Barringer

M. W.

(2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5(4), 337–361. doi:10.1177/109442802237115

22.

Kline

R. B.

(2016). Principles and practice of structural equation modeling (4th ed.). New York, NY: The Guilford Press.

23.

Kristof-Brown

A. L.

Jansen

K. J.

Colbert

A. E.

(2002). A policy-capturing study of the simultaneous effects of fit with jobs, groups, and organizations. Journal of Applied Psychology, 87(5), 985–993. doi:10.1037//0021-9010.87.5.985

24.

LaHuis

D. M.

Hartman

M. J.

Hakoyama

Clark

P. C.

(2014). Explained variance measures for multilevel models. Organizational Research Methods, 17(4), 433–451. doi:10.1177/1094428114541701

25.

Lance

C. E.

Butts

M. M.

Michels

L. C.

(2006). The sources of four commonly reported cutoff criteria—What did they really say? Organizational Research Methods, 9(2), 202–220. doi:10.1177/1094428105284919

26.

Lance

C. E.

Noble

C. L.

Scullen

S. E.

(2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7(2), 228–244. doi:10.1037/1082-989X.7.2.228

27.

D. S.

Correll

Wittenbrink

(2015). The Chicago face database: A free stimulus set of faces and norming data. Behavior Research Methods, 47(4), 1122–1135. doi:10.3758/s13428-014-0532-5

28.

Maydeu-Olivares

Coffman

D. L.

(2006). Random intercept item factor analysis. Psychological Methods, 11(4), 344–362. doi:10.1037/1082-989X.11.4.344

29.

Mulaik

S. A.

(2009). Linear causal modeling with structural equations. Washington, DC: CRC Press.

30.

Muthén

L. K.

Muthén

B. O.

(2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599–620. doi:10.1207/S15328007SEM0904_8

31.

Newman

D. A.

Lyon

J. S.

(2009). Recruitment efforts to reduce adverse impact: Targeted recruiting for personality, cognitive ability, and diversity. Journal of Applied Psychology, 94(2), 298–317. doi:10.1037/a0013472

32.

Olsen

J. E.

Martins

L. L.

(2016). Racioethnicity, community makeup, and potential employees’ reactions to organizational diversity management approaches. Journal of Applied Psychology, 101(5), 657–672. doi:10.1037/apl0000080

33.

Patzelt

Shepherd

D. A.

(2008). The decision to persist with underperforming alliances: The role of trust and control. Journal of Management Studies, 45(7), 1217–1243. doi:10.1111/j.1467-6486.2008.00791.x

34.

Paulhus

D. L.

(1991). Measurement and control of response bias. In Robinson

J. P.

Shaver

P. R.

Wrightsman

L. S.

(Eds.), Measures of personality and social psychological attitudes (pp. 17–60). San Diego, CA: Academic Press.

35.

Ployhart

R. E.

Oswald

F. L.

(2004). Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods, 7(1), 27–65.

36.

Reb

Cropanzano

(2007). Evaluating dynamic performance: The influence of salient gestalt characteristics on performance ratings. Journal of Applied Psychology, 92(2), 490–499. doi:10.1037/0021-9010.92.2.490

37.

Rhemtulla

(2016). Population performance of SEM parceling strategies under measurement and structural model misspecification. Psychological Methods, 21(3), 348–368. doi:10.1037/met0000072

38.

Rosman

Lubatkin

O’Neill

(1994). Rigidity in decision behaviors: A within-subject test of information acquisition using strategic and financial informational cues. Academy of Management Journal, 37(4), 1017–1033. doi:10.2307/256610

39.

Rotundo

Sackett

P. R.

(2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87(1), 66–80. doi:10.1037/0021-9010.87.1.66

40.

Sauer

Auspurg

Hinz

Liebig

(2011). The application of factorial surveys in general population samples: The effects of respondent age and education on response times and response consistency. Survey Research Methods, 5(3), 89–102. doi:10.18148/srm/2011.v5i3.4625#sthash.YYreWVGE.dpuf

41.

Shepherd

D. A.

Patzelt

Baron

R. A.

(2013). “I Care about nature, but…”: Disengaging values in assessing opportunities that cause harm. Academy of Management Journal, 56(5), 1251–1273. doi:10.5465/amj.2011.0776

42.

Tomassetti

A. J.

Dalal

R. S.

Kaplan

S. A.

(2016). Is policy capturing really more resistant than traditional self-report techniques to socially desirable responding? Organizational Research Methods, 19(2), 255–285. doi:10.1177/1094428115627497

43.

Vandenberg

R. J.

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. doi:10.1177/109442810031002

44.

Weijters

Baumgartner

Schillewaert

(2013). Reversed item bias: An integrative model. Psychological Methods, 18(3), 320–334. doi:10.1037/a0032121

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB