Construct Development and Validation in Three Practical Steps: Recommendations for Reviewers,Editors,and Authors *

Abstract

We review contemporary best practice for developing and validating measures of constructs in the organizational sciences. The three basic steps in scale development are: (a) construct definition, (b) choosing operationalizations that match the construct definition, and (c) obtaining empirical evidence to confirm construct validity. While summarizing this 3-step process [i.e., Define-Operationalize-Confirm], we address many issues in establishing construct validity and provide a checklist for journal reviewers and authors when evaluating the validity of measures used in organizational research. Among other points, we pay special attention to construct conceptualization, acknowledging existing constructs, improving existing measures, multidimensional constructs, macro-level constructs, and the need for independent samples to confirm construct validity and measurement equivalence across subpopulations.

Keywords

construct development construct measurement scale development construct validity

Get full access to this article

View all access options for this article.

References

Ackerman

P. L.

Beier

M. E.

Boyle

M. O.

(2005). Working memory and intelligence: The same or different constructs? Psychological Bulletin, 131(1), 30-60. Retrieved from https://doi.org/10.1037/0033-2909.131.1.30

Aguinis

Vandenberg

R. J.

(2014). An ounce of prevention is worth a pound of cure: Improving research quality before data collection. Annual Review of Organizational Psychology and Organizational Behavior, 1, 569-595.

Alwin

D. F.

Krosnick

J. A.

(1991). The reliability of survey attitude measurement: The influence of question and respondent attitudes. Sociological Methods and Research, 20(1), 139-181. https://doi.org10.1177_0049124191020001005

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing (2014). Standards for educational and psychological testing. AERA.

Anderson

B. S.

Kreiser

P. M.

Kuratko

D. F.

Hornsby

J. S.

Eshima

(2015). Reconceptualizing entrepreneurial orientation. Strategic Management Journal, 36(10), 1579-1596. Retrieved from https://doi.org/10.1002/smj.2298

Anderson

J. C.

Gerbing

D. W.

(1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103(3), 411-423. https://doi.org/10.1037/0033-2909.103.3.411

Anderson

J. C.

Gerbing

D. W.

(1991). Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities. Journal of Applied Psychology, 76(5), 732-740. Retrieved from https://doi.org/10.1037/0021-9010.76.5.732

Asparouhov

Muthén

(2009). Exploratory structural equation modeling. Structural Equation Modeling, 16(3), 397-438. https://doi.org10.1080/10705510903008204

Bagozzi

R. P

. (1981). Evaluating structural equation models with unobservable variables and measurement error: A comment. Journal of Marketing Research (pre-1986), 18(000003), 375. Retrieved from https://doi.org/10.1177/002224378101800312

10.

Bandalos

D. L.

(2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9(1), 78-102. Retrieved from https://doi.org/10.1207/S15328007SEM0901_5

11.

Bennett

R. J.

Robinson

S. L.

(2000). Development of a measure of workplace deviance. Journal of Applied Psychology, 85(3), 349-360. Retrieved from https://doi.org/10.1037/0021-9010.85.3.349

12.

Bergh

D. D.

Fairbank

J. F.

(2002). Measuring and testing change in strategic management research. Strategic Management Journal, 23(4), 359-365. https://doi.org/10.1002/smj.232

13.

Bliese

P. D.

(2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349-381). Jossey-Bass.

14.

Bollen

K. A.

Diamantopoulos

(2017). In defense of causal–formative indicators: A minority report. Psychological Methods, 22(3), 581-596. Retrieved from https://doi.org/10.1037/met0000056

15.

Boyd

B. K.

(1994). Board control and ceo compensation. Strategic Management Journal, 15(5), 335-344. Retrieved from https://doi.org/10.1002/smj.4250150502

16.

Boyd

B. K.

Gove

Hitt

M. A.

(2005). Construct measurement in strategic management research: Illusion or reality? Strategic Management Journal, 26(3), 239-257. Retrieved from https://doi.org/10.1002/smj.444

17.

Bozeman

D. P.

Perrewé

P. L.

(2001). The effect of item content overlap on organizational commitment questionnaire–turnover cognitions relationships. Journal of Applied Psychology, 86(1), 161-173. Retrieved from https://doi.org/10.1037/0021-9010.86.1.161

18.

Bradburn

Sudman

Wansink

(2004). Asking questions: The definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires (First edition ed.). Jossey-Bass.

19.

Brady

D. L.

Brown

D. J.

Liang

L. H.

(2017). Moving beyond assumptions of deviance: The reconceptualization and measurement of workplace gossip. Journal of Applied Psychology, 102(1), 1-25. Retrieved from https://doi.org/10.1037/apl0000164

20.

Brannick

M. T.

Spector

P. E.

(1990). Estimation problems in the block-diagonal model of the multitrait-multimethod matrix. Applied Psychological Measurement, 14(4), 325-339. Retrieved from https://doi.org/10.1177/014662169001400401

21.

Brayfield

A. H.

Rothe

H. F.

(1951). An index of job satisfaction. Journal of Applied Psychology, 35(5), 307-311. Retrieved from https://doi.org/10.1037/h0055617

22.

Bromiley

Rau

Zhang

(2017). Is R&D risky? Strategic Management Journal, 38(4), 876-891. Retrieved from https://doi.org/10.1002/smj.2520

23.

Brown

Maydeu-Olivares

(2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36-52. Retrieved from https://doi.org/10.1037/a0030641

24.

Brown

T. A.

(2015). Confirmatory factor analysis for applied research. Guilford Press.

25.

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. https://doi.org/10.1037/h0046016

26.

Campbell

J. P.

Gasser

M. B.

Oswald

(1996). The substantive nature of job performance variability. In K. Murphy (ed.), Individual differences and behavior in organizations (1st ed, pp. 258–299).

27.

Cardinal

L. B.

Sitkin

S. B.

Long

C. P.

(2010). A configurational theory of control. Organizational Control, 51, 79, 85-100.

28.

Carpenter

N. C.

Son

Harris

T. B.

Alexander

A. L.

Horner

M. T.

(2016). Don’t forget the items:Item-level meta-analytic and substantive validity techniques for reexamining scale validation. Organizational Research Methods, 19(4), 616-650. https://doi.org/10.1177/1094428116639132

29.

Cattell

R. B.

(1947). Confirmation and clarification of primary personality factors. Psychometrika, 12, 197-220. https://doi.org/10.1007/BF02289253

30.

Certo

S. T.

Busenbark

J. R.

Kalm

LePine

J. A.

(2020). Divided we fall: How ratios undermine research in strategic management. Organizational Research Methods, 23(2), 211-237. https://doi.org/10.1177/1094428118773455

31.

Chan

(1998a). The conceptualization and analysis of change over time: An integrative approach incorporating longitudinal mean and covariance structures analysis (LMACS) and multiple indivator latent growth modeling (MLGM). Organizational Research Methods, 2(4), 421-483 Retreived from 10.1177/109442819814004.

32.

Chan

(1998b). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83(2), 234-246. Retrieved from doi.org/10.1037/0021-9010.83.2.234

33.

Chang

C.-H.

Ferris

D. L.

Johnson

R. E.

Rosen

C. C.

Tan

J. A.

(2012). Core self-evaluations:A review and evaluation of the literature. Journal of Management, 38(1), 81-128. https://doi.org/10.1177/0149206311419661

34.

Chen

Bliese

P. D.

Mathieu

J. E.

(2005). Conceptual framework and statistical procedures for delineating and testing multilevel theories of homology. Organizational Research Methods, 8(4), 375-409. Retrevied from https://doi.org//10.1177/1094428105280056

35.

Cho

(2016). Making reliability reliable: A systematic approach to reliability coefficients. Organizational Research Methods, 19(4), 651-682. https://doi.org/10.1177/1094428116656239

36.

Cho

Kim

(2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207-230. https://doi.org/10.1177/1094428114555994

37.

Clark

L. A.

Watson

(1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309-319. https://doi.org/10.1037/1040-3590.7.3.309

38.

Clark

L. A.

Watson

(2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412-1427. https://doi.org/10.1037/pas0000626

39.

Cole

D. A.

Ciesla

J. A.

Steiger

J. H.

(2007). The insidious effects of failing to include design-driven correlated residuals in latent-variable covariance structure analysis. Psychological Methods, 12(4), 381-398. https://doi.org/10.1037/1082-989x.12.4.381

40.

Colquitt

J. A.

(2001). On the dimensionality of organizational justice: A construct validation of a measure. Journal of Applied Psychology, 86(3), 386-400. Retrieved from 10.1037/0021-9010.86.3.386

41.

Colquitt

J. A.

Sabey

T. B.

Rodell

J. B.

Hill

E. T.

(2019). Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness. Journal of Applied Psychology, 104(10), 1243-1265. https://doi.org/10.1037/apl0000406. 10.1037/apl0000406.supp (Supplemental)

42.

Conway

J. M.

Huffcutt

A. I.

(2003). A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods, 6(2), 147-168. https://doi.org/10.1177/1094428103251541

43.

Cornwell

J. M.

Dunlap

W. P.

(1994). On the questionable soundness of factoring ipsative data: A response to Saville & Willson (1991). Journal of Occupational & Organizational Psychology, 67(2), 89-100. https://doi.org/10.1111/j.2044-8325.1994.tb00553.x

44.

Cortina

J. M.

(1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104. https://doi.org/10.1037/0021-9010.78.1.98

45.

Cortina

J. M.

Green

J. P.

Keeler

K. R.

Vandenberg

R. J.

(2016). Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizational Research Methods, 20(3), 350-378. https://doi.org/10.1177/1094428116676345

46.

Cortina

J. M.

Sheng

Keener

S. K.

Keeler

K. R.

Grubb

Schmitt

Tonidandel

Summerville

K. M.

Heggestad

E. D.

Banks

(2020). From alpha and omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the journal of applied psychology. Journal of Applied Psychology, 105(12), 1351-1381. https://doi.org/10.1037/apl0000815

47.

Credé

Tynan

M. C.

Harms

P. D.

(2017). Much ado about grit: A meta-analytic synthesis of the grit literature. Journal of Personality and Social Psychology, 113(3), 492-511. https://doi.org/10.1037/pspp0000102

48.

Cronbach

L. J.

(1960). Essentials of psychological testing (2nd ed). Harper.

49.

Cronbach

L. J.

Gleser

G. C.

Nanda

Rajaratnam

(1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley & Sons.

50.

Cronbach

L. J.

Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 4, 281-302. https://doi.org/10.1037/h0040957

51.

Danneels

(2016). Survey measures of first- and second-order competences. Strategic Management Journal, 37(10), 2174-2188. https://doi.org/10.1002/smj.2428

52.

DeShon

R. P.

(2002). Generalizability theory. In Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 189-220). Jossey-Bass.

53.

DeVellis

R. F.

(2003). Scale development: Theory and applications (2 ed., Vol. 26). Sage Publications.

54.

DeVellis

R. F

. (2017). Scale development : theory and applications (Fourth edition. ed.). Thousand Oaks, CA: Sage.

55.

Drasgow

Kanfer

(1985). Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70(4), 662-680. https://doi.org/10.1037/0021-9010.70.4.662

56.

Edwards

J. R.

(2001). Multidimensional constructs in organizational behavior research. Organizational Research Methods, 4(2), 144-192. https://doi.org/10.1177/109442810142004

57.

Edwards

J. R.

(2002). Alternatives to difference scores: Polynomial regression analysis and response surface methodology. In Drasgow

Schmitt

(Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 350-400). Jossey-Bass.

58.

Edwards

J. R.

(2003). Construct validation in organizational behavior research. In Greenberg

(Ed.), Organizational behavior: the state of the science (2nd edition ed.). Erlbaum.

59.

Edwards

J. R.

(2011). The fallacy of formative measurement. Organizational Research Methods, 14 (2), 370-388. https://doi.org/10.1177/1094428110378369

60.

Edwards

J. R.

Bagozzi

. (2000). Relationships between constructs and measures. Psychological Methods, 5(2), 155–174. https://doi.org/10.1037/1082-989X.5.2.155

61.

Edwards

J. R.

Berry

J. W.

(2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13(4), 668-689. https://doi.org/10.1177/1094428110380467

62.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299. https://doi.org/10.1037/1082-989X.4.3.272

63.

Fan

Sivo

S. A.

(2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42(3), 509-529. https://doi.org/10.1080/00273170701382864

64.

Finkel

S. E.

(1995). Causal analysis with panel data. Sage Publications.

65.

Fiske

D. W.

(1949). Consistency of the factorial structures of personality ratings from different sources. Journal of Abnormal and Social Psychology, 44(3), 329-344. https://doi.org/10.1037/h0057198

66.

Ford

J. K.

MacCallum

R. C.

Tait

(1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39(2), 291-314. https://doi.org/10.1111/j.1744-6570.1986.tb00583.x

67.

Fornell

Larcker

D. F.

(1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. Journal of Marketing Research, 18(3), 382-388. https://doi.org/10.2307/3150980

68.

Gerbing

D. W.

Anderson

J. C.

(1985). The effects of sampling error and model characteristics on parameter estimation for maximum likelihood confirmatory factor analysis. Multivariate Behavioral Research, 20(3), 255-271. https://doi.org/10.1207/s15327906mbr2003_2

69.

Grégoire

D. A.

Shepherd

D. A.

Schurer Lambert

(2010). Measuring opportunity-recognition beliefs: Illustrating and validating an experimental approach. Organizational Research Methods, 13(1), 114-145. https://doi.org/10.1177/1094428109334369

70.

Heggestad

E. D.

Scheaf

D. J.

Banks

G. C.

Monroe Hausfeld

Tonidandel

Williams

E. B.

(2019). Scale adaptation in organizational science research: A review and best-practice recommendations. Journal of Management, 45(6), 2596-2627. https://doi.org/10.1177/0149206319850280

71.

Helm

J. L.

Castro-Schilo

Oravecz

(2017). Bayesian Versus maximum likelihood estimation of multitrait–multimethod confirmatory factor models. Structural Equation Modeling, 24(1), 17-30. https://doi.org/10.1080/10705511.2016.1236261

72.

Hicks

L. E.

(1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167-184. https://doi.org/10.1037/h0029780

73.

Hinkin

T. R.

(1995). A review of scale development practices in the study of organizations. Journal of Management, 21(5), 967-988. https://doi.org/10.1177/014920639502100509

74.

Hinkin

T. R.

(1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104-121. https://doi.org/10.1177/109442819800100106

75.

Hinkin

T. R.

Tracey

J. B.

(1999). An analysis of variance approach to content validation. Organizational Research Methods, 2(2), 175-186. https://doi.org/10.1177/109442819922004

76.

Hoelter

J. W.

(1983). The analysis of covariance structures:Goodness-of-fit indices. Sociological Methods & Research, 11(3), 325-344. https://doi.org/10.1177/0049124183011003003

77.

Horn

J. L.

McArdle

J. J.

(1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3-4), 117-144. https://doi.org/10.1080/03610739208253916

78.

Howell

R. D.

Breivik

Wilcox

J. B.

(2007). Reconsidering formative measurement. Psychological Methods, 12(2), 205-218. https://doi.org/10.1037/1082-989X.12.2.205

79.

L. t.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

80.

Ironson

G. H.

Smith

P. C.

Brannick

M. T.

Gibson

W. M.

Paul

K. B.

(1989). Construction of a job in general scale: A comparison of global, composite, and specific measures. Journal of Applied Psychology, 74(2), 193-200. https://doi.org/10.1037/0021-9010.74.2.193

81.

Jackson

D. L

. (2001). Sample size and number of parameter estimates in maximum likelihood confirmatory factor analysis: A monte carlo investigation. Structural Equation Modeling, 8(2), 205–223. doi:10.1207/S15328007SEM0802_3

82.

Jackson

D. L.

(2003). Revisiting sample size and number of parameter estimates: Some support for the N:q hypothesis. Structural Equation Modeling: A Multidisciplinary Journal, 10(1), 128-141. https://doi.org/10.1207/S15328007SEM1001_6

83.

Jackson

D. L.

(2007). The effect of the number of observations per parameter in misspecified confirmatory factor analytic models. Structural Equation Modeling: A Multidisciplinary Journal, 14(1), 48-76. https://doi.org/10.1080/10705510709336736

84.

Jackson

D. L.

Gillaspy

J. A.

Jr. Purc-Stephenson

(2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6-23. https://doi.org/10.1037/a0014694

85.

James

L. R.

Jones

A. P.

(1974). Organizational climate: A review of theory and research. Psychological Bulletin, 81(12), 1096-1112. https://doi.org/10.1037/h0037511

86.

Jex

S. M.

Bliese

P. D.

(1999). Efficacy beliefs as a moderator of the impact of work-related stressors: A multilevel study. Journal of Applied Psychology, 84(3), 349-361. https://doi.org/10.1037/0021-9010.84.3.349

87.

Judge

T. A.

Piccolo

R. F.

(2004). Transformational and transactional leadership: A meta-analytic test of their relative validity. Journal of Applied Psychology, 89(5), 755-768. https://doi.org/10.1037/0021-9010.89.5.755

88.

Kellermanns

F. W.

Walter

Lechner

Floyd

S. W.

(2005). The lack of consensus about strategic consensus: Advancing theory and research. Journal of Management, 31(5), 719-737. https://doi.org/10.1177/0149206305279114

89.

Kelley

T. L.

(1927). Interpretation of educational measurements. World Book.

90.

Ketchen

D. J.

Ireland

R. D.

Baker

L. T.

(2013). The use of archival proxies in strategic management studies: Castles made of sand? Organizational Research Methods, 16(1), 32-42. https://doi.org/10.1177/1094428112459911

91.

Klein

H. J.

Cooper

J. T.

Molloy

J. C.

Swanson

J. A.

(2014). The assessment of commitment: Advantages of a unidimensional, target-free approach. Journal of Applied Psychology, 99(2), 222-238. https://doi.org/10.1037/a0034751

92.

Klein

H. J.

Molloy

J. C.

Brinsfield

C. T.

(2012). Reconceptualizing workplace commitment to redress a stretched construct: Revisiting assumptions and removing confounds. Academy of Management Review, 37(1), 130-151. https://doi.org/10.5465/arma.2010.0018

93.

Klein

K. J.

Kozlowski

S. W. J.

(2000a). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In Klein

K. J.

Kozlowski

S. W. J.

(Eds.), Multilevel theory, research, and methods in organizations; foundations, extensions, and new directions (pp. 3-90). Jossey-Bass.

94.

Klein

K. J.

Kozlowski

S. W. J.

(Eds.). (2000b). Multilevel theory, research, and methods in organizations; foundations, extensions, and new directions. Jossey-Bass.

95.

Kronmal

R. A.

(1993). Spurious correlation and the fallacy of the ratio standard revisited. Journal of the Royal Statistical Society, 156(3), 379-392. https://doi.org/10.2307/2983064

96.

Krosnick

J. A.

Presser

(2010). Questions and questionanaire design. In Wright

J. D.

Marsden

P. V.

(Eds.), Handbook of survey research. Emerald Group.

97.

Lance

C. E.

Vandenberg

R. J.

(2002). Confirmatory factor analysis. In Drasgow

Schmitt

(Eds.), Measuring and analyzing behavior in organizations: Advances in measurement and data analysis (pp. 221-254). Jossey-Bass.

98.

Lance

C. E.

Vandenberg

R. J.

(2009). Statistical and methodological myths and urban legends : Doctrine, verity and fable in the organizational and social sciences. Routledge.

99.

Landy

F. J.

(1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41(11), 1183-1192. https://doi.org/10.1037/0003-066X.41.11.1183

100.

Schmidt

F. L.

Harter

J. K.

Lauver

K. J.

(2010). The problem of empirical redundancy of constructs in organizational research: An empirical investigation. Organizational Behavior and Human Decision Processes, 112(2), 112-125. https://doi.org/10.1016/j.obhdp.2010.02.003

101.

LeBreton

J. M.

Senter

J. L.

(2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815-852. https://doi.org/10.1177/1094428106296642

102.

Little

T. D.

Rhemtulla

Gibson

Schoemann

A. M.

(2013). Why the items versus parcels controversy needn’t be one. Psychological Methods, 18(3), 285-300. https://doi.org/10.1037/a0033266

103.

Loevinger

(1957). Objective tests as instruments of psychological theory. Psychological Reports, 3(3), 635-694. https://doi.org/10.2466/pr0.1957.3.3.635

104.

Lord

F. M.

Novick

M. R

. (1968). Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.

105.

MacCallum

R. C.

Browne

M. W.

Sugawara

H. M.

(1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130-149. https://doi.org/10.1037/1082-989X.1.2.130

106.

MacCallum

R. C.

Lee

Browne

M. W.

(2010). The issue of isopower in power analysis for tests of structural equation models. Structural Equation Modeling, 17(1), 23-41. https://doi.org/10.1080/10705510903438906

107.

MacCallum

R. C.

Roznowski

Necowitz

L. B.

(1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490-504. https://doi.org/10.1037/0033-2909.111.3.490

108.

MacKenzie

S. B.

Podsakoff

P. M.

Jarvis

C. B.

(2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90(4), 710-730. https://doi.org/10.1037/0021-9010.90.4.710

109.

MacKenzie

S. B.

Podsakoff

P. M.

Podsakoff

N. P.

(2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing technologies. MIS Quarterly, 35(2), 293-A295. Retrieved from http://ezproxy.gsu.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=60461934&site=ehost-live&scope=site.

110.

Marsh

H. W.

Hau

K.-T.

Wen

(2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Structural Equation Modeling, 11(3), 320-342. https://doi.org/10.1207/s15328007sem1103_2

111.

Marsh

H. W.

Lüdtke

Nagengast

Morin

A. J. S.

Von Davier

(2013). Why item parcels are (almost) never appropriate: Two wrongs do not make a right—camouflaging misspecification with item parcels in CFA models. Psychological Methods, 18(3), 257-284. https://doi.org/10.1037/a0032773. 10.1037/a0032773.supp (Supplemental)

112.

Mathieu

J. E.

Farr

J. L.

(1991). Further evidence for the discriminant validity of measures of organizational commitment, job involvement, and job satisfaction. Journal of Applied Psychology, 76(1), 127-133. https://doi.org/10.1037/0021-9010.76.1.127

113.

Mayer

Roberts

R. D.

Barsade

S. G.

(2008). Emerging research in emotional intelligence. Annual Review of Psychology, 59, 507-536. https://doi.org/10.1146/annurev.psych.59.103006.093646

114.

McDonald

R. P.

(2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64-82. https://doi.org/10.1037/1082-989x.7.1.64

115.

McKenny

A. F.

Aguinis

Short

J. C.

Anglin

A. H.

(2018). What doesn’t get measured does exist: Improving the accuracy of computer-aided text analysis. Journal of Management, 44(7), 2909-2933. https://doi.org/10.1177/0149206316657594

116.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412-433. https://doi.org/10.1037/met0000144

117.

Meade

A. W.

(2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77(4), 531-552. https://doi.org/10.1348/0963179042596504

118.

Meade

A. W.

Kroustalis

C. M.

(2006). Problems with item parceling for confirmatory factor analytic tests of measurement invariance. Organizational Research Methods, 9(3), 369-403. https://doi.org/10.1177/1094428105283384

119.

Messick

(1981). Constructs and their vicissitudes in educational and psychological measurement. Psychological Bulletin, 89(3), 575-588. https://doi.org/10.1037/0033-2909.89.3.575

120.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741

121.

Miller

C. C.

Washburn

N. T.

Glick

W. H.

(2013). PERSPECTIVE—The myth of firm performance. Organization Science, 24(3), 948-964. https://doi.org/10.1287/orsc.1120.0762

122.

Morin

A. J. S.

Arens

A. K.

Marsh

H. W.

(2016). A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Structural Equation Modeling: A Multidisciplinary Journal, 23(1), 116-139. https://doi.org/10.1080/10705511.2014.961800

123.

Mowday

R. T.

Steers

R. M.

Porter

L. W.

(1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14(2), 224-247. https://doi.org/10.1016/0001-8791(79)90072-1

124.

Muthen

B. O.

(1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22(3), 376-398. https://doi.org/10.1177/0049124194022003006

125.

Muthén

L. K.

Muthén

B. O.

(2002). How to use a Monte Carlo study to decide on sample size and determine power. Structural Equation Modeling, 9(4), 599-620. https://doi.org/10.1207/S15328007SEM0904_8

126.

Newman

D. A.

Harrison

D. A.

Carpenter

N. C.

Rariden

S. M.

(2016). Construct mixology: Forming new management constructs by combining old ones. Academy of Management Annals, 10(1), 943-995. https://doi.org/10.1080/19416520.2016.1161965

127.

Newman

D. A.

Joseph

D. L.

Hulin

C. L.

(2010). Job attitudes and employee engagement: Considering the attitude “A-factor”. In Handbook of employee engagement: Perspectives, issues, research and practice (pp. 43-61). Edward Elgar Publishing.

128.

Newman

D. A.

Sin

H.-P.

(2020). Within-group agreement (rWG): Two theoretical parameters and their estimators. Organizational Research Methods, 23(1), 30-64. https://doi.org/10.1177/1094428118809504

129.

Norman

W. T.

(1963). Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. The Journal of Abnormal and Social Psychology, 66(6), 574-583. https://doi.org/10.1037/h0040291

130.

Nunnally

J. C.

(1970). Introduction to psychological measurement. McGraw-Hill.

131.

Nunnally

J. C

. (1978). Psychometric theory. New York: McGraw Hill.

132.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory. McGraw-Hill Inc.

133.

O'Boyle

E. H.

Jr. Williams

L. J.

(2011). Decomposing model fit: Measurement vs. Theory in organizational research using latent variables. Journal of Applied Psychology, 96(1), 1-12. https://doi.org/10.1037/a0020539

134.

O'Neill

O. A.

Rothbard

(2017). Is love all you need? The effects of emotional culture, suppression, and work–family conflict on firefighter risk-taking and health. Academy of Management Journal, 60(1), 78-108. https://doi.org/10.5465/amj.2014.0952

135.

Ones

D. S.

Viswesvaran

Reiss

A. D.

(1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81(6), 660-679. https://doi.org/10.1037/0021-9010.81.6.660

136.

Ostroff

(1993). Comparing correlations based on individual-level and aggregated data. Journal of Applied Psychology, 78(4), 569-582. https://doi.org/10.1037/0021-9010.78.4.569

137.

Pandey

S. K.

(2019). Applying natural language processing capabilities in computerized textual analysis to measure organizational culture. Organizational Research Methods, 22(3), 765-797. https://doi.org/10.1177/1094428117745648

138.

Pedhazur

E. J.

Schmelkin

L. P.

(1991). Measurement, design, and analysis: An integrated approach. Lawrence Erlbaum Associates, Publishers.

139.

Petrenko

O. V.

Aime

Ridge

Hill

(2016). Corporate social responsibility or CEO narcissism? CSR motivations and organizational performance. Strategic Management Journal, 37(2), 262-279. https://doi.org/10.1002/smj.2348

140.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J.-Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879. https://doi.org/10.1037/0021-9010.88.5.879

141.

Podsakoff

P. M.

MacKenzie

S. B.

Podsakoff

N. P.

(2016). Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences. Organizational Research Methods, 19(2), 159-203. https://doi.org/10.1177/1094428115624965

142.

Richardson

H. A.

Simmering

M. J.

Sturman

M. C.

(2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12(4), 762. https://doi.org/10.1177/1094428109332834

143.

Rönkkö

Cho

(2020). An updated guideline for assessing discriminant validity. Organizational Research Methods, 0(0), 1094428120968614. https://doi.org/10.1177/1094428120968614

144.

Rousseau

D. M.

(1985). Issues of level in organizational research: Multi-level and cross-level perspectives. Research in Organizational Behavior, 7, 1-37.

145.

Scarpello

Campbell

J. P.

(1983). Job satisfaction: Are all the parts there? Personnel Psychology, 36(3), 577-600. https://doi.org/10.1111/j.1744-6570.1983.tb02236.x

146.

Schmidt

F. L.

Ilies

(2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs. Psychological Methods, 8(2), 206-224. https://doi.org/10.1037/1082-989X.8.2.206

147.

Schmitt

(1982). The use of analysis of covariance structures to assess beta and gamma change. Multivariate Behavioral Research, 17(3), 343-358. https://doi.org/10.1207/s15327906mbr1703_3

148.

Schriesheim

C. A.

Castro

S. L.

Cogliser

C. C

. (1999). Leader–member exchange (LMX) research: A comprehensive review of theory, measurement, and data-analytic practices. The Leadership Quarterly, 10(1), 63–113. 10.1016/S1048-9843(99)80009-5

149.

Schriesheim

C. A.

Hinkin

T. R.

Podsakoff

P. M.

(1991). Can ipsative and single-item measures produce erroneous results in field studies of French and Raven's (1959) five bases of power? An empirical investigation. Journal of Applied Psychology, 76(1), 106-114. https://doi.org/10.1037/0021-9010.76.1.106

150.

Schriesheim

C. A.

Powers

K. J.

Scandura

T. A.

Gardiner

C. C.

Lankau

M. J.

(1993). Improving construct measurement in management research: Comments and a quantitative approach for assessing the theoretical content adequacy of paper-and-pencil survey-type instruments. Journal of Management, 19(2), 385-417. https://doi.org/10.1177/014920639301900208

151.

Schwab

D. P.

(1980). Construct validity in organizational behavior. Research in Organizational Behavior, 2, 03-43.

152.

Schwab

D. P.

(2005). Research methods for organizational studies (second ed.). Lawrence Erlbaum Associates.

153.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company.

154.

Shaffer

J. A.

DeGeest

(2016). Tackling the problem of construct proliferation: A guide to assessing the discriminant validity of conceptually related constructs. Organizational Research Methods, 19(1), 80-110. https://doi.org/10.1177/1094428115598239

155.

Shipp

A. J.

Edwards

J. R.

Lambert

L. S.

(2009). Conceptualization and measurement of temporal focus: The subjective experience of the past, present, and future. Organizational Behavior and Human Decision Processes, 110(1), 1-22. https://doi.org/10.1016/j.obhdp.2009.05.001

156.

Tay

Jebb

A. T.

(2018). Establishing construct continua in construct validation: The process of continuum specification. Advances in Methods and Practices in Psychological Science, 1(3), 375-388. https://doi.org/10.1177/2515245918775707

157.

Tay

Woo

S. E.

Vermunt

J. K.

(2014). A conceptual and methodological framework for psychometric isomorphism: Validation of multilevel construct measures. Organizational Research Methods, 17(1), 77-106. https://doi.org/10.1177/1094428113517008

158.

Tepper

B. J.

Dimotakis

Lambert

L. S.

Koopman

Matta

F. K.

Park

H. M.

Goo

(2018). Examining follower responses to transformational leadership from a dynamic, person–environment fit perspective. Academy of Management Journal, 61(4), 1343-1368. https://doi.org/10.5465/amj.2014.0163

159.

Tourangeau

Conrad

F. G.

Couper

M. P.

(2013). The science of web surveys. Oxford University Press.

160.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press.

161.

Van Knippenberg

Sitkin

S. B.

(2013). A critical assessment of charismatic—transformational leadership research: Back to the drawing board? Academy of Management Annals, 7(1). https://doi.org/10.1080/19416520.2013.759433

162.

Vandenberg

R. J.

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-69.

163.

Widaman

K. F.

(1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9(1), 1-26. https://doi.org/10.1177/014662168500900101

164.

Williams

L. J.

McGonagle

(2016). Four research designs and a comprehensive analysis strategy for investigating common method variance with self-report measures using latent variables. Journal of Business & Psychology, 31(3), 339-359. https://doi.org/10.1007/s10869-015-9422-9

165.

Williams

L. J.

Vandenberg

R. J.

Edwards

J. R.

(2009). Structural equation modeling in management research: A guide for improved analysis. The Academy of Management Annals, 3, 543-604. Retrieved from http://www.informaworld.com/10.1080/19416520903065683.

166.

Willis

G. B.

(2005). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.

167.

Wiseman

(2009). On the use and misuse of ratios in strategic management research. In Research methods in strategy and management (Vol. Vol. 5, pp. 75-110). Emerald Group Publishing Limited.

168.

Woehr

D. J.

Putka

D. J.

Bowler

M. C.

(2012). An examination of G-theory methods for modeling multitrait–multimethod data: Clarifying links to construct validity and confirmatory factor analysis. Organizational Research Methods, 15(1), 134-161. https://doi.org/10.1177/1094428111408616

169.

Wolfson

M. A.

Tannenbaum

S. I.

Mathieu

J. E.

Maynard

M. T.

(2018). A cross-level investigation of informal field-based learning and performance improvements. Journal of Applied Psychology, 103(1), 14-36. https://doi.org/10.1037/apl0000267

170.

Zhang

Sun

Drasgow

Chernyshenko

O. S.

Nye

C. D.

Stark

White

L. A.

(2020). Though forced, still valid: Psychometric equivalence of forced-choice and single-statement measures. Organizational Research Methods, 23(3), 569-590. https://doi.org/10.1177/1094428119836486

171.

Zickar

M. J.

(2020). Measurement development and evaluation. Annual Review of Organizational Psychology and Organizational Behavior, 7(1), 213-232. https://doi.org/10.1146/annurev-orgpsych-012119-044957