Construct validity continues to pose challenges in the organizational sciences. To capture difficult-to-measure constructs of interest, researchers have often relied on content analysis. One content analysis technique, computer-aided text analysis (CATA), is particularly attractive because of the ability to process large samples with high speeds and reliabilities. Unfortunately, inconsistent guidance exists to guide researchers through the use of this tool in a manner compatible with accepted methods used to validate constructs in a rigorous manner. The authors review research using content analysis to examine the extent to which such studies integrate methods for assessing content, external, discriminant, and predictive validity. To provide direction for organizational researchers interested in using CATA to measure theoretically based constructs relevant to the management field, they suggest a number of possible procedures to enhance construct validity. They illustrate these procedures using the construct of entrepreneurial orientation.
Abrahamson E., & Park C. ( 1994). Concealment of negative organizational outcomes: An agency theory perspective. Academy of Management Journal, 37, 1302-1334.
2.
Amernic J., Craig R., & Tourish D. ( 2007). The transformational leader as pedagogue, physician, architect, commander, and saint: Five root metaphors in Jack Welch’s letters to stockholders of General Electric. Human Relations , 60, 1839-1872.
3.
Andrews A.O., & Welbourne T.M. (2000). The people/performance balance in IPO firms: The effect of the chief executive officer’s financial orientation. Entrepreneurship Theory and Practice, 25, 93-106.
4.
Atuahene-Gima K., & Ko A. ( 2001). An empirical investigation of the effect of market orientation and entrepreneurship orientation alignment on product innovation. Organization Science, 12, 54-74.
5.
Bagozzi R.P., Yi Y., & Phillips L.W. ( 1991). Assessing construct validity in organizational research . Administrative Science Quarterly, 36, 421-458.
Bartholomew S., & Smith A.D. ( 2006). Improving survey response rates from chief executive officers in small firms: The importance of social networks. Entrepreneurship Theory and Practice, 30, 83-96.
8.
Baruch Y. ( 1999). Response rate in academic studies-A comparative analysis . Human Relations, 52, 421-438.
9.
Bettis R.A., & Hitt M.A. ( 1995). The new competitive landscape. Strategic Management Journal, 16, 7-19.
10.
Bligh M.C., Kohles J.C., & Meindl J.R. ( 2004a). Charisma under crisis: Presidential leadership, rhetoric, and media responses before and after the September 11th terrorist attacks . The Leadership Quarterly, 15, 211-239.
11.
Bligh M.C., Kohles J.C., & Meindl J.R. ( 2004b). Charting the language of leadership: A methodological investigation of President Bush and the crisis of 9/11. Journal of Applied Psychology, 89, 562-574.
12.
Bowman E.H. ( 1984). Content analysis of annual reports for corporate strategy and risk. Interfaces, 14, 61-71.
13.
Boyd B.K., Gove S., & Hitt M.A. ( 2005). Construct measurement in strategic management research: Illusion or reality?Strategic Management Journal, 26, 239-257.
14.
Campbell D.T., & Fiske D.W. ( 1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
15.
Carley K.M. ( 1997). Extracting team mental models through textual analysis . Journal of Organizational Behavior, 18, 533-558.
16.
Chandler G.N., & Lyon D. W. (2001). Issues of research design and construct measurement in entrepreneurship research: The past decade. Entrepreneurship Theory and Practice, 25, 101-113.
17.
Chen M. J., & Hambrick D.C. ( 1995). Speed, stealth and selective attack: How small firms differ from large firms in competitive behavior. Academy of Management Journal, 38, 453-482.
18.
Cheng Y., Liu M. H., & Qian J. ( 2006). Buy-side analysts, sell-side analysts, and investment decisions of money managers. Journal of Financial and Quantitative Analysis, 4, 51-83.
19.
Cook T. D., & Campbell D.T. ( 1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin .
20.
Courtis J.K. ( 1982). Private shareholder response to corporate annual report . Accounting and Finance, 22, 53-72.
21.
Covin J.G., & Slevin D.P. ( 1988). The influence of organization structure on the utility of an entrepreneurial top management style. Journal of Management Studies, 25, 217-234.
22.
Covin J.G., & Slevin D.P. ( 1989). Strategic management of small firms in hostile and benign environments. Strategic Management Journal, 10, 75-87.
23.
Cronbach L.J. ( 1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (pp. 443-507). Washington, DC: American Council on Education.
Daily C.M., Certo S.T., & Dalton D.R. ( 2005). Investment bankers and IPO pricing: Does prospectus information matter?Journal of Business Venturing, 20, 93-111.
26.
D’Aveni R., & MacMillan I. ( 1990). Crisis and the content of managerial communications: A study of the focus of attention of top managers in surviving and failing firms. Administrative Science Quarterly, 35, 634-657.
27.
Deephouse D.L. ( 1996). Does isomorphism legitimate?Academy of Management Journal, 39, 1024-1039.
28.
Deephouse D.L. ( 2000). Media reputation as a strategic resource: An integration of mass communication and resource-based theories. Journal of Management, 26, 1091-1112.
29.
Doucet L., & Jehn K.A. ( 1997). Analyzing harsh words in a sensitive setting: American expatriates in communist China. Journal of Organizational Behavior, 18, 559-582.
30.
Dowling G.R., & Kabanoff B. ( 1996). Computer-aided content analysis: What do 240 advertising slogans have in common? Marketing Letters, 7, 63-75.
31.
Duriau V.J., Reger R.K., & Pfarrer M.D. ( 2007). A content analysis of the content analysis literature in organizational studies: Research themes, data sources, and methodological refinements. Organizational Research Methods, 10, 5-34.
32.
Dyer W. G., & Whetten D.A. ( 2006). Family firms and social responsibility: Preliminary evidence from the S&P 500. Entrepreneurship Theory and Practice , 30, 785-802.
33.
Edwards J.R. ( 2000). Multidimensional constructs in organizational behavior research: An integrative analytical framework. Organizational Research Methods, 4, 144-192.
34.
Ellis L. ( 1994). Research methods in the social sciences. Madison, WI: WCB Brown & Benchmark.
35.
Emrich C.G., Brower H.H., Feldman J.M., & Garland H. ( 2001). Images in words: Presidential rhetoric, charisma, and greatness. Administrative Science Quarterly, 46, 527-557.
36.
Fendt J., & Sachs W. ( 2008). Grounded theory method in management research. Organizational Research Methods, 11, 430-455.
37.
Ferrier W.J. ( 2001). Navigating the competitive landscape: The drivers and consequences of competitive aggressiveness. Academy of Management Journal, 44, 858-877.
38.
George G., Wiklund J., & Zahra S.A. ( 2005). Ownership and the internationalization of small firms . Journal of Management, 31, 210-233.
39.
Gioia D. A., & Chittipeddi K. (1991). Sensemaking and sensegiving in strategic change initiation. Strategic Management Journal , 12, 433-448.
40.
Godfrey P.C., & Hill C. W. L. (1995). The problem of unobservables in strategic management research. Strategic Management Journal, 16, 519-533.
41.
Goodman R. ( 1980). Annual reports serving a dual marketing function-Report as survey. Public Relations Quarterly, 36, 21-24.
42.
Hair J.F., Anderson R.E., Tatham R.L., & Black W.C. ( 1998). Multivariate data analysis. Upper Saddle River, NJ: Prentice Hall.
43.
Hambrick D.C. ( 2007). The field of management’s devotion to theory: Too much of a good thing? Academy of Management Journal, 50, 1346-1352.
Hitt M. A., & Ireland R.D. ( 2005). The Blackwell encyclopedia of management: Entrepreneurship (2nd ed.). Oxford, UK: Blackwell.
46.
Holsti O.R. ( 1969). Content analysis for the social sciences and humanities . Reading, MA: Addison-Wesley.
47.
Hughes M., & Morgan R.E. ( 2007). Deconstructing the relationship between entrepreneurial orientation and business performance at the embryonic stage of firm growth . Industrial Marketing Management, 36, 651-661.
48.
Hult G. T. M., Snow C. C., & Kandemir D. ( 2003). The role of entrepreneurship in building cultural competitiveness in different organizational types. Journal of Management, 29, 401-426.
49.
Ireland R.D., Reutzel C.R., & Webb J.W. ( 2005). Entrepreneurship research in AMJ: What has been published, and what might the future hold?Academy of Management Journal , 48, 556-564.
50.
Ireland R.D., Webb J.W., & Coombs J.E. ( 2005). Theory and methodology in entrepreneurship research. Research Methodology in Strategy and Management, 2, 111-141.
51.
Jauch L.R., Osborn R.N., & Martin T.N. ( 1980). Structured content analysis of cases: Complementary method for organizational research. Academy of Management Review , 5, 517-526.
52.
Kabanoff B.Waldersee, R., & Cohen M. ( 1995). Espoused values and organizational change themes. Academy of Management Journal, 38, 1075-1104.
53.
Karren R. J., & Barringer M.W. (2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5, 337-361.
54.
Kazanjian R.K. ( 1988). Relation of dominant problems to stages of growth in technology-based new ventures. Academy of Management Review , 31, 257-279.
55.
Kerlinger F.N., & Lee H.B. ( 2000). Foundations of behavioral research (4th ed.). Orlando, FL: Thomson Learning .
56.
Konrad A. M., & Mangel R. ( 2000). The impact of work-life programs on firm productivity . Strategic Management Journal, 21, 1225-1237.
57.
Kreiser P.M., Marino L.D., & Weaver K.M. ( 2002). Assessing the psychometric properties of the entrepreneurial orientation scale: A multi-country analysis. Entrepreneurship Theory and Practice, 26, 71-94.
58.
Krippendorff K. ( 2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA : Sage.
59.
Lance C.E., Butts M.M., & Michels L.C. ( 2006). The sources of four commonly reported cutoff criteria: What did they really say?Organizational Research Methods , 9, 202-220.
60.
Lumpkin G.T., & Dess G.G. ( 1996). Clarifying the entrepreneurial orientation construct and linking it to performance. Academy of Management Review , 21, 135-172.
61.
Lumpkin G.T., & Dess G.G. ( 2001). Linking two dimensions of entrepreneurial orientation to firm performance: The moderating role of environment and industry life cycle. Journal of Business Venturing, 16, 429-451.
62.
Lumpkin G.T., & Dess G.G. ( 2005). Entrepreneurial orientation. In M. A. Hitt & R. D. Ireland (Eds.), The Blackwell encyclopedia of management: Entrepreneurship (pp. 104-107). Oxford, UK: Blackwell.
63.
Lyon D.W., Lumpkin G.T., & Dess G.G. ( 2000). Enhancing entrepreneurial orientation research: Operationalizing and measuring a key strategic decision making process. Journal of Management, 26, 1055-1085.
64.
Maitlis S., & Lawrence T.B. ( 2007). Triggers and enablers of sensegiving in organizations . Academy of Management Journal, 50, 57-84.
65.
Marino K.E., Castaldi R.M., & Dollinger M.J. (1989). Content analysis in entrepreneurship research: The case of initial public offerings. Entrepreneurship Theory and Practice, 14, 51-66.
66.
Markel N. ( 1998). A semiotic contribution to libel action law. International Journal for the Semiotics of Law, 11, 49-56.
67.
McAdams D.P., & Zeldow P.B. ( 1993). Construct validity and content analysis. Journal of Personality Assessment, 61, 243-245.
68.
McGahan A.M., & Porter M.E. ( 1997). How much does industry matter, really?Strategic Management Journal, 18, 15-30.
69.
Mehran H. ( 1995). Executive compensation structure, ownership, and firm performance. Journal of Financial Economics, 38, 163-184.
70.
Meyer A. D. (1991). What is strategy’ s distinctive competence?Journal of Management, 17, 821-833.
71.
Michalisin M.D. ( 2001). Validity of annual report assertions about innovativeness: An empirical investigation. Journal of Business Research, 53, 151-161.
72.
Miller D. ( 1983). The correlates of entrepreneurship in three types of firms. Management Science, 29, 770-791.
73.
Morgan R.E., & Strong C.A. ( 2003). Business performance and dimensions of strategic orientation . Journal of Business Research, 56, 163-176.
74.
Morris R. ( 1994). Computerized content analysis in management research: A demonstration of advantages & limitations. Journal of Management, 20, 903-931.
75.
Mossholder K.W., Settoon R.P., Harris S. G., & Armenakis A.A. (1995). Measuring emotion in open-ended survey responses: An application of textual data analysis. Journal of Management, 21, 335-355.
76.
Naldi L., Nordquist M., Sjoberg K., & Wiklund J. ( 2007). Entrepreneurial orientation, risk taking, and performance in family firms. Family Business Review, 20, 33-58.
Nunnally J.C., & Bernstein I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
79.
Palmer T.B., & Short J.C. ( 2008). Mission statements in U.S. colleges of business: An empirical examination of their content with linkages to configurations and performance . Academy of Management Learning and Education, 7, 454-470.
80.
Pearce J. A., & David F. ( 1987). Corporate mission statements: The bottom line. Academy of Management Executive, 1, 109-116.
81.
Pennebaker J.W., Mehl M. R., & Niederhoffer K.G. (2003). Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54, 547-577.
82.
Porac J.F., Wade J.B., & Pollock T.G. ( 1999). Industry categories and the politics of the comparable firm in CEO compensation. Administrative Science Quarterly , 44, 112-144.
83.
Potter W. J., & Levine-Donnerstein D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
84.
Rauch A., Wiklund J., Frese M., & Lumpkin G.T. ( 2004). Entrepreneurial orientation and business performance: Cumulative empirical evidence. In S. A. Zahra, C. G. Brush, P. Davidsson, J. Fiet , P. G. Greene, R. T. Harrison, M. Lerner , C. Mason, G. D. Meyer, J. Sohl & A. Zacharakis (Eds.), Frontiers of Entrepreneurship Research (pp. 164-177). Wellesley, MA: Babson.
85.
Reilly F.K., & Wright D.J. ( 2002). Alternative small-cap stock benchmarks. Journal of Portfolio Management, 28, 82-95.
86.
Riffe D., & Lacy S. & Fico, F.G. ( 2005). Analyzing media messages: Using quantitative content analysis in research (2nd ed.). Mahwah, NJ: Erlbaum.
Rogelberg S.G., & Stanton J.M. ( 2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195-209.
89.
Rosenberg S.D., Schnurr P.P., & Oxman T.E. ( 1990). Content analysis: A comparison of manual and computerized systems. Journal of Personality Assessment, 54, 298-310.
90.
Rumelt R. ( 1991). How much does industry matter? Strategic Management Journal, 13, 119-134.
91.
Russell Investments. (2008a). Russell 3000® Index Fact Sheet. Retrieved March 23, 2009 , from http://www.russell.com/indexes/characteristics_fact_sheets/us/Russell_3000_Index.asp
92.
Russell Investments. (2008b). Russell 2000® Index Fact Sheet. Retrieved May 20, 2008, from http://www.russell.com/indexes/characteristics_fact_sheets/US/Russell_2000_Index.asp
Savall H., Zardet V., Bonnet M., & Péron M. (2008). The emergence of implicit criteria actually used by reviewers of qualitative research articles: Case of aEuropean journal. Organizational Research Methods, 11, 510-540.
95.
Scandura T.A., & Williams E.A. ( 2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248-1264.
96.
Schnurr P.P., Rosenberg S.D., Oxman T.E., & Tucker G.J. ( 1986). A methodological note on content analysis: Estimates of reliability. Journal of Personality Assessment, 50, 601-609.
97.
Shane S. ( 2004). Encouraging university entrepreneurship? The effect of the Bayh-Dole Act on university patenting in the United States. Journal of Business Venturing, 19, 127-151.
98.
Short J.C., Ketchen D.J., & Palmer T.B. ( 2002). The role of sampling in strategic management research on performance: A two-study analysis. Journal of Management , 28, 363-385.
99.
Short J.C., Ketchen D.J., Palmer T.B., & Hult T.M. ( 2007). Firm, strategic group, and industry influences on performance . Strategic Management Journal, 28, 147-167.
100.
Short J.C., & Palmer T.B. ( 2003). Organizational performance referents: An empirical examination of their content and influences. Organizational Behavior and Human Decision Processes, 90, 209-224.
101.
Short J.C., & Palmer T.B. ( 2008). The application of DICTION to content analysis research in strategic management. Organizational Research Methods, 11, 727-752.
102.
Singh K., Ang S.H., & Leong S.M. ( 2003). Increasing replication for knowledge accumulation in strategy research. Journal of Management, 29, 533-549.
103.
Skalski, P.D. ( 2002). Computer content analysis software. In K. A. Neuendorf (author), The content analysis guidebook. Thousand Oaks, CA: Sage .
104.
Slater S.F., & Narver J.C. ( 2000). The positive effect of a market orientation on business profitability: A balanced replication. Journal of Business Research, 48, 69-73.
105.
Smart D.T., & Conant J.S. ( 1994). Entrepreneurial orientation, distinctive marketing competencies and organizational performance. Journal of Applied Business Research, 10, 28-38.
106.
Sonpar K., & Golden-Biddle K. (2008). Using content analysis to elaborate adolescent theories of organization. Organizational Research Methods, 11, 795-814.
107.
Stam W., & Elfring T. ( 2008). Entrepreneurial orientation and new venture performance: The moderating role of intra- and extraindustry social capital. Academy of Management Journal, 51, 97-111.
108.
Stevenson R.L. ( 2001). In praise of dumb clerks: Computer-assisted content analysis . In M. D. West (Ed.), Theory, method and practice in computer content analysis (pp. 3-12). Portsmouth, NH: Greenwood.
109.
Tang J., Tang Z., Marino L.D., Zhang Y., & Li Q. ( 2008). Exploring an inverted u-shape relationship between entrepreneurial orientation and performance in Chinese ventures. Entrepreneurship Theory and Practice, 30, 219-239.
110.
Tsang E. W. K., & Kwan K. ( 1999). Replication and theory development in organizational science: A critical realist perspective. Academy of Management Review, 24, 759-780.
111.
Van De Ven A. H., & Johnson P.E. ( 2006). Knowledge for theory and practice. Academy of Management Review, 31, 802-821.
112.
Van Maanen J. ( 1979). Reclaiming qualitative methods for organizational research: A preface. Administrative Science Quarterly, 24, 520-526.
113.
Wade J.B., Porac J.F., & Pollock T.G. ( 1997). Worth, words, and the justification of executive pay . Journal of Organizational Behavior, 18, 641-664.
114.
Walter A., Auer M., & Ritter T. ( 2006). The impact of network capabilities and entrepreneurial orientation on university spin-off performance. Journal of Business Venturing, 21, 541-567.
Zahra S.A. ( 1991). Predictors and financial outcomes of corporate entrepreneurship: An exploratory study. Journal of Business Venturing, 6, 259-285.
117.
Zahra S.A. ( 1996). Governance, ownership, and corporate entrepreneurship: The moderating impact of industry technological opportunities. Academy of Management Journal, 39, 1713-1735.
118.
Zahra S.A., & Covin J.G. ( 1995). Contextual influences on the corporate entrepreneurship performance relationship: A longitudinal analysis. Journal of Business Venturing, 10, 43-58.
