Sage Journals: Discover world-class research

Abstract

Organizational scientists must capitalize on the big data revolution to better understand the nomothetic, idiographic, multilevel, and/or dynamic processes that make up today’s workplace. Simultaneously, researchers must collect high-quality data and be careful, diligent, and deliberate during data wrangling and data analysis so that all results can be replicated and all inferences are appropriate. Unfortunately, big data create many uncommon challenges during data acquisition and data wrangling that must be considered and overcome to fulfill the promise and potential of big data. Specifically, during acquisition, organizational scientists must become familiar with concepts like web scraping and databases, determine how to divide big data files into manageable chunks for cleaning and analysis, all while ensuring not to violate data usage rules and regulations. Likewise, once acquired, to effectively wrangle data so that they are ready for analysis researchers must be able to handle multiple file formats and data encoding standards, utilize a variety of software to visualize and diagnose data structure, and be adept at using functions and algorithms to determine variable structure and evaluate records and variables for missing or erroneous information. The current article provides a concise definition of big data and addresses each of these novel challenges and concepts related to big data acquisition and wrangling, specifically focusing on providing guidance and recommendations. Finally, a detailed big data example, team development using play-by-play basketball data, is provided. Each step of the process of scraping the data from the web as well as wrangling the multilevel big data into tidy data form is discussed, accompanied by a supplemental R file that contains all of the code necessary for researchers to replicate the described procedure.

Keywords

big data data acquisition data wrangling data cleaning

Get full access to this article

View all access options for this article.

References

Arbuckle

J. L.

(1996). Full information estimation in the presence of incomplete data. In Marcoulides

G. A.

Schumacker

R. E.

(Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243–277). Mahwah, NJ: Lawrence Erlbaum.

Braun

M. T.

Kuljanin

(2015). Big data and the challenge of construct validity. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 521–527.

Braun

M. T.

Kuljanin

DeShon

R. P.

(2013). Spurious results in the analysis of longitudinal data in organizational research. Organizational Research Methods, 16, 302–330.

Codd

E. F.

(1990). The relational model for database management: Version 2. Boston, MA: Addison-Wesley.

Culpepper

S. A.

Aguinis

(2011). R is for revolution: A cutting-edge, free open source statistical package. Organizational Research Methods, 14, 735–740.

Dasu

Johnson

(2003). Exploratory data mining and data cleaning. Hoboken, NJ: John Wiley.

Enders

C. K.

Bandalos

D. L.

(2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430–457.

Ganster

D. C.

Fox

M. L.

Dwyer

D. J.

(2001). Explaining employees’ health care costs: A prospective examination of stressful job demands, personal control, and physiological reactivity. Journal of Applied Psychology, 86, 954–964.

Graham

J. W.

(2003). Adding missing-data-relevant variables to FIML-based structural equation models. Structural Equation Modeling, 10, 80–100.

10.

Graham

J. W.

Hofer

S. M.

(2000). Multiple imputation in multivariate research. In Little

T. D.

Schnabel

K. U.

Baumert

(Eds.), Modeling longitudinal and multilevel data: Practical issues, applied approaches, and specific examples (pp. 201–218). Mahwah, NJ: Lawrence Erlbaum.

11.

Graham

J. W.

Hofer

S. M.

MacKinnon

D. P.

(1996). Maximizing the usefulness of data obtained with planned missing value patterns: An application of maximum likelihood procedures. Multivariate Behavioral Research, 31, 197–218.

12.

Grand

J. A.

Braun

M. T.

Kuljanin

Kozlowski

S. W. J.

Chao

G. T.

(2016). The dynamics of team cognition: A process-oriented theory of knowledge emergence in teams. Journal of Applied Psychology, 101, 1353–1385.

13.

Guzzo

R. A.

Fink

King

Tonidandel

Landis

(2015). Big data recommendations for industrial-organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 491–508.

14.

Hernandez

Newman

Jeon

(2016). Twitter analysis: Methods for data management and validation of a word count dictionary to measure city-level job satisfaction. In Tonidandel

King

Cortina

(Eds.), Big data at work: The data science revolution and organizational psychology (pp. 64–114). New York, NY: Routledge.

15.

King

E. B.

Tonidandel

Cortina

J. M.

Fink

A. A.

(2016). Building understanding of the data science revolution and I-O psychology. In Tonidandel

King

Cortina

(Eds.), Big data at work: The data science revolution and organizational psychology (pp. 1–15). New York, NY: Routledge.

16.

Kozlowski

S. W. J.

Chao

G. T.

(2012). The dynamics of emergence: Cognition and cohesion in work teams. Managerial and Decision Economics, 33, 335–354.

17.

Kozlowski

S. W. J.

Chao

G. T.

Chang

C. D.

Fernandez

(2016). Using big data to advance the science of team effectiveness. In Tonidandel

King

Cortina

(Eds.), Big data at work: The data science revolution and organizational psychology (pp. 272–309). New York, NY: Routledge.

18.

Kozlowski

S. W. J.

Chao

G. T.

Grand

J. A.

Braun

M. T.

Kuljanin

(2013). Advancing multilevel research design: Capturing the dynamics of emergence. Organizational Research Methods, 16, 581–615.

19.

Kozlowski

S. W. J.

Chao

G. T.

Grand

J. A.

Braun

M. T.

Kuljanin

(2016). Examining the dynamics of multilevel emergence: Learning and knowledge building in teams. Organizational Psychology Review, 6, 3–33.

20.

Little

(2002). Statistical analysis with missing data. Hoboken, NJ: John Wiley.

21.

Mathieu

J. E.

Chen

(2011). The etiology of the multilevel paradigm in management research. Journal of Management, 37, 610–641.

22.

Matlin

(2015, 3 4). Watch me write this article. Retrieved from http://fivethirtyeight.com/features/watch-me-write-this-article/

23.

McAfee

Brynjolfsson

(2012). Big data: The management revolution. Harvard Business Review, 90(10), 60–66.

24.

National Institutes of Health. (2003, 2 26). Final NIH statement on sharing research data. Retrieved from http://grants.nih.gov/grants/guide/notice-files/NOT-OD-03-032.html

25.

National Science Foundation. (n.d.). Open data at NSF. Retrieved from http://www.nsf.gov/data/

26.

Olguin

D. O.

Gloor

P. A.

Pentland

(2009, 3). Capturing individual and group behavior with wearable sensors. Proceedings of the AAAI Spring Symposium on Human Behavior Modeling, Stanford, CA.

27.

Olguin

D. O.

Waber

B. N.

Kim

Mohan

Ara

Pentland

(2009, 2). Sensible organizations: Technology and methodology for automatically measuring organizational behavior. IEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics, 39, 43–55.

28.

R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

29.

Rubin

D. B.

(1976). Inference with missing data. Biometrika, 63, 581–592.

30.

Smith

P. C.

Budzeika

K. A.

Edwards

N. A.

Johnson

S. M.

Bearse

L. N.

(1986). Guidelines for clean data: Detection of common mistakes. Journal of Applied Psychology, 71, 457–460.

31.

Tuckman

B. W.

(1965). Developmental sequence in small groups. Psychological Bulletin, 63, 384–399.

32.

Uzzi

Mukherjee

Stringer

Jones

(2013). Atypical combinations and scientific impact. Science, 342, 468–472.

33.

Vancouver

J. B.

Weinhardt

J. M.

Schmidt

A. M.

(2010). A formal, computational theory of multiple-goal pursuit: Integrating goal-choice and goal-striving processes. Journal of Applied Psychology, 95, 985–1008.

34.

Wickham

(2007). Reshaping data with the reshape package. Journal of Statistical Software, 21, 1–20.

35.

Wickham

(2009). ggplot2: Elegant graphics for data analysis. New York, NY: Springer-Verlag.

36.

Wickham

(2011). The split-apply-combine strategy for data analysis. Journal of Statistical Software, 40, 1–29.

37.

Wickham

(2014). Tidy data. Journal of Statistical Software, 59, 1–23.

38.

Wickham

(2015, 8 2). Interview with P. Coyle.

39.

Witt

L. A.

West

A. L.

Martir

A. B.

(2016, 4). Big data, big business, and big brother? Panel presented at the 31st conference of the Society for Industrial and Organizational Psychology, Anaheim, CA.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

Special Considerations for the Acquisition and Wrangling of Big Data

Abstract

Keywords

Get full access to this article

References

Supplementary Material