Best Practices in Data Collection and Preparation: Recommendations for Reviewers,Editors,and Authors

Abstract

We offer best-practice recommendations for journal reviewers, editors, and authors regarding data collection and preparation. Our recommendations are applicable to research adopting different epistemological and ontological perspectives—including both quantitative and qualitative approaches—as well as research addressing micro (i.e., individuals, teams) and macro (i.e., organizations, industries) levels of analysis. Our recommendations regarding data collection address (a) type of research design, (b) control variables, (c) sampling procedures, and (d) missing data management. Our recommendations regarding data preparation address (e) outlier management, (f) use of corrections for statistical and methodological artifacts, and (g) data transformations. Our recommendations address best practices as well as transparency issues. The formal implementation of our recommendations in the manuscript review process will likely motivate authors to increase transparency because failure to disclose necessary information may lead to a manuscript rejection decision. Also, reviewers can use our recommendations for developmental purposes to highlight which particular issues should be improved in a revised version of a manuscript and in future research. Taken together, the implementation of our recommendations in the form of checklists can help address current challenges regarding results and inferential reproducibility as well as enhance the credibility, trustworthiness, and usefulness of the scholarly knowledge that is produced.

Keywords

quantitative research qualitative research research design

Get full access to this article

View all access options for this article.

References

Aguinis

(2019). Performance management (4th ed.). Chicago, IL: Chicago Business Press.

Aguinis

Edwards

J. R.

(2014). Methodological wishes for the next decade and how to make wishes come true. Journal of Management Studies, 51, 143–174.

Aguinis

Gottfredson

R. K.

Joo

(2013). Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods, 16, 270–301.

Aguinis

Henle

C. A.

Ostroff

(2001). Measurement in work and organizational psychology. In Anderson

Ones

D. S.

Sinangil

H. K.

Viswesvaran

(Eds.), Handbook of industrial, work and organizational psychology (Vol. 1, pp. 27–50). London: Sage.

Aguinis

Y. H.

Joo

(2018). Gender productivity gap among star performers in STEM and other scientific fields. Journal of Applied Psychology, 103, 1283–1306.

Aguinis

Joo

(2015). Debunking myths and urban legends about how to identify influential outliers. In Lance

C. E.

Vandenberg

R. J.

(Eds.), More statistical and methodological myths and urban legends (pp. 206–223). New York, NY: Routledge.

Aguinis

Lawal

S. O.

(2012). Conducting field experiments using eLancing’s natural environment. Journal of Business Venturing, 27, 493–505.

Aguinis

O’Boyle

(2014). Star performers in twenty-first-century organizations. Personnel Psychology, 67, 313–350.

Aguinis

Pierce

C. A.

Bosco

F. A.

Muslin

I. S.

(2009). First decade of Organizational Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69–112.

10.

Aguinis

Ramani

R. S.

Alabduljader

(2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12, 83–110.

11.

Aguinis

Ramani

R. S.

Villamor

(2019). The first 20 years of Organizational Research Methods: Trajectory, impact, and predictions for the future. Organizational Research Methods. Advance online publication. doi:10.1177/1094428118786564

12.

Aguinis

Solarino

A. M.

(2019). Transparency and replicability in qualitative research: The case of interviews with elite informants. Strategic Management Journal. Advance online publication. doi:10.1002/SMJ.3015

13.

Aguinis

Vandenberg

R. J.

(2014). An ounce of prevention is worth a pound of cure: Improving research quality before data collection. Annual Review of Organizational Psychology and Organizational Behavior, 1, 569–595.

14.

Antonakis

House

R. J.

Simonton

D. K.

(2017). Can super smart leaders suffer from too much of a good thing? The curvilinear effect of intelligence on perceived leadership behavior. Journal of Applied Psychology, 102, 1003–1021.

15.

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73, 3–25.

16.

Baker

(2016). 1,500 scientists lift the lid on reproducibility. Nature, 533, 452–454.

17.

Banks

G. C.

O’Boyle

E. H.

Pollack

J. M.

White

C. D.

Batchelor

J. H.

Whelpley

C. E.

… Adkins

C. L.

(2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 5–20.

18.

Becker

T. E.

Atinc

Breaugh

J. A.

Carlson

K. D.

Edwards

J. R.

Spector

P. E.

(2016). Statistical control in correlational studies: 10 essential recommendations for organizational researchers. Journal of Organizational Behavior, 37, 157–167.

19.

Becker

T. E.

Robertson

M. M.

Vandenberg

R. J.

(2019). Nonlinear transformations in organizational research: Possible problems and potential solutions. Organizational Research Methods. Advance online publication. doi:10.1177/1094428118775205

20.

Bergh

D. D.

Sharp

B. M.

Aguinis

(2017). Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings. Strategic Organization, 15, 423–436.

21.

Bernerth

Aguinis

(2016). A critical review and best-practice recommendations for control variable usage. Personnel Psychology, 69, 229–283.

22.

Bettis

R. A.

Ethiraj

Gambardella

Helfat

Mitchell

(2016). Creating repeatable cumulative knowledge in strategic management. Strategic Management Journal, 37, 257–261.

23.

Bliese

P. D.

(2018). Editorial. Organizational Research Methods, 21, 3–5.

24.

Butler

Delaney

Spoelstra

(2017). The grey zone: Questionable research practices in the business school. Academy of Management Learning & Education, 16, 94–109.

25.

Byington

E. K.

Felps

(2017). Solutions to the credibility crisis in management science. Academy of Management Learning & Education, 16, 142–162.

26.

Camerer

C. F.

Dreber

Forsell

T-H.

Huber

Johannesson

… Wu

(2016). Evaluating replicability of laboratory experiments in economics. Science, 351, 1433–1436.

27.

Chen

Bamberger

P. A.

Song

Vashdi

D. R.

(2018). The effects of team reflexivity on psychological well-being in manufacturing teams. Journal of Applied Psychology, 103, 443–462.

28.

Chiu

Balkundi

Weinberg

F. J.

(2017). When managers become leaders: The role of manager network centralities, social power, and followers’ perception of leadership. Leadership Quarterly, 28, 334–348.

29.

Cohen

West

S. G.

Aiken

L. S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.

30.

Courtright

S. H.

Gardner

R. G.

Smith

T. A.

Mccormick

B. W.

Colbert

A. E.

(2016). My family made me do it: A cross-domain, self-regulatory perspective on antecedents to abusive supervision. Academy of Management Journal, 59, 1630–1652.

31.

Crawford

G. C.

Aguinis

Lichtenstein

Davidsson

McKelvey

(2015). Power law distributions in entrepreneurship: Implications for theory and research. Journal of Business Venturing, 30, 696–713.

32.

Dineen

B. R.

Duffy

M. K.

Henle

C. A.

Kiyoung

(2017). Green by comparison: Deviant and normative transmutations of job search envy in a temporal context. Academy of Management Journal, 60, 295–320.

33.

Follmer

E. H.

Talbot

D. L.

Kristof-Brown

A. L.

Astrove

S. L.

Billsberry

(2018). Resolution, relief, and resignation: A qualitative study of responses to misfit at work. Academy of Management Journal, 61, 440–465.

34.

Gilbert

(October 3, 2016). Rosetta landing: Comet-chasing probe makes final touchdown. Retrieved from https://www.cnn.com/2016/09/30/health/rosetta-landing-comet-philae/index.html

35.

Grand

J. A.

Rogelberg

S. G.

Banks

G. C.

Landis

R. S.

Tonidandel

(2018). From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. Perspectives on Psychological Science, 13, 448–456.

36.

Honig

Lampel

Baum

J. A. C.

Glynn

M. A.

Jing

Lounsbury

… van Witteloostuijn

(2018). Reflections on scientific misconduct in management: Unfortunate incidents or a normative crisis? Academy of Management Perspectives, 32, 412–442.

37.

Huang

J. L.

Ryan

A. M.

Zabel

K. L.

Palmer

(2014). Personality and adaptive performance at work: A meta-analytic investigation. Journal of Applied Psychology, 99, 162–179.

38.

Joo

Aguinis

Bradley

K. J.

(2017). Not all non-normal distributions are created equal: Improved theoretical and measurement precision. Journal of Applied Psychology, 102, 1022–1053.

39.

Kalaignanam

Kushwaha

T., M.

Steenkamp

J. E.

Tuli

K. R.

(2013). The effect of CRM outsourcing on shareholder value: A contingency perspective. Management Science, 59, 748–769.

40.

Kaul

Nary

Singh

(2018). Who does private equity buy? Evidence on the role of private equity from buyouts of divested businesses. Strategic Management Journal, 39, 1268–1298.

41.

Ketchen

D. J.

Ireland

R. D.

Baker

L. T.

(2013). The use of archival proxies in strategic management studies: Castles made of sand? Organizational Research Methods, 16, 32–42.

42.

Schmidt

F. L.

Putka

D. J.

(2009). The multifaceted nature of measurement artifacts and its implications for estimating construct-level relationships. Organizational Research Methods, 12, 165–200.

43.

Levitt

H. M.

Bamberg

Creswell

J. W.

Frost

D. M.

Josselson

Suárez-Orozco

(2018). Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73, 26–46.

44.

McDonnell

M-H.

Brayden

(2013). Keeping up appearances: Reputational threat and impression management after social movement boycotts. Administrative Science Quarterly, 58, 387–419.

45.

Neckebrouck

Schulze

Zellweger

(2018). Are family firms good employers? Academy of Management Journal, 61, 553–585.

46.

Newman

D. A.

(2014). Missing data: Five practical guidelines. Organizational Research Methods, 17, 372–411.

47.

Ryu

McCann

B. T.

Reuer

J. J.

(2018). Geographic co-location of partners and rivals: Implications for the design of R&D alliances. Academy of Management Journal, 61, 945–965.

48.

Schafer

J. L.

Graham

J. W.

(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177.

49.

Schmidt

F. L.

Hunter

J. E.

(2014). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.

50.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.

51.

Shaffer

J. A.

Postlethwaite

B. E.

(2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65, 445–494.

52.

Spector

P. E.

Brannick

M. T.

(2011). Methodological urban legends: The misuse of statistical control variables. Organizational Research Methods, 14, 287–305.

53.

Stanley

M. H. R.

Buldyrev

S. V.

Havlin

Mantegna

R. N.

Salinger

M. A.

Stanley

H. E.

(1995). Zipf plots and the size distribution of firms. Economics Letters, 49, 453–457.

54.

Stone-Romero

E. F.

(2011). Research strategies in industrial and organizational psychology: Nonexperimental, quasi-experimental, and randomized experimental research in special purpose and nonspecial purpose settings. In Zedeck

(Ed.), APA handbook of industrial and organizational psychology (Vol. 1, pp. 37–72). Washington, DC: American Psychological Association.

55.

Teddlie

(2007). Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research, 1, 77–100.

56.

Truelove

Kellogg

K. C.

(2016). The radical flank effect and cross-occupational collaboration for technology development during a power shift. Administrative Science Quarterly, 61, 662–701.

57.

Tsui

A. S.

(2013). The spirit of science and socially responsible scholarship. Management and Organization Review, 9, 375–394.

58.

Van Iddekinge

C. H.

Aguinis

Mackey

J. D.

DeOrtentiis

P. S.

(2018). A meta-analysis of the interactive, additive, and relative effects of cognitive ability and motivation on performance. Journal of Management, 44, 249–279.