Developing a brief data literacy measure using CFA and IRT

Abstract

Data literacy is essential in today’s workforce, yet existing data literacy assessments lack brevity, limiting their practical use. This study develops a brief data literacy instrument (BDLI) by streamlining the 25-item Global Data Literacy Benchmark (GDLB). Two studies were conducted. The first study (N = 408) explored item reduction and factor consistency using student participants, while the second study (N = 388) replicated findings using a diverse professional sample. Results indicated that a 12-item BDLI achieved similar factor structure and reliability compared to the original GDLB. Confirmatory factor analysis and item response theory confirmed the BDLI’s consistency and discriminatory power. These results suggest that the BDLI offers a practical tool for assessing an individual’s data literacy skills, which can be used to develop targeted learning opportunities at the individual or organizational level. The value and limitations of the BDLI are discussed.

Keywords

Data literacy item response theory CFA

Get full access to this article

View all access options for this article.

References

AICPA (2021) CPA Evolution Model Curriculum. AICPA.

Almgerbi

De Mauro

Kahlawi

, et al (2022) A systematic review of data analytics job requirements and online-courses. Journal of Computer Information Systems 62(2): 422–434.

Alpaydin

(2016) Machine Learning. The New AI, Cambridge, MA, MIT Press.

Appelbaum

Showalter

Sun

, et al (2021) A framework for auditor data literacy: A normative position. Accounting Horizons 35(2): 5–25.

Baker

(2001) The Basics of Item Response Theory. ERIC.

Bean

Bowen

(2021) Item response theory and confirmatory factor analysis: Complementary approaches for scale development. Journal of Evidence-Based Social Work 18(4): 597–618.

Bean

(2021) Using R for Social Work Research. Columbus, OH: Ohio State University Press.

Bean

(2018) Big data executive survey 2018.

Benet-Martínez

Leu

Lee

, et al (2002) Negotiating biculturalism: Cultural frame switching in biculturals with oppositional versus compatible cultural identities. Journal of Cross-Cultural Psychology 33(5): 492–516.

10.

Bennett

(2016) How Chief Data Officers Can Tackle Formidable Roadblocks Including People, Culture and Internal Resistance. Gartner: Gartner Website.

11.

Bhargava

Deahl

Letouzé

, et al (2015) Beyond data literacy: Reinventing community engagement and empowerment in the age of data.

12.

Bond JR

F C

Depaulo

(2006) Accuracy of deception judgments. Personality and Social Psychology Review 10(3): 214–234.

13.

Bonikowska

Sanmartin

Frenette

(2019) Data Literacy: What It Is and How to Measure It in the Public Service. Statistics Canada.

14.

Brown

Herring

, et al. (2019) The Analytics Academy: Bridging the Gap Between Human and Artificial Intelligence. McKinsey.

15.

Brynjolfsson

Mitchell

(2017) What can machine learning do? Workforce implications. Science 358(6370): 1530–1534.

16.

Buchanan

(2005) A (very) brief history of artificial intelligence. AI Magazine, 26, 53.

17.

Burisch

(1984) Approaches to personality inventory construction: A comparison of merits. American Psychologist 39(3): 214–227.

18.

Burisch

(1997) Test length and validity revisited. European Journal of Personality 11(4): 303–315.

19.

Campbell

Converse

Rodgers

(1976) The Quality of American Life: Perceptions, Evaluations, and Satisfactions. New York, NY: Russell Sage Foundation.

20.

Carver

(1997) You want to measure coping but your protocol's too long: Consider the brief COPE. International Journal of Behavioral Medicine 4(1): 92–100.

21.

Chalmers

(2012) Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software 48(6): 1–29.

22.

Columbus

(2014) 84% Of enterprises see big data analytics changing their industries’ competitive landscapes in the next year. Available at: https://www.forbes.com/sites/louiscolumbus/2014/10/19/84-of-enterprises-see-big-data-analytics-changing-their-industries-competitive-landscapes-in-the-next-year/#6a5aa2317de1

23.

Corrall

(2019) Repositioning data literacy as a mission-critical competence.

24.

Crofts

(2018) Global data literacy benchmark.

25.

Cui

Chen

Lutsyk

, et al (2023) Data literacy assessments: A systematic literature review. Assessment in Education Principles Policy and Practice 30(1): 76–96.

26.

D'ignazio

(2017) Creative data literacy: Bridging the gap between the data-haves and data-have nots. Information Design Journal 23: 6–18.

27.

DATA-REVOLUTION-GROUP (2014) Accessible data: open data, accountability and data literacy [Online]. Available at: https://www.undatarevolution.org/data-revolution/ (accessed 22 February 2020).

28.

DATATOTHEPEOPLE.ORG (2018) We’re data to the people [Online]. Available at: https://www.datatothepeople.org/ (accessed 22 February 2020).

29.

Davenport

Harris

(2017) Competing on Analytics: Updated, With a New Introduction: The New Science of Winning. Boston, MA: Harvard Business Press.

30.

Davenport

(2006) Competing on analytics. Harvard Business Review 84: 98.

31.

Deloitte (2015) Your guide to a successful ERP journey.

32.

Downes

Nunes

(2013) Big bang disruption. Harvard Business Review 91(1): 44–56.

33.

Dykes

(2019) Data Curiosity: How To Cultivate An Inquisitive Workforce [Online]. Forbes. Available at: https://www.forbes.com/sites/brentdykes/2019/10/10/data-curiosity-how-to-cultivate-an-inquisitive-workforce/#7d72b1735471 (accessed 11 October 2019).

34.

Feeny

Willcocks

(1998) Core IS capabilities for exploiting information technology. Sloan Management Review 39: 9–21.

35.

Friedman

(2017) Thank You for Being Late: An Optimist's Guide to Thriving in the Age of Accelerations (Version 2.0, With a New Afterword). New York, NY: Picador/Farrar Straus and Giroux.

36.

Gartner (2016) Gartner Estimates That 90 Percent of Large Organizations Will Have a Chief Data Officer by 2019 [Online]. Stamford, CN: Gartner. Available at: https://www.gartner.com/en/newsroom/press-releases/2016-01-26-gartner-estimates-that-90-percent-of-large-organizations-will-have-a-chief-data-officer-by-2019 (accessed 15 August 2019).

37.

George

Carlson

Valacich

(2013) Media selection as a strategic component of communication. MIS Quarterly 37(4): 1233–1251.

38.

Graham

Nosek

Haidt

, et al (2011) Mapping the moral domain. Journal of Personality and Social Psychology 101(2): 366–385.

39.

Gray

Chambers

Bounegru

(2012) The Data Journalism Handbook: How Journalists Can Use Data to Improve the News. Sebastopol, CA: O’Reilly Media, Inc.

40.

Gray

Gerlitz

Bounegru

(2018) Data infrastructure literacy. Big Data & Society 5(2): 2053951718786316.

41.

Gummer

Mandinach

(2015) Building a conceptual framework for data literacy. Teachers College Record 117(4): n4–22.

42.

Gupta

George

(2016) Toward the development of a big data analytics capability. Information Management 53(8): 1049–1064.

43.

Holeni

(2020) Data is the new oil of the 21st century. CIGFARO Journal (Chartered Institute of Government Finance Audit and Risk Officers) 21(4): 12–14.

44.

Hunt

(2005) The challenges of integrating data literacy into the curriculum in an undergraduate institution. IASSIST Quarterly/International Association for Social Science Information Service and Technology 28(2): 12–12.

45.

Johnson

Maurer

Torres

, et al (2024) A preview of the 2023 SIM IT trends study. MIS Quarterly Executive 22: 83–124.

46.

Johnson

Torres

Maurer

, et al (2023b) The 2022 SIM IT issues and trends study. MIS Quarterly Executive 22: 55–98.

47.

Kılıç

Koyuncu

Uysal

(2023) Scale development based on item response theory: A systematic review. International Journal of Psychology and Educational Studies 10(1): 209–223.

48.

Kirby

(2021) Demystifying and developing a framework for “data analytics”. Economics and Business Journal: Inquiries and Perspectives 12(1): 54–68.

49.

Kirby

(2020) The impact of functional TMT technology leadership roles on firm financial outcomes. In: UNIVERSITY, C. (ed.).

50.

Koltay

(2017) Data literacy for researchers and data librarians. Journal of Librarianship and Information Science 49(1): 3–14.

51.

Landi

(2019) Amazon plans to spend $700M to retrain a third of its workforce for data, analytics roles [Online]. Available at: https://www.cnbc.com/2019/07/11/amazon-plans-to-spend-700-million-to-retrain-a-third-of-its-workforce-in-new-skills-wsj.html (accessed December 2020)

52.

Lavalle

Lesser

Shockley

, et al (2011) Big data, analytics and the path from insights to value. MIT Sloan Management Review 52: 21–32.

53.

Lawson

RHT

Desroches

(2019) The Impact of Big Data on Finance Now and in the Future. Montvale, NJ: Institute of Management Accountants.

54.

Letouze

(2016) Should ‘data literacy’ be promoted? [Online]. Available at: https://unstats.un.org/unsd/undataforum/blog/should-data-literacy-be-promoted/ (accessed 20 February 2020).

55.

Lin

(2022) Confirmatory Factor Analysis (CFA) in R with lavaan. UCLA: Statistical Consulting Group.

56.

Manyika

Chui

Brown

, et al (2011) Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute.

57.

Markham

(2020) Taking data literacy to the streets: Critical pedagogy in the public sphere. Qualitative Inquiry 26(2): 227–237.

58.

Mcafee

Brynjolfsson

Davenport

, et al (2012) Big data: The management revolution. Harvard Business Review 90: 60–68.

59.

McKinsey (2009) Hal Varian on how the Web Challenges Managers. McKinsey Technology, Media & Telecommunications.

60.

McKinsey Analytics (2019) Catch Them if you can: How Leaders in Data and Analytics Have Pulled Ahead. McKinsey Analytics.

61.

Muck

Hell

Gosling

(2007) Construct validation of a short five-factor model instrument. European Journal of Psychological Assessment 23(3): 166–175.

62.

Nath

Kirby

(2022) An empirical examination of the factors of data literacy. Journal of Digital Science 4(1): 3–20.

63.

Noble

(1984) Computer literacy and ideology. Teachers College Record 85(4): 602–614.

64.

Pothier

Condon

(2020) Towards data literacy competencies: Business students, workforce needs, and the role of the librarian. Journal of Business & Finance Librarianship 25(3–4): 123–146.

65.

R-CORE-TEAM (2023) A language and environment for statistical computing [Computer software].

66.

Ridsdale

Rothwell

Smit

, et al (2015) Strategies and best practices for data literacy education: Knowledge synthesis report.

67.

Robinson

Gibson

Khan

, et al (2003) Validity of two oral health-related quality of life measures. Community Dentistry and Oral Epidemiology 31(2): 90–99.

68.

Robins

Hendin

Trzesniewski

(2001) Measuring global self-esteem: Construct validation of a single-item measure and the Rosenberg self-esteem scale. Personality and Social Psychology Bulletin 27(2): 151–161.

69.

Robins

Trzesniewski

Tracy

, et al (2002) Global self-esteem across the life span. Psychology and Aging 17(3): 423–434.

70.

Rosseel

(2012) Lavaan: An R package for structural equation modeling. Journal of Statistical Software 48(2): 1–36.

71.

Ross

Beath

Goodhue

(1996) Develop long-term competitiveness through IT assets. Sloan Management Review 38: 31–42.

72.

Ross

Beath

Quaadgras

(2013) You may not need big data after all. Harvard Business Review 91: 90+.

73.

Schield

(2005) Information literacy, statistical literacy, data literacy. IASSIST Quarterly 28: 6.

74.

Schield

(2006) Statistical literacy survey analysis: Reading graphs and tables of rates and percentages. In: Proceedings of the Sixth International Conference on Teaching Statistics, 2006. International Statistical Institute and International Association for for Statistical Education [Online]. Available at: http://www.stat.auckland.ac.nz/~iase

75.

Schüller

(2022) Data and AI literacy for everyone. Statistical Journal of the IAOS 38(2): 477–490.

76.

Shibu

(2019) New law mandates chief data officers at federal agencies. govloop.

77.

Shrestha

(2021) Factor analysis as a tool for survey analysis. American Journal of Applied Mathematics and Statistics 9(1): 4–11.

78.

Simms

(2008) Classical and modern methods of psychological scale construction. Social and Personality Psychology Compass 2(1): 414–433.

79.

Spitzer

Eisenberg

Lowe

(1998) Information Literacy: Essential Skills for the Information Age. Syracuse University, New York: ERIC Clearinghouse on information and technology.

80.

Stephenson

Schifter Caravello

(2007) Incorporating data literacy into undergraduate information literacy programs in the social sciences. Reference Services Review 35(4): 525–540.

81.

Strauss

(2016) These are the Skills Bosses Say New College Grads do not have. Forbes.

82.

Tabrizi

Lam

Girard

, et al (2019) Digital transformation is not about technology. Harvard Business Review 13: 1–6.

83.

UNESCO (2019) UNESCO - Literacy Website [Online]. Available at: https://www.unesco.org/en/literacy (accessed 22 February 2020).

84.

Waid

(2019) Solving The Last Mile Problem For Data Science Project Success. Forbes.

85.

Weill

Stephanie

Shah

(2021) Does your C-Suite have enough digital smarts. MIT Sloan Management Review, 62(3).

86.

West

Taylor

(2012) Model fit and model selection in structural equation modeling. In: Hoyle

(ed.) Handbook of Structural Equation Modeling. New York, NY: Guilford Press, pp. 209–231.

87.

Windsor

(2019) What is the uncanny? British Journal of Aesthetics 59(1): 51–65.

88.

Zhan

Huang

, et al (2016) The Value of Chief Data Officer presence on Firm Performance. PACIS, p.213.

89.

Zhang

Lee

Wang

, et al (2017) Chief data officer appointment and origin: A theoretical perspective.