Sage Journals: Discover world-class research

Abstract

Objective

We explored the Investment construct to quantify human utilization of AI and examined respect, trust, workload, and self-confidence as factors influencing utilization within the Investment Framework.

Background

The Investment construct is a new approach proposed to quantify human utilization of AI-enabled agents based on the measurable opportunity costs incurred by humans. An experimental study was conducted to operationalize the construct and examine its relationships with influencing factors.

Method

Thirty participants performed a maintenance task while an AI assisted them. Participants were split into three groups with different allocated completion times to simulate different task-load levels, under the assumption that a lower task load would lead to higher Investment. We collected typing data as the objective measure of Investment, and responses to a six-item custom scale as the subjective measure. Additionally, we collected data for the influencing factors listed above.

Results

There was a significant difference between groups in terms of both Investment measures, and Investment was higher with a lower task load, as expected. Furthermore, these two Investment measures correlated. We also revealed a correlation between respect and the subjective measure of Investment. No other significant correlations were revealed.

Conclusion

This study presents the first empirical evidence of AI utilization conceptualized through incurred opportunity costs within the Investment Framework, explored using two independent measures and influencing factors.

Application

The Investment Framework can guide the design of effective human AI interaction by helping to balance human effort and AI benefit, prevent underutilization, and enhance both team performance and AI improvement.

Keywords

artificial intelligence intelligent systems trust in automation human systems team collaboration

Get full access to this article

View all access options for this article.

References

BenSaïda

(2025). Shapiro-Wilk and Shapiro-Francia normality tests. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/13964-shapiro-wilk-and-shapiro-francia-normality-tests

Chiou

E. K.

Lee

J. D.

(2015). Beyond reliance and compliance: Human-automation coordination and cooperation [MAG ID: 2570671207 S2ID: 8264f344a2be5db59347d81d679b8985ce40f0eb]. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 195–199. https://doi.org/10.1177/1541931215591040

Chong

Zhang

Goucher-Lambert

Kotovsky

Cagan

(2022). Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice. Computers in Human Behavior, 127(1), Article 107018. https://doi.org/10.1016/j.chb.2021.107018

Dunning

Anderson

J. E.

Schlösser

Ehlebracht

Fetchenhauer

(2014). Trust at zero acquaintance: More a matter of respect than expectation of reward. Journal of Personality and Social Psychology, 107(1), 122–141. https://doi.org/10.1037/a0036673

Fong

(2001). Collaborative control: A robot-centric model for vehicle teleoperation (Publication No. CMU-RI-TR-01-34) [Doctoral dissertation]. Carnegie Mellon University.

Fong

Scholtz

Shah

J. A.

Fluckiger

Kunz

Lees

Schreiner

Siegel

Hiatt

L. M.

Nourbakhsh

Simmons

Antonishek

Bugajska

Ambrose

Burridge

Schultz

Trafton

J. G.

(2006). A preliminary study of peer-to-peer human-robot interaction [ISSN: 1062-922X]. In 2006 IEEE international conference on systems, man and cybernetics (Vol. 41, pp. 3198–3203). IEEE. https://doi.org/10.1109/ICSMC.2006.384609

Fong

Thorpe

Baur

(2003). Robot, asker of questions. Robotics and Autonomous Systems, 42(3), 235–243. https://doi.org/10.1016/S0921-8890(02)00378-0

Frank

J. D.

McGuire

Moses

H. R.

Stephenson

(2016). Developing decision aids to enable human spaceflight autonomy [Number: 4]. AI Magazine, 37(4), 46–54. https://doi.org/10.1609/aimag.v37i4.2683

Gao

Lan

Yuan

Ding

Zhou

(2024). Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities and Social Sciences Communications, 11(1), 1–24. https://doi.org/10.1057/s41599-024-03611-3

10.

Gombolay

Bair

Huang

Shah

(2017). Computational design of mixed-initiative human–robot teaming that considers human factors: Situational awareness, workload, and workflow preferences. The International Journal of Robotics Research, 36(5-7), 597–617. https://doi.org/10.1177/0278364916688255

11.

Gutzwiller

R. S.

Chiou

E. K.

Craig

S. D.

Lewis

C. M.

Lematta

G. J.

Hsiung

C.-P.

(2019). Positive bias in the ‘trust in automated systems survey’? an examination of the Jian et al.(2000) scale. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 63(1), 217–221. https://doi.org/10.1177/1071181319631201

12.

Hart

S. G.

Staveland

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Hancock

P. A.

Meshkati

(Eds.), Advances in psychology (Vol. 52, pp. 139–183). North-Holland. https://doi.org/10.1016/S0166-4115(08)62386-9

13.

Hoff

K. A.

Bashir

(2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570

14.

Jian

J.-Y.

Bisantz

Drury

(2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71. https://doi.org/10.1207/S15327566IJCE0401∖textunderscore04

15.

Johnson

Vera

(2019). No AI is an island: The case for teaming intelligence [Number: 1]. AI Magazine, 40(1), 16–28. https://doi.org/10.1609/aimag.v40i1.2842

16.

Johnson

R. E.

Rosen

C. C.

Chang

C.-H.

(2011). To aggregate or not to aggregate: Steps for developing and validating higher-order multidimensional constructs. Journal of Business and Psychology, 26(3), 241–248. https://doi.org/10.1007/s10869-011-9238-1

17.

Jones

H. M. F.

(2002). Respecting Respect: Exploring a great deal. Educational Studies, 28(4), 341–352. https://doi.org/10.1080/0305569022000042381

18.

Kahneman

(1973). Attention and effort. Prentice-Hall.

19.

Krausman

Neubauer

Forster

Lakhmani

Baker

A. L.

Fitzhugh

S. M.

Gremillion

Wright

J. L.

Metcalfe

J. S.

Schaefer

K. E.

(2022). Trust measurement in human-autonomy teams: Development of a conceptual toolkit. J. Hum.-Robot Interact, 11(3), 1–58. https://doi.org/10.1145/3530874

20.

Kurzban

Duckworth

Kable

J. W.

Myers

(2013). An opportunity cost model of subjective effort and task performance. Behavioral and Brain Sciences, 36(6), 661–679. https://doi.org/10.1017/S0140525X12003196

21.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50∖_30392

22.

Lee

J. D.

Stanton

N. A.

Moray

(1994). Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-computer Studies International Journal of Man-machine Studies, 40(1), 153–184. https://doi.org/10.1006/ijhc.1994.1007

23.

Leontitsis

(2025). Cronbach’s alpha. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/7829-cronbach-s-alpha

24.

Liaw

Driver

Fraune

M. R.

(2019). Robot sociality in human-robot team interactions. In Stephanidis

Antona

(Eds.), HCI International 2019 – Late breaking posters (pp. 434–440). Springer International Publishing. https://doi.org/10.1007/978-3-030-30712-7_53

25.

Lichacz

F. M. J.

Cain

Patel

(2003). Calibration of confidence in situation awareness queries. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 47(1), 222–226. https://doi.org/10.1177/154193120304700147

26.

Malle

B. F.

Ullman

(2021). Chapter 1 - A multidimensional conception and measure of human-robot trust. In Nam

C. S.

Lyons

J. B.

(Eds.), Trust in human-robot interaction (pp. 3–25). Academic Press. https://doi.org/10.1016/B978-0-12-819472-0.00001-0

27.

Marquez

J. J.

Landon

L. B.

Salas

(2023). The Next Giant Leap for Space Human Factors: The Opportunities. Hum Factors, 1547-818165(6), 1279–1288. https://doi.org/10.1177/00187208231174955

28.

MATLAB . (2025a). Isoutlier. https://www.mathworks.com/help/matlab/ref/filloutliers.html

29.

MATLAB . (2025b). Meaneffectsize. https://www.mathworks.com/help/stats/meaneffectsize.html

30.

MATLAB . (2025c). Multcompare. https://www.mathworks.com/help/stats/multcompare.html

31.

MATLAB . (2025d). Vartestn. https://www.mathworks.com/help/stats/vartestn.html

32.

Merriam-Webster . (2025). Respect definition & meaning [website]. https://www.merriam-webster.com/dictionary/respect

33.

Meyer

Lee

J. D.

(2013). Trust, reliance, and compliance [MAG ID: 1642805147 S2ID: 2d77d62d8db0ce57ad7db0e85af2e7b6fdd83b6a]. https://doi.org/10.1093/oxfordhb/9780199757183.013.0007

34.

O’Connell

(2025). The true cost of investing: Opportunity cost - Forbes advisor [website].

35.

O’Meara

Jenks

Stevens

Mindock

Robinson

(2021). Methodology to quantify accuracy for procedure execution analysis. In Harris

W.-C.

(Eds.), Engineering psychology and cognitive ergonomics (pp. 292–307). Springer International Publishing. https://doi.org/10.1007/978-3-030-77932-0_24

36.

Otto

A. R.

Daw

N. D.

(2019). The opportunity cost of time modulates cognitive effort. Neuropsychologia, 123(1), 92–105. https://doi.org/10.1016/j.neuropsychologia.2018.05.006

37.

Ouyang

Jiang

Almeida

Wainwright

Mishkin

Zhang

Agarwal

Slama

Ray

Schulman

Hilton

Kelton

Miller

Simens

Askell

Welinder

Christiano

Leike

Lowe

(2022). Training language models to follow instructions with human feedback. arXiv. https://doi.org/10.48550/arXiv.2203.02155

38.

Parasuraman

Manzey

D. H.

(2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

39.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Occupational Health and Industrial Medicine, 4(37), 157–253. https://doi.org/10.1518/001872097778543886

40.

Piedmont

R. L.

(2014). Inter-item correlations. In Michalos

A. C.

(Ed.), Encyclopedia of quality of life and well-being research (pp. 3303–3304). Springer. https://doi.org/10.1007/978-94-007-0753-5_1493

41.

Razin

Y. S.

Feigh

K. M.

(2023). Converging measures and an emergent model: A meta-analysis of human-automation trust questionnaires. [arXiv:2303.13799 [cs]]. https://doi.org/10.48550/arXiv.2303.13799

42.

Sarker

I. H.

(2021). Machine learning: Algorithms, real-world applications and research directions. SN Computer Science, 2(3), 160. https://doi.org/10.1007/s42979-021-00592-x

43.

Seymour

Van Kleek

Binns

Murray-Rust

(2022). Respect as a lens for the design of AI systems. In Proceedings of the 2022 AAAI/ACM conference on AI, ethics, and society (pp. 641–652). [arXiv:2206.07555 [cs]]. https://doi.org/10.1145/3514094.3534186

44.

Smith

E. R.

Sherrin

Fraune

M. R.

Sabanovic

(2020). Positive emotions, more than anxiety or other negative emotions, predict willingness to interact with robots. Personality and Social Psychology Bulletin, 46(8), 1270–1283. https://doi.org/10.1177/0146167219900439

45.

Gore

(2024). Tlx @ nasa ames - nasa tlx app. [Curator: Phil So; NASA Official: Brian Gore]. https://humansystems.arc.nasa.gov/groups/tlx/tlxapp.php

46.

Stevens

Karasinski

Jenks

O’Meara

Buchner

Potnis

Rocha

Robinson

(2021). Effectiveness of self-customized refresher and just-in-time-training for mechanical repair task performance. In Stephanidis

Antona

Ntoa

(Eds.), HCI International 2021 - Posters (pp. 158–165). Springer International Publishing. https://doi.org/10.1007/978-3-030-78635-9_23

47.

Tavakol

Dennick

(2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2(1), 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd

48.

Ulusoy

(2024). Typing-time-data-collection. https://github.com/ulubilgeulusoy/typing-time-data-collection

49.

Ulusoy

(2025). The investment construct for human autonomy teaming and task execution support for deep space habitats. University of Southern California. [Ph.D. dissertation]. https://digitallibrary.usc.edu/asset-management/2A3BF1M3670KN

50.

Ulusoy

Kaufman

Reisman

(2024). Human factors experiment design process in the context of deep space habitat maintenance operations with autonomous agents. In Proceedings of the 75th International Astronautical Congress (IAC).

51.

Ulusoy

Reisman

G. E.

(2025). Investment construct in human autonomy teaming for deep space habitat operations. Acta Astronautica.

52.

Wenzel

E. M.

(2021). Standard measures for use in analog studies, ISS, and research for long-duration exploration missions (tech. rep. No. NASA/TM-20210021987). https://ntrs.nasa.gov/citations/20210021987

53.

Wilson

R. C.

Shenhav

Straccia

Cohen

J. D.

(2019). The Eighty Five Percent Rule for optimal learning. Nature Communications, 10(1), 4646. https://doi.org/10.1038/s41467-019-12552-4

Exploring the Investment Construct for Quantifying AI Utilization

Abstract

Objective

Background

Method

Results

Conclusion

Application

Keywords

Get full access to this article

References