Sensitivity of Self-Report Measures of Trust in Human-Autonomy Teaming

Abstract

As autonomous agents are integrated into human teams, measuring trust in Human–Autonomy Teams (HATs) becomes essential. Existing trust surveys, developed for human or automation contexts, may not capture HAT-specific dynamics. This study evaluated the sensitivity of three trust measures to trust manipulations in a simulated HAT environment. Thirty-six participants completed five missions with confederates via a Wizard-of-Oz approach, while trust was manipulated through team composition and communicative versus behavioral trust/distrust spread. Surveys assessed trust in teammates and systems after each mission. Results showed varying sensitivity across measures: McAllister’s scales captured communicative cues, Jian’s measures reflected mission and communication effects, and only McAllister’s cognitive scale detected behavioral trust spread. These findings suggest that trust instruments vary in responsiveness across trust types and manipulation sources. Combining instruments based on research goals is critical for capturing trust dynamics in human-autonomy teams.

Keywords

human-autonomy teaming trust measurement self-report measures of trust

Get full access to this article

View all access options for this article.

References

Chen

J. Y.

(2018). Human-autonomy teaming in military settings. Theoretical Issues in Ergonomics Science, 19(3), 255–258.

Chiou

E. K.

Lee

J. D.

(2023). Trusting automation: designing for responsivity and resilience. Human Factors, 65(1), 137–165.

Cooke

N. J.

Shope

S. M

. (2004). Synthetic task environments for teams: CERTT’s UAV-STE. In Handbook of human factors and ergonomics methods (pp. 476–483). CRC Press.

Costa

A. C.

Fulmer

C. A.

Anderson

N. R.

(2018). Trust in work teams: An integrative review, multilevel model, and future directions. Journal of Organizational Behavior, 39(2), 169–184.

Duan

Zhou

Scalia

M. J.

Yin

Weng

Zhang

Freeman

McNeese

N. J.

Gorman

J. C.

Tolston

. (2024). Understanding the evolvement of trust over time within Human-AI teams. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2), 1–31.

Feitosa

Grossman

Kramer

W. S.

Salas

(2020). Measuring team trust: A critical and meta-analytical review. Journal of Organizational Behavior, 41(5), 479–501.

Fritz

C. O.

Morris

P. E.

Richler

J. J.

(2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2–18.

Fulmer

C. A.

Gelfand

M. J.

(2012). At what level (and in whom) we trust: Trust across multiple organizational levels. Journal of Management, 38(4), 1167–1230.

Guo

Yang

X. J.

Shi

. (2023, March). TIP: a trust inference and propagation model in multi-human multi-robot teams. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (pp. 639–643).

10.

Hancock

P. A.

Billings

D. R.

Schaefer

K. E.

Chen

J. Y.

De Visser

E. J.

Parasuraman

(2011). A meta-analysis of factors affecting trust in human-robot interaction. Human Factors, 53(5), 517–527.

11.

Hou

Cai

(2023). Exploring trust in human–AI collaboration in the context of multiplayer online games. Systems, 11(5), 217.

12.

Jarvenpaa

S. L.

Knoll

Leidner

D. E.

(1998). Is anybody out there? Antecedents of trust in global virtual teams. Journal of Management Information Systems, 14(4), 29–64.

13.

Jian

J. Y.

Bisantz

A. M.

Drury

C. G.

(2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71.

14.

Kelley

J. F

. (1983, December). An empirical methodology for writing user-friendly natural language computer applications. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 193–196). Association for Computing Machinery.

15.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.

16.

Mayer

R. C.

Davis

J. H.

Schoorman

F. D.

(1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709–734.

17.

McAllister

D. J.

(1995). Affect-and cognition-based trust as foundations for interpersonal cooperation in organizations. Academy of Management Journal, 38(1), 24–59.

18.

McNeese

N. J.

Demir

Cooke

N. J.

Myers

(2018). Teaming with a synthetic teammate: Insights into human-autonomy teaming. Human Factors, 60(2), 262–273.

19.

Peterson

Kaplan

D. A.

(2016). The 10 laws of trust: Building the bonds that make a business great. AMACOM.

20.

Richardson

J. T.

(2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6(2), 135–147.

21.

Schelble

B. G.

Flathmann

Scalia

Zhou

Myers

McNeese

N. J.

Gorman

Freeman

(2022). Addressing the spread of trust and distrust in distributed human-AI teaming constellations. In Proceedings of the CHI TRAIT Workshop (2022). Association for Computing Machinery (ACM).

22.

Sullivan

G. M.

Feinn

(2012). Using effect size—or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279–282.

23.

Waytz

Heafner

Epley

(2014). The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. Journal of Experimental Social Psychology, 52, 113–117.

24.

Yagoda

R. E.

Gillan

D. J.

(2012). You want me to trust a ROBOT? The development of a human–robot interaction trust scale. International Journal of Social Robotics, 4, 235–248