Systematic Review of the Use and Modification of the “Trust in Automated Systems Scale”

Abstract

Subjective trust in automation measurement is used prolifically due to its high availability and practical ease in implementation, and remains close to the state-of-the-art. Yet, it is also easy to violate a validated survey measure by altering components of the measure, such as changing wording, altering the number of items, changing the scale of response, and translating into new languages. An exhaustive review of the Jian et al. scale’s trust in automated systems measure as it was used across 1,480 cited works. More than half altered the scale in ways which, on average, lowered the scale’s internal validity. Increasing the number of modifications further reduced internal validity. Guidance for the future use of subjective scales in trust measurement is provided. We retain hope of encouraging adherence to best practices and enabling continued use of subjective trust measurement in human-centered AI system development efforts.

Keywords

trust review trust in AI trust measurement trust in automated systems scale

Get full access to this article

View all access options for this article.

References

AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.

Chiou

E. K.

Lee

(2021). Trusting automation: designing for responsivity and resilience. Human Factors, 65(1), 137–165.

Chiou

E. K.

Lee

(2019). Negotiated and reciprocal exchange structures in human-agent cooperation. Computers in Human Behavior, 90, 288–297.

Crocker

Algina

(2006). Introduction to classical and modern test theory. Cengage Learning.

de Visser

E. J.

Beatty

P. J.

Estepp

J. R.

Kohn

Abubshait

Fedota

J. R.

McDonald

C. G.

(2018). Learning from the slips of others: Neural correlates of trust in automated agents. Frontiers in Human Neuroscience, 12(August), 1–15.

Dixon

S. R.

Wickens

C. D.

(2006). Automation reliability in unmanned aerial vehicle control: A reliance-compliance model of automation dependence in high workload. Human Factors, 48(3), 474–486.

Downing

S. M.

Haladyna

T. M.

(2004). Validity threats: overcoming interference with proposed interpretations of assessment data. Medical Education, 38(3), 327–333.

Duan

Flathmann

McNeese

N. J.

Scalia

M. J.

Zhang

Gorman

J. C.

Freeman

Zhou

Hauptman

A. I.

Yin

(2025). Trusting autonomous teammates in human-AI teams: A literature review. In Proceedings of the CHI Conference on Human Factors in Computing Systems.

10.

Duan

Mcneese

(2025). Gender stereotypes toward non-gendered generative AI: The role of gendered expertise and gendered linguistic cues. Proc. of the ACM on Human-Computer Interaction, 9(1), 1–35.

11.

Gutzwiller

R. S.

Chiou

E. K.

Craig

S. D.

Lewis

C. M.

Lematta

G. J.

Hsiung

(2019). Positive bias in the ‘Trust in Automated Systems Survey’? An examination of the Jian et al. (2000) scale. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 217–221.

12.

Gutzwiller

R. S.

Reeder

(2020). Dancing with algorithms: Interaction creates greater preference and trust in machine-learned behavior. Human Factors, 63(5), 854–867.

13.

Haladyna

T. M.

Downing

S. M.

(2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17–27.

14.

Halpern

(2022, January). The rise of A.I. Fighter pilots. The New Yorker. https://www.newyorker.com/magazine/2022/01/24/the-rise-of-ai-fighter-pilots

15.

Hoff

K. A.

Bashir

(2015). Trust in automation: integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434.

16.

Hussein

Elsawah

Abbass

H. A.

(2020). The reliability and transparency bases of trust in human-swarm interaction: principles and implications. Ergonomics, 63(9), 1116–1132.

17.

Jian

Bisantz

Drury

(2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71.

18.

Kohn

S. C.

Kluck

Shaw

(2020). A brief review of frequently used self-report measures of trust in automation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 1436–1440.

19.

Lee

J. R.

Gutzwiller

R. S.

(2025). Do the eyes have it? A review of using eye tracking for automation trust measurement. Human Factors. Advance online publication. https://doi.org/10.1177/00187208251348395

20.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.

21.

Sarter

(2019). Eye tracking: a process-oriented method for inferring trust in automation as a function of priming and system reliability. IEEE THMS, 49(6), 560–568.

22.

Merritt

S. M.

Ilgen

D. R.

(2008). Not all trust is created equal: Dispositional and history-based trust in human-automation interactions. Human Factors, 50(2), 194–210.

23.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.

24.

Patton

C. E.

Wickens

C. D.

(2024). The relationship of trust and dependence. Ergonomics, 67(11), 1535–1552.

25.

Perrig

S. A. C.

Scharowski

Brühlmann

(2023). Trust issues with trust scales: Examining the psychometric quality of trust measures in the context of AI. In Extended abstracts of the 2023 CHI conference on human factors in computing systems (pp. 1–7). Association for Computing Machinery.

26.

Razin

Y. S.

Feigh

K. M.

(2020). Hitting the road: exploring human-robot trust for self-driving vehicles. Proceedings of the IEEE ICHMS (pp. 1–6). IEEE.

27.

Safar

J. A.

Turner

C. W.

(2005). Validation of a two factor structure for system trust. Proceedings of the Human Factors and Ergonomics Society, 49, 497–501.

28.

Schaefer

K. E.

Chen

J. Y. C.

Szalma

J. L.

Hancock

P. A.

(2016). A meta-analysis of factors influencing the development of trust in automation: implications for understanding autonomy in future systems. Human Factors, 58(3), 377–400.

29.

Scholz

D. D.

Kraus

Miller

(2025). Measuring the propensity to trust in automated technology: examining similarities to dispositional trust in other humans and validation of the PTT-A scale, International Journal of Human–Computer Interaction, 41(2), 970–993.

30.

Spain

Bustamante

Bliss

(2008). Towards an empirically developed scale for system trust: Take two. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52(19), 1335–1339.

31.

Wei

Bolton

M. L.

Humphrey

(2019). Subjective measurement of trust: is it on the level? Proc of the Human Factors and Ergonomics Society, 63, 212–216.

32.

Wei

Bolton

M. L.

Humphrey

(2020). The level of measurement of trust in automation. Theoretical Issues in Ergonomics Science, 22(3), 274–295.

33.

Wickens

C. D.

Hollands

Banbury

Parasuraman

(2013). Engineering psychology and human performance (4th ed.). Pearson.

34.

Wickens

C. D.

McCarley

Gutzwiller

R. S.

(2022). Applied attention theory (2nd ed.). Taylor & Francis.