Subjective trust in automation measurement is used prolifically due to its high availability and practical ease in implementation, and remains close to the state-of-the-art. Yet, it is also easy to violate a validated survey measure by altering components of the measure, such as changing wording, altering the number of items, changing the scale of response, and translating into new languages. An exhaustive review of the Jian et al. scale’s trust in automated systems measure as it was used across 1,480 cited works. More than half altered the scale in ways which, on average, lowered the scale’s internal validity. Increasing the number of modifications further reduced internal validity. Guidance for the future use of subjective scales in trust measurement is provided. We retain hope of encouraging adherence to best practices and enabling continued use of subjective trust measurement in human-centered AI system development efforts.
AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.
2.
BraunV.ClarkeV. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
3.
ChiouE. K.LeeJ. (2021). Trusting automation: designing for responsivity and resilience. Human Factors, 65(1), 137–165.
4.
ChiouE. K.LeeJ.SuT. (2019). Negotiated and reciprocal exchange structures in human-agent cooperation. Computers in Human Behavior, 90, 288–297.
5.
CrockerL.AlginaJ. (2006). Introduction to classical and modern test theory. Cengage Learning.
6.
de VisserE. J.BeattyP. J.EsteppJ. R.KohnS.AbubshaitA.FedotaJ. R.McDonaldC. G. (2018). Learning from the slips of others: Neural correlates of trust in automated agents. Frontiers in Human Neuroscience, 12(August), 1–15.
7.
DixonS. R.WickensC. D. (2006). Automation reliability in unmanned aerial vehicle control: A reliance-compliance model of automation dependence in high workload. Human Factors, 48(3), 474–486.
8.
DowningS. M.HaladynaT. M. (2004). Validity threats: overcoming interference with proposed interpretations of assessment data. Medical Education, 38(3), 327–333.
9.
DuanW.FlathmannC.McNeeseN. J.ScaliaM. J.ZhangR.GormanJ. C.FreemanG.ZhouS.HauptmanA. I.YinX. (2025). Trusting autonomous teammates in human-AI teams: A literature review. In Proceedings of the CHI Conference on Human Factors in Computing Systems.
10.
DuanW.McneeseN.LiL. (2025). Gender stereotypes toward non-gendered generative AI: The role of gendered expertise and gendered linguistic cues. Proc. of the ACM on Human-Computer Interaction, 9(1), 1–35.
11.
GutzwillerR. S.ChiouE. K.CraigS. D.LewisC. M.LemattaG. J.HsiungC. (2019). Positive bias in the ‘Trust in Automated Systems Survey’? An examination of the Jian et al. (2000) scale. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), 217–221.
12.
GutzwillerR. S.ReederJ. (2020). Dancing with algorithms: Interaction creates greater preference and trust in machine-learned behavior. Human Factors, 63(5), 854–867.
13.
HaladynaT. M.DowningS. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17–27.
HoffK. A.BashirM. (2015). Trust in automation: integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434.
16.
HusseinA.ElsawahS.AbbassH. A. (2020). The reliability and transparency bases of trust in human-swarm interaction: principles and implications. Ergonomics, 63(9), 1116–1132.
17.
JianJ.BisantzA.DruryC. (2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53–71.
18.
KohnS. C.KluckM.ShawT. (2020). A brief review of frequently used self-report measures of trust in automation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 1436–1440.
19.
LeeJ. R.GutzwillerR. S. (2025). Do the eyes have it? A review of using eye tracking for automation trust measurement. Human Factors. Advance online publication. https://doi.org/10.1177/00187208251348395
20.
LeeJ. D.SeeK. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.
21.
LuY.SarterN. (2019). Eye tracking: a process-oriented method for inferring trust in automation as a function of priming and system reliability. IEEE THMS, 49(6), 560–568.
22.
MerrittS. M.IlgenD. R. (2008). Not all trust is created equal: Dispositional and history-based trust in human-automation interactions. Human Factors, 50(2), 194–210.
23.
ParasuramanR.RileyV. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.
24.
PattonC. E.WickensC. D. (2024). The relationship of trust and dependence. Ergonomics, 67(11), 1535–1552.
25.
PerrigS. A. C.ScharowskiN.BrühlmannF. (2023). Trust issues with trust scales: Examining the psychometric quality of trust measures in the context of AI. In Extended abstracts of the 2023 CHI conference on human factors in computing systems (pp. 1–7). Association for Computing Machinery.
26.
RazinY. S.FeighK. M. (2020). Hitting the road: exploring human-robot trust for self-driving vehicles. Proceedings of the IEEE ICHMS (pp. 1–6). IEEE.
27.
SafarJ. A.TurnerC. W. (2005). Validation of a two factor structure for system trust. Proceedings of the Human Factors and Ergonomics Society, 49, 497–501.
28.
SchaeferK. E.ChenJ. Y. C.SzalmaJ. L.HancockP. A. (2016). A meta-analysis of factors influencing the development of trust in automation: implications for understanding autonomy in future systems. Human Factors, 58(3), 377–400.
29.
ScholzD. D.KrausJ.MillerL. (2025). Measuring the propensity to trust in automated technology: examining similarities to dispositional trust in other humans and validation of the PTT-A scale, International Journal of Human–Computer Interaction, 41(2), 970–993.
30.
SpainR.BustamanteE.BlissJ. (2008). Towards an empirically developed scale for system trust: Take two. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52(19), 1335–1339.
31.
WeiJ.BoltonM. L.HumphreyL. (2019). Subjective measurement of trust: is it on the level?Proc of the Human Factors and Ergonomics Society, 63, 212–216.
32.
WeiJ.BoltonM. L.HumphreyL. (2020). The level of measurement of trust in automation. Theoretical Issues in Ergonomics Science, 22(3), 274–295.
33.
WickensC. D.HollandsJ.BanburyS.ParasuramanR. (2013). Engineering psychology and human performance (4th ed.). Pearson.
34.
WickensC. D.McCarleyJ.GutzwillerR. S. (2022). Applied attention theory (2nd ed.). Taylor & Francis.
