Sage Journals: Discover world-class research

Abstract

Unchecked overreliance on AI systems in human-AI teaming (HAT) can erode critical human judgment and compromise accountability, especially in high-stakes decisions. This paper introduces two integrated studies aimed at preserving balanced human agency in effective AI-assisted decision-making. First, it presents the Human-AI-System Concordance (HASC) Matrix, a diagnostic framework designed to systematically identify vulnerabilities to overreliance within human-AI collaboration (HAC) scenarios. Second, it then empirically evaluates Cognitive Forcing Functions (CFFs)—targeted interventions designed to mitigate overreliance and maintain calibrated trust—through a controlled experimental setting. These studies deliver actionable insights for developing balanced HAC environments, highlighting the conditions under when overreliance occurs and demonstrating how and which cognitive interventions can effectively mitigate it. This comprehensive approach can help to advance understanding of designing AI systems to better support effective human-centered decision-making.

Keywords

human-AI teaming AI-assisted decision-making overreliance trust calibration decision support systems automation bias cognitive forcing functions human-AI-system concordance (HASC) matrix explainable AI interaction design

Get full access to this article

View all access options for this article.

References

Arrieta

A. A.

Díaz-Rodríguez

Del Ser

Bennetot

Tabik

Barbado

Garcia

Gil-Lopez

Molina

Benjamins

Chatila

Sapena

Martínez-Plumed

Figueira

Á.

Gervás

Millán

J. d. R.

Cornelio

Herrera

Ching

. . . Herrera

(2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115.

Cacioppo

J. T.

Petty

R. E.

(1982). The Need for Cognition. Journal of Personality and Social Psychology, 42(1), 116–131.

Chiriatti

Venza

Grimaldi

(2024). System 0: The emergent challenges of cognitive automation. AI & Society, 39(2), 347–360.

Dzindolet

M. T.

Pierce

L. G.

Beck

H. P.

Dawe

L. A.

(2003). The perceived utility of human and automated aids in decision making. Human Factors, 45(3), 349–359.

Green

Chen

(2019). Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments. Proceedings of the ACM on human-computer interaction, 3(CSCW) (pp. 1–25). Association for Computing Machinery (ACM).

Gunning

Aha

(2019). DARPA’s explainable artificial intelligence (XAI) Program. AI Magazine, 40(2), 44–58.

Hoffman

R. R.

Mueller

S. T.

Klein

Litman

(2021). Metrics for explainable AI: challenges and prospects. arXiv preprint arXiv:1812.04608v3.

Kahneman

(2011). Thinking, fast and slow. Farrar, Straus and Giroux.

Lai

Tan

(2021). On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT) (pp. 29–39). Association for Computing Machinery (ACM).

10.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.

11.

Madhavan

Wiegmann

D. A.

(2007). Similarities and differences between human-human and human-automation trust: An integrative review. Theoretical Issues in Ergonomics Science, 8(4), 277–301.

12.

Mosier

K. L.

Skitka

L. J.

(1996). Human decision makers and automated decision aids: Made for each other? In Parasuraman

Mouloua

(Eds.), Automation and human performance: Theory and applications (pp. 201–220). CRC Press.

13.

Parasuraman

Riley

(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.

14.

Swets

J. A.

(1996). Signal detection theory and ROC analysis in psychology and diagnostics. Lawrence Erlbaum Associates.

15.

Topol

(2019). Deep medicine: How artificial intelligence can make healthcare human again. Basic Books.

16.

Zhang

Liao

Q. V.

Bellamy

R. K. E.

Shah

(2022). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1–13). Association for Computing Machinery (ACM).

Beyond Overreliance: The Human-AI-System Concordance (HASC) Matrix and the Cognitive Dynamics of AI-Assisted Decision-Making

Abstract

Keywords

Get full access to this article

References