Supporting Bayesian Reasoning to Mitigate Over-Reliance in AI-Assisted Decision Making

Over-reliance on AI systems in high-stakes decision-making leads to avoidable errors and raises concerns about human oversight, responsibility, and bias. This paper proposes a theoretical framework for understanding over-reliance as a biased form of reasoning under uncertainty. We argue that decision-makers may systematically ignore relevant prior information when assessing AI suggestions, thus conflating the system’s reliability for the probability of reaching a correct decision. This “base rate neglect” fallacy violates sound Bayesian reasoning and is well-documented in empirical studies from cognitive science. By reanalysing experimental data from a study on forensic decision-making, we show that biased reasoning leading to over-reliance may occur on a case-by-case basis rather than as a stable tendency of the decision-maker. An interesting implication of our framework is that improving AI accuracy alone may worsen rather than reduce over-reliance. Looking for alternative solutions, we draw on the cognitive science literature to present an interface prototype that supports Bayesian reasoning by helping users understand uncertainty, properly combine different pieces of evidence, and rationally adjust their opinions in AI-assisted evaluative judgments.