AI Explanations Could Make Humans Stop Thinking Independently

When AI explains its decision, humans may stop thinking independently, according to new research from Harvard Business School, MIT, and the University of Washington. The study found that AI recommender tools are persuasive enough to convince evaluators to reject decisions made by independent human experts, which causes them to pass on promising innovations. Evaluators also went along with AI approval of ideas that the human experts found sub-par. The researchers observed that reviewers were more inclined to defer to an incorrect AI decision when the model explained itself.

Narrative explanations degraded human judgment, rather than enhancing it. People did better when they weren’t given a reason for the AI’s decision. The researchers explained in their findings that “Effective human-AI collaboration requires designs that preserve rather than supplant independent human judgment.” Every enterprise screens proposed projects before pursuing them, but there is always uncertainty. The risk of trade-offs involves false positives, where a company goes forward with projects that ultimately fail, or false negatives, where the company rejects ideas that might have succeeded.

Because decision-makers have limited time and only basic information, they are increasingly turning to large language models. These models use predictive algorithms to generate recommendations and rationales based on context. The researchers set out to explore AI’s role in what they called “early-stage innovation screening.” They judged how human evaluators were influenced by LLM recommendations, both with and without explanations from the model on how and why it reached its decision.

The experiment asked 228 experienced evaluators to assess nearly 50 submissions to an MIT challenge. They tested three different scenarios: human-only proposals with no AI assistance; LLM evaluations with a written rationale for the decision; and black-box AI pass-fail recommendations with no accompanying explanation. Evaluators’ decisions were then compared to those made by four human experts. Those decisions were considered the ‘correct’ baseline. They were judged on whether they outright complied with the LLM’s recommendations, overrode them, or productively overrode them, meaning they independently verified persuasive model outputs before making a decision.

The decisions were classified as correct (agreeing with human experts’ positive/negative decisions), false positive (supporting submissions that experts would reject), and false negative (rejecting submissions experts would move forward with). Overall, the evaluators accepted LLM recommendations 67% of the time. They agreed with both black-box and narrative LLM decisions roughly 75% of the time, but only agreed with human decisions 54% of the time.

Seemingly counterintuitively, black-box recommendations improved the quality of decisions, aligning them with human experts, but recommendations with narratives did not. When given an LLM recommendation to reject a submission and an accompanying reason why, evaluators disproportionately agreed. This reduced false positives but “substantially” increased false negatives. The researchers posit that this is because narrative explanations “suppress” productive overrides. LLMs provide a convincing argument that is easy to accept, essentially discouraging independent human verification. This contradicts a common assumption that LLM explanations augment human decision-making.

The researchers pointed out that people are cognitively predisposed to weigh negative information more heavily than positive information; the phenomenon is known as ‘negativity bias.’ “Rejection is an active, eliminative decision that feels more consequential and accountable than preserving optionality,” they wrote. It also maintains the status quo, avoids risk and bias, and requires no resource commitment. LLM explanations provide “ready-made justifications” for going along with rejection decisions without independently verifying them. Humans effectively offload their thinking to AI, researchers explained. Evaluators often rely on surface cues such as fluency, coherence, and seeming credibility. LLMs are particularly well-suited to exploit this because they are linguistically fluent and expert-like, creating an “illusion of explanatory depth.” Thus, “individuals tend to overestimate their understanding of a decision despite limited insight into its reasoning,” the researchers wrote.

While narrative explanations can lead to more errors, the researchers noted that simple, unexplained recommendations improved decision quality. This suggests that hiding the reasoning behind an AI’s decision might actually preserve human discretion and verification in some contexts. The findings have “clear implications” for enterprises designing AI-assisted evaluation systems. Enterprises should be cautious with LLM explanations in high-stakes decision-making. AI recommendations should not be taken at face value; they should always be tested before any associated deployment. This helps improve accuracy and encourages human reviewers to detect errors and learn how models operate, or potentially can even increase human-AI agreement.

In decision contexts such as quality control, compliance screening, or fraud detection, LLM explanations could support conservative human decision-making. On the other hand, in tasks like early-stage screening, LLM narratives could undermine performance by “discouraging independent judgment and suppressing productive human override.” In this context, simpler or more opaque recommendations may preserve human discretion and verification. Future design of explanation systems should factor in the nature of the task and the potential cost of errors made by AI. Enterprises could experiment with models that support contrasting narratives or uncertainty disclosures based on a fixed threshold, rather than on purely binary decisions. Systems could also be structured to invite human disagreement.