The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
This paper defines the “LLM fallacy” as a bias where users credit their AI-assisted outputs to their own skill rather than the model’s contribution. It analyzes how fluent, opaque interactions with large language models blur human-machine boundaries, offers a framework for its mechanisms, and discusses impacts on education, hiring, and AI literacy.
- People using LLMs tend to internalize the AI's output as evidence of their own skill, not just trust the tool's answer (unlike classic automation bias).
- This happens because model reasoning is invisible, the prose feels fluently "authored," there's no clear handoff moment, and positive feedback loops reinforce the false belief.
- It shows up concretely: coding, academic writing, data analysis, and creative work all get passed off as personal competence when AI did much of the work.
- Real consequences include students graduating without real mastery, hiring managers misjudging AI-reliant candidates, and skewed performance reviews at work.
When people lean on large language models to draft text, write code or analyze data, they often mistake the model’s fluent output for their own skill. The authors call this the “LLM fallacy.” It springs from three factors: the models’ high fluency hides mistakes, the interfaces make interaction feel seamless, and users lack visibility into how proposals are generated. As a result, someone might believe they’re a better writer or programmer than they really are—because they credit themselves for the model’s work.
The paper lays out how this bias differs from classic automation bias or simple overreliance on tools. Instead of merely trusting a machine’s answer, users internalize the outcome as proof of personal expertise. The authors outline four core mechanisms: attribution opacity (you can’t see the model’s chain of thought), fluency illusion (smooth prose feels authored), frictionless handoff (no clear handoff between user and AI), and reinforcement loops (positive feedback reinforces misperception). They then map examples across four domains: coding (auto-generated functions passed off as own work), academic writing (AI-polished paragraphs mistaken for genuine research skill), data analysis (model-suggested queries misread as analytical insight), and creative tasks (story ideas credited to the writer).
They close by flagging real-world stakes. In education, students might graduate thinking they mastered topics they never truly learned. Hiring managers could misjudge candidates who lean heavily on AI-generated résumés. Even within teams, employees might overstate capabilities, skewing performance reviews. The authors propose experiments—eye-tracking to see where users look in AI interfaces, think-aloud studies to trace attribution—and suggest interface tweaks like transparent model provenance indicators and prompts that force users to annotate and reflect on AI contributions.