Releasing AI-generated software
Short summary
Mark Seemann argues that fully LLM-driven software engineering could work—producing code as reliable as human-written systems—but legal systems will struggle to assign blame when things go wrong, likely scapegoating whoever signed off on the release rather than the AI that created the defect.
- Code review and human inspection were never foolproof either; the real question isn't whether LLM code is perfect, but whether it's better than human-driven processes, which it plausibly could be.
- Even if LLM-driven systems produce fewer bugs than human teams, courts and liability frameworks aren't equipped to handle a world where no individual "wrote" the problematic code—judges will still need someone to prosecute.
- People will adopt AI-driven systems aggressively (like Tesla FSD drivers ignoring the road) because the odds look good, then face severe legal consequences when rare failures occur, regardless of how robust their quality assurance actually was.
Full summary
Mark Seemann examines Robert C. Martin's claim that he no longer reads AI-generated code, instead relying entirely on automated tests and quality gates to validate it. Seemann's initial skepticism—how can you trust code you haven't seen?—gives way to a more nuanced view. He acknowledges that human code review was never foolproof either. Programmers make mistakes constantly. Off-by-one errors, null-reference exceptions, state mutations slip through even rigorous inspection. Research shows code reviews catch bugs, but not all of them. The real question becomes whether LLM-driven development with comprehensive automated testing could outperform human-centered processes, not whether it achieves perfection. Seemann argues there's no inherent barrier to a fully AI-driven software engineering pipeline—one where humans oversee quality metrics but don't read code.
The harder problem is what happens when defects reach production. Even a theoretically superior AI-driven system will still ship bugs. Most are benign: misaligned buttons, wrong language text, duplicate messages. But some are catastrophic. The Mars Climate Orbiter cost $327.6 million. Self-driving cars could kill people. Medical software could deliver lethal radiation doses. When serious failures occur, courts and legal systems will demand accountability. The challenge is that judges can't imprison an LLM. They can sentence the person who signed off on the release. A developer's defense—"the AI wrote it"—won't hold up in court against a prosecutor pointing at the signature on the deployment authorization.
Seemann predicts that people will exploit the gap between technical capability and legal responsibility. He uses Tesla's Full Self-Driving as an example: Danish authorities approved it in June 2026 on the condition that drivers remain responsible. As the system proves reliable, drivers will stop paying attention. They'll scroll their phones for weeks without incident. Then one accident happens, someone dies, and the driver faces involuntary manslaughter charges. Organizations considering full AI automation need to grapple with this reality: the better the system works, the more likely humans will abdicate responsibility, and the more likely they'll face prison time when something goes wrong.