What does it mean to understand what we built?
AWS engineer Marc Brooker argues that software engineering's focus on reading code gets in the way of understanding what really matters: the behavior of the system as built. He says we need new tools, like the Bode plots or Nyquist criterion other engineering fields use, to grasp emergent properties such as metastable failures.
- Code-level understanding is a poor fit for system-level properties. Things like "a SELECT never shows uncommitted data" or "makes the user's life easier" emerge from the whole system, not individual lines.
- Emergent behaviors like metastable failures can't realistically be extracted from source code, in the same way the Lorenz attractor or a Julia set resists analysis from its equations alone.
- Software lacks the analytical tools other engineering disciplines have, and Brooker sees AI as a way to help build them. He also thinks the "No AI Slop" manifesto doesn't fully work out what "understanding" should mean.
Marc Brooker, an engineer at AWS working on agentic AI safety and policy (previously EC2, EBS, databases, and serverless), is responding to the No AI Slop Manifesto, which lists "understanding over speed" as a value. He agrees with the sentiment but thinks the manifesto uses a narrow definition of understanding. Its example, that teams explore "resources, code, and prior art" before starting work, treats understanding as understanding code. Brooker's objection is that this privileges program text, a habit software has carried for about sixty years, and that the habit is now holding the field back from understanding what actually matters.
What matters, in his view, is the behavior of the as-built system. Some of those properties are precise, like "a SELECT never shows results from uncommitted transactions," and others are fuzzy, like "makes the user's life easier." Neither kind lives in any single line of code. They emerge from the whole, and tracing how they are synthesized from individual lines is, in his words, a mammoth task. That is why the field built abstractions in the first place. The hardest cases are the emergent ones, such as metastable failures, where behavior arises from components interacting. Like the Lorenz attractor or a Julia set, these are generally intractable to derive from the code, so reading the code will not explain them.
His central complaint is that software engineering has no equivalents yet of the tools other engineering disciplines use to reason about complex systems. He points to Bode plots and the Nyquist stability criterion as examples of what is missing. Engineers still examine each screw, each grain of aggregate, and each PCB trace and try to grasp the whole by sheer concentration. He thinks the field should look up, toward system-level tools, and he sees AI as a way to build them. He likes the manifesto but argues its authors have not fully worked out what it means to be for understanding rather than merely against not understanding.