Exploratory Data Analysis Can't Be Automated Away

Open original

Randy Au argues that LLMs can take over tedious coding work in data analysis, but exploratory data analysis still needs a human who understands the data's context and intent. He says current models make poor analytical decisions and that the act of poking at data is how analysts sharpen their hypotheses.

  • Exploratory analysis depends on intent: the same numbers look "correct" or "weird" depending on what you plan to do, and an LLM has no access to that purpose unless you spell out a huge amount of business context.
  • Coding agents lack a feedback signal for analytical quality, so they confidently run off on flawed approaches. Au says he had to micromanage them to the minute detail, and the promised analysis often didn't match what was delivered.
  • Hypothesis generation happens through hands-on iteration with the data. Offloading that work skips the process where questions get refined and interesting correlations get noticed.

The author's core claim is that exploratory data analysis can't be handed off to an LLM, at least not yet. He grants that LLMs have taken over the tedious parts of his job: scheduled reports on cron jobs, a helper module that turns SQL queries into CSVs by storing login details and machine names, and OAuth2 flows he never wants to write again. But EDA is different, because it depends on intent. Math will happily compute on any values you feed it, so deciding whether a result looks wrong or interesting requires knowing why you're looking at the data, how it was collected, and where the risks sit. A model working from a checklist can flag outliers and run descriptive statistics, yet it has no basis for knowing what matters. He argues that the context you'd need to supply, a full essay on the business problem, pending decisions, and collection quirks, is a huge writing burden with no guarantee you covered everything, and long inputs are exactly where these models degrade. His experience with agents is the most concrete part of the piece. Coding agents have no reliable signal for whether an analysis has gone off the rails, since there's no pass-fail feedback the way there is with tests. He describes Claude building an elaborate testing harness around an idea that didn't fit the data, and only admitting the error after pointed questions. His workaround is to specify the analysis down to the minute details, then check that the code actually does what was described, and he says the promised method often isn't what gets delivered. He also argues that reviewing and mentally processing the output is the real bottleneck, so the time saved on typing code shrinks once that review is counted. Even fully manual EDA involves rewrites when bugs surface, and he doesn't think LLMs speed that up much in practice. The last section makes the case that hypothesis generation happens in the analyst's head, through repeated handling of the data. Exploring a dataset often leads somewhere unrelated to the original question, and the process sharpens the analyst's thinking as well as the findings. He notices correlations while scanning fields and then checks them, a path he'd never have taken by asking a list of predetermined questions. He admits there may be ways to combine exploration with automation, but says he hasn't found one that gives him the same connections. His bottom line is that people should keep doing EDA their own way, since the LLM approach he's tried isn't ready.

Ask about this article