LLM Classification Is Feature Engineering
Using LLMs directly as classifiers is frustrating because they can't be calibrated, don't reliably use all available data, and are hard to improve systematically. Instead, treat LLM outputs as features in a logistic regression or other ML model, which solves these problems while keeping the LLM's signal.
- Direct LLM classification fails on three critical fronts: you can't calibrate confidence scores or adjust precision-recall tradeoffs, the model may ignore structured data or context you provide, and you have no visibility into what the LLM actually used to make its decision.
- Wrapping an LLM verdict in logistic regression automatically fixes calibration (empirically matching true probabilities), lets you add other features, and gives interpretability about how much the LLM contributes to the final decision.
- On an irony detection task, logistic regression on the raw LLM verdict improved Brier score from 0.259 to 0.175 just by calibrating; adding more LLM-extracted features (token probabilities, multiple reasoning passes) provides a clear path to further improvement that prompt-tweaking doesn't offer.
Using LLMs as direct classifiers—feeding in text and getting back a label—sounds appealing but breaks down in practice. The author identifies three core problems: LLM confidence scores aren't calibrated, so you can't reliably trade off precision and recall; LLMs ignore structured data and may not actually use all the information in a prompt; and you get no real visibility into what the model is actually doing. These aren't flaws in the LLM itself—it was never designed for classification—but they make LLM-as-classifier a frustrating approach when you need something that actually works.
The fix is treating LLM outputs as features rather than final predictions. Wrap the LLM verdict into a logistic regression, and suddenly everything works: you get calibrated probabilities you can threshold however you want, you can add other structured features alongside the LLM output, and you understand how much the LLM verdict actually contributes to your final decision. When performance isn't good enough, you have a clear path forward instead of guessing at prompt tweaks. You can collect training data, screen features for consistency, extract multiple signals from the LLM (like log probabilities or sub-verdicts), or swap in a different model architecture entirely.
The author demonstrates this on irony detection in tweets using the SemEval 2018 dataset. A basic prompt gets 74.7% F1 and 96.5% true positive rate, but the Brier score of 0.259 shows the predictions aren't calibrated—barely better than random guessing. By treating the LLM output as a feature and fitting a logistic regression on training data, you'd expect to fix this calibration problem and improve overall performance, all while keeping the LLM's actual signal intact.