Using LLMs as direct classifiers—feeding in text and getting back a label—sounds appealing but breaks down in practice. The author identifies three core problems: LLM confidence scores aren't calibrated, so you can't reliably trade off precision and recall; LLMs ignore structured data and may not actually use all the information in a prompt; and you get no real visibility into what the model is actually doing. These aren't flaws in the LLM itself—it was never designed for classification—but they make LLM-as-classifier a frustrating approach when you need something that actually works. The fix is treating LLM outputs as features rather than final predictions. Wrap the LLM verdict into a logistic regression, and suddenly everything works: you get calibrated probabilities you can threshold however you want, you can add other structured features alongside the LLM output, and you understand how much the LLM verdict actually contributes to your final decision. When performance isn't good enough, you have a clear path forward instead of guessing at prompt tweaks. You can collect training data, screen features for consistency, extract multiple signals from the LLM (like log probabilities or sub-verdicts), or swap in a different model architecture entirely. The author demonstrates this on irony detection in tweets using the SemEval 2018 dataset. A basic prompt gets 74.7% F1 and 96.5% true positive rate, but the Brier score of 0.259 shows the predictions aren't calibrated—barely better than random guessing. By treating the LLM output as a feature and fitting a logistic regression on training data, you'd expect to fix this calibration problem and improve overall performance, all while keeping the LLM's actual signal intact.