1 link tagged with all of: calibration + llm-classification
Click any tag below to further narrow down your results
Links
Using LLMs directly as classifiers is frustrating because they can't be calibrated, don't reliably use all available data, and are hard to improve systematically. Instead, treat LLM outputs as features in a logistic regression or other ML model, which solves these problems while keeping the LLM's signal.
- Direct LLM classification fails on three critical fronts: you can't calibrate confidence scores or adjust precision-recall tradeoffs, the model may ignore structured data or context you provide, and you have no visibility into what the LLM actually used to make its decision.
- Wrapping an LLM verdict in logistic regression automatically fixes calibration (empirically matching true probabilities), lets you add other features, and gives interpretability about how much the LLM contributes to the final decision.
- On an irony detection task, logistic regression on the raw LLM verdict improved Brier score from 0.259 to 0.175 just by calibrating; adding more LLM-extracted features (token probabilities, multiple reasoning passes) provides a clear path to further improvement that prompt-tweaking doesn't offer.