1 link tagged with all of: llm + vulnerability-analysis + context-windows
Click any tag below to further narrow down your results
Links
The author reruns security vulnerability triage experiments across 26 combinations of Claude and GPT-5 models with varying reasoning effort and context sizes. A four-model “council” achieved 86.2% unanimous votes, and GPT-5.4 at medium/high effort led overall performance, though full-chain solutions remained rare. The study also found higher reasoning sometimes backfires and function-level inputs outperformed whole-file analysis.
- More reasoning effort doesn't reliably mean better vulnerability detection—GPT-5.5-high/xhigh underperformed its own medium setting, and low-effort modes always trailed.
- A 4-model voting council hit 86.2% unanimous decisions and cut no-majority cases to 2.8%, far outperforming any single model.
- Feeding models function-level code instead of whole files dramatically boosted results.
- Even the best setups rarely solved the full vulnerability chain (just 1.9% full solves overall), showing partial pattern-matching rather than genuine deep understanding.