FrameBench:A Language Understanding Benchmark Based on Frame Semantics

Open original

This arXiv paper introduces FrameBench, a multiple-choice benchmark that checks whether language models can tell apart the semantic frames a verb evokes in different sentences, the implicit background inferences humans make when reading. It covers English and Japanese, built from FrameNet-style resources with native-speaker verification.

  • The benchmark tests whether models distinguish the frames the same verb evokes across contexts, probing implicit enrichment rather than surface-level language skill.
  • Small models struggle noticeably, while several large models score above the human reference baseline.
  • The dataset and construction and evaluation code are publicly released, so the results can be reproduced and extended.

FrameBench is a benchmark for testing whether language models understand sentences the way people do, by filling in what the text leaves unsaid. The idea comes from frame semantics: when you read a verb, you pull in background knowledge about the situation it evokes. Hearing that someone "bought" something implies money changed hands, a seller existed, and so on. The authors want to know whether LLMs make these implicit inferences, or whether their strong scores on standard tasks hide a gap. The benchmark is multiple-choice, and each question asks the model to tell apart the frames evoked by the same verb in different contexts. The English and Japanese versions were built from FrameNet-style resources through a generation-and-verification pipeline, with native speakers checking the items. The results split by model size. Small models struggle noticeably on the task, which suggests the frame-level distinctions are not something they pick up easily. Several large models, though, beat the human reference scores, so on this particular measure they are not just approximating human comprehension but outperforming the human baseline. That is a striking claim, and it is worth asking how the human reference was collected, how many annotators were involved, and how much the multiple-choice format rewards pattern matching over genuine understanding. The paper's abstract doesn't settle those questions, so they belong on the list for the full PDF. The practical value is that the authors released both the dataset and the construction and evaluation code. For anyone working on semantic evaluation, multilingual benchmarks, or the question of whether benchmark wins translate into real comprehension, FrameBench gives a concrete, reusable test in two languages. Read it if you care about that question. Skip it if you're mainly after new modeling techniques, since the paper is about evaluation.

Ask about this article