← All Tags
# optimization → tldr-a-byte-sized-daily-tech-newsletter

1 link tagged with all of: optimization + tldr-a-byte-sized-daily-tech-newsletter

Click any tag below to further narrow down your results

+ tokenization (1) + integer-linear-programming (1) + cutting-planes (1) + byte-pair-encoding (1)

Links

Finding Optimal Tokenizers

The author frames tokenizer design as an integer linear program, relaxes it to a continuous LP, and uses cutting planes to close the gap between fractional and integral solutions. They automate cut discovery with Codex, apply cycle constraints on overlapping token edges, and report provably optimal tokenizers on small pretokenized datasets.
Last saved Jun 18, 2026 · 7 min read
+ tokenization + integer-linear-programming + cutting-planes + byte-pair-encoding optimization tldr-a-byte-sized-daily-tech-newsletter