The author frames tokenizer design as an integer linear program, relaxes it to a continuous LP, and uses cutting planes to close the gap between fractional and integral solutions. They automate cut discovery with Codex, apply cycle constraints on overlapping token edges, and report provably optimal tokenizers on small pretokenized datasets.

• BPE, despite being greedy, actually gets within 1% of the true optimal tokenizer, as revealed by casting tokenization as an ILP with a provable lower bound.
• The LP relaxation plus rounding gives a feasible upper bound, and cutting planes (constraints violated by fractional solutions but satisfied by all integral ones) tighten the gap between the two.
• Codex was used to automatically discover these cutting planes by brute-forcing small word pairs/triplets and solving auxiliary LPs to find maximally violated constraints, rather than deriving them by hand.
• Results are only provably optimal under a fixed pretokenization scheme, since dropping that assumption would make the LP intractably large.

The author describes creating a 340 M-parameter Llama-based model trained exclusively on English texts published before 1900. They built custom data pipelines, tokenization, base-training and fine-tuning scripts, handled deduplication and filtering of historical sources, and trained locally and on cloud GPUs for about $80. The result is a toy “Victorian” chatbot that can hallucinate and isn’t aligned for modern safety.

• Built a 340M-parameter LLaMA-style model trained exclusively on pre-1900 English texts, doing every step (data pipeline, tokenization, base-training, fine-tuning) from scratch
• Total cloud GPU cost was about $80, after prototyping a smaller 14M-parameter version locally
• Cleaning and deduplicating ~90GB of historic sources (Gutenberg, Internet Archive, British Library, Harvard) was the hardest and most time-consuming part
• The model has no hard 1900 knowledge cutoff or safety alignment, so it can hallucinate and let anachronistic facts (like Einstein) slip through

OpenAI released Privacy Filter, an open-weight model that detects and redacts eight categories of personally identifiable information in text using context-aware language understanding rather than simple pattern matching. It runs locally on long inputs in a single pass and uses BIOES span tags for precise masking. The model was built by converting a pretrained language model into a token classifier and trained on a mix of public and synthetic data.

• Uses context-aware language understanding instead of regex, so it distinguishes public figures from private individuals and catches hidden credentials like passwords
• Small enough to run entirely locally, processing long documents in a single pass
• Built by converting a pretrained language model into a bidirectional token classifier trained on public plus synthetic data, with model-assisted annotation and human review filling gaps
• Tags eight PII categories using BIOES span tagging for clean, precise redaction boundaries

This article explores an unconventional method for classifying text by leveraging compression algorithms. The author demonstrates how to concatenate labeled documents, compress them, and use the compressed sizes to predict labels for new texts. While the method shows promise, it is computationally expensive and generally underperforms compared to traditional classifiers.

• Compression-based classification (concatenate labeled texts, measure compressed-size increase) hits a 0.749 macro F1 with gzip on 4 categories of 20 Newsgroups, versus 0.88 for multinomial Naive Bayes.
• lzma pushes accuracy up to 0.897, beating Naive Bayes, but takes 32 minutes versus over 5 minutes for gzip on just 1,353 test cases—far too slow to be practical.
• The technique reframes text classification as an information-theory problem, showing compression algorithms implicitly model word probability distributions.

Pingkit is a toolkit designed for training reproducible, capacity-aware models using transformer activations. It offers features for extracting embeddings, training neural architectures, and creating custom probes tailored to specific research needs. The toolkit is integrated with Hugging Face models and provides various utilities for data processing and model training.

• Pingkit extracts embeddings directly from transformer activations for downstream probing/analysis rather than relying on final-layer outputs alone.
• It's built to train "capacity-aware" probes, letting researchers control model complexity to avoid overstating what information is linearly/nonlinearly decodable from activations.
• Integrates with Hugging Face models, so it can plug into existing pretrained transformer workflows without custom extraction code.
• Emphasizes reproducibility, aiming to standardize probing experiments that are often ad hoc and hard to replicate across papers.