Three months ago, the author started building a “Vintage LLM” trained solely on texts from before 1900. They wrote custom scripts for data ingestion, tokenization, base-training and fine-tuning. The final model follows the LLaMA architecture with 340 million parameters and lives on HuggingFace (croqaz/vintage-LLM-340m-v1-base). All code and datasets are open source on GitHub, and bigger variants are covered in an earlier post. The hardest part was assembling and cleaning historic texts. They pulled from sources like Project Gutenberg, the Internet Archive, The British Library and Harvard’s collections—roughly 90 GB of raw data. Filtering out unreadable OCR dumps and removing duplicates ate up weeks of work. A side project, the Book-Metadata HF dataset, tags titles and publication years for 5,300 Gutenberg books. Training happened in two phases. Small experiments ran on the author’s PC (Ryzen 7 9700X, 64 GB RAM, RX 9070 GPU), where they built a 14 million-parameter prototype. For the 340 M model, they rented GPU time on RunPod, ThunderCompute and Vast.ai, spending about \$80 total. Tokenization used a standard byte-level approach. Base-training taught the model to predict text; fine-tuning shaped conversational turns, without any alignment or safety filters. The result is a toy LLM that speaks like a Victorian-era writer but will hallucinate and reflect 19th-century biases. It doesn’t enforce a hard 1900 knowledge cutoff—testing for figures like Einstein can still slip through. Still, it’s a hands-on dive into every step of LLM construction, from raw text wrangling to distributed GPU training.