The author describes creating a 340 M-parameter Llama-based model trained exclusively on English texts published before 1900. They built custom data pipelines, tokenization, base-training and fine-tuning scripts, handled deduplication and filtering of historical sources, and trained locally and on cloud GPUs for about $80. The result is a toy “Victorian” chatbot that can hallucinate and isn’t aligned for modern safety.

Organizations are increasingly faced with the decision of whether to implement Retrieval-Augmented Generation (RAG) or fine-tuning for their AI initiatives. RAG connects large language models to external databases, allowing access to real-time information, reducing inaccuracies, and enhancing security and traceability. However, implementing RAG comes with its own technical challenges that require careful planning and maintenance.