Bitnet.cpp is a framework designed for efficient inference of 1-bit large language models (LLMs), offering significant speed and energy consumption improvements on both ARM and x86 CPUs. The software enables the execution of large models locally, achieving speeds comparable to human reading, and aims to inspire further development in 1-bit LLMs. Future plans include GPU support and extensions for other low-bit models.

Tokasaurus is a newly released LLM inference engine designed for high-throughput workloads, outperforming existing engines like vLLM and SGLang by more than 3x in benchmarks. It features optimizations for both small and large models, including dynamic prefix identification and various parallelism techniques to enhance efficiency and reduce CPU overhead. The engine supports various model families and is available as an open-source project on GitHub and PyPI.

ANEMLL is an open-source project designed to facilitate the porting of Large Language Models (LLMs) to Apple Neural Engine (ANE) with features like model evaluation, optimized conversion tools, and on-device inference capabilities. The project includes support for various model architectures, a reference implementation in Swift, and automated testing scripts for seamless integration into applications. Its goal is to ensure privacy and efficiency for edge devices by enabling local model execution.