GitHub - Tufalabs/tufazip: We shrink 1 GB of Wikipedia to 105.39 MB, losslessly.

Open original

tufazip is a Transformer-based compressor for the enwik9 benchmark, reported at 105,388,353 bytes total once the decoder archive is counted. The repo gives the exact commands, file hashes, and hardware needed to reproduce the decode.

  • Compressed enwik9 is 100,958,324 bytes, with a 4,430,029-byte decoder ZIP, for a ranked total of 105,388,353 bytes.
  • It is an online Transformer initialized from a quantized offline Cycle2 prior, and needs two NVIDIA B200 GPUs.
  • Compression takes about 631,548 seconds and decompression about 645,748 seconds, so it is far from a practical everyday tool.

This is a decoder for tufazip v2, a neural compressor built by Tommy He at Tufa Labs. The job is to reconstruct the 1,000,000,000-byte enwik9 text file (the Wikipedia dump used in Matt Mahoney's Large Text Compression Benchmark) from a 100,958,324-byte compressed artifact plus a 4,430,029-byte ZIP holding the decoder, weights, and vocabulary. The ranked total, the number that matters for the benchmark, is 105,388,353 bytes. The method is an online Transformer that starts from a quantized offline "Cycle2" prior, so the model keeps adapting as it decodes. The cost is time: compression took roughly 631,548 seconds and decompression about 645,748 seconds, excluding offline training. That is close to two hundred hours each way, which is the real trade-off here. You get a strong compression ratio at the price of a very slow, GPU-bound pipeline. The setup is strict. You need Linux x86-64, exactly two visible NVIDIA B200 GPUs, GCC, CUDA 13.0, cuDNN 9.20, Python 3.12.3, and uv. The run command is a single uv invocation that takes the compressed file, an output path for enwik9, and an optional state directory for resumable checkpoints. If the job dies partway, you rerun the identical command and it picks up from the last checkpoint. The article includes a table of SHA-256 hashes for the input artifact, the weights file (4,334,741 bytes), and the decoded output, so you can verify each stage. Runtime pins are Python 3.12.3, NumPy 2.4.6, and PyTorch 2.12.0+cu130. The README also flags that v1 remains at commit 794dda3 with its original decoder and instructions, and it credits Jerome Sieber for mentorship and Tufa Labs for compute. It also thanks Matt Mahoney, whose benchmark and online textbook Data Compression Explained are the reference points this project is measured against. If you are curious about neural compression on enwik9 specifically, the commit diff between v1 and v2 and the ranked-total arithmetic are the parts worth reading closely.

Ask about this article