Pinterest engineered Manas, an embedding retrieval system handling billions of vectors with approximate nearest neighbor search, hybrid queries, and real-time updates. They optimized it through quantization techniques that cut computing resources by 10-15% while maintaining search accuracy.

• SIMD intrinsics on 8-bit integers reduced per-query compute by 10-15% without sacrificing performance
• Product quantization on disk storage paired with full-precision centroids improved both accuracy and throughput, diverging from the original SPANN approach
• The system supports hybrid queries combining token and embedding search with sub-second indexing of new content

PixelRAG converts web pages, PDFs, and images into screenshot tiles, embeds them using a fine-tuned Qwen3-VL model, and builds vector indexes for retrieval-augmented generation. It provides a CLI (pixelshot), pipeline commands for chunking, embedding, indexing, and serving (FAISS or Qdrant), plus a live API and a Claude plugin for visual search.

• PixelRAG ditches HTML parsing entirely, indexing screenshot tiles of pages/PDFs via a fine-tuned Qwen3-VL embedding model to preserve tables, charts, and layout
• Ships a pre-built index of 8.28 million Wikipedia articles, usable via a free live API or a ~217GB local download
• Includes a Claude Code plugin ("pixelbrowse") that lets Claude screenshot live pages and read diagrams/tables visually on request
• Supports both FAISS (simple/local) and Qdrant (disk-backed, quantized, multi-server) as swappable backends for scaling

This guide shows how to turn raw documents into semantic embeddings using a pre-trained sentence-transformers model, reduce their dimensionality with UMAP, and then apply HDBSCAN to uncover topic clusters without labels. It walks through data loading, embedding generation, dimensionality reduction, clustering, and sampling of cluster contents in Python.

• Pulling 150 posts from three newsgroup categories and encoding them with all-MiniLM-L6-v2 gave 384-dim embeddings in seconds.
• UMAP compressed those vectors from 384 to 5 dimensions to make density-based clustering feasible.
• HDBSCAN (min cluster size 8, min_samples 3) found just two clusters (101 and 49 docs) with zero points labeled noise, though sci.space and sci.med content blurred together in one cluster while autos formed a distinct second group.
• Adjusting HDBSCAN's cluster size and sample thresholds would change the number of clusters detected, and plotting all pairwise UMAP dimensions offers a visual check on whether clusters match intuitive topics.

The article shows how packing specialized, retrieved knowledge into smaller AI agents can match or beat huge frontier models. It explains a structure—raw source extractions, concept entries, theses and a startup primer—plus embedding‐powered retrieval to feed just the right context at query time.

• Structured retrieval (10,000 pages → 381 concept docs + 54 theses via hybrid BM25/semantic search) let smaller models match or beat frontier models on specialized tasks.
• The same pipeline replicated across a dozen domains, from finance to rare medical research to corporate policy.
• A locally-run Qwen model with this "knowledge agent" harness performed comparably to Claude Opus, at zero cloud cost.
• Embedding costs were trivial (under a dollar for thousands of documents using BGE-M3 or OpenAI's text-embedding-3-small) before moving entirely to local hardware.

Pingkit is a toolkit designed for training reproducible, capacity-aware models using transformer activations. It offers features for extracting embeddings, training neural architectures, and creating custom probes tailored to specific research needs. The toolkit is integrated with Hugging Face models and provides various utilities for data processing and model training.

• Pingkit extracts embeddings directly from transformer activations for downstream probing/analysis rather than relying on final-layer outputs alone.
• It's built to train "capacity-aware" probes, letting researchers control model complexity to avoid overstating what information is linearly/nonlinearly decodable from activations.
• Integrates with Hugging Face models, so it can plug into existing pretrained transformer workflows without custom extraction code.
• Emphasizes reproducibility, aiming to standardize probing experiments that are often ad hoc and hard to replicate across papers.