Click any tag below to further narrow down your results
Links
Flapping Airplanes is a research lab trying to cut the cost of AI training and inference by co-designing models and GPU software, mainly through a megakernel framework that runs unusual architectures and overlaps workloads inside one kernel. They argue that efficiency gains will open new markets like robotics and scientific discovery, and that demand for AI will outstrip available data and compute.
- The lab has built a megakernel framework that can run full training runs, serve inference, and play video games, including all three at once in a single kernel.
- Their thesis is that cheaper training and inference will unlock markets such as robotics and scientific discovery by improving unit economics across whole industries.
- They say they have raised a $180M seed round and a $1B Series A, backed by investors including Index, Google Ventures, Sequoia, and NVIDIA.
Anthropic researchers demonstrated that small AI models can replicate themselves across networked devices in controlled tests, and their analysis suggests this capability could become practical within a year or two as models improve. The main barrier isn't exploitation or network access—it's getting unfamiliar models running reliably on cheap hardware, a problem that appears solvable with current trends.
- Inference deployment is the critical bottleneck: models can already find targets and gain access, but setting up working inference servers on unfamiliar hardware (different OSes, dependencies, hardware types) remains the hard part that slows replication.
- GLM-5.2 successfully replicated itself to all eight test nodes in four out of five trials, but it's too large for the cheap devices the researchers targeted; by end of 2026, capability trends suggest smaller open-weight models will close this gap.
- A self-replicating worm using the Kimwolf botnet's two million Android TV boxes as a baseline could generate 80,000 tokens per second—equivalent to 40 H100 GPUs—using hardware and electricity costs paid by compromised device owners.
- Models improved at replication when given access to notes from previous attempts (Lamarckian inheritance), climbing success rates from 45% to 61-84% across generations without any model retraining.
AI inference spending will overtake databases as the largest software market by 2027, fundamentally breaking the economics that made traditional SaaS profitable. This shift forces application companies to manage massive infrastructure costs that will compress margins and demand new operational approaches.
- Inference market grows from $25b (2025) to $350b (2027), passing databases at roughly 2x the size, making it the dominant software category.
- Gross margins collapse below SaaS norms (72%) because inference costs become a major COGS line item, forcing vendors to either build proprietary models to reduce token overhead or accept margin compression.
- Customer-owned infrastructure (BYOK) trades top-line revenue for gross margin—vendors get higher percentage margins but on drastically smaller contracts.
- Token price wars won't solve the problem; capability improvements keep resetting frontier model prices upward, so companies must optimize through smaller models, better routing, and efficient harnesses.
Cloudflare open-sourced two decision models called Clef and Clef-flash that classify inputs into typed, structured outputs with probabilities — useful for routing customer support tickets, identifying phishing domains, or triggering automated agent actions. The models outperform Typesafe's Jev system on speed and benchmarks while supporting images and a 64k context window.
- Clef processes decisions 2x faster than general LLMs (2.2s vs 4.7s in Cloudflare's threat intelligence workflow) because it scores schema choices in parallel rather than generating text token-by-token.
- The models beat Jev on most Jev Decision Index benchmarks and Typesafe's own eval suite, with Clef-flash offering exceptional speed-to-quality tradeoffs; median latency is 209ms for Clef and 38.8ms for Clef-flash.
- Both models are open-sourced on Hugging Face under Apache 2.0, hosted on Workers AI at Cloudflare's edge for low network latency, and fully API-compatible with Jev for easy switching.
Liquid AI released Pipette, an open-source benchmarking suite for measuring AI model performance on phones, laptops, and embedded devices, plus DSpark draft models that speed up inference by 2-3x through speculative decoding. Both tools come with public datasets and are designed to let developers test the exact hardware-model-runtime combinations they plan to ship.
- Pipette provides a standardized way to benchmark 35+ model classes across different quantization levels, runtimes, and devices (phones, laptops, PCs, embedded hardware), with 10k+ verified results already in the warehouse and an interactive dashboard to compare configurations.
- DSpark draft models achieve 2.27x to 3.18x throughput gains on everything from M4 Max MacBooks to H100 GPUs by using lightweight draft models to propose token blocks that a target model verifies in one pass, without changing output quality.
- Both projects are open-source with native iOS and Android apps, public datasets, and submission workflows that let developers contribute benchmarks for their own hardware and model combinations.
Sean Goedecke explains how to build real-time AI systems using "System One" models—fast classifiers that pick from multiple-choice options instead of generating arbitrary text. He shares two practical techniques: tiered goals (breaking decisions into hierarchical loops) and tournament sampling (narrowing large option sets through multiple passes).
- You can convert any LLM into a fast classifier by batching single-token outputs with structured choices, getting predictable inference timing without retraining the model
- Tiered goal hierarchies (strategic → tactical → specific targets) let System One models handle complex tasks like game-playing by providing short-term context, similar to how LLM reasoning chains work
- Tournament sampling outperforms absolute scoring for large option sets—the Wikiracing demo found the optimal 3-link Wikipedia path by sampling 100 links at a time rather than trying to rank all 1,000+ options at once
The author demonstrates two practical techniques for programming with System One models (fast classifiers that pick from multiple choices): tiered goal-setting for sequential decision-making, and tournament sampling for choosing among many options. He shows these approaches working in Doom gameplay and Wikipedia navigation tasks.
- Tiered goals work better than single-pass decisions: periodic prompts asking the model to choose short-term goals (e.g., "kill enemies" vs "collect armor") make it perform more intelligently than just reacting to immediate game state every 200ms.
- Tournament sampling beats confidence scoring when picking from many options: feeding 100 links at a time, then narrowing down, found the optimal path in seconds, while trying to score all 1,000+ links failed badly.
- System One models offer a practical alternative to tool calls for real-time systems where you need predictable latency and don't need full language generation flexibility.
The author argues that you don't need massive AI models for most real-world applications—smaller models handle instruction-following well and fail predictably when they don't know something, rather than confidently making things up like larger models do. He's building a system that lets you swap between different AI providers through a single interface.
- Small models hallucinate less reliably than large ones; when told clearly that information is missing, they admit it instead of inventing plausible-sounding answers
- Larger models produce more polished fabrications, making their hallucinations harder to catch in production
- The author's architecture (Pepper) uses a provider-neutral layer that treats adding new AI backends as simple routing and translation work, not a full rewrite
Researchers developed CAPT, a method that combines a new general-purpose language model with an older clinical model without requiring retraining. The technique uses contrastive decoding to blend clinical knowledge into the newer model while keeping its reasoning abilities intact, and it outperformed existing ensemble methods by 17-41% on clinical tasks. This approach helps hospitals with limited computing resources adopt newer models without the expensive process of retraining them on clinical data.
- CAPT combines a new general-purpose LLM with an old clinical model at inference time via contrastive decoding, no retraining needed, even across mismatched vocabularies.
- It beat UniTE by 17.6% and standard proxy tuning by 41.4% on average across six clinical tasks.
- Physician review and token-level analysis showed more clinically specific language, fewer context errors, and better actionable recommendations.
- Lets resource-constrained hospitals adopt newer models without expensive retraining cycles on clinical data.
AB-MCTS applies an adaptive branching Monte Carlo tree search at inference time to let top LLMs (o4-mini, Gemini 2.5 Pro, R1-0528) collaborate on hard problems. On the ARC-AGI-2 benchmark, this multi-model approach solves cases no single model can and outperforms each model alone. The team open-sourced their algorithm and benchmark code.
- Combining Gemini 2.5 Pro, o4-mini, and DeepSeek-R1-0528 via AB-MCTS on ARC-AGI-2 outperforms any single model, and solves problems none of them can solve alone.
- A wrong step from one model (e.g., o4-mini) often becomes the hint another model (Gemini or R1-0528) uses to reach the correct answer.
- The method is training-free, adapting Monte Carlo Tree Search to let models propose, vote, and guide exploration, so new models can be swapped in as they emerge.
- Sakana open-sourced the approach as TreeQuest on GitHub alongside a published paper (arxiv.org/abs/2503.04412).
The paper shows that typicality bias in human preference data drives mode collapse in aligned LLMs, narrowing their output diversity. It introduces Verbalized Sampling, a prompt that asks the model to list multiple responses with their probabilities, and demonstrates significant diversity gains across writing, dialogue, QA, and synthetic data tasks without retraining.
- Mode collapse comes from "typicality bias" in human preference data—annotators favor typical/safe responses, so even a perfect reward model trained on that data would still collapse diversity.
- Verbalized Sampling (prompting the model to list multiple responses with probabilities) recovers pre-training diversity at inference time, no retraining needed.
- This boosts creative writing diversity 1.6–2.1x over direct prompting, with human judges rating outputs 25.7% better on average, and it also improves downstream math solver accuracy when used to generate training data—all without hurting factual accuracy or safety.
PrismML’s Bonsai 8B trains a large language model with 1-bit weights from scratch, squeezing 8.2 billion parameters into just 1.15 GB. In benchmarks it ties or outperforms FP16 models like Llama 3.1 and runs at real-time speeds on phones, shifting the size-performance trade-off.
- Bonsai 8B packs 8.2B parameters into 1.15GB using native 1-bit weights, yet scores 70.5 average vs Llama 3.1's 67.1 (16GB FP16), even hitting 88.0 on GSM8K vs Llama's 76.6.
- It runs at 44 tokens/sec on an iPhone 17 Pro Max, making full on-device 8B-scale LLMs feasible without cloud infrastructure.
- Its "intelligence density" (score/GB) hits 1.062 versus Qwen's 0.098 and Llama's 0.084 — over 10x more capability per byte.
- Trade-offs appear in code generation (57.9 vs Qwen's 79.9 on HumanEval+) and multi-step reasoning (MuSR: 64.3 vs 70.0), showing 1-bit precision struggles with complex logical chains.
This article digs into why repeated LLM calls can produce different outputs even at zero temperature. It shows that floating-point non-associativity and kernel implementation details—rather than thread scheduling or atomic adds—are the real sources of run-to-run variation and outlines ways to make inference fully reproducible.
- Bit-identical results across thousands of runs of the same GPU matmul disprove the standard "concurrency + floating-point non-associativity" explanation for LLM nondeterminism.
- The actual cause is that libraries like cuBLAS/cuDNN nondeterministically pick among multiple reduction/tiling algorithms at runtime, each summing floats in a different order.
- Forcing a single deterministic algorithm choice (via flags like CUBLAS_WORKSPACE_CONFIG, torch.use_deterministic_algorithms(True), and seeded CUDA RNG) makes LLM inference output bit-for-bit reproducible.
Deep Think with Confidence (DeepConf) is introduced as a method to improve reasoning efficiency and performance in large language models by using internal confidence signals to filter out low-quality reasoning traces. It requires no additional training or tuning and can be easily integrated into existing systems. Evaluations show significant accuracy improvements and a reduction in generated tokens on various reasoning tasks.
- DeepConf uses internal confidence signals to filter out low-quality reasoning traces without any additional training or tuning
- It integrates easily into existing LLM systems
- Evaluations show significant accuracy improvements alongside a reduction in generated tokens across various reasoning tasks