Click any tag below to further narrow down your results
Links
Google Quantum AI ran a series of tests on its superconducting chips showing that quantum systems can't be assigned fixed answers in advance, and that this property can be measured and used for computation. The team reports results from up to 105 qubits, and argues the same tests can serve as a benchmark for hardware errors.
- On a 3×3 "magic square" game, the qubits won 98.30% of rounds, beating the 88.89% ceiling any pre-written classical strategy can reach.
- A Bell–Kochen–Specker style test gave χ = 5.590 ± 0.008, above the classical limit of 4 and near the quantum maximum of 6.
- Shallow quantum circuits solved hidden-linear-function problems on grids up to 105 qubits, and the same tasks exposed subtle coherent errors that random-circuit benchmarks miss.
A developer tested whether DuckDB could compete with Spark and Trino at large scales and found that a single powerful machine running DuckDB outperforms distributed clusters on 10TB queries—faster than Trino and half the cost of multi-node setups. The key insight: most time is spent waiting on S3 reads, not computation, so distribution overhead doesn't help.
- At 10TB scale, single-node DuckDB on an r7gd.4xlarge beats Trino on identical queries by 1.7-2.1x, and costs half as much as 5-13 node clusters despite being 2-3x slower than them
- S3 read rate, not compute speed, determines performance at scale—DuckDB's default single-threaded S3 reads leave CPUs 90% idle, but running 4x more threads than cores achieves 3x faster read rates
- DuckDB v2.0 (alpha) expects 2-3x faster S3 reads, potentially making single-node DuckDB the cheapest option for datasets up to 10TB
Liquid AI released Pipette, an open-source benchmarking suite for measuring AI model performance on phones, laptops, and embedded devices, plus DSpark draft models that speed up inference by 2-3x through speculative decoding. Both tools come with public datasets and are designed to let developers test the exact hardware-model-runtime combinations they plan to ship.
- Pipette provides a standardized way to benchmark 35+ model classes across different quantization levels, runtimes, and devices (phones, laptops, PCs, embedded hardware), with 10k+ verified results already in the warehouse and an interactive dashboard to compare configurations.
- DSpark draft models achieve 2.27x to 3.18x throughput gains on everything from M4 Max MacBooks to H100 GPUs by using lightweight draft models to propose token blocks that a target model verifies in one pass, without changing output quality.
- Both projects are open-source with native iOS and Android apps, public datasets, and submission workflows that let developers contribute benchmarks for their own hardware and model combinations.
AI agents that succeed 77% of the time on average only succeed consistently on 53% of tasks — a reliability gap hidden by standard benchmarks. IBM Research built a diagnostic tool that identifies unstable decision points in an agent's reasoning and converts them into guidelines that cut this gap in half without sacrificing overall accuracy.
- Standard metrics (Mean@k) mask a consistency problem: tasks an agent can solve sometimes but not others, with nothing about the task changing. On hard tasks, the gap between average success rate and consistent success rate reaches 30 percentage points.
- The Consistency Analyzer pinpoints flip-prone steps by resampling each decision point in a recorded trace with a single model call (requesting 5 completions), identifying where the model's output distribution is too flat to be reliable.
- Consistency guidelines derived from these unstable steps cut the gap roughly in half (24.4pp → 12.0pp) and generalize to related tasks, lifting Pass^5 by +16pp on the same task and +13pp on similar tasks without reducing average accuracy.
Hex built a suite of analytical evals to test data-analysis models and found Claude Fable 5 outperforms its Opus 4.x predecessors by 10–15%, nailing both semantically modeled and raw-data tasks with fewer mistakes. They’ve also designed a tougher “Frontier” benchmark for long-horizon, open-ended scenarios, where Fable 5’s careful assumptions and cross-checks boost its pass rate to around 58%.
- Claude Fable 5 beats Opus 4.7 by 10-15 points on Hex's core benchmarks, scoring 93%+ on Analytical Hard/Semantically Modeled tests and 65% on Semantically Unmodeled tasks, versus prior Opus versions' single-digit gains
- Fable's advantage comes from following a "golden workflow" (starting in the semantic layer, cross-checking raw SQL) and transparently stating assumptions, which lets it catch errors like a cents-for-dollars mistake that Opus misses
- On Hex's new "Frontier" benchmark for long-horizon, open-ended tasks, Fable at Max Effort hits 58% pass rate, notably outperforming other setups
The article argues that as AI handles more tasks, companies must assess human skills like critical thinking and creativity. It introduces MindForm, a SaaS platform using game-like scenarios to measure and benchmark these un-automatable cognitive abilities. The piece covers its revenue model, go-to-market plan, and technical stack for a quick MVP.
- Existing hiring tools (HackerRank tests, case studies) fail to measure creative problem-solving, mental flexibility, or critical thinking—leaving companies "flying blind" as automation removes routine tasks
- MindForm's competitive moat comes from proprietary data linking cognitive profiles to specific roles (e.g., ML engineer at fintech vs. PM at CPG), built up through every platform interaction
- Revenue model combines per-user tiered subscriptions, API access, and benchmark report sales, with a free "Cognitive Agility Score" and open-source engine driving both developer trials and HR licensing deals
- MVP is buildable in weeks using Next.js, Vercel, Supabase, and Framer Motion
DuckDB has proven to be superior to Polars when handling large datasets, particularly 1TB of data. While DuckDB effectively manages memory and execution with a robust design, Polars struggles with large data processing, leading to out-of-memory errors.
- DuckDB handled the 1TB benchmark successfully while Polars crashed with out-of-memory errors
- DuckDB's memory management and execution engine scale more effectively to large datasets than Polars'
- Polars is optimized more for in-memory, smaller-scale workloads and breaks down at larger scales like 1TB