Building an Ultra-High Throughput AI-SQL Engine
Short summary
Quail is a new inference engine that runs AI-SQL queries (SQL with LLM function calls) much faster than existing approaches by co-optimizing query planning and GPU inference. It reduces wasted compute from CPU overhead and KV cache mismanagement, cutting costs by 4-5x compared to naive implementations.
- Running AI-SQL queries on vLLM naively is 27x slower than the theoretical optimum because of CPU scheduling overhead and unnecessary KV cache evictions that force reprocessing of tokens.
- Quail jointly plans queries and inference execution to minimize KV cache waste and keep the GPU busy, achieving 360 documents/second on a medical adverse event filtering task that would cost $0.37 on an H100 versus $1.75 with GPT-5 nano.
- The system handles real workloads like filtering 100,000 IMDB reviews through two AI predicates in under 6 minutes for under $0.37, demonstrating practical viability for production AI-SQL pipelines.
Full summary
AI-SQL lets you run LLM classifiers directly inside databases through SQL queries, which sounds great until you actually try it at scale. The problem is brutal: filtering 100,000 rows means 100,000 LLM calls. Joining two tables with 5,000 and 4,144 rows means millions of calls. Existing systems from Berkeley, Stanford, MIT, and Cornell have tried to cut costs by eliminating redundant calls or using cheaper models, but you still end up with hundreds of thousands or millions of inference requests. Database vendors like Snowflake, BigQuery, Databricks, and MotherDuck all support AI-SQL now, but the economics are terrible.
The researchers found that using vLLM—a popular general-purpose inference engine—to handle these millions of requests is wasteful. On a medical records query called BIO-4 with 5,000 reports and 4,144 reaction terms, vLLM took 6.84 hours when the theoretical minimum was 14.9 minutes. The GPU spent huge chunks of time idle while the CPU handled request scheduling overhead. vLLM also kept discarding key-value cache data it needed again later, forcing it to recompute 50 million extra tokens. The researchers calculated this should have been a straightforward prefill-only workload where the H100 stayed busy, but the general-purpose design of vLLM wasn't built for the access patterns AI-SQL creates.
Quail is their purpose-built inference engine that treats the entire query plan as a single optimization problem instead of sending millions of separate requests to a generic system. You write normal SQL with AI functions, and Quail handles the execution. The code example shows filtering 100,000 IMDB reviews through two AI predicates—checking if reviews discuss the ending and whether they recommend the movie. Quail's physical plan tracks things like KV cache budgets, token chunk sizes, and rewind strategies to squeeze efficiency out of the GPU. The actual numbers from running this workload show it processes the data in stages, with the first filter passing 28% of rows through to the second filter, completing in a couple of minutes rather than hours.