2 links tagged with all of: structured-output + real-time-ai
Click any tag below to further narrow down your results
Links
System One models are stripped-down LLMs that only output multiple-choice answers, trading flexibility for speed and predictability. The author shows two practical techniques for building real-time systems with them: layered goal-setting for sequential decision-making and tournament sampling for choosing among many options.
- You can convert any LLM into a fast classifier by batching single-token outputs with structured prompts—no model retraining needed, just inference-level changes.
- Tiered goals (asking the model to pick short-term objectives before making immediate decisions) dramatically improves performance in tasks like game-playing by giving the model more compute to reason about strategy.
- Tournament sampling—splitting large choice sets across multiple rounds rather than trying to rank everything at once—works better than absolute scoring because LLMs judge relative quality more reliably than absolute confidence.
The author demonstrates two practical techniques for programming with System One models (fast classifiers that pick from multiple choices): tiered goal-setting for sequential decision-making, and tournament sampling for choosing among many options. He shows these approaches working in Doom gameplay and Wikipedia navigation tasks.
- Tiered goals work better than single-pass decisions: periodic prompts asking the model to choose short-term goals (e.g., "kill enemies" vs "collect armor") make it perform more intelligently than just reacting to immediate game state every 200ms.
- Tournament sampling beats confidence scoring when picking from many options: feeding 100 links at a time, then narrowing down, found the optimal path in seconds, while trying to score all 1,000+ links failed badly.
- System One models offer a practical alternative to tool calls for real-time systems where you need predictable latency and don't need full language generation flexibility.