Click any tag below to further narrow down your results
Links
Notion argues that teams get little from AI until they build a shared context system that people and agents work in together, then shared "factories" that run recurring tasks for everyone. It's a sales-flavored field guide, so read the product pitch with that in mind.
- Chat-based AI is stateless and single-player; value compounds only when agents read from and write to a central, well-structured knowledge store.
- Shared "factories" (specific, shared, scrutinized agent systems) beat everyone hand-rolling personal agents, because fixed setup costs get paid once and improvements compound.
- The article's concrete build process: notice a repeatable task, plan with a simple version, run it in shadow mode first, then improve it against written evals.
Dylan Martin, a PostHog engineer, explains why he left the feature flags team for AI product work and what that shift has taught him, mainly about writing evals and deciding where agents should act versus where humans should see and decide. It's a short personal account of the transition rather than a technical deep dive.
- Much of the skill set carries over: systems thinking, failure modes, and proving correctness still matter, so the move felt smaller than the job postings suggested.
- Evals are the main new skill. Unlike unit tests, they need to accept several good answers to the same request and catch subtle regressions, like counting signup events instead of unique users.
- The hardest open question is interface design: when MCP and skills should replace clicking through a UI, and when a human needs visual previews to make the call, such as seeing a flag's blast radius.
Daniel Miessler argues that nearly all AI work follows one loop: define the ideal state, measure progress with evals, build, evaluate, and iterate. He says the main bottleneck isn't AI's ability to execute but people's ability to clearly describe what they want.
- AI failures often come from incomplete or conflicting descriptions of what the user wants, not from poor execution. The AI can only work from the written artifact, not the idea in your head.
- Evals are simple checks that score work against defined criteria, such as accuracy, format, or passing tests. Failing criteria give you specific things to improve.
- The system is meant to be general, applying to software, writing, sales, and business planning alike, with the evals themselves refined over time when a result passes but still feels wrong.
The author compares newer decision models against LLM judges for automated evals on a leasing-agent failure task. The main takeaway is that the default 0.50 threshold often leaves performance on the table, and that the scores need tuning and calibration before you trust them.
- The default 0.50 decision threshold can miss nearly all failures; tuning it per model, using F1 with an 80% precision/recall floor, improves results.
- OpenAI Decisions had the best separation (F1 about 0.87) but cost about 4.5 times more than Perplexity Decider (F1 about 0.79) per 100 eval runs.
- Raw model probabilities are often poorly calibrated; Platt scaling helped Perplexity, Liquid d1, and Clef Flash, but made OpenAI's slightly worse.
@wulfie_bain_ on X
An OpenAI Startups applied AI lead argues that prompts in even the best AI companies grow contradictory over time because teams only add lines and never remove them. He proposes treating prompts like code and product decisions, using a modular, MECE structure (Background, Behaviour, Output).
- Prompts evolve accretively, so contradictions and ambiguity pile up. Conditional prompts spread across files make the whole thing hard for anyone to review end to end.
- He reports results from prompt rewrites: one startup cut agent speed-ups of about 50%, another saw 40% higher 7-day retention, and another cut costs by 30%.
- Smarter models don't fix ambiguity. Someone still has to specify what the product should do, and evals force that clarity.
Claire Gouze lays out a practical process to build and test a context layer your data agents can trust. She shows how to start small with dbt docs and markdown files, write evals, clean up data models, and iterate until you reach around 90% reliability.
- Adding query logs and profiling alone got the agent stuck around 40% accuracy; reworking the data model and documentation pushed it to 90%.
- Treat context like code: store it in files/markdown, version in Git, add CI/CD evals, and iterate on failure cases rather than plugging agents directly into raw databases.
- Start small—10 to 20 core tables with existing dbt docs—rather than trying to cover the whole warehouse at once.
- Evals should come from real BI query history or autogenerated question sets, reviewed by teams and tracked over time to build trust in the agent.