38 things we've learned about building AI products
Short summary
PostHog shares concrete lessons from shipping 10 AI products used by 100,000 monthly users, covering when to build AI, how to structure agents, and what fails in practice. The core insight: most AI failures are about reliability and context, not capability, and you need to rethink product architecture for agents rather than bolt them on.
- Most AI product failures come from poor reliability, unclear limits, and insufficient context—not lack of capability. PostHog's install wizard improved activation 2x and conversion 5x because it automated a 2-hour task, not because it was technically impressive.
- Agents need complete tool coverage and flexible abstractions to work. Forcing agents through preset tools or leaving out endpoint coverage (like missing experiment-start when experiment-create exists) makes them abandon tasks and ask users to finish manually.
- Context matters more than reasoning power. PostHog improved wizard effectiveness more by giving agents better context than by adding multi-step planning, subagents, or better models—and model upgrades bulldoze your assumptions anyway, so don't over-invest in harnesses.
- Build shared infrastructure and let product teams own agent features. PostHog's MCP server has 5x more daily users than their AI product because it's easier to maintain and lets teams integrate agents into their own products rather than using a monolithic assistant.
Full summary
PostHog built ten AI products used by nearly 100,000 people monthly, and they've learned that most AI features shouldn't exist. The core insight: AI makes things worse when it's slow, unreliable, or solves problems nobody has. Instead of asking "how do we add AI?" they ask "what's the agent-first version of this?" The real wins come from targeting specific jobs—tasks taking 30+ seconds, using unfamiliar interfaces like SQL, or repeated 20+ times. Their install wizard does two hours of work in eight minutes and drives paid conversion 5x higher. The actual moat isn't the AI itself but your product's context: functionality, user data, and built tools. Their MCP server gets 5x more daily usage than their main AI product, suggesting people often prefer purpose-built infrastructure over general assistants.
The practical mechanics matter more than model choice. PostHog AI went through three different LLM harnesses in a year—the harness itself isn't where you innovate. What matters is letting agents run actual code (sandboxed SQL, Python scripts, even PRs), building shared infrastructure primitives, and treating every docs-search call as evidence of a missing skill or tool. They logged 28,000 such calls monthly, each one a signal that agents need better recipes. Model upgrades bulldoze your assumptions constantly—cheap reasoning and reliable tool use changed their entire architecture. The wizard costs $6.67 per run, but improved activation justifies it. Subagents drain resources without shared cache benefits, and many agent failures are actually context failures, not reasoning failures. Giving agents more context improved effectiveness more than multi-step planning or better models.
Reliability beats capability—early complaints weren't about what PostHog AI couldn't do but inconsistent performance and unclear limits. They set up observability from day one, traced every LLM call with inputs, outputs, latency, and cost. Agents work better meeting them at their abstraction level: instead of forcing preset query tools, they let agents write SQL directly. Transparency helps—agents can read your codebase, handbook, and docs. But watch for prompt injection from public inputs. Use evals starting small: manually read traces, find failures, build targeted checks. Example code beats documentation for teaching agents. They ship fast enough that context drifts, so they run agents checking merged PRs for doc updates and auto-filing pull requests. Blank text boxes terrify users, so add suggestions and nudges. Show every step of reasoning instead of hiding it.