Click any tag below to further narrow down your results
Links
Dylan Martin, a PostHog engineer, explains why he left the feature flags team for AI product work and what that shift has taught him, mainly about writing evals and deciding where agents should act versus where humans should see and decide. It's a short personal account of the transition rather than a technical deep dive.
- Much of the skill set carries over: systems thinking, failure modes, and proving correctness still matter, so the move felt smaller than the job postings suggested.
- Evals are the main new skill. Unlike unit tests, they need to accept several good answers to the same request and catch subtle regressions, like counting signup events instead of unique users.
- The hardest open question is interface design: when MCP and skills should replace clicking through a UI, and when a human needs visual previews to make the call, such as seeing a flag's blast radius.
PostHog shares concrete lessons from shipping 10 AI products used by 100,000 monthly users, covering when to build AI, how to structure agents, and what fails in practice. The core insight: most AI failures are about reliability and context, not capability, and you need to rethink product architecture for agents rather than bolt them on.
- Most AI product failures come from poor reliability, unclear limits, and insufficient context—not lack of capability. PostHog's install wizard improved activation 2x and conversion 5x because it automated a 2-hour task, not because it was technically impressive.
- Agents need complete tool coverage and flexible abstractions to work. Forcing agents through preset tools or leaving out endpoint coverage (like missing experiment-start when experiment-create exists) makes them abandon tasks and ask users to finish manually.
- Context matters more than reasoning power. PostHog improved wizard effectiveness more by giving agents better context than by adding multi-step planning, subagents, or better models—and model upgrades bulldoze your assumptions anyway, so don't over-invest in harnesses.
- Build shared infrastructure and let product teams own agent features. PostHog's MCP server has 5x more daily users than their AI product because it's easier to maintain and lets teams integrate agents into their own products rather than using a monolithic assistant.