Moving into AI Product Engineering

Open original

Dylan Martin, a PostHog engineer, explains why he left the feature flags team for AI product work and what that shift has taught him, mainly about writing evals and deciding where agents should act versus where humans should see and decide. It's a short personal account of the transition rather than a technical deep dive.

  • Much of the skill set carries over: systems thinking, failure modes, and proving correctness still matter, so the move felt smaller than the job postings suggested.
  • Evals are the main new skill. Unlike unit tests, they need to accept several good answers to the same request and catch subtle regressions, like counting signup events instead of unique users.
  • The hardest open question is interface design: when MCP and skills should replace clicking through a UI, and when a human needs visual previews to make the call, such as seeing a flag's blast radius.

Dylan Martin left a traditional product engineering role leading PostHog's feature flags team to work on AI product engineering. His view is that the move feels smaller than job postings imply, since software engineering is already shifting under everyone and systems thinking (failure modes, proving correctness) matters in any engineering role. He had already spent time using coding agents on the flags team, learning where they help and where they break, so the switch was about building products that deliver those productivity gains without the traps he had to climb out of himself. He wanted to work on what he calls "frontier-shaped" problems, interfaces where the industry hasn't settled on what good looks like yet. The biggest new skill he points to is writing evals. Traditional tests assume deterministic output, but LLM responses vary, and several answers can be correct. His example is an analytics agent asked "How many users signed up yesterday?" An eval should catch a regression where the tool starts counting signup events instead of unique users when the same question is broken down by country. His advice is to learn evals alongside building the product, and he points readers to a teammate's post on benchmarking agents. The hardest problem for him right now is UI design, specifically how to split work between a human-facing interface and the same capabilities exposed through MCP (Model Context Protocol) and skills for agents. His working rule is that when someone already knows the outcome they want, an agent should handle the steps, which puts more weight on tools to pick the right action and avoid double-applying a change on retry. When a human needs to make a judgment call, such as rolling out a feature flag and checking its blast radius, or seeing which users an error affects, he thinks a visual preview beats text, because people catch things in pictures that they would miss in paragraphs. He admits he has no settled answer on how much agents should do on their own versus how much humans should see and decide, and that "human in the loop" leaves open where the human actually belongs.

Ask about this article