AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance

The core problem, as this InfoQ webinar panel frames it, is that AI systems fail in a different way from ordinary software. They rarely crash. They produce confident wrong answers. Lan Chu described a financial compliance agent that blocked prohibited investment advice using vector similarity thresholds. An unannounced background update to the embedding model shifted the distance distribution, causing spikes in both false positives and false negatives in live traffic, while every infrastructure health check stayed green. Zichuan Xiong pointed to a quieter version of the same problem he calls semantic drift: text-to-SQL agents that flatten domain terms, treating "member" the same way in sales and customer support, producing analytics errors that pass standard tracing. Both examples share a lesson. Monitoring uptime tells you almost nothing about correctness. On the code side, the panelists agreed that human line-by-line review cannot keep pace with coding agents. Prem Chandrasekaran of ThoughtWorks argued for automating style and syntax checks in CI/CD and reserving human attention for architecture, business intent, and non-functional requirements. He also called for a return to mutation testing, deliberately injecting bugs to see whether a test suite actually checks logic or just hits coverage numbers. Xiong added that intent should be evaluated at the harness level, before a pull request even exists. Chu's position on evals was that they belong in production as living infrastructure: online span-level tracing, LLM-assisted categorization of failure patterns, and a pipeline that feeds real edge cases back into offline benchmarks. Security and privacy got the most pointed warnings. Chandrasekaran noted that RAG pipelines can retrieve documents users should never see, and Katharine Jarmul said models memorize training data, which can enable de-anonymization or leak API keys. Jarmul was blunt that prompt engineering is not a defense; she recommended red teaming, continuous threat modeling, and strict functional scopes for agents. Chandrasekaran's practical advice was to treat each agent like a new engineering hire: give it its own non-human identity, separate read-only permissions from reversible writes, pin versions of third-party MCP dependencies, and keep secrets out of the agent's runtime entirely. He also referenced the GitSpawn vulnerability, where a malicious repository configuration ran unauthorized code during routine commands. The panel's final point was organizational rather than technical: engineering and security can validate guardrails, but whether to ship is a business call.