At re:Invent 2025, AWS introduced the DevOps Agent, designed to resolve and prevent incidents while enhancing system reliability and performance. The development team focused on creating a robust incident response system capable of accurate root cause analysis for AWS applications. The agent employs a multi-agent architecture with a lead agent coordinating tasks and specialized sub-agents that manage specific investigations. For instance, when dealing with extensive log data, a sub-agent filters out irrelevant information, allowing the lead agent to concentrate on significant findings. Transitioning from a prototype using large language models (LLMs) to a reliable product involves several key mechanisms. The team identified five essential components: evaluations (or evals) to pinpoint failures and establish quality baselines, a visualization tool for debugging, a rapid feedback loop for local scenario testing, intentional changes to avoid bias, and regular reviews of production samples for understanding customer experiences. Evals, akin to test suites in traditional software development, are crucial for building confidence in the agent’s performance. They involve creating realistic scenarios that the agent navigates, assessing whether it can correctly identify root causes and provide accurate outputs. Creating effective eval scenarios presents challenges, particularly in authoring realistic faults and maintaining quick feedback loops. The team recognized that developing a high-fidelity microservice environment is resource-intensive, so they focused on reusing a few foundational environments while layering various fault scenarios on top. Slow feedback loops hinder the testing process, as lengthy deployment times can lead developers to release changes prematurely, risking regressions. This highlights the importance of efficient testing processes in product development to ensure high-quality outcomes.