LLM-as-judge is gaining traction as a method for evaluating AI outputs, especially as development teams rush to deploy AI agents. As of March 2025, 40% of AI projects are already in production. Traditional data quality monitoring isn't effective for the non-deterministic nature of AI, which can produce varied outputs from the same inputs. LLM-as-judge aims to address this by using one AI model to assess the performance of another, focusing on subjective metrics like relevance and helpfulness. However, this approach isn't without its challenges, particularly when both systems can generate inaccurate outputs. The article emphasizes that while LLM-as-judge can be a powerful tool, it should be used judiciously. It’s effective for monitoring complex outputs where traditional methods, such as ROUGE or BLEU scores, fall short. These conventional metrics often fail to capture the nuanced quality of AI-generated text. There are scenarios where deterministic methods are more suitable—like ensuring outputs adhere to specific formats or simple conditions. For example, a pharmaceutical company may require outputs in a valid postal code format, which is best handled by traditional monitoring. Real-world applications illustrate the benefits and pitfalls of LLM-as-judge. A case from Monte Carlo highlights how this method identified a reliability issue in their Monitoring Agent, catching a recommendation that could have gone unnoticed. The piece also outlines best practices for implementing LLM-as-judge to avoid costly mistakes. Missteps in this evaluation method can waste resources and erode trust in AI systems, hindering innovation. The article stresses that while LLM-as-judge is not foolproof, it can effectively detect declines in output quality when monitored over time.