Using large language models (LLMs) as judges raises important questions about bias and decision-making. The authors designed a structured evaluation pipeline to explore these issues specifically, focusing on how judges might favor their own models or display biases based on task type and model identity. They conducted experiments using the MT-Bench benchmark, which consists of 80 questions across various categories, to analyze the performance of six models from three vendors: Claude, GPT, and Gemini. Each vendor contributed both a “fast” and a “thinking” tier, allowing for a detailed comparison of how model characteristics influence judging outcomes. In their first experiment, they conducted blind evaluations where judges ranked anonymized answers. The results showed that GPT judges exhibited a significant self-preference bias, selecting their own model 80% of the time, while Gemini judges were more impartial. Claude performed well overall, receiving high rankings from both Claude and Gemini judges. The second experiment expanded the scope by analyzing how bias varied across different task types. They found that self-preference rates differed by category, with GPT judges consistently favoring their own answers more than non-GPT judges. The evaluation framework included scripts for generating answers, judging, and analyzing results. The authors also created utilities for regenerating answers and retrying failed judgments, ensuring robustness in their findings. Their structured approach allows for a reproducible analysis of LLM behavior, providing insights into how different models perform under controlled conditions. Overall, the experiments highlight substantial biases in LLM judges, particularly in how they assess outputs based on their affiliations and the type of tasks presented.