The article explores how large language models (LLMs) act as judges in evaluating other LLMs. It examines potential biases, the impact of model identity on outcomes, and differences in performance between "fast" and "thinking" tiers across various tasks. Experiments reveal insights into self-preference among judges and how hinting can influence their decisions.

This article discusses how fine-tuning open-source LLM judges using Direct Preference Optimization (DPO) can lead to performance that matches or exceeds GPT-5.2 in evaluating model outputs. The authors trained models like GPT-OSS 120B and Qwen 3 235B on human preference data, achieving better accuracy and efficiency at a lower cost.

This article outlines the LLM-as-judge evaluation method, which uses AI to assess the quality of AI outputs. It discusses its advantages, limitations, and offers best practices for effective implementation based on recent research and practical experiences.

LLM-SRBench is a new benchmark aimed at enhancing scientific equation discovery using large language models, featuring comprehensive evaluation methods and open-source implementation. It includes a structured setup guide for running and contributing new search methods, as well as the necessary configurations for various datasets. The benchmark has been recognized for its significance, being selected for oral presentation at ICML 2025.

The article evaluates various language models (LLMs) to determine which one generates the most effective SQL queries. It compares the performance of these models based on their accuracy, efficiency, and ease of use in writing SQL code. The findings aim to guide users in selecting the best LLM for their SQL-related tasks.

Evaluating large language model (LLM) systems is complex due to their probabilistic nature, necessitating specialized evaluation techniques called 'evals.' These evals are crucial for establishing performance standards, ensuring consistent outputs, providing insights for improvement, and enabling regression testing throughout the development lifecycle. Pre-deployment evaluations focus on benchmarking and preventing performance regressions, highlighting the importance of creating robust ground truth datasets and selecting appropriate evaluation metrics tailored to specific use cases.

LLMs are being developed to generate CAD models for simple 3D mechanical parts, leveraging techniques like OpenSCAD for programmatic CAD design. Initial tests show promising results, with evaluations revealing that LLMs have recently improved their capabilities in generating accurate solid models and understanding mechanical design principles. A GitHub repository is available for further exploration of the evaluation processes and tasks involved.

ZeroSumEval is a framework designed for evaluating large language models (LLMs) through competitive games, dynamically scaling in difficulty as models improve. It features multi-agent simulations with clear win conditions to assess various capabilities such as knowledge, reasoning, and planning, while enabling easy extension for new games and integration with optimization tools. The framework supports multiple games including chess, poker, and math quizzes, and provides comprehensive logging and analysis tools for performance evaluation.