To build effective product evaluations, follow three key steps: label a small dataset, align LLM evaluators, and run experiments while adjusting configurations. Start by sampling input and output from LLM requests and labeling them based on evaluation criteria like faithfulness and relevance. Use a simple spreadsheet format with clear binary labels (pass/fail) for objective criteria and win/lose/tie comparisons for subjective ones. Avoid numeric labels or Likert scales, as they complicate consistency among human annotators and LLM evaluators. A balanced dataset is critical, aiming for at least 50-100 failure cases out of 200+ total samples, as these “fails” are vital for understanding trust issues in the model outputs. Generating fail cases can be done by using simpler models that produce organic failures or leveraging active learning to prioritize human annotation on likely failures. Once the samples are labeled, create a prompt template to label new inputs and outputs. It’s essential to split the dataset into training and testing sets to avoid overfitting and ensure generalization. Each evaluator should focus on one dimension. Avoid creating a single evaluator that tries to assess multiple dimensions simultaneously, as this complicates calibration and performance analysis. Instead, use separate evaluators for different criteria and combine results through heuristics. When evaluating outputs, consider position bias by running evaluations with swapped orders. Consistency is key; if an evaluator flips its judgment, mark these as ties rather than forcing a decision. Measure evaluator performance using precision, recall, and Cohen’s Kappa. Human performance is the benchmark, but remember that human annotators often struggle with consistency and accuracy. The real advantage of LLM evaluators lies in their scalability, enabling quick, consistent assessments across many samples. Finally, integrate all evaluators into an evaluation harness that processes input-output pairs and aggregates results, making it easy to track performance and identify improvements.