Define quality, build representative datasets, and combine multiple evaluation methods.
◷ Estimated reading time: 7 min
Define Quality Before Choosing a Metric
"Good" is too vague to evaluate. Translate product intent into observable dimensions such as correctness, groundedness, task completion, policy compliance, latency, cost, and user satisfaction. Weight them according to the risk of the use case: a creative-writing assistant and a payment agent should not share the same release bar.
Offline and Online Evaluation
Offline evaluation
Runs before release on a curated dataset
Fast, repeatable, and useful for regression gates
Cannot fully reproduce real users or production dependencies
Online evaluation
Uses production signals, experiments, and feedback
Reveals real behavior and distribution shift
Requires privacy, safety, and rollback controls
Golden Datasets and Regression Suites
A useful evaluation set represents common tasks, high-risk cases, edge cases, and known historical failures. Keep inputs, expected properties, rubrics, and relevant context versioned. Whenever production reveals a meaningful failure, add a sanitized version to the regression suite.
Use a Portfolio of Evaluators
Deterministic checks: schemas, exact values, permissions, citations, and tool arguments.
Human review: nuanced quality, domain correctness, and policy interpretation.
Model-based graders: scalable rubric scoring or pairwise comparison, calibrated against human labels.
Production signals: completion, escalation, correction, abandonment, latency, and cost.
Common mistake
An LLM judge is not ground truth. It can be biased by verbosity, ordering, or style. Validate the grader on representative human-labeled examples and keep deterministic checks for facts the application can verify directly.
Key Takeaways
Quality begins with explicit, use-case-specific dimensions.