loading…
Why one successful demo cannot prove that a probabilistic system is dependable.
A traditional software is mostly judged against explicit rules: given the same input and state, the same code path should produce the same result. A GenAI system contains a probabilistic component, so quality must be understood across many inputs, repeated runs, users, and operating conditions.
A polished demo proves only that one path worked once. The same system may fail when the wording changes, retrieved documents conflict, a tool returns an unusual payload, or the model receives a longer conversation. Reliability is therefore a property of a distribution of behavior, not a screenshot.
| Layer | Example failure |
|---|---|
| Model output | A fluent but incorrect answer |
| Retrieval | The correct document was never retrieved |
| Workflow | A required validation step was skipped |
| Agent trajectory | The final answer is acceptable, but the agent used an unauthorized tool |
| Operations | Latency or cost grows until the feature is unusable |
Reliable teams define acceptable behavior, build representative test sets, run repeatable evaluations, enforce runtime controls, and observe production outcomes. Chapter 4 treats evaluation, security, guardrails, observability, and governance as one connected system rather than separate checklists.
Do not ask only, "Did the answer look right?" Ask, "Did the system use the right evidence, follow the permitted path, stay within budget, and produce an acceptable outcome?"