01 · Mental model
Evaluate the behavior, controls, and operating envelope
Deterministic tests validate schemas, permissions, required fields, and known invariants. Human annotation captures expert judgment and ambiguous quality. Model-based evaluators scale rubric application but must be calibrated. Adversarial datasets test injection, overreach, leakage, and unsafe persistence. Operational gates cover latency, cost, error rate, observability, ownership, and rollback.
The release decision belongs to accountable humans and policy, supported by versioned datasets, evaluator definitions, thresholds, exceptions, and evidence from the exact candidate being promoted.
02 · Visual explanation
03 · Compare and decide
Use complementary evidence
| Decision lens | Evaluation method | Best use |
|---|---|---|
| Deterministic | Schema, exact match, policy, tool and calculation checks | Fast gates with reproducible pass/fail |
| Human | Nuance, impact, usefulness, and expert acceptance | Gold labels, calibration, consequential review |
| LLM-as-judge | Rubric-based semantic comparison at scale | Regression detection after calibration |
| Adversarial | Injection, leakage, overreach, denial, and unsafe persistence | Security behavior and safe failure |
04 · Cybersecurity example
Release gate for the review agent
A new model and skill version claim better architecture findings.
Run the frozen representative dataset.
Compare quality, security, cost, and latency to baseline.
Review regressions and adversarial failures.
Promote only the exact candidate with rollback to the previous version.
Outcome: The upgrade decision is based on repeatable evidence instead of a stronger-looking demonstration.
05 · What to remember
The 60-second recall
Freeze representative datasets and version every evaluator.
Calibrate model-based judges against expert labels.
Release readiness includes operations, security, ownership, and rollback—not only answer quality.
Teach-back prompt: Explain this concept to a teammate using the diagram, then name one failure mode and the control that stops it.
06 · Questions people ask
FAQ
07 · Primary sources