Evaluation Lab
Measure the behavior you need. Inspect failure slices and keep critical policy failures visible.
Classification metrics
Enter confusion-matrix counts. Undefined metrics remain undefined rather than becoming misleading zeros.
| Metric | Value |
|---|---|
| precision | 80.0% |
| recall | 66.7% |
| f1 | 72.7% |
| accuracy | 80.0% |
Release decision exercise
Quality gains are necessary, but critical failures have their own gate.
Hold release
expert review pending
Illustrative decision logic, not a live release approval or deployment.
Evaluate an agent at every boundary
| Stage | Check | Evidence |
|---|---|---|
| Intent | Correct metric, population and time range | Canonical task contract |
| Retrieval | Relevant, fresh, authorized sources | Source IDs, versions, recall@k |
| Tool use | Valid arguments and resource authority | Tool schema, policy decision, trace |
| Result | Correct values and safe aggregation grain | Reference query, fixture, tolerance |
| Answer | Claims supported; uncertainty visible | Claim-to-source mapping |
| Publication | Intended recipient and verified effect | Approval plus target-side receipt |
Retrieval and regression metrics
Recall@k = relevant documents retrieved in top k / all relevant documents. Precision@k = relevant results in top k / k. Evaluate reranking independently from generation. For regression, MAE averages absolute errors; MSE emphasizes larger deviations. Always inspect error distributions and important slices.
Calibrate human and AI judges
Use independent reviewers, anchored examples, blind output identities and counterbalanced order. Keep ties, abstentions and critical failures. Measure agreement by dimension; adjudicate contested cases. Test an AI judge against human labels and position/verbosity perturbations before trusting it.
Do not treat repeated ratings of the same case as independent model-performance samples. For a 12-case pilot, report uncertainty and use the results to refine the rubric rather than claiming a robust quality gain.