Skip to workspace

Evaluation Lab

Measure the behavior you need. Inspect failure slices and keep critical policy failures visible.

Classification metrics

Enter confusion-matrix counts. Undefined metrics remain undefined rather than becoming misleading zeros.

MetricValue
precision80.0%
recall66.7%
f172.7%
accuracy80.0%

Release decision exercise

Quality gains are necessary, but critical failures have their own gate.

Hold release

expert review pending

Illustrative decision logic, not a live release approval or deployment.

Evaluate an agent at every boundary

StageCheckEvidence
IntentCorrect metric, population and time rangeCanonical task contract
RetrievalRelevant, fresh, authorized sourcesSource IDs, versions, recall@k
Tool useValid arguments and resource authorityTool schema, policy decision, trace
ResultCorrect values and safe aggregation grainReference query, fixture, tolerance
AnswerClaims supported; uncertainty visibleClaim-to-source mapping
PublicationIntended recipient and verified effectApproval plus target-side receipt
Retrieval and regression metrics

Recall@k = relevant documents retrieved in top k / all relevant documents. Precision@k = relevant results in top k / k. Evaluate reranking independently from generation. For regression, MAE averages absolute errors; MSE emphasizes larger deviations. Always inspect error distributions and important slices.

Calibrate human and AI judges

Use independent reviewers, anchored examples, blind output identities and counterbalanced order. Keep ties, abstentions and critical failures. Measure agreement by dimension; adjudicate contested cases. Test an AI judge against human labels and position/verbosity perturbations before trusting it.

Do not treat repeated ratings of the same case as independent model-performance samples. For a 12-case pilot, report uncertainty and use the results to refine the rubric rather than claiming a robust quality gain.

Bring this lab to your teamOriginal teaching examples · No production training claims