BENCHMARK · LIVE RUN 2026-09-26
Every run, every candidate, one fixed judge.
Bars show the share of eval cases each policy passed. A candidate ships only at 100% with nothing uncertain.
- Active policy
- Rejected
- Promoted
- Screened
hash 3b7e04 · active during order #1
Passes 2 of 4 cases before any fix: it has no answer for a timeout after saving.
rules · candidate
Fails order.transient — 0 orders; that timeout needed a retry.
guardrails · promoted as v2 · hash 9f2c1a
Passes every case, including the one where the lookup is down too.
context · screened
Named the incident's customer and SKU (c_2041, OAT-1L), so it never ran.
hash 5d0e93 · 5 refund cases added
Passes 7 of 9 before any refund runs.
tools · candidate
Fails refund.partial_x2 — blocked a second, legitimate refund.
guardrails · promoted as v4 · hash c81d77
Passes all 9 cases. Shipped before the first refund call.