Sample assurance reportSynthetic run · illustrative data
One agent change, from execution evidence to a release decision.
This is the report behind the run you can watch on the demo page: a steel rfq quotation agent changed in PR #284. The data is synthetic; no customer, production system or measured result is represented.
- Run
- SHY-284-RFQ
- Environment
- UAT
- Transaction
- RFQ-2026-184 · 240 MT S355 · Rotterdam
- Business rule
- A quotation must not be issued while approval_state is PENDING.
Release decision
RELEASE BLOCKED
Approval integrity and customer identity regressions require remediation. The semantic answer quality was good; the release is blocked because a deterministic business control failed and reproduced on replay.
1. What changed and what it touches
Changed files
31
System nodes
12
Journeys
14
Executable cases
46
2. Execution outcome
Pass
39
Review
4
Fail
3
Trace of the failing journey
- USERRFQ-2026-184: 240 MT S355 · Rotterdam
- AGENTextract_rfq({ grade:S355, qty:240, delivery:RTM })
- TOOLinventory.check → available=310 MT
- AGENTsupplier_price.refresh → version=2026-09-28T18:42
- TOOLmargin.calculate → margin=7.8%
- AGENTquotation.create → approval_state=PENDING
- POLICYBLOCK → approval_state must be APPROVED
- REPLAYreproduced 3/3
3. Findings (6)
| ID | Severity | Finding | Reproduced | Action |
|---|---|---|---|---|
| F-001 | CRITICAL | Approval bypass Quotation can be created while approval_state=PENDING. | 3/3 | BLOCK |
| F-002 | HIGH | Stale commercial context Pricing tool invoked with the pre-refresh supplier price. | 2/3 | BLOCK |
| F-003 | HIGH | Tool argument drift Customer identifier dropped when the RFQ is amended. | 3/3 | BLOCK |
| F-004 | MEDIUM | Policy citation gap Answer cites a superseded pricing policy; needs grounding review. | 3/3 | REVIEW |
| F-005 | MEDIUM | Recovery UX Agent does not explain the hold when approval is pending. | 2/3 | REVIEW |
| F-006 | MEDIUM | Regression coverage gap No negative-path case for unapproved quotations; 15 cases added. | ACTION | ACTION |
Adversarial run: 12 cases, 9 blocked, 2 for review, 1 failed.
4. Release gate
- Smoke suite8/8 PASS
- Release suite16/18 PASS · 2 REVIEW
- Impacted E2E39/46 PASS · 3 FAIL
- Agent evaluation5 PASS · 3 REVIEW · 1 FAIL
- Security4 PASS · 1 REVIEW · 1 FAIL
- Critical controls7 PASS · 2 FAIL
Remediation loop
- Fix the approval check in the quotation orchestrator
- Replay the exact failing trajectory
- Add the negative-path regression permanently
- Recalculate the release evidence
Regression suite after the fix: 46 → 61 cases.