The Problem With Green Checkmarks on Broken Conversations

A regression suite runs overnight. In the morning, the dashboard is green. The release ships. Two days later, a customer posts a screenshot of a conversation that fell apart halfway through — the agent froze, looped, or handed off to nowhere. Someone goes back to check the test run. It passed. The scores were fine. Nothing about the report said this was coming.

This isn't a rare edge case. It's the predictable result of how most conversational AI testing tools score a run, and once you see the mechanism, you can't unsee it in every "green" dashboard that's ever surprised you.

Scoring what happened, not what was supposed to happen

Most evaluation tools work the same way: run the conversation, collect whatever transcript comes out, score it. That sounds reasonable until you ask what happens when the conversation doesn't finish. The agent times out at turn 6 of a planned 14. The session drops. A tool call throws and the conversation dead-ends. The scorer doesn't know the plan was 14 turns — it only sees the 6 that happened, and it scores exactly those.

If those 6 turns were polite, on-topic, and grounded, they'll score well. The report says PASS. Nobody lied. The tool did precisely what it was built to do: score the transcript it was given. The problem is that transcript was never the whole story.

Why the truncated run often scores better, not worse

Here's the part that makes this genuinely dangerous rather than just an edge case: a conversation that stops early frequently scores higher than one that runs to completion, not lower. A full 14-turn conversation about a damage claim has 14 chances to say something slightly off-policy, misread an amount, or handle an edge case clumsily. A conversation that dies at turn 6 has only had 6 chances. Fewer turns means fewer opportunities to violate anything — so the average score often looks cleaner, not worse. The bot didn't behave better. It just didn't get far enough to behave worse.

A scoring model that doesn't distinguish "this conversation succeeded cleanly" from "this conversation never got the chance to fail" will systematically reward the second case. That's not a tuning problem you fix with a stricter threshold. It's a structural gap in what's being measured.

Completion is a gate, not a metric

The fix isn't a smarter quality score. It's a different question, asked first: did this conversation actually complete? Not "did it look fine while it lasted" — did it reach the goal, or terminate in a real resolution state, rather than stalling, erroring, or timing out partway through.

That question has to be answered before quality scoring counts for anything. If the answer is no, the verdict is FAIL — full stop, regardless of how clean the partial transcript looked. Execution integrity is evaluated ahead of quality, not folded into the same average.

This doesn't mean throwing away the partial score. It stays visible, attached to the run, because it's genuinely useful for diagnosis — it tells you exactly how the agent was behaving in the turns it did complete, which is often the fastest way to find where things started going wrong. What it can't do is stand in for a passing verdict. A broken conversation should never look like a passing one, even when the broken part was well-behaved.

What to check in your own pipeline

If you're running any kind of automated evaluation on a conversational agent, it's worth asking directly: does a truncated, timed-out, or errored run get scored as if it were a complete one? If the answer is yes — even implicitly, because nothing in the pipeline explicitly checks for completion before scoring — there's a real chance your "green" dashboard has been quietly rewarding conversations for not getting far enough to fail.

See it against your own agent

Shyena treats execution integrity as a hard gate, checked before any quality score is allowed to count. A conversation that didn't complete is capped at FAIL, with the raw score preserved alongside it for diagnosis — never presented as a pass. If you want to see what your current pipeline would say about a run like this, we'll run one against your live agent and show you.