Automated AI Evaluation. Trusted Every Time.
Shyena is built to evaluate any AI system before it ships — live today for conversational and voice AI. It runs real, agent-driven conversations against your live bot, judges the quality of every turn with LLM-based evaluation and deterministic checks, and makes it structurally impossible for a broken conversation to report a green pass.
Automated
End-to-end automated AI evaluation at scale.
Accurate
Precise, consistent and unbiased results.
Trusted
Reliable insights. Confident decisions.
Evaluation — live today
Security — live today, via Ziran
The problem
Conversational agents don't fail like normal software.
Non-determinism breaks your test signal
LLM-driven agents answer differently every run. The same scenario passes on Tuesday and fails on Wednesday, so teams stop trusting the suite and start ignoring red builds.
Manual QA can't cover open-ended dialogue
A realistic agent has thousands of viable conversation paths. Reviewing transcripts by hand covers a handful per release, and the ones you skip are the ones that reach customers.
Test tools assume scripted click-paths
Traditional automation asserts that a selector exists. It cannot judge whether the agent handled an angry renewal request correctly, stayed in policy, or actually resolved the issue.
31
metrics evaluated on every case, by default
117
metrics in the full catalog, including custom ones you define
0
false green passes — the gate structurally prevents it
Default depth for a standard case; accessibility-gated runs evaluate more.
Where Shyena fits
Not a prompt tester. Not an observability tool.
Tools like Promptfoo and DeepEval test a single prompt. Arize Phoenix watches what already happened in production. Botium scripts a chatbot's expected path. Shyena is the only one that executes a full live conversation, judges it on semantics and orchestration as well as wording, and refuses to let a broken run report a pass.
They're not mutually exclusive — teams often unit-test prompts with tools like these before Shyena runs the full conversation as the release gate none of them cover.
How it works
One run. 31 metrics. A verdict you can defend.
Every regression run follows the same four stages, and each stage produces evidence the next one is allowed to trust — from a single persona definition to a case scored against 31 metrics by default, spanning LLM-judged quality, deterministic assertions, semantic state-transition validity, and orchestrator-level decision analysis.
Define agentic test personas
Describe a goal, a persona and a playbook — not brittle scripted steps. Shyena improvises like a real customer would.
Execute real conversations
A real browser or API session drives your live agent end to end, across chat and voice, with retries and backpressure built in.
Evaluate against 31 metrics
LLM-as-judge scoring, deterministic assertions, six-construct semantic assurance, and orchestrator-level decision analysis — 31 metrics evaluated by default, from your quality pillars down to whether the agent dispatched the right tool call.
Gated, trustworthy verdicts
If the execution didn't complete, the verdict is capped at FAIL regardless of score. No false green passes, ever.
Platform
Everything a release gate for conversational AI needs.
Built first for conversational and voice AI. RAG evaluation is next — the judge model already includes five RAG-specific quality dimensions.
Agentic Test Personas
Model the customers who actually call you: confused, impatient, multilingual, off-script. Each persona pursues a goal instead of replaying a transcript.
Real Conversation Execution
Runs against your live conversational AI platform through the same surface your customers use — no mocks, no simulated backends.
LLM-as-Judge Metrics
Turn-level scoring for grounding, resolution, tone, policy adherence and escalation quality, with the reasoning stored alongside each score.
Deterministic Assertions
Hard checks for the things that must never be fuzzy: refund amounts, disclosure text, redaction, handoff targets and latency budgets.
Semantic Assurance
Six-construct state-transition validity model — intent integrity, context memory, dialogue state correctness, business compliance, tool decisions, and recovery — with a causal root-cause taxonomy behind every violation.
Orchestrator Quality
Scores the agent's internal decisions, not just its replies: correct tool/route dispatch, missed invocations, and decision oscillation across a conversation — weighted and traceable to the exact turn.
Execution-Integrity Gate
Incomplete, timed-out or errored runs can never be scored into a pass. Integrity is evaluated before quality, not after.
Automated Bug Report Generation
Every FAIL gets an LLM-generated root-cause report — a 5-Whys chain, severity, and duplicate detection — rendered as Jira-ready markdown, automatically, no manual write-up required.
Custom Metrics
Extend the 31-metric default catalog with your own — subclass a documented SDK, register it, and it runs alongside the built-ins with the same exception isolation and latency tracking.
Full Audit Trail
Every prompt, judge call, assertion and retry is recorded and replayable, so a verdict can be defended in a release review or an audit.
A broken conversation should never look like a passing one.
Most tools score whatever transcript they collected. If the agent stalled at turn 17, they grade the first sixteen turns and call it green. Shyena evaluates execution integrity first.
turn 17 · session terminated before goal resolution
quality score 0.81 · integrity check FAILED
verdict capped → FAIL (execution incomplete)
Scored honestly and capped. The team sees exactly which turn broke, with the full judge reasoning attached.
16 turns collected · no assertion errors raised
average score 0.81 → threshold 0.75
verdict → PASS
A false green: nothing crashed loudly, so the run reports healthy — and the regression reaches production.
Automated
No manual effort, continuous evaluation.
Measurable
Quantitative metrics that matter.
Reliable
Consistent, repeatable and unbiased.
Explainable
Transparent results with clear evidence.
Actionable
Insights that drive real improvement.
Continuous
Always learning, always improving.
See it evaluate your own agent
Bring one real scenario. We'll run it against your live conversational AI agent and walk through every judged turn with you.