Why Conversational AI Needs a Different Testing Model

A conversational agent is not a UI script with a larger input field. Its next action depends on context, previous turns, orchestration decisions, tools, retrieval and the user's response. That makes a frozen transcript a weak oracle for production assurance.

The transcript problem

A traditional test often looks like this:

USER: I need a refund
AGENT: Please provide your order number
USER: 12345
AGENT: Your refund has been issued
PASS

The test is easy to replay, but it encodes one exact path. If the agent instead verifies the customer's identity first, retrieves the order, and then issues the refund, the test may fail even though the journey is correct.

The reverse is more dangerous: the agent can reproduce the expected wording while failing to change the authoritative order state.

The test therefore needs a stronger oracle than text equality.

Define the journey contract

Start with the business goal and the conditions around it.

goal: resolve a damaged-order request
persona: customer with a damaged parcel
allowed_outcomes:
  - refund issued
  - replacement accepted
required_constraints:
  - verify order ownership
  - do not expose another customer's data
success_evidence:
  - authoritative order state changed
  - final response accurately describes the result

The goal defines what success means. Constraints define what cannot be violated. Evidence defines how the evaluator will know that the goal was actually achieved.

This gives the executor freedom to follow the agent's real behaviour without giving the evaluator permission to accept arbitrary behaviour.

Separate flexible behaviour from hard contracts

Not every property needs the same kind of assertion.

This separation is important because a language model is a poor replacement for an authoritative database or execution trace.

If the expected amount is €75, the evaluator should not accept €74 because the explanation "sounds right." If a required identity check did not execute, a fluent confirmation should not turn the journey green.

Test trajectories, not wording

A good conversational test allows multiple valid paths.

                 CUSTOMER GOAL
                       |
          +------------+------------+
          |                         |
       PATH A                    PATH B
   verify -> refund        identify -> retrieve
          |                     -> refund
          +------------+------------+
                       |
                       v
                SAME VALID OUTCOME

The paths can differ while the contract remains stable.

This is especially useful when agents are allowed to ask clarifying questions, select tools dynamically, or hand off to specialist flows. The test should explore that variation rather than suppress it by forcing a script.

But non-determinism is not a free pass

Acceptable variation still needs boundaries.

A journey should fail when the agent:

• accesses another customer's information;
• skips a required authorization step;
• invokes a privileged tool without permission;
• claims a state change that did not happen;
• enters an unsupported handoff;
• terminates before the customer's goal is complete.

The practical rule is simple: allow variation inside the contract; enforce the contract boundaries.

What should be generated from the system?

For an enterprise agent, test generation should start from what the system can actually do.

The flow or agent configuration can reveal intents, nodes, conditions, tools, APIs, handoffs and termination paths. Those structures can seed a test universe. Personas and goals then turn the structural model into executable journeys.

That is stronger than asking an LLM to invent generic customer prompts because it keeps test design connected to the real system surface.

Evaluate the journey after execution

The report should preserve at least four layers:

1. Goal outcome — was the customer objective achieved?
2. Execution path — what intents, nodes, tools and handoffs occurred?
3. Contract checks — which deterministic and security conditions passed or failed?
4. Language quality — was the response relevant, grounded and clear?

A semantic score can be high while the first three layers expose a failure. That is not a contradiction; the metrics are answering different questions.

The release implication

A conversational test should produce an explainable verdict rather than a transcript comparison.

TEST INTENT
    |
    v
LIVE CONVERSATION
    |
    +--> trajectory
    +--> orchestration
    +--> tools / APIs
    +--> deterministic state
    +--> security observations
    |
    v
EVALUATION
    |
    v
EVIDENCE-BACKED VERDICT

This also makes regression analysis more useful. A release can change the preferred path without changing the business outcome. Conversely, a release can preserve the wording while breaking a tool or state transition.

Conclusion

Conversational AI requires a different testing abstraction because the system is allowed to behave differently while still being correct. The test should therefore define the goal, boundaries and evidence rather than prescribe every sentence.

The transcript records one execution. The journey contract defines what success means.