DeepEval Rubric Scores: How to Make LLM Evaluation More Consistent

A semantic evaluation score is only useful when the scoring model understands what each score actually means.

That is why rubric-based evaluation matters.

DeepEval's G-Eval documentation supports an optional rubric argument that constrains the evaluation model to defined score ranges and expected outcomes. DeepEval documents rubric ranges from 0 to 10, with non-overlapping ranges, while its metrics expose a final score from 0 to 1 and use a configurable threshold for pass/fail decisions.

For AI assurance teams, the important distinction is this:

A rubric should define the meaning of a score. It should not manufacture confidence that the underlying agent execution was correct.

Why a rubric is better than an unconstrained score

Consider an evaluator asked to score an answer for correctness.

Without a rubric, two runs might produce scores such as:

• 0.71 — mostly correct
• 0.76 — good answer
• 0.81 — very good answer

Those numbers are difficult to interpret consistently. What exactly separates 0.71 from 0.76? What changed between 0.81 and a release threshold of 0.80?

A rubric gives the evaluator explicit semantic boundaries.

For example:

These bands are a Shyena recommended example, not DeepEval's default rubric. The important design principle is that each band has a distinct, observable outcome and the ranges do not overlap.

What DeepEval recommends

DeepEval's G-Eval implementation supports three complementary controls that are particularly useful together:

1. Criteria

Define what the evaluator is judging.

criteria="Determine whether the actual output is factually correct based on the expected output."

Criteria are useful when exploring an evaluation problem and allowing G-Eval to generate evaluation steps.

2. Evaluation steps

For more controllable scoring, explicitly define how the evaluator should reason about the criterion.

evaluation_steps=[
    "Check whether the actual output contradicts the expected output.",
    "Penalize omission of critical information.",
    "Do not penalize wording differences when the meaning is preserved."
]

DeepEval's documentation notes that providing explicit evaluation steps avoids regenerating the steps from the criteria on every run and gives you more control over the metric. criteria and evaluation_steps are alternatives: supply one or the other, not both. Once you lock the steps, the criterion is expressed through them.

3. Rubric

Use rubric bands to constrain the final judgement to defined outcomes.

from deepeval.metrics import GEval
from deepeval.metrics.g_eval import Rubric
from deepeval.test_case import LLMTestCase, SingleTurnParams

metric = GEval(
    name="Response Correctness",
    evaluation_params=[
        SingleTurnParams.ACTUAL_OUTPUT,
        SingleTurnParams.EXPECTED_OUTPUT,
    ],
    # Pass evaluation_steps OR criteria, not both. Steps are used here so the
    # judge's procedure is fixed rather than regenerated on every run.
    evaluation_steps=[
        "Check for factual contradictions.",
        "Check for missing critical information.",
        "Ignore wording differences when meaning is preserved.",
    ],
    rubric=[
        Rubric(score_range=(0, 2), expected_outcome="Factually incorrect or materially misleading."),
        Rubric(score_range=(3, 4), expected_outcome="Major correctness gaps remain."),
        Rubric(score_range=(5, 6), expected_outcome="Generally correct but materially incomplete."),
        Rubric(score_range=(7, 8), expected_outcome="Correct and complete with minor issues."),
        Rubric(score_range=(9, 10), expected_outcome="Fully correct, relevant and complete."),
    ],
    threshold=0.8,
)

The key design rule is not the number of bands. It is the semantic contract behind each band.

The 0–10 rubric and the 0–1 metric score are not the same thing

This distinction is important when designing dashboards and release gates.

DeepEval documents rubric score_range values on a 0–10 scale, while its metrics expose a 0–1 score. The metric threshold is applied to the metric score, not directly to the rubric's 0–10 labels.

That means an engineering dashboard should avoid mixing these values without labeling them clearly.

A production report can show:

This separation prevents a common mistake: treating a semantic score as if it were a complete system verdict.

Recommended rubric design for AI agents

For agentic systems, score the property that actually matters to the journey.

A useful rubric should answer five questions:

1. What is being judged?
2. What evidence should the judge inspect?
3. What makes an outcome materially wrong?
4. What separates acceptable from strong?
5. What evidence makes the result release-relevant?

Avoid vague bands such as:

• 0–3 = bad
• 4–6 = okay
• 7–10 = good

These labels leave too much interpretation to the evaluator.

Prefer observable outcomes:

• required information missing
• unsupported claim introduced
• correct answer but wrong business action
• correct answer and correct action
• policy requirement satisfied
• critical safety constraint violated

The more concrete the outcome, the more useful the rubric becomes.

Rubrics should not replace deterministic checks

This is where AI assurance differs from simple LLM evaluation.

Suppose an agent handles an address-change request.

A semantic evaluator may decide that the final response is helpful and clear. But the system could still have:

• called the wrong API
• used the wrong customer identifier
• skipped authentication
• changed the wrong field
• violated a mandatory business rule
• failed to complete the requested transaction

A high semantic score cannot prove those conditions were correct.

The assurance model should therefore layer the evidence:

Deterministic contracts
        ↓
Execution integrity
        ↓
Semantic / rubric evaluation
        ↓
Security and policy checks
        ↓
Evidence-backed release decision

The rubric belongs in the semantic evaluation layer.

A practical Shyena rubric pattern

For an AI agent journey, Shyena can structure evaluation around five evidence dimensions:

A rubric can score the semantic dimensions, while deterministic and runtime checks constrain the final release verdict.

This is particularly important for multi-turn and agentic systems where the final message can look correct even when the path taken to reach it was wrong.

Recommended threshold strategy

Do not choose a threshold simply because 0.5 is the library default.

DeepEval documents 0.5 as the default threshold for metrics, but production teams should define thresholds based on the risk and purpose of the metric.

For example:

GEval(
    name="Customer Response Quality",
    criteria="Evaluate whether the response satisfies the customer request accurately and completely.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
    rubric=[
        Rubric(score_range=(0, 2), expected_outcome="Incorrect or unsafe."),
        Rubric(score_range=(3, 5), expected_outcome="Partially correct with material gaps."),
        Rubric(score_range=(6, 7), expected_outcome="Acceptable but not consistently complete."),
        Rubric(score_range=(8, 9), expected_outcome="Strong response with minor issues."),
        Rubric(score_range=(10, 10), expected_outcome="Fully correct and complete."),
    ],
    threshold=0.8,
)

The threshold should then be validated against representative evaluation data rather than selected arbitrarily.

How to make rubric scores more reproducible

DeepEval notes that G-Eval can become inconsistent when it regenerates evaluation steps from criteria. A practical approach is:

Explore
  ↓
Define criteria
  ↓
Inspect generated evaluation steps
  ↓
Lock explicit evaluation steps
  ↓
Define non-overlapping rubric bands
  ↓
Calibrate against human-labelled examples
  ↓
Set the release threshold
  ↓
Track score + reason + evidence over time

This turns a qualitative evaluation idea into an engineering control.

Rubric scoring for agentic AI: go beyond the final answer

For agentic systems, the evaluator should not be limited to actual_output.

Depending on the metric, the test case can include evidence such as:

• expected output
• retrieval context
• tools called
• tool arguments
• tool outputs
• intermediate steps
• full trajectory
• conversation history

DeepEval provides agent-oriented trajectory metrics such as task completion, step efficiency, plan adherence and plan quality. Its documentation also distinguishes trajectory evaluation from metrics that score individual interactions.

The right evaluation input depends on the question being asked.

If the question is "Was the final response well written?", a semantic rubric may be sufficient.

If the question is "Did the agent complete the customer's task correctly?", the evaluator needs execution and trajectory evidence as well.

A release-gate pattern for Shyena

A practical Shyena release gate can combine the rubric score with non-semantic controls:

                    AI AGENT RUN
                         │
          ┌──────────────┼──────────────┐
          │              │              │
   Deterministic     Execution      Semantic
     checks          integrity       rubric
          │              │              │
          └──────────────┼──────────────┘
                         ↓
                  Security checks
                         ↓
                 Evidence package
                         ↓
                  Release verdict

A rubric can answer:

How good was this outcome according to the defined semantic criteria?

It should not independently answer:

Is the entire AI system safe and correct to release?

That requires evidence from the complete assurance stack.

The engineering rule

Use criteria to define the evaluation objective.

Use evaluation steps to make the judge's process more controllable.

Use rubrics to define what score bands mean.

Use thresholds to establish metric-level success.

Use deterministic, execution and security evidence to constrain the final release decision.

That combination turns a number into an engineering signal rather than a decorative dashboard score.

DeepEval rubric score: practical checklist

Before putting a rubric into CI/CD, verify:

• Every score range has a clear expected outcome.
• Score ranges do not overlap.
• The rubric uses observable evidence rather than vague adjectives.
• Evaluation parameters contain only the information required by the criteria.
• Explicit evaluation steps are used when reproducibility matters.
• The threshold is calibrated against representative examples.
• The semantic score is not treated as proof of deterministic correctness.
• Agent trajectory and tool behaviour are evaluated when they affect the business outcome.
• Security and policy checks remain independent controls.
• The release decision retains the evaluator's reason and supporting evidence.

Final takeaway

DeepEval's rubric capability is useful because it gives semantic evaluation a more explicit scoring contract.

The strongest implementation is not "use a rubric and trust the score."

It is:

Define the rubric → control the evaluation steps → calibrate the threshold → capture the reason → connect the result to execution evidence → make the release decision from the complete evidence set.

For AI agents, that is the difference between measuring answer quality and building an assurance system.

Limits of this approach

G-Eval uses an LLM as the judge, so even with locked evaluation steps and a rubric its scores remain non-deterministic and depend on the judge model. For checks that must be exactly repeatable, DeepEval documents a decision-tree metric (DAGMetric) that branches on explicit conditions. Rubric thresholds should be calibrated against human-labelled examples, and agreement between the judge and human reviewers should be measured before a score gates a release. The rubric bands and the 0.8 threshold shown above are illustrative, not validated values.

References

1. DeepEval documentation, "G-Eval" (LLM-as-a-judge metrics, criteria, evaluation steps, rubric): <https://deepeval.com/docs/metrics-llm-evals>
2. Liu, Y. et al. (2023). "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment." EMNLP 2023. <https://arxiv.org/abs/2303.16634>
3. Zheng, L. et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023. <https://arxiv.org/abs/2306.05685>