How to read this guide

Start with the visual assembly above. Each piece maps to a concept in this guide; the sections below provide the engineering detail, rules, examples and evidence needed to implement it correctly.

A retrieval-augmented generation system can retrieve a relevant passage and still produce an incorrect answer. Evaluate the path from source ingestion through retrieval and context assembly to the claims in the final answer.

The fictional Northstar Metrics SDK is an illustrative versioned-documentation assistant, not a real product or customer deployment.

System boundaries

Layer	Responsibility	Evidence to preserve
Source registry	Track authority, version, freshness and access	Source ID, revision, metadata
Indexing	Parse, chunk, embed and link entities	Index version, chunk IDs, entity links
Retrieval	Select passages and traverse relationships	Query, candidates, scores, graph paths
Context resolution	Apply authority and version rules	Selected context and conflicts
Answer generation	Produce an answer from approved context	Model, prompt revision, raw output
Claim verification	Check facts, grounding and contradictions	Claims, source links, assertion results
Evidence and gate	Preserve the run and apply policy	Manifest, verdict, policy revision

Define the contract first

For an SDK v4 question, specify the expected version and parameter, required source, forbidden v3 assumptions and abstention conditions before execution.

case:
  id: sdk-v4-batch-flush
  source_set: northstar-docs-2026-10
  required_facts:
    sdk_version: "v4"
    parameter: "flushIntervalMs"
  fail_if:
    - wrong_version
    - unsupported_claim
    - missing_required_source
    - material_contradiction

The example is fictional. Use deterministic assertions for exact values and semantic evaluation for claim grounding. Review generated test cases before promoting their expected answers to ground truth.

Route failures by evidence

Find the earliest stage where evidence diverges from the contract. Do not change the model first if the correct source never reached context; do not tune retrieval if the evidence is present but the answer contradicts it.

Evaluate independently

Answer correctness, including version and entity.
Evidence coverage and retrieval quality.
Claim-level grounding and contradiction detection.
Conflict resolution and abstention.
Security invariants against untrusted retrieved content.
Latency, cost and evidence completeness.

Do not collapse these into one score. A high average must not override a critical unsupported claim or security failure.

Compare configurations fairly

Compare vector, graph and hybrid retrieval with the same benchmark, source snapshot, generation model and evaluation rules. Change one dimension at a time, preserve the retrieved context, repeat stochastic runs where needed and report by query class. Let evidence—not architecture labels—decide.

Example release policy

release_policy:
  critical_unsupported_claims: 0
  wrong_version_answers: 0
  security_invariant_failures: 0
  required_evidence_complete: true
  human_review_for_policy_changes: true

This is illustrative, not a measured Shyena result. Set and version thresholds according to product risk.

Validate the Shyena website

These commands validate and build the website content; they do not run a RAG benchmark:

npm install
npm run content:validate
npm run content:generate
npm run build

Source note

The public Graph-RAG versus standard RAG experiment is a limited synthetic experiment. Its numeric-string matching metric does not prove claim-level grounding or a universal architecture ranking. This guide uses the source only for that methodological context and presents an independent design.

Back to Docs Evaluation Model

Primary sources are listed inside each page.

Trace RAG failures from source to claim.