DeepEval + GEval, LLM-as-judge

The eval stack

Nothing generated ships unexamined. The lanes, cheapest first:

1. Deterministic gates — citation presence, structure, readability, length budgets. Free, instant, zero variance. They run before any LLM judge; a judge is the expensive last line, not the first.
2. LlamaIndex response evals — faithfulness, relevancy, correctness on RAG answers; retrieval metrics (MRR, hit rate) over pinned query→expected-source sets, so retriever changes are measured, not felt.
3. A DeepEval lane — runs in its own venv (make eval-deepeval) because deepeval's pydantic pin conflicts with the service image; eval dependencies never leak into production.
4. Eight GEval-style metrics on three axes (audio experience, system design, content) judge every generated audio chapter. Chapters that fail don't ship.

Rule 1: a verdict without evidence is not a verdict

An audit of judge output found 54% of "supported" claims couldn't actually be cited back to a source — the judge said yes without receipts. The fix is structural, not prompt-level: any asserted verdict must persist the evidence it used. A "supported" with no stored citation is treated as unverified, full stop. Judges are pinned, too — scores are only comparable within a judge+prompt version, so the judge-stack identity is folded into the cache key.

Rule 2: automated repair is worse than no repair

The grounding watchdog could auto-"refute" claims it thought were wrong. In practice it produced roughly two bogus refutations for every real one — an auto-repair loop that vandalizes good content faster than it fixes bad content. Refutations now require a human in the loop; the automated lane runs with repair off.

Testing the non-deterministic

The same philosophy handles agent and RAG testing generally: deterministic checks wherever a property can be deterministic (structure, citations, tool-call shapes, scripted-LLM integration tests), LLM-as-judge only for qualities that genuinely need judgment — with evidence persisted and refutation reviewed. Related pages: the serving layer ships a scripted LLM for zero-model integration tests; the agents page covers deterministic trajectory testing; the full evaluation architecture — dataset contracts, trajectory evaluators, the coverage gate — is the Evaluation & Feedback guide.

Numbers

• Eight GEval-style metrics on three axes judge every generated audio chapter; the judge bar is ≥0.80.
• The audit that forced Rule 1: 54% of "supported" verdicts couldn't be cited back to a source.
• The audit that forced Rule 2: ~2 bogus refutations per real one from automated repair.
• Eval dependencies live in their own venv — zero eval packages in the production image.