Evaluation

Measure retrieval and answers on your corpus. A generic benchmark is not a substitute.

type EvaluationCase = { question: string; expectedSourceIds: string[];
  requiredAnswerFacts: string[] };
type EvaluationResult = { retrievedRelevantSources: number; contextPrecision: number;
  answerFactCoverage: number; unsupportedClaims: number;
  selectorInputCharacters: number; answerInputCharacters: number; latencyMs: number };

const variants = [
  { name: 'A: direct', retrieval: 'topK 5 → answer model' },
  { name: 'B: gate', retrieval: 'topK 40 → selector → top 6 → answer model' },
  { name: 'C: semantic', retrieval: 'semantic topN 50 → topK 8 → answer model' },
];

Run every variant over the same cases. Score outputs against corpus-grounded facts. Record abstentions separately. Do not invent benchmark numbers for a landing page or a sales deck.

Track retrieval recall, context precision, source coverage, answer fact coverage, unsupported claims, selector cost, answer-model token or character usage, latency, and abstention quality.

Reflex ships its own evaluation harness. A six-task fixture there is exploratory and cannot support certification. See Reflex. KAR conformance vectors check the evaluator. They are not a claim that your documents are true.