Evaluation services

The idea in one minute#

An evaluation is the test suite of an AI system: fixed cases, a way to grade each output, and a pass rate. When one team owns one feature, a script and a folder of JSON cases are enough. When a platform serves dozens of teams, and every prompt edit, model upgrade, route change, and retrieval setting can change behavior, evaluation becomes a shared service with the properties of any other piece of infrastructure: versioned inputs, reproducible runs, capacity planning, and an answer that people act on.

The service is only useful if its verdicts can be trusted, and that is mostly a statistics problem. Pass rates on a few hundred cases are noisy; the system is non-deterministic; a model judge can agree with humans 88% of the time and still miss most real failures. The consequence to act on: pin and record every version in a run, compare candidates to baselines on the same cases, report intervals and flipped cases instead of single numbers, and validate every judge with chance-corrected agreement before it gates anything.

A picture#

flowchart LR
  CH["<b>Change</b><br/><small>prompt, model, route,<br/>retrieval, tools</small>"] --> RUN
  DS[("<b>Dataset registry</b><br/><small>versioned cases,<br/>held-out split, slices</small>")] --> RUN
  RUN["<b>Runner</b><br/><small>batch lane, pinned versions,<br/>k runs per case</small>"] --> GR["<b>Graders</b><br/><small>code, judge, human</small>"]
  GR --> RS[("<b>Results store</b><br/><small>keyed by system, dataset,<br/>grader versions</small>")]
  RS --> GATE{"<b>Gate</b><br/><small>paired test, slices,<br/>must-pass, cost</small>"}
  GATE -->|"pass"| ROLL["<b>Canary rollout</b>"]
  GATE -->|"fail"| BACK["<b>Flipped cases<br/>to the author</b>"]
  ROLL --> TR["<b>Sampled production traces</b>"]
  TR -->|"new cases after review"| DS
  CAL[("<b>Human labels</b><br/><small>judge calibration set</small>")] -.-> GR
  class CH neutral
  class DS,RS,CAL memory
  class RUN,GR compute
  class GATE good
  class ROLL,TR io
  class BACK queue
classDef neutral fill:#f3f1ea,stroke:#5f5e55,color:#141413,stroke-width:1.5px
classDef compute fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef memory fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef queue fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef io fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef default fill:#f5d195,stroke:#b0420c,color:#141413,stroke-width:1.5px
classDef warn fill:#ef94ad,stroke:#b3154f,color:#141413,stroke-width:2.2px,stroke-dasharray:6 4
classDef good fill:#8dbb6f,stroke:#2b6a24,color:#141413,stroke-width:2.2px

Every result is keyed by the three versions that produced it: system, dataset, and grader. Without that key, two numbers cannot be compared.

How it really works#

The parts of the service#

The results store is what turns evaluation from an event into a history. With it, a question like “did the June prompt change hurt Spanish-language refunds?” is a query, and re-grading old outputs with a new grader does not require re-running the system.

Reproducibility is a key#

A run is reproducible only if everything that affected its output is recorded. That is the same list as the key of an exact response cache (see caches): model and version, prompt and template version, tool definitions, sampling parameters, retrieval index or retrieved chunk IDs, and the dataset version. The grader adds its own: the judge model and version, the rubric version, and code grader commit. Store system outputs keyed this way, so a grader change re-uses outputs instead of paying for them again.

Capacity and cost#

Evaluation is a large batch workload, and it is easy to underestimate. Take 2,000 cases, 3 runs each, a system call of 4,000 input and 500 output tokens, and a judge call of 5,000 input and 200 output tokens, at hypothetical prices of $3 and $15 per million input and output tokens:

system calls   6,000 x (4,000 x $3 + 500 x $15) / 1M   = $117
judge calls    6,000 x (5,000 x $3 + 200 x $15) / 1M   = $108
one full run                                           = $225
on a 50%-discount batch API                           ~= $112
20 changes a day, full run each                        = $2,250 to $4,500 per day

So the service needs tiers: a small smoke set (a few hundred cases, one run) on every pull request, the full set with repeated runs before release, and a scheduled nightly run against production to catch drift. Runs go through the batch lane from queues and batch lanes, never through interactive quota.

Comparing two versions: pair the cases#

The common mistake is comparing two pass rates with overlapping intervals and concluding nothing, or comparing two point estimates and concluding too much. Because the baseline and the candidate run on the same cases, the right comparison is paired. Count the cases that flipped:

• b: passed on the baseline, failed on the candidate (broke)
• c: failed on the baseline, passed on the candidate (fixed)

Under the hypothesis of no difference, each flip is equally likely to go either way, so b follows a binomial distribution with n = b + c and p = 1/2. That is the exact McNemar test (McNemar, 1947). Pairing removes the variation in case difficulty, which is most of the variance, and the flipped cases are also the list a reviewer should read. Miller (2024) recommends the same structure for model evaluations generally: report standard errors, compare on paired cases, and use clustered errors when cases share a source document.

Non-determinism: run cases more than once#

The same case can pass one run in three. A single run per case turns that randomness into apparent regressions and fixes. Choose an aggregation that matches the product:

How many cases#

Small differences need large sets. For an unpaired two-proportion test at significance 0.05 and power 0.8, detecting a drop from 90% to 85% needs about 690 cases per arm, and from 90% to 88% needs about 3,800. Pairing lowers these numbers, sometimes a lot, but the message stands: a 200-case suite cannot detect a two-point regression. Decide the smallest regression the gate must catch, then size the set, and use slices with their own thresholds for the groups that matter, since an overall average hides a small slice’s collapse.

Validating a judge#

A model judge is a classifier, and its labels are only useful once compared with human labels on a calibration set. Raw agreement is misleading when most answers pass: a judge that says “pass” almost always agrees with humans most of the time and catches few failures. Use Cohen’s kappa (Cohen, 1960), which corrects for the agreement expected by chance:

kappa = (p_observed - p_chance) / (1 - p_chance)
p_chance = P(human pass) x P(judge pass) + P(human fail) x P(judge fail)

Also report the judge’s recall on real failures, since that is what a gate needs. Known judge biases (Zheng et al., 2023) include position bias in pairwise comparisons, verbosity bias toward longer answers, and self-preference for outputs from the same model family. Controls:

• Binary, specific rubric questions (“does the answer contain a claim not in the sources?”) rather than 1 to 10 scores.
• Swap the order in pairwise judgments and count only consistent verdicts.
• A judge from a different model family than the system under test.
• Pinned judge version; re-calibrate when it changes.

The gate#

A gate is a list of conditions that must all hold:

1. No significant regression on the paired comparison, overall and per slice, or a non-inferiority margin: the candidate is not worse than the baseline by more than a stated amount, with confidence.
2. Every must-pass case passes all runs: safety rules, past incidents.
3. Cost and latency within budget: tokens per case and p95 latency.
4. A human reads the broken cases before an override.

Keeping evaluation data clean#

Evaluation cases must not leak into training or fine-tuning data, or the suite measures memorization. Keep a held-out split, mark evaluation data with a canary string so it can be found in corpora, and run contamination checks in the data platform, as covered in a data platform. Cases built from production traces are customer data: they need the same access control, retention, and redaction as the traces themselves.

Designing the cases themselves is covered in designing evals, and judges in depth in model graders.

Code#

The program runs three checks an evaluation service makes. A baseline and a candidate are run three times on the same 400 cases; the candidate fixes some cases and breaks others. Two judges are compared with 200 human labels. Then it computes the cases needed to detect a drop.

"""evalsvc.py: the statistics an evaluation service must get right.

1. A baseline and a candidate run on the same 400 cases, 3 times each.
   Report Wilson intervals, the paired flips, and an exact McNemar test.
2. A model judge is checked against 200 human labels: raw agreement
   versus Cohen's kappa, and what a lenient judge does to the pass rate.
3. How many cases a gate needs to detect a given regression.
Simulated systems; standard library only.
"""

from __future__ import annotations

import math
import random


def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = k / n
    c = (p + z * z / (2 * n)) / (1 + z * z / n)
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return c - h, c + h


def mcnemar_exact(b: int, c: int) -> float:
    """Two-sided exact p-value: under no difference, b ~ Binomial(b + c, 1/2)."""
    n, k = b + c, min(b, c)
    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)


def paired_demo(rng: random.Random) -> None:
    n_cases, runs = 400, 3
    # each case has a latent pass probability; the candidate fixes some, breaks some
    base_p = [min(1, max(0, rng.betavariate(6, 1.3))) for _ in range(n_cases)]
    cand_p = []
    for p in base_p:
        r = rng.random()
        if r < 0.06:
            cand_p.append(max(0.0, p - 0.5))   # regressed on 6% of cases
        elif r < 0.10:
            cand_p.append(min(1.0, p + 0.3))   # improved on 4%
        else:
            cand_p.append(p)

    def majority(ps: list[float]) -> list[bool]:
        return [sum(rng.random() < p for _ in range(runs)) * 2 > runs for p in ps]

    a, b = majority(base_p), majority(cand_p)
    ka, kb = sum(a), sum(b)
    print(f"1. paired comparison, {n_cases} cases, majority of {runs} runs per case")
    for name, k in (("baseline", ka), ("candidate", kb)):
        lo, hi = wilson(k, n_cases)
        print(f"   {name:<10} {k}/{n_cases} = {k/n_cases:.1%}  (95% interval {lo:.1%} to {hi:.1%})")
    broke = sum(x and not y for x, y in zip(a, b))
    fixed = sum(y and not x for x, y in zip(a, b))
    print(f"   intervals overlap; the paired view: {broke} cases broke, {fixed} cases fixed")
    print(f"   exact McNemar p = {mcnemar_exact(broke, fixed):.4f}"
          f"  -> {'regression is real' if mcnemar_exact(broke, fixed) < 0.05 else 'not distinguishable from noise'}")


def kappa_demo(rng: random.Random) -> None:
    n = 200
    human = [rng.random() < 0.85 for _ in range(n)]  # 85% of answers really pass
    # judge A: lenient, says pass 97% of the time regardless
    judge_a = [True if h else rng.random() < 0.80 for h in human]
    # judge B: calibrated with a rubric, makes symmetric 7% errors
    judge_b = [h if rng.random() > 0.07 else not h for h in human]
    print(f"\n2. judge validation against {n} human labels (humans pass {sum(human)/n:.0%})")
    for name, j in (("lenient judge", judge_a), ("rubric judge", judge_b)):
        agree = sum(x == y for x, y in zip(human, j)) / n
        ph, pj = sum(human) / n, sum(j) / n
        pe = ph * pj + (1 - ph) * (1 - pj)
        kappa = (agree - pe) / (1 - pe)
        fail_caught = sum((not h) and (not x) for h, x in zip(human, j)) / max(1, n - sum(human))
        print(f"   {name:<14} agreement {agree:.0%}  kappa {kappa:+.2f}  reports pass {pj:.0%}"
              f"  catches {fail_caught:.0%} of real failures")


def sample_size(p: float, drop: float, alpha: float = 0.05, power: float = 0.8) -> int:
    """Cases per arm for a two-proportion test (unpaired, normal approximation)."""
    za, zb = 1.959964, 0.841621
    q = p - drop
    pbar = (p + q) / 2
    num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p * (1 - p) + q * (1 - q))) ** 2
    return math.ceil(num / drop ** 2)


def main() -> None:
    rng = random.Random(11)
    paired_demo(rng)
    kappa_demo(rng)
    print("\n3. cases needed to detect a drop at alpha 0.05, power 0.8 (unpaired)")
    for p, d in ((0.90, 0.05), (0.90, 0.02), (0.95, 0.01), (0.80, 0.04)):
        print(f"   from {p:.0%} to {p-d:.0%}: {sample_size(p, d):,} cases")


if __name__ == "__main__":
    main()

Run it with python3 evalsvc.py:

1. paired comparison, 400 cases, majority of 3 runs per case
   baseline   347/400 = 86.8%  (95% interval 83.1% to 89.7%)
   candidate  354/400 = 88.5%  (95% interval 85.0% to 91.3%)
   intervals overlap; the paired view: 33 cases broke, 40 cases fixed
   exact McNemar p = 0.4828  -> not distinguishable from noise

2. judge validation against 200 human labels (humans pass 84%)
   lenient judge  agreement 88%  kappa +0.32  reports pass 96%  catches 22% of real failures
   rubric judge   agreement 94%  kappa +0.78  reports pass 80%  catches 94% of real failures

3. cases needed to detect a drop at alpha 0.05, power 0.8 (unpaired)
   from 90% to 85%: 686 cases
   from 90% to 88%: 3,841 cases
   from 95% to 94%: 8,158 cases
   from 80% to 76%: 1,683 cases

The candidate’s pass rate is 1.7 points higher and the intervals overlap, which looks like “slightly better”. The paired view says something more useful: 33 cases broke and 40 were fixed, and the exact McNemar test cannot distinguish that from noise. The change is not an improvement until someone reads the 33.

The lenient judge agrees with humans on 88% of answers, which sounds acceptable, but its kappa is 0.32 and it catches 22% of real failures; as a gate it would pass almost everything. The rubric judge has kappa 0.78 and catches 94%. Raw agreement hid a fourfold difference in the property that matters.

Where it breaks#

• The suite measures only what is in it. Offline evaluation cannot contain failures nobody imagined. Sampled online grading and trace review cover the rest.
• Datasets go stale. Product scope, user behavior, and policy change. A dataset without an owner and a refresh schedule decays into a regression test for last year’s product.
• Judges drift with their providers. A judge served by a hosted API can change when the provider updates it. Pin versions, keep the calibration set, and re-measure kappa on every judge change.
• Statistical tests answer narrow questions. McNemar says whether flips are balanced, not whether the broken cases matter. Severity lives in tags and must-pass lists, not in p-values.
• The program’s systems and judges are simulated. The pass probabilities and judge error rates are invented to show the arithmetic.

Remember this#

• Key every result by system version, dataset version, and grader version.
• Compare on paired cases; report flipped cases and an exact test, not two overlapping intervals.
• Run cases several times and choose the aggregate the product needs.
• A judge gates nothing until its kappa and failure recall against human labels are known.
• Evaluations are batch workloads with real cost: tier them, and run them on the batch lane.

Try it#

1. In evalsvc.py, raise the regression share from 6% to 10%. At what share does McNemar call the regression real?
2. Change runs to 1. How many more cases flip between baseline and candidate, and does the verdict change?
3. Add a third judge with position bias: on half the cases it always says pass. Compute its agreement and kappa.
4. For a feature you know, write the gate: slices, must-pass cases, the smallest regression it must detect, and the dataset size that implies.

Check yourself#

1. Why compare a candidate to the baseline with a paired test rather than two independent pass rates?

   Show answer

   Both versions run on the same cases, and case difficulty varies far more than the effect of a change. Pairing cancels that variation and focuses on the cases that flipped, which gives a more sensitive test and the exact list of cases to read.

2. A judge agrees with human labels 90% of the time. Why is that not enough to trust it?

   Show answer

   If most answers pass, a judge that nearly always says pass agrees most of the time by chance. Kappa corrects for chance agreement, and recall on real failures shows whether the judge would catch regressions. Both must be measured on a human-labeled calibration set.

3. What must be recorded to reproduce an evaluation run?

   Show answer

   The system version (model and version, prompt and template version, tool definitions, sampling parameters, retrieval index version), the dataset version, and the grader version (judge model and version, rubric version, code grader commit), plus seeds and the number of runs.

4. Why can a 200-case suite not gate a two-point regression?

   Show answer

   The sampling noise on a pass rate from 200 cases is several points wide. Detecting a drop from 90% to 88% with standard power needs thousands of cases per arm unpaired. A smaller suite can only catch large regressions, or regressions concentrated in tagged slices and must-pass cases.

5. How can a system under evaluation attack its own gate?

   Show answer

   Its output is fed to a model judge. Text in the output that instructs the judge to pass it is a prompt injection. The judge prompt must delimit graded content as data, and the judge’s calibration set should include injection attempts.

Sources#

• Q. McNemar, Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika, 1947.
• J. Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement, 1960.
• E. B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference, Journal of the American Statistical Association, 1927.
• E. Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024.
• L. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS Datasets and Benchmarks, 2023.
• A. Srivastava et al., Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (BIG-bench canary string), 2022.
• UK AI Security Institute, Inspect, evaluation framework documentation, read October 2026.