The first question I ask about any A/B test is how big the sample needs to be. The answer depends on the smallest change worth detecting and how noisy the metric is. But before I trust that answer, I check two things: the setup and the metric. An A/A test, where both groups get the identical experience, should show no real difference. If it does, the setup is biased, for example users landing in groups unevenly. Passing it doesn’t prove the metric is right, though. If revenue double-counts orders that ship in two parcels, both groups double-count equally and the A/A test passes. So I also reconcile the metric against known transactions. A metric can pass an A/A test while consistently counting the wrong thing.

Evals raise the same two questions. The first is sample size. An eval doesn’t test a system on every question users will ever ask; it tests a sample. Pick a different 200 questions and you’d get a different score, just as a different set of users gives a different A/B result. The eval needs enough questions for a real improvement to stand out from that luck of the draw.

The second is the measurement itself. Every answer has to be graded, and graders make mistakes. When I started building evals for an AI analytics chatbot, I saw this quickly. An answer key accepts only one reading of an ambiguous date range. Two human raters disagree on whether a revenue number should pass when it silently excludes refunds. An LLM judge accepts a confident explanation without noticing that the SQL double-counts orders.

Those mistakes change the sizing question. An eval can be correctly sized to detect a 3-point change in the grader’s pass rate and still be badly sized to detect a 3-point change in answer quality. Under a simple model, a grader that flips one verdict in ten roughly doubles the questions you need. This piece takes the two problems in order: sample size first, then the grader.

where an eval score’s noise comes from · 2 problems

How big does an eval need to be?

By eval I mean something specific: an eval set of questions, an answer from each system version for every question, and a grader that marks each answer pass or fail. The score is the share that pass. Two separate problems make that score unreliable, and sizing an eval means dealing with both:

1. Sample size. The eval set is a sample. A different set of questions would give a different score, and a small set can’t tell a real improvement apart from luck.

2. Grading. The grader is a measuring instrument. Some of its verdicts are wrong, so the score tracks what the grader thinks, which isn’t always what’s true.

This section covers the first problem, assuming a perfect grader. The numbers below follow one worked example: an analytics chatbot at 82% accuracy, compared with a new prompt that reaches 85%.

Say your analytics chatbot answers 82% of questions correctly, and a new prompt reaches 85%. How many questions do you need to tell a real 3-point improvement apart from luck?

It’s the same calculation as sizing an A/B test. Pick the smallest improvement worth detecting, a significance level (5%, two-sided), and power, the chance of detecting the improvement if it’s real (80%). If the two versions answer separate sets of questions, you need about 2,400 questions per version. Most internal eval sets have 50 to 300. With 100 questions per version, the smallest improvement you can reliably detect is around 13 points.

You can do much better by running both versions on the same questions. Their results are correlated: both tend to pass easy questions and fail hard ones. Comparing them question by question removes that shared difficulty from the noise. What matters then is how often the two versions disagree, one passing while the other fails:

You’d need to estimate that disagreement rate too, ideally from an earlier comparison.

Evan Miller’s Adding Error Bars to Evals develops paired analysis, power calculations, and clustered error bars for questions that come in related groups, such as several questions about the same table. His framework also allows for random variation within a single question, so ordinary error bars already absorb some random grading noise.

What error bars can’t tell you is whether the grader’s pass rate tracks actual correctness. That’s the second problem, and the rest of this piece is about it.

Every grader makes mistakes

First, make sure the task has a right answer. In analytics, many questions hinge on a definition nobody wrote down. “Last month” can mean the calendar month or the last 30 days. Revenue may include or exclude refunds. An active customer might be someone who bought, logged in, or did something else. If the definition is open, a mismatch with the answer key isn’t a grading mistake. The eval is mixing correctness with whether the chatbot guessed the key’s interpretation. Fix the task before judging the grader.

Golden sets. A golden set grades by comparison with a reference answer, and the reference can be wrong. Northcutt and colleagues found label errors across ten widely used ML test sets, averaging at least 3.3% (Northcutt et al.). That isn’t an error rate for analytics evals, but it’s a good reason to treat every answer key as a measurement that needs checking.

Human raters. People handle ambiguity better than an answer key, but they make judgment calls. Does the right number with one wrong sentence of explanation pass? Is an unstated assumption acceptable? Raters also change their standards as they go. Shankar and colleagues call this criteria drift: people refine what “good” means after seeing outputs (Shankar et al.). The refined rubric is often better, but answers graded before the change may need regrading.

LLM judges. Zheng and colleagues found position, verbosity, and self-preference biases in the judges they tested (Zheng et al.). For an analytics chatbot, the bigger question is whether the judge can check the calculation at all. A judge that sees only the final answer, without the query, data, or execution result, can’t establish correctness. Asking it again won’t supply evidence it never had.

Random or systematic. Some mistakes look random: they hit both versions about equally. Their main cost is sample size, though they also distort the absolute score. Others take sides: a judge that rewards longer answers favors whichever version writes more. Random mistakes need more questions. Systematic ones need a different fix. The next two sections take each in turn.

What random mistakes do to sample size

Start with a deliberately simple model. Assume the grader:

• wrongly fails 10% of correct answers,

• wrongly passes 10% of incorrect answers,

• makes each mistake independently, and

• behaves the same way for both versions.

That’s a stronger set of assumptions than “the grader is 90% accurate,” and it’s worth stating them, because the results depend on them.

Under this model, the observed pass rate is 0.1 + 0.8 × the true accuracy. The 82% baseline shows up as 75.6%, and the 85% candidate as 78%. The real 3-point improvement shrinks to 2.4 points.

The required eval roughly doubles. Notice the absolute scores dropped too: random mistakes aren’t harmless just because they’re random.

The doubling is specific to this baseline. Around a 50% pass rate, the same model raises the requirement by about 56%. The general point holds: the shrinking gap costs more than the extra noise.

calculated for this article · normal approximation, symmetric independent grader errors

The cost accelerates: each extra 5 points of grader error adds more questions than the last.

Pairing makes it worse. If the two versions truly disagree on 10% of questions, independent flips on each verdict raise the observed disagreement to 24.4%, because the grader creates disagreements that aren’t real. The shared questions needed rise from about 870 to about 3,320.

So a single accuracy figure isn’t enough to plan with. You need to know how the grader errs on correct and incorrect answers separately, and whether its mistakes on the two versions are related.

When mistakes take sides

The model above treats both versions the same. Real graders often don’t. Their mistakes can follow an answer style, a task type, or one version.

Suppose the new version writes longer explanations, and the judge rewards length even when the extra material is wrong. The measured advantage now exceeds the real improvement in correctness. A different bias could hide a real improvement or reverse the ranking.

More questions don’t fix this. They give you an increasingly precise estimate of the grader’s preference. In A/B terms, it’s a broken metric: no sample size rescues a test that’s measuring the wrong thing. You need evidence about the grading itself.

Measuring your grader

That evidence comes from a grading audit, the eval equivalent of reconciling a metric against known transactions.

Take a random sample of answers from both versions. For each one, keep everything needed to check it: the question, the metric definitions, the query, a data snapshot, and the execution result. Have two reviewers grade independently against a versioned rubric. Compare their verdicts with the answer key and the LLM judge, then settle disagreements by going back to the underlying evidence.

Record more than an agreement percentage. Count three things separately: correct answers marked wrong, incorrect answers marked right, and cases that are still ambiguous or can’t be verified. Then check where the errors fall. If they concentrate in one task type or favor one version, you have bias, not noise.

The audit also shows where to cut errors at the source:

• Golden sets: state the definition in the question, or accept every defensible answer.

• Human raters: sharpen the rubric on the cases where they disagree, and regrade after it changes.

• LLM judges: don’t ask the judge to verify what it can’t see. For an analytics chatbot, run the query and compare the result with the correct number in code, and use the judge only for what code can’t check, such as whether the explanation is clear.

One caution: two reviewers can share the same wrong assumption. The audit corrects toward your most careful grading, not toward perfect truth, and it’s worth saying so when you report results.

The sizing recipe

Putting it together, sizing an eval looks like this:

1. Define the tasks so each question has a right answer.

2. Audit the grader to estimate how often it fails correct answers and passes incorrect ones, and whether its mistakes favor either version.

3. Choose the design. Run both versions on the same questions where you can.

4. State the smallest improvement worth detecting in correctness, not in the grader’s pass rate.

5. Compute the questions needed using the grader’s error rates, not a perfect-grader formula.

6. If the audit found bias, fix the grader or correct the estimate before trusting the result.

For step 6, one statistical option is Prediction-Powered Inference. It combines a large sample graded by the LLM judge with a smaller random sample graded carefully by people. For a mean score, the idea is simple:

Corrected estimate = the judge’s average on the large sample + the average (careful grade − judge grade) on the small sample.

The small sample measures the judge’s average error, and the confidence interval accounts for the uncertainty in that correction. For a comparison between two versions, apply the correction to the difference, not to each score separately.

It has limits. It corrects the overall estimate, not individual verdicts. The calibration sample has to be drawn at random from the same questions, not picked from conspicuous disagreements. A weak judge adds little, and a small calibration sample leaves wide intervals. And everything above assumes you know the grader’s error rates, when in practice you’ve estimated them from an audit, so that uncertainty belongs in the final analysis too.

Before trusting the number

The useful sizing question isn’t “how many questions does an eval need?” It’s “how many questions do I need to detect this improvement in correctness, with this grader, this design, and these dependencies between questions?”

Before I trust that number, I want to look at an answer the grader passed when it shouldn’t have, understand why, and check whether the same mistake would hit both versions equally. In an A/B test, that’s validating the metric before sizing the experiment. In an eval, the grader is the metric.