Comparing Versions¶

You changed the model, or you rewrote the prompt. The eval suite that was passing at 41% is now at 85%. Two questions follow immediately:

Did behavior actually change? Or is this run-to-run jitter from a non-deterministic agent?
Is the change real or noise? A 44-point jump is obviously real. A 3-point jump on 20 trials usually isn't. Where's the line?

This page answers both with one workflow: run the same eval set against two configurations, stamp each with a DecisionSpec so results are attributable, and use a bootstrap comparison to decide whether the difference is statistically real.

The full runnable example is examples/version_compare.py. It uses a seeded, simulated agent so it runs with no API keys; swap in your real adapter and the comparison machinery is identical.

The setup: one ruler, two things to measure¶

The cardinal rule of a fair comparison is to change only the thing under test. There are two valid axes:

Model change — same prompt, swap ModelConfig (e.g. gpt-4o-mini → gpt-4o).
Prompt change — same model, swap the prompt text in PromptSpec.

Everything else stays fixed, and critically, the grader and the eval set are the ruler — they must be identical across both versions. If you change the grader and the prompt at the same time, you can't attribute the delta to either. What you vary is the adapter (the agent invocation) and the DecisionSpec (the attributable description of what produced the result).

Step 1: parameterize and stamp each version¶

Wire one adapter per configuration, and build a DecisionSpec that records the model and prompt behind it. ModelConfig requires a provider; model_id is the version you're pinning. PromptSpec.from_prompts hashes the prompt text so two different prompts produce two different fingerprints without storing the raw text in your baselines.

from tracelens import DecisionSpec, ModelConfig, PromptSpec

def spec_for(version: str, prompt_text: str) -> DecisionSpec:
    return DecisionSpec(
        model=ModelConfig(provider="openai", model_id="gpt-4o-mini", temperature=0.7),
        prompts=PromptSpec.from_prompts(system_prompt=prompt_text, prompt_version=version),
    )

v1_spec = spec_for("v1", "Reply to the support ticket.")
v2_spec = spec_for(
    "v2", "Reply concisely, cite the relevant policy, and propose concrete next steps."
)

The two specs now have different fingerprints because the prompt hashes differ. That difference is what makes a result attributable: a transcript stamped with v2_spec.fingerprint provably came from the v2 prompt and not from v1.

print(v1_spec.fingerprint_short)            # 20a1b674339b
print(v2_spec.fingerprint_short)            # cc545403c02b
print(v1_spec.fingerprint != v2_spec.fingerprint)  # True

If you change the model instead, swap the ModelConfig and the fingerprint moves the same way. Either way, each result carries a fingerprint of exactly the configuration that produced it. See Reproducibility & DecisionSpec for what goes into a fingerprint and what's deliberately left out.

Step 2: run both versions on the same eval set¶

Pass the spec to EvaluationRunner via decision_spec=; the runner stamps it onto every transcript that doesn't already carry one. Use num_runs > 1 — a single run per task tells you nothing about a non-deterministic agent's variance.

from tracelens import EvaluationRunner, RunnerConfig, SimpleAdapter

async def run_version(make_adapter, spec: DecisionSpec):
    runner = EvaluationRunner(
        make_adapter(),
        [ReplyQualityGrader("reply_quality")],   # SAME grader for both versions
        RunnerConfig(num_runs=10, max_concurrency=1),
        decision_spec=spec,
    )
    batch = await runner.run(TASKS)              # SAME eval set for both versions
    scores = [o.score for trial in batch.trials for o in trial.outcomes]
    return batch, scores

b1, s1 = await run_version(lambda: SimpleAdapter(make_agent(0.66)), v1_spec)
b2, s2 = await run_version(lambda: SimpleAdapter(make_agent(0.82)), v2_spec)

scores is the flattened list of per-trial grader scores (12 tasks × 10 runs = 120 values per version). That per-trial granularity is what the significance test in the next step consumes.

Printing the per-version summary gives you the headline numbers:

version comparison
------------------
  v1 [20a1b674339b]  pass_rate=41%  mean_quality=0.656  n=120
  v2 [cc545403c02b]  pass_rate=85%  mean_quality=0.832  n=120

The fingerprint in brackets is your audit trail: every row is tied to a specific model + prompt.

Step 3: the verdict — `tracelens compare`¶

The means say v2 looks better. The comparison tells you whether to believe it, and whether it matters. With both runs saved (tracelens run --save-trials), it is one command:

tracelens compare v1-trials.json v2-trials.json --metric mean_score --threshold 0.05

or, in Python, compare_runs(b1, b2, metric="mean_score", threshold=0.05) on the two TrialBatch objects (this is what the example does). Either way the statistics are the ones fixed in the statistical contract:

The task is the sampling unit. Each task's mean score under v1 and under v2 is paired, and the interval comes from resampling tasks. Twelve tickets of different difficulty therefore cancel out instead of looking like noise, and ten repeated trials of one ticket never masquerade as ten independent samples.
Comparability is checked, not assumed. The runs' provenance must show the same task content and graders; a task edited between the runs, or a changed grader, is refused rather than matched by id.
Enough tasks to decide. With the task as the unit, the sign-flip test behind the p-value cannot go below 2 / 2^T on T tasks, so at the default 95% confidence there is no verdict on fewer than six tasks: the command prints the numbers, says what p-value the test cannot get below and how many tasks a verdict needs, and exits 2. Six is the floor, not a target: at six, one ticket that does not move with the rest already leaves the result inconclusive, which is why this example uses twelve.
Three readings, one verdict. Significance (the interval excludes 0 and the p-value agrees), practical relevance (|delta| against --threshold), and evidence (the interval's extent against the threshold). A small significant change passes as "below the threshold" only when the interval's lower end also stays above -threshold. The verdict is one of improvement, regression, significant but below the threshold, equivalent within the threshold, inconclusive, or insufficient evidence; exit codes are 0, 1, and 2 respectively for "no regression", "regression", and "cannot tell".

The example prints the same summary the command does:

Compared v2 vs v1 on mean_score (higher is better): paired task bootstrap over 12 task(s)
  v1: 120 trials, 120 gradable, 12 task(s) with values
  v2: 120 trials, 120 gradable, 12 task(s) with values
  tasks: 12 task(s) compared, aligned by content
  delta = +0.1763  95% CI [+0.1488, +0.2049]  p = 0.0005 (exact)  (B = 10000, seed = 0)
  readings: significant, |delta| >= threshold 0.05, interval beyond +0.05
  changed beyond trial noise (p <= 0.05 after Holm over 12 tasks): ticket-0 0.677 -> 0.858 (p < 0.0001), ticket-6 0.598 -> 0.869 (p < 0.0001), ...
  Verdict: IMPROVEMENT (exit 0)
  What changed: DecisionSpec prompts (attribution evidence, not proof of cause)
  What moved (largest first): ticket-6 +0.271 (n 10/10), ticket-1 +0.254 (n 10/10), ...

Read it top to bottom: what was compared (metric, tasks, how they were aligned), what the data say (delta, interval, p-value, and the three readings), which tasks changed beyond their own trial noise (seven of the twelve tickets, each tested on its own trials), the verdict, then what changed (from the two DecisionSpecs; here only the prompts) next to what moved (the tasks with the largest paired differences, the ones to read first). "What changed" is attribution evidence, not proof of cause.

Every field in that summary is also in --output compare.json, so a CI job can branch on the exit code and archive the record. An inconclusive comparison exits 2 on purpose: not enough evidence must never read as "no regression"; pass --observe for dashboards that only want the numbers.

When the comparison is inconclusive¶

On realistic suites most comparisons end inconclusive, so the summary also says what would decide one. Twenty pass/fail tasks at five trials a side, where the candidate broke three tasks and left the other seventeen alone, read (the case is in the unit tests):

  delta = -0.1200  95% CI [-0.2600, +0.0000]  p = 0.2464  (B = 10000, seed = 0)
  readings: not significant, |delta| >= threshold 0.03, interval reaches -0.03
  resolution: changes within about ±0.1300 (the interval's half-width) are inside this comparison's noise; ±0.03 would take about 380 tasks at this per-task spread
  changed beyond trial noise (p <= 0.05 after Holm over 20 tasks): t02 1.000 -> 0.000 (p = 0.0020)
  worth a recheck (p <= 0.1 before adjustment): t03 1.000 -> 0.200 (p = 0.0215), t00 1.000 -> 0.400 (p = 0.0618)
  Verdict: inconclusive: more runs or tasks needed (exit 2)
  Next: rerun those tasks on both versions with more trials (tracelens run ... --task-id t02 t03 t00 --num-runs 20 --save-trials <file>), then compare the two reruns

The verdict is about the mean over tasks, and three tasks that moved against seventeen that did not leave the sign-flip test only three signs to go on. The lines around the verdict say what would decide it:

resolution is the interval's half-width, about the smallest change this comparison can tell from none. It shrinks as one over the square root of the number of tasks, so the task count is a planning figure for the suite: at this spread, resolving ±0.03 takes about 380 tasks. More trials per task shrink only the trial-noise part of it; when the tasks themselves moved by different amounts, as here, only more tasks narrow it.
changed beyond trial noise tests each task on its own trials, with the gate's test for the metric (Boschloo's exact test for pass_rate, Welch's t-test for scores), Holm-adjusted over the compared tasks, so the chance of naming a task that did not change is held to 5%. t02 went from 5/5 to 0/5: that task changed, whatever the suite verdict says. The line prints for every verdict and never changes it; it is about these tasks, not tasks in general. Five trials a side cannot give a p-value below 2/1024, so on more than 25 tasks no single task can be named this way; the recheck, which adjusts only over the tasks it reruns, can.
worth a recheck lists the tasks whose own p-value is at most twice the level before the adjustment, and Next reruns them with the named ones: both versions, four times the trials (at least 20, at most 200). The screen is lenient on purpose. Only the fresh trials decide, so a task flagged by chance costs a rerun, not a false alarm.

To follow Next, run each version on those tasks and compare the two reruns:

tracelens run --config tracelens.yaml --task-id t02 t03 t00 --num-runs 20 \
  --output recheck/v1-results.json --report recheck/v1-report.md \
  --html-report recheck/v1-report.html --save-trials recheck/v1-trials.json
# switch to v2 (model, prompt, or code), then the same command with v2 paths
tracelens compare recheck/v1-trials.json recheck/v2-trials.json

A rerun is a separate run. When its settings come from a config file, point every output the file names, and its checkpoint if it sets one, somewhere else: otherwise the rerun overwrites the full run's results, and the full run's checkpoint refuses the smaller run. Three tasks are too few for a suite verdict, so the recheck ends insufficient evidence; its changed beyond trial noise line is the answer for those tasks.

Trial-level comparison with `compare_metrics`¶

compare_metrics is the older, lower-level tool: it compares two flat lists of values with an unpaired bootstrap and treats every value as an independent sample. Use it when the two samples really are independent draws (two unrelated sets of trials, latency samples), not for two runs of the same eval set, where compare_runs pairs the tasks. It returns a ComparisonResult:

from tracelens.statistics.inference import compare_metrics

res = compare_metrics(s1, s2, confidence=0.95, compute_p_value=True)

compute_p_value=True adds a permutation test (it's off by default because it's the expensive part). The result fields:

Field	Meaning
`delta`	`current.mean - baseline.mean` (here, v2 − v1)
`relative_delta`	`delta / abs(baseline.mean)`; `None` when the baseline mean is 0, and `unavailable_fields` then lists it
`ci_lower`, `ci_upper`	bootstrap 95% CI of the difference
`is_significant`	`True` when the CI excludes 0
`cohens_d`	effect size (standardized magnitude)
`p_value`	permutation-test p-value (`None` unless `compute_p_value=True`)

On the example's two score lists this reads:

  quality delta (v2 - v1) = +0.176  95% CI [+0.147, +0.206]  cohens_d=1.50  p=0.000

How to read this, in order of what matters:

95% CI [+0.147, +0.206] excludes 0 → the difference is real, not noise. This is the single most important line. If the CI had been [-0.02, +0.34], the improvement would be plausibly zero and you should not ship on it. The is_significant flag is exactly this check.
cohens_d=1.50 → the effect is large (Cohen's conventions: <0.2 negligible, <0.5 small, <0.8 medium, ≥0.8 large; exposed as res.effect_magnitude). A change can be statistically significant but tiny; d tells you whether it's worth caring about. 1.50 means the two distributions barely overlap.
p=0.000 → the permutation test agrees the distributions differ. Treat the CI as primary and the p-value as corroboration, not the other way around.

Convenience properties wrap the same logic: res.is_improvement (significant and positive) and res.is_regression (significant and negative) read better than hand-checking is_significant and delta > 0.

If the CI had straddled zero, the honest verdict is "no detectable difference at this sample size" — collect more runs before concluding either way. The full statistical toolkit is in Statistical Comparison.

Step 4: don't stop at the mean — capability vs reliability¶

A version that wins on mean_quality can still be flakier. The mean hides the distribution. Look at both axes per version:

pass@k (capability) — can this version solve the task at all within k attempts? A higher pass@k means a higher ceiling.
pass^k (reliability) — does it solve the task on every one of k attempts? A version with a better mean but a worse pass^k is more capable on a good day and less trustworthy in production.

Compute these per version and compare them side by side, not just the means. See pass@k vs pass^k for the full treatment.

Two ways to run a comparison¶

This page covers ad-hoc head-to-head: you have two configs in hand right now and you want a verdict. That's tracelens compare on two saved runs (or compare_runs on two batches).

The other mode is baseline-gated comparison for CI: you store a baseline once, then every future run is compared against it automatically and the build fails on a regression. That's the job of RegressionDetector — see the Baseline Regression Tutorial.

The two modes share the DecisionSpec machinery. RegressionDetector.compare_with_specs takes both runs' specs and can distinguish "the model or prompt changed" (an intentional, attributable difference) from "only infra changed" (resource budget, concurrency — noise you don't want to gate on). That spec-aware comparison is demonstrated in examples/noise_aware_regression.py, and explained in Reproducibility & DecisionSpec.