An AI speed-up claim is a prediction until you measure it

When a coding agent says a change makes your code faster, measure it on the commit before and after the change. Run the same realistic workload several times on one machine, and compare latency percentiles, memory and CPU time. The agent can write and run that benchmark, and the claim stands when the numbers from those runs back it.

The tool behaviour below comes from each tool’s official docs, last verified on 11 October 2026; none of it was tested for this post. This is part 19 of the series.

What a performance claim needs before you accept it

A performance claim from a coding agent is a prediction until something measures it. An agent can read a loop, spot a cheaper pattern and predict a speed-up without running anything. Nine checks turn that prediction into a number you can repeat:

Start from a baseline and a workload that looks like production

A baseline is the measurement of the code before the change. Without one, “faster” has nothing to be faster than. Run the benchmark on the commit before the change, save the raw output to a file, and record next to it:

• the commit
• the runtime and its version
• the machine
• the exact command and the input

Keep both commits checked out side by side if that helps, for example in two Git worktrees. Part 13 covers worktrees.

The workload decides what the numbers mean. A benchmark on a tiny input tells you little about the large one your users send. Match production on the things that change cost: input size, the shape of the data, how many requests arrive together, and whether caches start warm or cold.

Cache state matters even for a command-line tool. The hyperfine README says that for programs doing a lot of disk I/O, results “can be heavily influenced by disk caches and whether they are cold or warm”. It uses --warmup for warm-cache runs and --prepare to clear caches before each cold run. Pick one state, and use it on both sides.

Profile before anyone optimises

A profiler shows where a program spends its time, so the change goes where the time is. Python has a sampling profiler for that job. Its profiling.sampling docs say: “Statistical profiling excels at answering the question, “Where is my program spending time?”” A profile is evidence, in the same sense as the stack traces and logs in Part 9.

A profiler doesn’t measure the size of a speed-up well. The same docs warn that when two versions differ by a small amount, “sampling noise can obscure real differences”, and point to timeit for micro-benchmarks. Profile to choose the change; benchmark to measure it.

Profilers and benchmark tools by language

Most languages have their own profiler and benchmark harness. A few, from their docs:

Each row comes from that tool’s official docs, last verified on 11 October 2026; none was tested for this post. BenchmarkDotNet’s docs add that it “will only run benchmarks if the application is built in the Release configuration.”

Benchmark the before and after the same way

A fair benchmark runs both versions under the same conditions, enough times to see the noise. For a command-line program, hyperfine runs both commands and compares them. An example, with made-up paths for the two builds:

# Example: warm-up runs, a fixed run count, raw results saved
hyperfine --warmup 3 --runs 20 --export-json results.json \
  './before/report --input data/sample.csv' \
  './after/report --input data/sample.csv'

The first command is the reference: per the README, “each subsequent result shows its change from that reference”. From version 2.0.0, hyperfine runs commands without a shell by default, per its changelog. Use -S for pipes or wildcards.

For Go, a benchmark lives next to the tests. An example, where BuildReport and loadSample stand in for your own code:

// Example: setup before the loop is not timed
func BenchmarkBuildReport(b *testing.B) {
	rows := loadSample("testdata/sample.csv")
	for b.Loop() {
		BuildReport(rows)
	}
}

The testing docs say b.Loop resets the timer the first time it is called, so the setup “does not count toward the benchmark measurement”. For example, run it 10 times on each commit, then compare:

go test -run='^$' -bench=BuildReport -benchmem -count=10 > old.txt
# switch to the new commit, then:
go test -run='^$' -bench=BuildReport -benchmem -count=10 > new.txt
benchstat old.txt new.txt

Benchstat shows the median of each side with a confidence interval, and a ~ when it “did not detect a statistically significant difference”. Its tips cover the conditions:

• Run on “an otherwise idle machine, ideally one that isn’t running on battery”.
• “interleave before and after runs”, so noise falls evenly on both sides. For Go, the tips suggest pre-compiling each benchmark binary with go test -c to speed this up.
• “Pick a number of benchmark runs (at least 10, ideally 20) and stick to it.”

Benchstat warns against “simply rerunning your benchmarks until it reports a significant change”. An agent told to prove a speed-up can do exactly that, run after run, until the numbers agree with it. Fix the run count in the prompt.

Benchstat also notes that “statistically significant” is not the same as “large”. A real difference can still be too small to matter.

The harness can hide part of the cost

Every harness makes choices, and a benchmark the agent writes inherits them. Python’s timeit docs say it “temporarily turns off garbage collection during the timing”. If garbage collection is part of the cost, the same page shows how to turn it back on in the setup string. The JMH README says “Your benchmarks should be peer-reviewed”, and “Do not assume that a nice harness will magically free you from considering benchmarking pitfalls.”

Review an agent’s benchmark like any other code. Does it call the code that changed? Is setup outside the timed part? Do both sides get the same input?

Compare latency percentiles, memory and resource use together

A single average hides the slow requests. Google’s SRE book says “a high-order percentile, such as the 99th or 99.9th, shows you a plausible worst-case value”. The 50th percentile, the median, “emphasizes the typical case”. It also warns that “we cannot assume that the mean and the median are the same”.

Report the median and p95 or p99 for both versions, side by side. Then add the costs a speed-up can quietly move somewhere else:

• peak memory, from hyperfine, tracemalloc or -benchmem
• allocations, which feed garbage collection
• CPU time, which can grow while wall-clock time shrinks
• error rate under load

A cache that lowers the median and raises peak memory is a trade-off. The requirement decides whether it is a good one.

For a service, a load-testing tool reports percentiles directly. Grafana k6’s thresholds are “the pass/fail criteria that you define for your test metrics”. An example, with made-up limits:

// Example: the run fails if p95 latency or the error rate is too high
export const options = {
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<200'],
  },
};

Per the same page, if a threshold fails, “k6 would exit with a non-zero exit code”. That makes a threshold usable as a performance budget in CI. Part 10 covers blocking gates.

For a web page, Lighthouse scores move between runs. Its variability docs say “The median Lighthouse score of 5 runs is twice as stable as 1 run”. They also say not to collect several reports “at the same time on the same machine”.

Two prompts for performance work with a coding agent

Neither prompt relies on a tool-specific feature. The agent writes and runs the benchmark; every number in its answer has to come from a run.

Prompt 1: profile and baseline before optimising

An example prompt for the start of performance work, before any code changes.

Task: <describe the slow code path or the performance goal>
Workload: <input size, data shape, concurrency, warm or cold cache>

Measure before changing any code.
1. Find the project's benchmark for this code path, or write
   one with the language's standard harness.
2. Run it on the current commit: warm-up runs first, then
   at least 10 measured runs. Save the raw output to a file.
3. Record the commit, runtime version, machine and command.
4. Profile the same workload. List the top functions by
   time, and save the profile file.
5. Propose one change aimed at the top hotspot. Say which
   number it should move, and which numbers must not get worse.

Do not change application code yet.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or
make external changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the
failure rather than repeatedly guessing.

Prompt 2: check a performance claim

An example prompt for after the change, when the agent says it is faster.

Task: check this performance claim: <paste the claim>
Baseline: <the commit and the saved results from before>

1. Run the same benchmark, workload and command on the new
   commit, on the same machine, with the same run count.
   Interleave before and after runs if the harness allows.
2. Report for both versions, side by side: median, p95 or
   p99 if available, spread, peak memory and CPU time.
3. Paste the exact commands and the raw output.
4. Say whether the difference is larger than the run-to-run
   noise. If not, report "no measurable change".
5. Confirm the benchmark calls the changed code, and that
   the tests still pass.

Every number must come from a run. Do not estimate.
Do not rerun the benchmark until the result looks better.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or
make external changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the
failure rather than repeatedly guessing.

Read the raw output yourself before you repeat any number from the report.

A performance claim checklist

An example checklist for any pull request that claims a speed-up:

PERFORMANCE CLAIM CHECKLIST

□ Baseline measured on the commit before the change
□ Raw output saved, with commit, runtime, machine and command
□ Workload matches production in size, shape and cache state
□ Profile shows where the time goes before optimising
□ Warm-up runs before the measured runs
□ Run count fixed in advance, at least 10
□ Same idle machine for both sides, runs interleaved
□ Spread reported, and the difference beats the noise
□ Median and p95 or p99, not the mean alone
□ Peak memory, allocations and CPU time compared
□ Benchmark calls the changed code, setup not timed
□ Tests still pass after the change
□ A performance budget in CI for the paths that matter

Quick answers

How do you check if AI-generated code is actually faster?

Checking AI-generated code for a speed-up means running the same benchmark on the commit before and after the change. Use a realistic workload, warm-up runs and at least 10 measured runs on one idle machine, then compare latency percentiles and peak memory.

Should you profile or benchmark first?

Profiling comes first, because a profile shows where the program spends its time and so where a change can help. A benchmark comes next, to measure how much the change moved the numbers. Python’s docs suggest timeit for measuring small differences.

How many benchmark runs are enough?

Benchmark runs should number at least 10 per side, ideally 20, chosen in advance, according to Go’s benchstat docs. Hyperfine runs at least 10 by default. Rerunning until a difference looks significant is what benchstat calls “multiple testing”, a common statistical error.

Why use p95 or p99 latency instead of the average?

An average latency hides the slow requests in the tail. Google’s Site Reliability Engineering (SRE) book says a high percentile such as the 99th “shows you a plausible worst-case value”. It also warns that the mean and the median can differ.

Can a coding agent write its own performance benchmark?

A coding agent can write and run a benchmark, and save time doing it. The benchmark still needs a review, for example whether it calls the changed code and keeps setup out of the timing. The numbers must come from the agent’s runs.

More on AI coding is in the AI category.