I took one storage optimization, changed nothing about it, and read its cost six different ways. The answers ranged from 66,895 gas to 93,885 gas, and one of the six said the optimized contract was more expensive than the contract it replaced.
That is the problem with gas benchmarks. The measurement is cheap to produce and expensive to trust.
This article is the experiment behind that claim. I built a small Foundry fixture with six paired contracts — a baseline and a supposedly optimized version of each — and measured every pair across compiler versions, optimizer settings, compilation pipelines, EVM targets, contract state, and workload size. Every number below comes from that run. The harness, the raw JSON, and the plotting code are in the repository under examples/articles/how-do-i-measure-solidity-gas-optimizations-without-misleading-benchmarks.
The short version: a gas saving is not a property of a code change. It is a property of a code change plus five settings, and if you do not state all five, your number is not reproducible.
The measurement moved 28.9% without touching the code
Start with the part that surprised me most, because it has nothing to do with Solidity.
Foundry 1.8.0 made --isolate the default for forge test. The release notes say so plainly: "Forge tests now run in isolate mode by default… The default can change gas snapshots, exact traces and gas costs, and CREATE-derived addresses." Isolation runs each top-level call as its own transaction, so accounts and storage slots are cold again rather than staying warm from earlier calls in the same test function. The Foundry book's testing guide describes it the same way: "repeated calls to the same contract can be charged as cold again under isolation."
Here is the same test file, the same contract, the same pinned build, run under both releases:
forge test, gasleft() around one call |
Foundry 1.5.1 | Foundry 1.8.3 |
|---|---|---|
| Default flags | 72,821 | 93,885 |
--isolate |
93,885 | 93,885 |
--no-isolate |
not a flag yet | 72,821 |
Upgrading Foundry changed the measured cost of an unchanged call by 21,064 gas, or 28.9%. Nothing about the contract moved. The two releases agree exactly once you name the isolation mode, which is the useful part: the disagreement is entirely about what the default means this month.
So the first rule of a gas benchmark is that the tool version and its flags belong in the write-up, next to the compiler version.
One call, six numbers
I used one pair for the method comparison. SumStorage.accumulate() walks a 16-element array and adds each item into a storage accumulator. SumCached.accumulate() reads the length and the accumulator into memory, loops, and writes back once. This is the textbook version of the optimization.
Every value is a real measurement from this experiment. Negative change means the optimized version used less gas. Solidity 0.8.37, optimizer on with 200 runs, EVM target osaka, Anvil 1.5.1 for the receipts.
Five of the six methods agree on the size of the saving to within 51 gas: the two receipts say 4,774 and the three in-test readings say 4,825. They disagree on the base, so the same change reads as a 5.1% saving or a 6.7% saving depending on where you measured it.
The sixth method is the interesting one. forge test --gas-report reported a mean of 83,883 gas for the baseline and 83,995 for the optimized contract, making the optimization look 112 gas worse.
That is not a bug. It is arithmetic. The gas report aggregates every call to a function across the whole test run, and my test file called the two contracts a different number of times in different states:
SumStorage.accumulate()was called seven times: five on cold storage, two as warm repeats.SumCached.accumulate()was called three times, all cold.
Cold calls cost 88,769 and warm repeats cost 71,669. Five cold plus two warm averages to 83,883, which is exactly what the report printed. The optimized contract's three cold calls average 83,995. The report is telling the truth about the calls it saw; those calls were simply not comparable.
The report's own spread says so: min 71,669, median 88,769, max 88,769, seven calls. A function whose min and max differ by 17,100 gas is a function measured in two different states, and neither the mean nor the median of that pool means anything.
vm.startSnapshotGas has a sharp edge too
The cheatcode pair looks like the careful option, and it is the one Foundry's gas snapshot documentation points at for CI. It also has a documented warning worth repeating: "Gas snapshots are not accurate unless using isolated test mode."
I measured three consecutive empty windows — startSnapshotGas immediately followed by stopSnapshotGas, nothing in between:
empty.1 = 0 empty.2 = 754 empty.3 = 754
With a console2.log between the windows instead of nothing, under Foundry 1.8.3 defaults:
spaced.1 = 0 spaced.2 = 6,352 spaced.3 = 6,353
The first window reads zero. Every later window in the same test function picks up the gas spent since the previous window closed. That matches an open Foundry issue describing exactly this, and I reproduced it on 1.8.3, the current stable release. The practical consequence: one snapshot window per test function, and treat a second window in the same function as unmeasured.
Pin the compiler, and check that it listened
Now the build settings. The compiler is the easy one to state and the easy one to get wrong.
Solidity's default EVM target has moved three times in the range most projects span. The changelog is explicit: 0.8.25 set the default to cancun, 0.8.30 set it to prague, and 0.8.31 set it to osaka, which is still the default in 0.8.37, released on 2026-09-10. An A/B across two compiler versions without an explicit evm_version is an A/B across two different EVMs.
Pinning it is not quite enough either, because the pin can be silently downgraded. I asked six compilers for five targets each and read back what they actually compiled for, from solc's own metadata:
| Requested target | solc 0.8.24 and 0.8.25 | solc 0.8.30 and later |
|---|---|---|
| paris, shanghai, cancun | compiled as requested | compiled as requested |
| prague | compiled as cancun | compiled as requested |
| osaka | compiled as cancun | compiled as requested |
Those builds exited 0 and printed no warning. The downgrade happens above solc, not inside it: run the same request against the compiler binary directly and it refuses.
$ solc --evm-version prague --bin T.sol
Error: Invalid option for --evm-version: prague
Foundry resolves the requested target down to something the pinned compiler supports and gets on with the build. That is a reasonable thing for a build tool to do and a terrible thing for a benchmark to inherit. The fix is one assertion: after building, read metadata.settings.evmVersion out of the artifact and compare it with what you asked for. My harness does that for every configuration it builds, and throws if the compiler, optimizer runs, pipeline, or EVM target came back different:
const probe = JSON.parse(readFileSync(resolve(out, "Pack.sol", "PackWide.json"), "utf8"));
const md = probe.metadata;
const actual = {
solc: md.compiler.version,
optimizer: md.settings.optimizer,
evmVersion: md.settings.evmVersion,
viaIR: md.settings.viaIR === true,
};
if (actual.evmVersion !== cfg.evm) {
throw new Error(`evm mismatch for ${cfg.id}: ${actual.evmVersion}`);
}
With the target held at cancun, the compiler version itself mattered less than I expected. Across 0.8.24, 0.8.30, 0.8.33, 0.8.36 and 0.8.37, five of the six pairs produced exactly the same saving, to the gas unit. The calldata pair drifted: on 0.8.24 it saved 5,260 gas on a 64-element call, and on 0.8.30 and later it saved 5,581. A 6% difference in the saving from a compiler upgrade alone.
The optimizer decides whether your optimization exists
This is where most gas folklore comes from, and where the measurements are least kind to it.
Two things are worth stating before the numbers. Foundry's configuration reference lists optimizer as false by default and optimizer_runs as 200, which means an unconfigured project compiles unoptimized — the run count is sitting there looking configured while doing nothing. And runs is not the number of optimizer passes. The Solidity optimizer documentation is blunt about the confusion: the parameter "specifies roughly how often each opcode of the deployed code will be executed across the life-time of the contract," and "the optimizer will always run as many times as it can still improve the code."
Measured on an Anvil chain at EVM target osaka with Solidity 0.8.37. A negative bar means the "optimized" version used more gas than the baseline.
Three different shapes in one chart.
Storage packing is robust. Squeezing three fields into one slot saved 49.0% to 50.0% of a first write in every configuration. Optimizations that remove a SSTORE do not depend much on how the code around them was compiled, because the EVM's storage prices dominate everything else.
The calldata change is an optimizer artifact. Declaring an array parameter calldata instead of memory saved 22.4% on a 64-element call with the optimizer off, 7.1% with the optimizer on at 200 runs, and exactly 0 gas under via-IR — 44,260 gas either way, to the unit. Report the 22.4% and you are reporting your optimizer settings, not your code change.
The unchecked loop counter saved nothing at all, and under via-IR the version with unchecked was 192 gas more expensive over 64 iterations.
That last one deserved a second look, because "wrap the loop counter in unchecked" is probably the single most repeated gas tip in Solidity. So I hashed the deployed bytecode. With the optimizer on at 200 runs, these four contracts compile to byte-identical deployed bytecode:
for (uint256 i = 0; i < n; i++)for (uint256 i = 0; i < n; ++i)- the same loop with
unchecked { ++i; }in the body - and the
i++variant again under a different contract name
The compiler produces one program for all four spellings. There is no saving to measure because there is no difference to measure. With the optimizer off the four collapse into two groups rather than one, and the unchecked version is byte-identical to the plain ++i version.
Before concluding that unchecked does nothing, I checked that the effect exists at all. A checked and an unchecked a + b that the compiler cannot bound do compile differently, and with the optimizer off the unchecked version cost 179 gas less per call.
With the optimizer on, that control measured a saving of exactly zero — for a reason that turns out to be the next section's whole point.
The EVM target is a chain rule, not just a compile flag
Both versions of that control transaction cost precisely 21,860 gas. That number is not an execution cost. It is the EIP-7623 calldata floor: 21,000 intrinsic plus 10 gas per calldata token, and the call passes 62 zero bytes and 6 non-zero bytes, which is 86 tokens. 21,000 + 860 = 21,860.
EIP-7623 shipped with Pectra in May 2025. It charges a transaction the larger of its normal cost and a floor derived from its calldata, so a transaction that carries a lot of data and executes very little pays for the data regardless. Any execution saving below that floor is invisible on-chain. It is real in a gasleft() reading and worth nothing to a user.
The calldata optimization runs straight into it. Here is one artifact and one transaction, sent to a fresh local chain under five hardfork settings and two Anvil releases:
Anvil --hardfork |
Anvil 1.5.1 | Anvil 1.8.3 | EIP-7623 floor |
|---|---|---|---|
| paris | 42,089 | 44,260 | 44,260 |
| shanghai | 42,085 | 44,260 | 44,260 |
| cancun | 42,085 | 44,260 | 44,260 |
| prague | 44,260 | 44,260 | 44,260 |
| osaka | 44,260 | 44,260 | 44,260 |
Read the 1.5.1 column first, because it is the one that matches the specification. Under Cancun the optimized call costs 42,085 and the saving reads 11.7%. Under Prague and Osaka it lands exactly on the floor at 44,260 and the saving reads 7.1%. The deployed bytecode is identical across shanghai, cancun, prague and osaka — I hashed it — so the entire difference is the chain, not the compiler.
Benchmarking a calldata optimization against a Cancun chain in 2026 therefore overstates it by roughly two thirds, and mainnet has been past Prague since Pectra and past Osaka since Fusaka activated in December 2025.
Now read across. Anvil 1.8.3 applied the calldata floor under every hardfork flag I gave it, including forks that predate the rule by years. These were clean local nodes with no fork URL, so nothing was inferring a hardfork from a remote block header. A search of the Foundry issue tracker turned up no report of this, and I am describing what I measured rather than diagnosing it. The point for a benchmark is narrower and harder to dodge: --hardfork cancun did not mean the same thing to the two binaries. If your historical gas numbers came from a local chain, the local chain's version is part of the result.
State is a setting, and so is workload
Two more axes, both easy to state and easy to forget.
State. A storage write's price depends on what was in the slot. Packing three fields into one slot saved 44,040 gas on the first write, 9,840 on an overwrite, and 7,258 on a write that clears the fields back to zero. Same contract, same function, same arguments shape — three savings, from 24.6% to 49.9%, chosen entirely by which write you decided to benchmark.
Foundry gives you the controls for this. vm.cool(address) marks an account and its slots cold again; under a non-isolated 1.5.1 run, a repeated call read 12,415 gas warm and 48,406 gas after vm.cool, against 72,821 for the first call in a fresh test. Isolation gets you closer to a transaction boundary than any of them: the same repeat read 72,279 under --isolate.
Workload. The cached-accumulator optimization saved 94 gas over one element and 19,750 over 64 — 0.2% and 9.8% of the call. Both are true. Only one of them is relevant, and which one depends on the array length your users actually pass.
The same question applies to deployment. Storage packing cost 30,304 more gas to deploy and 140 more bytes of runtime code than the unpacked version. It pays for itself after one write, so the trade is obvious here, but it is a trade, and a benchmark that reports only the call is only reporting half of it.
Custom errors are the mirror image. Replacing a require string with a custom error saved 14,056 gas at deployment and 65 bytes of code size, saved 61 gas on the reverting path, and saved exactly zero on the successful path. A gas report that only exercises the happy path will conclude the change did nothing.
One more piece of honesty about that packing result: the version that saves 44,040 gas truncates uint256 inputs into uint128 and uint64 fields without checking them, which is not code I would ship. The version with the three range checks a production contract needs costs 44,393 gas on a first write — 102 gas more than the unsafe version, and still 43,938 gas below the baseline. Small, but it is the difference between a benchmark and a proposal.
What I would pin, and what I would write down
The settings that changed an answer in this experiment, in the order they bit me:
- The test tool and its isolation mode. Worth 28.9% on this workload, between two releases of the same tool.
- The measurement method. Receipt,
gasleft(), snapshot cheatcode, or gas report — and if it is the gas report, the call counts and states behind every column. - The compiler version, optimizer state, run count, and pipeline. Assert the artifact's metadata rather than trusting the config.
- The EVM target and the chain that executes it, including the local node's version.
- The state and the workload, stated as numbers, not as "a typical call."
Then report absolute gas and the percentage together, with the base you divided by. "Saves 4,825 gas on a 16-element call" survives a re-run. "6.6% faster" does not, because the denominator is a setting.
The habit that made the rest of this tractable was mechanical: every configuration builds into its own out/ and cache/ directory, gets its compiler settings verified from metadata, and runs against a chain started on the matching hardfork. Then run the whole thing twice. Both passes here were byte-identical JSON, which is what lets me claim the differences above are the settings rather than the noise. If your two passes differ, you have found something to fix before you have found an optimization.
This connects to two things I have written before. The gas number is only half of the cost question — the real cost of running an Ethereum app covers what those units are worth. And the discipline here is the same one in how do I publish reproducible benchmarks for open-source developer tools: the setup is the result. Once a contract is heading to mainnet, gas is one line in a longer list — from Solidity prototype to production is the rest of it.
Reproduce it
Foundry 1.5.1 and Node.js 22.17.0, from the fixture directory:
cd examples/articles/how-do-i-measure-solidity-gas-optimizations-without-misleading-benchmarks
node scripts/methods.mjs --tag=local # one call, every measurement method
node scripts/evm-target-probe.mjs # what each compiler actually targeted
node scripts/floor-probe.mjs local # the EIP-7623 floor, five hardforks
node scripts/unchecked-control.mjs # the unchecked positive control
for s in compiler optimizer viair evm; do node scripts/measure.mjs $s --pass=1; done
node scripts/summary.mjs optimizer 1 # readable digest of one sweep
Each sweep writes results/<sweep>-pass<n>.json with the configuration, the compiler metadata read back from the artifacts, per-contract deployment and call gas, and the machine description. The figures above are generated from those files by scripts/figures.mjs, so a re-measurement moves the charts with it.
What this does not settle
This is six small paired contracts, not a protocol. Nothing here touches proxies, libraries, inheritance depth, ERC-20 transfer paths, or a contract large enough for the optimizer's run count to change inlining decisions in interesting ways. A 1,000-line contract may well show optimizer effects that a 20-line one cannot.
The measurements are single deterministic observations, not distributions, which is appropriate because gas is deterministic — but it also means nothing here speaks to compilation time, and via-IR's compile cost is a real trade this experiment did not weigh.
I did not re-measure the transaction-level sweeps under Anvil 1.8.3. I tried: the compiler sweep completed and is the source of the two-version comparison above, but the remaining sweeps stalled when Anvil 1.8.3 accepted a transaction, consumed the nonce, and never mined it. Short probes and cast send against the same node worked, so the chain-level numbers for the optimizer, pipeline, and EVM sweeps come from Anvil 1.5.1 only.
Finally, everything above is a local chain. A real transaction also pays a base fee that is not in any of these numbers, competes for block space, and may hit an access list you did not model. The floor result is the reminder: the rules that decide what a saving is worth live on the chain, not in the compiler.