Seven one-line defects in one staking pool. The example tests caught one of them. The fuzz tests caught three. The invariants caught three. Two defects were found by two selections, three by exactly one, one by none of the three, and one only after I handed the pool a token that calls back into it.
The contract also had a defect I did not put there. The invariant suite found it before I had written a single mutation, and it was worth exactly one wei.
That is the short answer to the question in the title: these are not three levels of thoroughness, they are three different searches, and the one you skip is the one that decides what you ship with.
The pool and the three suites
The contract is a staking pool with pull-based reward accounting: stake, unstake, claim, an owner-only fund that shares new rewards across current stakers, and an owner-only skim that recovers tokens sent to the pool by mistake. Rewards are distributed by raising a per-share accumulator, and each account stores the share it has already been accounted for, so nobody has to iterate over holders. It is the same shape as the MasterChef accounting that most staking contracts were copied from.
121 lines of Solidity, three functions that move tokens, one that moves the index. Then three suites against it — 293 lines of tests plus 38 lines of shared setup:
- 14 unit tests. Deposits, proportional splits, a late staker earning nothing from the past, the revert conditions, the
Claimedevent, and the two owner-only functions refusing a stranger. - 5 fuzz properties. The same behaviour stated as properties — a full unstake returns exactly the principal, two stakers split a funding proportionally, nothing pays out more than was funded, a partial unstake settles rewards,
skimonly moves donations — each run 256 times with generated numbers. - 4 invariants, plus a handler. Solvency,
totalStakedmatching the sum of stakes, claimed plus claimable never exceeding funded, and a reward index that never goes backwards. The handler stakes, unstakes, claims, funds, donates and skims on behalf of three actors, with every amount bounded so that the call is legal, following the handler pattern in Foundry's own documentation.
Everything runs on Foundry's defaults: 256 fuzz runs, 256 invariant runs at a depth of 500, fail_on_revert off. Those defaults are identical in Foundry 1.5.1 and in the current 1.8.1 release. The numbers below are from Foundry 1.5.1 with solc 0.8.36, the optimizer at 200 runs, on an Apple M3 Pro. The project, the harness and the raw results are in the repository under examples/articles/unit-tests-vs-fuzz-tests-vs-invariant-tests-what-should-i-use-for-solidity.
What each selection varies, and how much of the space one run covered.
The budgets are worth sitting with. The unit suite executes 14 fixed call sequences. The fuzz suite executes 1,282 generated inputs through 5 call sequences I wrote — Forge reports 257 runs rather than 256 for two of the five properties. The invariant suite executes 128,000 calls in an order nobody wrote down. Four orders of magnitude end to end, and a different thing being varied at each step.
The invariant suite found a defect before I injected any
Before writing any mutations I ran all three suites against the contract as written. The unit and fuzz suites passed. The invariant suite failed:
[FAIL: claimed plus claimable exceeds what was funded: 67352459882248805486807 > 67352459882248805486806]
One wei. Forge shrank the run to six calls:
calldata=stake(uint256,uint256)
calldata=fund(uint256)
calldata=stake(uint256,uint256)
calldata=fund(uint256)
calldata=stake(uint256,uint256)
calldata=fund(uint256)
The arguments are elided here because they are the handler's raw fuzz inputs, which bound maps onto legal amounts before the pool ever sees them. The shape is the whole story: stake, fund, stake, fund, stake, fund.
The cause was one line:
return (stakeOf[account] * accRewardPerShare) / PRECISION - rewardDebt[account];
Dividing before subtracting means both terms get truncated separately, and floor(a + b) - floor(a) can be one larger than floor(b). Once an account's debt was recorded at a non-zero index, the next funding could credit it one wei more than had been deposited. Subtracting first and dividing once fixes it:
function pending(address account) public view returns (uint256) {
return (stakeOf[account] * accRewardPerShare - rewardDebt[account]) / PRECISION;
}
That divide-then-subtract expression is not an invention for this article. It is the shape in MasterChef — user.amount.mul(accSushiPerShare).div(1e12).sub(user.rewardDebt) — and in the many pools copied from it. Whether one wei per settlement matters depends on the contract; in a pool whose last withdrawal has to be covered by the balance, it is the difference between a final transfer succeeding and reverting.
What I want to draw out is why only one selection could have found it. I wrote a JavaScript model of the same accounting and generated 300,000 random call sequences at each length. No sequence of three or four calls over-distributes at all. At five calls, ten in 300,000 do.
Left: how often a random sequence of each length triggers the defect, in a model of the same accounting. Right: how often the real invariant suite caught it, by configured depth, over 20 seeds.
A unit test finds this only if you sit down and write stake, fund, stake, fund, stake, fund with values that do not divide evenly. A fuzz test finds it only if you write that six-call shape by hand and then let the generator pick the numbers. The invariant runner found it because it was writing the shapes itself.
Seven defects, one line each
Then I broke the contract deliberately. Seven mutations, each a single unique string replacement in the source, applied one at a time, with every suite run against each one under 20 different fuzz seeds.
| Defect | Unit | Fuzz | Invariant | With a hostile token |
|---|---|---|---|---|
M1 skim is callable by anyone |
caught | missed | missed | missed |
M2 skim treats staked tokens as excess |
missed | caught | caught | missed |
M3 fund rounds the reward index up |
missed | missed | missed | missed |
M4 stake credits the new stake for the old period |
missed | missed | caught | missed |
M5 unstake keeps the pre-unstake reward debt |
missed | caught | caught | missed |
M6 unstake pays out before updating state |
missed | missed | missed | caught |
M7 unclaimedRewards ignores what was already paid |
missed | caught | missed | missed |
Every cell in that table is the same across all 20 seeds: 20 out of 20 or 0 out of 20, never anything in between. At Foundry's default budgets, none of these results was a coin flip. That surprised me more than any individual row, and it is worth knowing before you reach for a bigger runs number: when a generated search misses, it usually misses every time, because the input region that fails is empty rather than rare.
Three defects, three different selections
M1 removes onlyOwner from skim, so anyone can send the pool's recoverable balance anywhere. The unit suite caught it on the one test written for exactly that:
function test_skim_revertsForNonOwner() public {
token.mint(address(pool), 7e18);
vm.prank(alice);
vm.expectRevert(RewardPool.NotOwner.selector);
pool.skim(alice);
}
Neither generated selection noticed, and the reason is structural rather than statistical. The fuzz properties are about arithmetic, so none of them varies the caller. The handler calls skim as the owner, because that is what a handler written to make legal calls does — and skim moving only donated tokens breaks no invariant even when a stranger triggers it. Authorisation is not a property of the pool's state; it is a property of who was allowed to change it.
That is not an argument that only unit tests can check authorisation. Adding one fuzz test with a generated caller catches it too:
function testFuzz_skim_revertsForNonOwner(address caller) public {
vm.assume(caller != owner);
token.mint(address(pool), 7e18);
vm.prank(caller);
vm.expectRevert(RewardPool.NotOwner.selector);
pool.skim(caller);
}
M4 settles rewards after increasing the stake, so staking again pays you for a period your new tokens were not staked. This one needs three calls in a particular order — stake, fund, stake — and nothing in the fuzz suite has that shape. The invariant suite caught it on every seed, through two properties at once: claimed plus claimable exceeds what was funded, and the pool no longer covering its stakes. Writing the three-call shape as a fuzz test catches it too. What mattered was not "generated versus written" but who chose the order.
M7 makes unclaimedRewards report the total funded instead of the amount still owed, which is a view function nothing but skim reads. The fuzz suite caught it on every seed. The invariant suite, on every seed, did not — and the way it failed to is the most useful thing in this experiment.
Under M7, skim either under-reports the excess or underflows and reverts. The invariant runner's default is fail_on_revert = false, so a reverting call is discarded and the run continues. On the unmutated contract my handler produces zero reverts in 128,000 calls, on all twenty seeds. Under M7 it produces about 9,500 — 7.5 percent of the campaign, silently thrown away — and all four invariants still pass, on every seed. The suite is green and one call in thirteen is being dropped on the floor.
M5 makes the same point from the other side. It corrupts an account's reward debt after a partial unstake, which bricks every later interaction for that account: about 94,000 calls out of 128,000 revert, 73 percent of the campaign. Two of the four invariants still passed. The two that failed did so because the invariant function itself panicked while reading pending, not because a property was evaluated and found false. Either way, the number to watch in an invariant run is not the assertion, it is the revert count.
The two defects the three suites missed
M3 rounds the reward index up instead of down in fund. Every selection passed, on every seed, and my first reaction was that I had found the article's punchline. The truth is duller and more useful: at the sizes the tests use, M3 is not a defect at all.
The index is scaled by PRECISION = 1e27. Rounding it up adds at most one unit, and an account only gains a wei from that extra unit once its stake exceeds PRECISION — a billion tokens at 18 decimals. My fuzz properties bound stakes to 1e23, a hundred thousand tokens, which looks generous and is nowhere near the threshold. Running the arithmetic directly over 200 reward amounts: at a stake of 1e26 the ceiling costs nothing at all, at 1e28 it costs at most 10 wei per funding, at 1e30 at most 1,000.
Raising the bounds is all it takes. The same property, the same assertions, one changed range:
a = bound(a, BIG_MIN, BIG_MAX);
b = bound(b, BIG_MIN, BIG_MAX);
reward = bound(reward, 1, MAX);
That test fails on M3 immediately. bound is not a formality to keep the fuzzer out of revert territory; it is the definition of the space being searched, and a defect outside it is invisible rather than unlikely.
M6 moves the token transfer in unstake ahead of the state updates. Textbook checks-effects-interactions violation, and a plain ERC-20 makes it completely inert: no callback, no reentry, nothing to observe. All three suites passed on every seed, which is the correct result for the world those suites live in.
So I changed the world. A token that calls its recipient — ERC-777 and ERC-1363 both do — plus an attacker contract that re-enters unstake once from the callback:
function onTokensReceived(uint256) external override {
if (!attacking) return;
attacking = false;
try pool.unstake(reenteredWith) {} catch {}
}
On the fixed contract the re-entrant call reverts and the attacker ends with exactly its own stake. On M6 it withdraws twice, and the pool is left holding nothing against a second staker's balance. Nothing about that test is a unit test, a fuzz test or an invariant test. It is a fourth question — what happens when a counterparty misbehaves — and none of the three selections asks it, because all three supply the same honest mock.
What each selection cost
Median wall clock of the whole command, over 15 measured runs after two discarded warmups, with the order rotated each round:
| Selection | Foundry 1.5.1 | Foundry 1.8.1 | What it executed |
|---|---|---|---|
| 14 unit tests | 182 ms | 110 ms | 14 fixed call sequences |
| 5 fuzz properties | 193 ms | 127 ms | 1,282 generated inputs |
| 4 invariants | 7.38 s | 15.13 s | 128,000 generated calls |
| 1 hostile-token test | 224 ms | 110 ms | 1 fixed call sequence |
The fuzz suite costs 11 milliseconds more than the unit suite while running ninety times as many test bodies. That is the part people underestimate: once you are inside one process and one compiled EVM, generated inputs are nearly free, and there is no good reason to leave a contract's arithmetic untested by a property.
The invariant campaign is a different order of cost — 7.38 seconds, roughly forty times either of the others. Two of these selections fit inside a save-and-rerun loop and one does not, and the one that does not is the one that found the defect I had not planted. That is an argument for running it in CI and on demand rather than for skipping it.
The current release makes that worse before it makes it better. On Foundry 1.8.1 the unit and fuzz suites are 1.7 and 1.5 times faster, while the same invariant campaign takes 15.13 seconds, 2.05 times as long as on 1.5.1. Isolate mode, which is on by default in 1.8.1 and off in 1.5.1, accounts for 870 milliseconds of that; holding it off on both, the campaign is 14.2 seconds against 7.8. I did not chase the rest, and the coverage-guided invariant fuzzing added in 1.7.0 is the obvious suspect. None of it changed a verdict: I re-ran the entire matrix on 1.8.1 and every cell matched.
When I benchmarked Foundry against Hardhat 3 last month, the closing caveat was that finishing the same number of runs faster does not automatically mean finding more vulnerabilities. This is the other half of that sentence. These selections differ by a factor of forty in cost, and the expensive one is not forty times better — it is looking somewhere else.
How I would choose
The measured answer is not a ranking. It is that each selection has a blind spot the others do not, and the blind spots are structural, so you can predict them:
Write unit tests for everything that is not a property of the state. Authorisation, revert conditions, event payloads, and the one or two examples that make the accounting legible to the next person. Generated tests are bad at these because there is nothing to generate: the interesting input is the caller, and the interesting assertion is a specific value you had to decide on.
Write fuzz properties for anything arithmetic, and treat the bound call as the specification it is. Every defect my fuzz suite caught was a defect in a number, and the one it missed was outside its range. If the contract will hold a billion tokens, fuzz it with a billion tokens.
Write invariants when the bug you are afraid of needs several calls to appear. That is where they earned their keep here: the reward-debt truncation needed at least five calls, M4 needed three in a specific order, and no amount of extra fuzz runs would have produced either shape. Then read the revert count. A campaign that discards 7 percent of its calls is not testing what its green tick suggests.
Then ask what your tests assume about everyone else. Every suite here used a well-behaved ERC-20, and that assumption hid a reentrancy bug from all three of them.
If you only have time for one, it depends on what the contract is. For something that mostly computes, fuzz properties find the most per line written. For something that mostly accumulates state across many users — which is most of DeFi — the invariant suite is the one that finds the bug you would not have thought of, provided you spend the effort on the handler rather than on more runs.
What this does not settle
I chose the seven defects, and I chose them to separate categories rather than to sample real bugs. A different seven would produce a different table. The honest version of this experiment is mutation testing, where a tool generates the defects; Foundry's documentation now describes a forge test --mutate MVP, but it is not in the 1.8.1 release I measured against, so every mutation here is one I wrote.
I also wrote all three suites, which makes each of them exactly as good as I am. A sharper unit suite would have caught M2 with one extra line of setup, and it would have been reasonable to write it that way; the flip tests in the repository show precisely which cells move with one added test.
Everything here is one contract, one handler, and Foundry's default budgets on one machine. The invariant results in particular are a property of my handler as much as of the runner: it bounds every call to be legal, which is why the control produces zero reverts, and a different handler would move several rows. And Hardhat 3 runs Solidity fuzz and invariant tests too — its testing guide calls Solidity tests "the only place where you can use built-in fuzzing and invariant testing" — so the selection question is not Foundry-specific even though this measurement is.
Finally, none of this measures the thing that determines most outcomes: the properties you think to state. Every defect that got caught here was caught by an assertion I had already written down. The invariant suite found the one bug I did not plant precisely because "never distribute more than was funded" was a sentence I wrote before I knew whether it was true.