back

Rick ·

From Solidity prototype to production: the checks that matter

Dev tooling

A green test suite is the weakest claim on a release checklist. It is also usually the only one a team can actually produce.

So I measured how weak it is. I built one small payout vault, wrote the suite a careful team would write — 21 unit tests, three fuzz tests, three stateful invariants, 98.51% line coverage on the contract — and then broke it twelve times, one line at a time. Each broken copy went through nine checks: the four you can run without a chain, and five that only exist once something is deployed.

Eight of the twelve defects passed the entire test suite.

Every one of the thirteen versions, including all twelve broken ones, deployed successfully.

And the check teams argue about most — the coverage threshold — caught nothing at all.

Which release check is the first to notice each defect. A matrix of thirteen rows, one per version of the contract, against nine columns: G1 unit tests, G2 fuzz tests, G3 invariant campaign, G4 coverage gate, G5 fork rehearsal, G6 config attestation, G7 bytecode attestation, G8 monitor, G9 incident drill. A vertical rule separates the first four columns, labelled a green test suite, from the last five, labelled after the suite is green. The control row with no defect injected passes all nine. D1, fee divides before it multiplies, is caught first by G2. D2, claim forgets to decrement totalOwed, by G3. D3, allocate loses its onlyAdmin modifier, by G1. D4, plain ERC-20 transfer with a bool return assumed, and D5, minAllocation hardcoded to 1e18, and D6, deploy leaves admin as the deployer EOA, and D7, deploy swaps admin and guardian, are each caught first by G5, the fork rehearsal. D8, deploy hardcodes 500 basis points where the config says 50, is caught first by G6. D9, deployed with optimizer_runs set to 1000, by G7. D10, claim is outside the pause, by G9. D11, the Claimed event stops indexing the account while the test keeps its own copy of the declaration, by G1. D11i, the same change where the test reuses the contract's declaration, by G8, the monitor. The coverage column contains no failures: it passes in nine rows and is blocked in the four rows where a test already failed, because forge coverage re-runs the failing suite.

The vault, and the property that matters

Everything below runs against PayoutVault, a 155-line contract that holds one ERC-20. Sponsors fund it, an admin allocates what a beneficiary is owed net of a fee, beneficiaries claim, and a guardian can pause every path that moves value.

It has one property worth stating in a sentence: the vault always holds at least what it owes.

    /// @notice The money property: the vault always holds at least what it
    /// owes. This is the same expression the post-deploy monitor evaluates.
    function invariant_solvent() public view {
        assertGe(
            token.balanceOf(address(vault)),
            vault.totalOwed() + vault.accruedFees(),
            "vault holds less than it owes"
        );
    }

That expression appears three times in this project on purpose: in the invariant campaign, in the post-deploy attestation, and in the production monitor. One property, checked before deployment, at deployment, and continuously afterwards. It is the cheapest thing in the whole exercise and the only one that spans all three.

The versions under test: Foundry 1.5.1, solc 0.8.36 pinned, Node.js 22.17.0, and viem 2.44.0, on an Apple M-series laptop. The deployment target is Ethereum mainnet, and the rehearsal target is a local Anvil fork of mainnet pinned to block 25,984,000. The real token is USDT. The contract, the harness, the raw results and the verifier that rebuilds every number in this article live in the repository behind this site under examples/articles/from-solidity-prototype-to-production-the-checks-that-matter/.

Nothing here touched a public network with a transaction, a real key, or real funds. The only private keys involved are Anvil's published development keys, on a chain that exists for the length of one run.

The twelve defects

Each defect is one realistic mistake, applied as an exact string replacement to a generated copy of the project, so an interrupted run cannot leave a mutant on disk. The harness asserts each target text appears exactly once before it touches anything.

what changed class first check that failed
D1 (amount / 10_000) * feeBps instead of (amount * feeBps) / 10_000 arithmetic fuzz tests
D2 claim() no longer decrements totalOwed accounting invariant campaign
D3 allocate() loses its onlyAdmin modifier access control unit tests
D4 plain token.transfer with a bool return assumed token integration fork rehearsal
D5 minAllocation hardcoded to 1e18 token integration fork rehearsal
D6 deploy script passes msg.sender as the admin deploy configuration fork rehearsal
D7 deploy script swaps the admin and guardian arguments deploy configuration fork rehearsal
D8 deploy script hardcodes 500 bps; the config says 50 deploy configuration config attestation
D9 deployed with optimizer_runs=1000; the repository pins 200 build drift bytecode attestation
D10 claim() is outside the pause incident response incident drill
D11 Claimed stops indexing the account; the test keeps its own copy of the event observability unit tests
D11i the same change, with the test reusing the contract's declaration observability monitor

Two passes of the full ladder — 117 cells — produced identical verdicts, so nothing below rests on a result that moves between runs.

The four checks that need no chain

The first three gates behave the way the textbooks promise, and each one catches something the others miss.

forge test --match-test ^test_
forge test --match-test ^testFuzz_
forge test --match-test ^invariant_
forge coverage --report summary

The unit suite catches D3. Removing onlyAdmin from allocate() makes test_allocate_revertsForNonAdmin stop reverting, which is exactly what a negative test is for.

The fuzz suite catches D1 and the unit suite does not, for a boring reason: every amount in the unit tests is a round number. 1000e18 * 50 / 10_000 and 1000e18 / 10_000 * 50 are the same value. The fuzzer reported a counterexample before completing a single run — runs: 0 — with the fee off by 13 units.

The invariant campaign catches D2, and it is the one case where the gate earned its keep as a gate rather than as a better-written test. Nothing in the unit suite asserts totalOwed() after a claim — the tests check the beneficiary's token balance and the per-account owed mapping, which are both correct. The invariant that adds up the per-account debts and compares them with the aggregate is what notices, after a randomized sequence of funds, allocations and claims, that the two have drifted apart by 13,111 tokens. An invariant is often just the assertion you forgot to write, run against sequences you would not have thought of.

The coverage gate is the interesting one, because it did nothing.

It caught no defect. Every version of the contract has the same tests, so it has almost the same coverage: 98.51% of lines on the control, 98.48% on the one defect that changed the line count. A 95% threshold passes in both cases.

Worse, it is not an independent check. forge coverage runs the suite, so in the four rows where a test had already failed, the coverage command failed too — for that reason, not because coverage dropped. I record those cells as blocked rather than counting them as catches, because a gate that only fails when an earlier gate already failed is not gating anything.

And 98.51% line coverage on this contract comes with 62.5% branch coverage. The lines all execute. Half the branches through them never do.

Everything deploys

All thirteen versions deployed on the first attempt.

That is not a surprise once you say it out loud, but it is worth saying, because "the deploy went through" is the moment a lot of teams treat as the finish line. Foundry's script runner is explicit about what it does: with an RPC URL and without --broadcast, it "executes and simulates without submitting transactions", and with --broadcast it does the signing and submission. What both phases prove is that the constructor runs. A deployment that reverts is a mistake you find in seconds. A deployment that succeeds with the wrong constants is the expensive kind.

Declarative deployment tools help with the mechanics of this — Hardhat Ignition tracks what has already been deployed and resumes, which is one of the things that made Hardhat worth re-evaluating — but resumability and idempotency are answers to "did it get deployed," not to "is what got deployed what we meant."

The rehearsal chain: a local Anvil fork of Ethereum mainnet at block 25,984,000, the deployed vault, and the five checks that read from it. Mainnet state at the pinned block, including the real USDT contract at 0xdAC17F958D2ee523a2206206994597C13D831ec7 with 6 decimals and a transfer that returns no value at all, plus a real USDT holder that is impersonated as the sponsor, is read through an Anvil fork running locally with chain id 1 for the length of one run. The deploy script deploys PayoutVault to that fork from config/deploy.json with a 50 basis point fee, using Anvil account 0 as the deployer, which holds no role in the deployed contract. Anvil account 1 stands in for the operations multisig as admin and account 2 is the pause-only guardian hot key. Five checks run against that deployment: G5 the fork rehearsal, four transactions as the configured roles; G6 the config attestation, eight eth_call reads compared with deploy.json; G7 the bytecode attestation, eth_getCode compared with the repository build; G8 the monitor, the Claimed filter from the ABI the monitor shipped with; and G9 the incident drill, in which the guardian pauses and the drain path must stop. A note records that every defect in the matrix deployed successfully, that the deploy transaction succeeding is not one of the nine checks, and that nothing in the figure touches a public network, a real key or real funds.

The rehearsal is where the real token shows up

The fork rehearsal is the single highest-yield check in the matrix: it caught four of the eight defects that survived the suite. It is also the easiest of the five to describe. Start an Anvil fork of the chain you are deploying to, pinned to a block. Run the real deploy script against it. Then run the happy path as the accounts the configuration names, against the token the configuration names.

That last clause is where the value is.

The test suite uses a mock ERC-20: 18 decimals, returns true from transfer, takes no fee. The well-behaved case. The production token is USDT, and USDT does not return anything from transfer at all. On the pinned fork, a raw transfer call to USDT returns 0x — zero bytes of return data — where USDC returns a 32-byte true.

EIP-20 is unambiguous that callers "MUST handle false from returns (bool success)" and "MUST NOT assume that false is never returned." It says nothing about tokens that return nothing, because those tokens are not conformant — and this one had 88,305,985,017 USDT in supply at the block I forked. That is why OpenZeppelin's SafeERC20 exists: it wraps ERC-20 operations so that "tokens that return no value (and instead revert or throw on failure) are also supported, non-reverting calls are assumed to be successful."

The control handles it directly:

    /// @dev Accepts the tokens that return nothing at all from `transfer`.
    function _safeTransfer(address to, uint256 amount) private {
        (bool ok, bytes memory ret) = address(token).call(abi.encodeCall(IERC20.transfer, (to, amount)));
        if (!ok || (ret.length != 0 && !abi.decode(ret, (bool)))) revert TransferFailed();
    }

D4 replaces those two lines with the one everybody writes first, if (!token.transfer(to, amount)) revert TransferFailed();. Against the mock, it is correct and all 27 tests pass. Against USDT, solc tries to decode a bool from an empty return and reverts. The first claim on the rehearsal chain fails, and with it the claim path the monitor and the drill depend on — which is why those two cells are blocked rather than green.

D5 is the same category with a different surface. The control derives the minimum allocation from the token:

        token = token_;
        admin = admin_;
        guardian = guardian_;
        feeBps = feeBps_;
        minAllocation = 10 ** token_.decimals();

D5 replaces the last line with minAllocation = 1e18;, which is "one whole token" for every token the test suite has ever seen. USDT has six decimals. The deployed vault's minimum allocation becomes one trillion USDT, and the rehearsal's first 1,000 USDT allocation reverts with BelowMinimum.

D6 and D7 are deploy-script mistakes rather than contract mistakes — the admin left as the deployer EOA, and the admin and guardian arguments swapped — and the rehearsal catches both, because the smoke sequence acts as the admin the configuration names. When the configured admin is not the admin on chain, the allocation reverts with NotAdmin before anything else happens.

That is a property of how I wrote the smoke test, not a law of nature. A rehearsal that only asserts "the deploy succeeded" catches neither of them, and a rehearsal that signs as whatever account is convenient catches neither of them either. Exercise the roles the configuration names, with the token the configuration names, or the rehearsal is only testing the compiler.

The attestation nobody writes

The fee defect, D8, sails through the rehearsal.

Everything works. The sponsor funds, the admin allocates, the beneficiary claims, and 950 USDT arrives instead of 995. A smoke test that asserts "value reached the beneficiary" — which is how smoke tests are written — sees a successful claim. The fee is ten times what the configuration says, and the only check that notices is the one that compares the deployment with its own configuration file.

That check is eight eth_call reads and a comparison:

  • the token address matches config/deploy.json
  • the admin matches
  • the guardian matches
  • the fee matches
  • the deployer holds no role
  • the admin and the guardian are not the same address
  • minAllocation equals 10 ** decimals() of the configured token
  • the vault starts unpaused

It caught D8, and it independently caught D5, D6 and D7 — the three the rehearsal had already stopped. Its median runtime across thirteen deployments was 11 milliseconds. It is the highest ratio of caught defects to engineering effort anywhere in this article, and nothing in the toolchain writes it for you.

One of those eight checks is doing more work than the others. "The deployer holds no role" is the one that turns a deploy script into a handoff: the EOA that pays for the transaction should not be able to change the fee afterwards. And if your handoff is a single-step ownership transfer, note what OpenZeppelin's two-step variant is for — it "can help prevent common mistakes, such as transfers of ownership to incorrect accounts, or to contracts that are unable to interact with the permission system." My vault's transferAdmin is single-step, which is a weakness the attestation can detect but not prevent.

"Verified" means the source matches the bytecode, not that the bytecode is right

D9 changes no source at all. The repository pins optimizer_runs = 200; the deployment is built with 1000, the way a deployer with a stale environment variable or a local profile would build it.

Every test passes, because the tests run from the repository's profile. The rehearsal passes. The attestation passes — the configuration is correct and every role is right. The contract works. It is simply not the contract in the repository: 3,913 bytes of runtime code on chain against 3,589 bytes compiled from the same source.

Comparing those two is less obvious than it sounds, because this contract has immutables, and immutable values are written into the runtime bytecode at construction. The artifact tells you where they are, and the check masks those eight slots before comparing, then checks the values separately against the configuration:

    const refs = Object.entries(artifact.deployedBytecode.immutableReferences ?? {});
    const mask = (hex) => {
      const chars = hex.split("");
      for (const [, slots] of refs) {
        for (const { start, length } of slots) {
          for (let i = start * 2; i < (start + length) * 2 && i < chars.length; i++) chars[i] = "0";
        }
      }
      return chars.join("");
    };

Foundry has a command with exactly this name, and it will not do this job. forge verify-bytecode verifies "the deployed bytecode against its source on Etherscan": pointed at the rehearsal chain with --rpc-url, it exits 1 with Error fetching creation data from verifier-url: InvalidApiKey. It needs a block explorer that has indexed the creation transaction. A rehearsal chain that lives for two minutes has no explorer, so on the chain where you most want this answer, you compute it yourself.

The word "verified" is doing a lot of work in this part of the stack, and it is worth being precise about it. Sourcify distinguishes an exact match, where "the bytecode of the deployed contract is byte-by-byte the same (excluding libraries and immutables)" including the metadata fingerprint, from a plain match, where the code agrees "except the metadata hash" and comments, variable names or source paths may differ. Both mean the source corresponds to the bytecode. Neither means the source is correct, and neither means it is the source in your repository at the commit you tagged.

That fingerprint is also the thing most release pipelines quietly turn off. The Solidity compiler "appends by default the IPFS hash of the metadata file to the end of the runtime bytecode", and the metadata covers the compiler version, the settings and the source files. So I built the same contract five ways, with the fingerprint on and off, and recorded what moved:

change to the build fingerprint on (Foundry's default) fingerprint off (bytecode_hash = "none")
one comment added to the source differs identical
the source moved to contracts/ differs identical
the same project built at a deeper absolute path identical identical
optimizer_runs raised to 1000 differs differs

With the fingerprint on, a comment changes the runtime bytecode: the length stays at 3,643 bytes and the trailing fingerprint does not, so nothing matches byte for byte. That is not a bug — it is the fingerprint doing its job, and it is why Sourcify can tell an exact match from a match. It is also why a pipeline that wants "the same source produces the same bytes on every machine" reaches for bytecode_hash = "none", which is what this repository pins, and which cost 54 bytes of metadata.

Both choices are defensible. What is not defensible is not knowing which one you made, because Foundry's defaults are the opposite of this repository's: measured against 1.5.1 in an empty directory outside any project, bytecode_hash is ipfs, cbor_metadata is true, and — worth a second look — optimizer is false while optimizer_runs sits at 200. The number is set; the optimizer that would use it is not. Foundry's configuration reference agrees on all three of the documented ones; cbor_metadata is not listed there, which is part of why measuring beats reading.

The monitor's contract with the chain is the ABI it shipped with

D11 is the change I expected to be invisible to everything:

    event Claimed(address indexed account, uint256 amount);

Drop the indexed. The signature is unchanged, so topic0 is unchanged. The account simply moves out of the topics and into the data. The compiler is happy, the contract is functionally identical, and every offchain filter that subscribes by indexed account stops matching.

The monitor in this project is a filter built from the ABI in config/monitor.json — pinned on purpose, because a monitor's contract with the chain is the ABI it shipped with, not whatever the repository compiles today:

    const filter = await publicClient.createEventFilter({
      address: vault,
      event: claimedEvent,
      args: { account: ACCOUNTS.ben2 },
      fromBlock: await publicClient.getBlockNumber(),
    });

Against the changed contract, that filter matched zero logs for a claim that really happened — the block contains the log, the filter cannot see it. Money moved, the dashboard stayed quiet.

But it was not invisible to the suite, and the reason is a detail worth keeping. Foundry's vm.expectEmit compares topics, and a test that declares its own copy of the event still expects the account in topics[1]. So the unit suite failed — log != expected Claimed — and D11's first catch is the unit tests, not the monitor.

Change one thing about the test, and that disappears. D11i is the same defect with the test reusing the contract's declaration, emit PayoutVault.Claimed(beneficiary, 995e18), which is how you would write it to avoid duplicating the event. Now the expectation recompiles along with the contract, both sides agree, all 27 tests pass, the rehearsal passes, the attestation passes, the bytecode attestation passes — and the monitor is the only check in the ladder that notices.

Two tests of the same behaviour, written two reasonable ways, and only one of them can see the defect. If your suite imports its event declarations from the contract under test, your event schema is unasserted, and the first thing that will tell you is an indexer that went quiet.

This is also the part of the stack that changed underneath everybody recently. The hosted monitoring product a lot of teams used, OpenZeppelin Defender, was announced in June 2025 to "officially retire" on July 1, 2026. What OpenZeppelin points at instead is an open-source, self-hosted service — openzeppelin-monitor, AGPL-3.0, at v1.6.0 since July 2026 — that watches for "specific events and transactions" and fires Slack, Discord, email, Telegram, webhook or script triggers. Whatever you run, it subscribes to topics, and D11i is a change that survives a green suite and blinds it.

The drill is the only check that asks whether the guardian can act

D10 removes whenNotPaused from claim().

The unit suite tests the pause — on fund(). The invariant campaign never pauses at all, because a paused vault makes no progress and would spend its budget on reverts. So the contract ships with a pause that covers funding and allocation, and does not cover the one function that moves tokens out.

The drill is a sequence, not an assertion: allocate to two beneficiaries, have one of them claim successfully so the path is known to work, have the guardian pause, then try to claim as the other. That claim has to revert, and it has to revert with Paused rather than for some unrelated reason. Then unpause and confirm the claim resumes.

On the control, the pause cost 28,191 gas and both value paths stopped. On D10, the claim went through while the vault was paused.

The drill is also where the blocked cells cluster. It could not run at all on five of the twelve defects: on D4 through D7 because the claim path never worked, and on D2 because its own accounting drift had left the vault unable to fund another allocation. A pause that "works" on a contract nobody can use is not evidence, so those five are recorded as blocked rather than scored as passes. The grey cells in that column are the ladder admitting what it did not test.

What the drill does not test, because a local fork cannot, is the part that has nothing to do with the contract: whether the key that can pause is reachable at 3am, whether the multisig can assemble a quorum, and whether anybody has run the sequence before. It measures the contract's side of the drill. The human side is rehearsable too, and nothing in this article measures it.

What the ladder costs

Per version of the contract, median times:

check median
G1 unit tests 1,236 ms
G2 fuzz tests 628 ms
G3 invariant campaign 1,272 ms
G4 coverage gate 1,118 ms
G5 fork rehearsal 742 ms
G6 config attestation 11 ms
G7 bytecode attestation 1,703 ms
G8 monitor 338 ms
G9 incident drill 153 ms

The whole ladder is 7.4 seconds of checks per version, plus 1.0 second to start the forked chain and 2.0 seconds to run the deploy script through it. Thirteen versions, nine checks each, two independent passes: 2 minutes 19 seconds per pass on a laptop.

The five checks that run after the suite is green cost 2.9 seconds of the 7.4. Four of the eight defects that survived a green suite were caught by the one that costs 742 milliseconds, and one more by the one that costs 11.

This is the part I did not expect going in. The post-deploy half of the ladder is cheaper than the test half, catches the defect classes the test half structurally cannot see, and is the half that usually does not exist.

What this does not settle

One contract on one machine, with defects I chose, and gates whose strictness I decided.

That last point matters most, and it cuts both ways. I wrote the smoke test to act as the configured roles, which is why it caught two configuration mistakes; a weaker smoke test would have pushed those into the attestation column. I wrote the smoke test to assert that value arrived rather than that exactly 995 USDT arrived, which is why the fee defect got past it; a stricter one would have caught D8 a step earlier. Every "first catch" in the matrix is a statement about a check as I implemented it, not about the category.

The defect set is also not a sample of anything. It is twelve mistakes I consider realistic, weighted deliberately towards the classes a test suite cannot see, so "eight of twelve survived the suite" describes this set and not Solidity defects in general. What the matrix supports is narrower and, I think, more useful: not one of those eight was caught by a test, and five of them were caught by exactly one check in the whole ladder.

Several things a production ladder needs are missing here. There is no static analysis in this matrix — forge lint and Slither belong in continuous integration beside the tests, and they would likely have caught D3 and possibly D4. There are no upgradeable contracts, so nothing about storage layout collisions or initializer mistakes is tested. There is no multisig: the admin is an Anvil account standing in for one, so the ladder never executes a real Safe transaction or rehearses a quorum. There is no gas or griefing analysis, no fee-on-transfer or rebasing token, no L2, and no mempool. And the rehearsal forks a public archive endpoint, so its timings include one network's latency on one afternoon.

Finally, the monitor here is a filter and a solvency expression in a test harness, not a deployed service with alerting, on-call and persistence. It demonstrates what a monitor can and cannot see. It is not a recommendation for how to run one.

The checks I would keep

If I could keep only four things from this exercise, in this order:

A rehearsal on a fork of the target chain, driven by the real deploy script, as the configured roles, against the real token. It caught half of what got past the suite. Everything about it is reusable: the same script, the same configuration file, one Anvil process.

An attestation that compares the deployment with its configuration file. Eleven milliseconds, eight reads, and it is the only thing between a fee typo and a production fee typo. Include "the deployer holds no role" and run it as the last step of the deploy, not as a task somebody remembers.

A bytecode comparison against a build from the committed profile. Not the explorer's badge — your own comparison, with the immutables masked and their values checked. And pin what the compiler emits, then decide deliberately whether you want the metadata fingerprint or reproducible bytes, because the default and the reproducible choice are not the same.

A pause drill on the rehearsal chain, run as the guardian. It is 153 milliseconds and it answers a question no test asks: whether the emergency switch covers the function that moves the money.

Then the test suite, which is where most of the effort goes today and where it should stay: the invariant that catches the accounting drift is worth more than the coverage number that catches nothing.

None of these are new ideas. What surprised me is the arithmetic. The five checks that run after the suite is green cost under three seconds between them, and they were the only thing standing between this contract and eight defects that a green suite, a passing coverage gate and a successful deployment all signed off on.