I took one upgradeable vault, wrote eleven broken versions of its next release, and ran six different checks over each of them.
The unit test suite passed all eleven.
That is the whole problem with testing an upgrade. A test suite deploys a fresh proxy, points it at the new implementation, and exercises it. Every variable it reads was written by the same version of the code that reads it, so the layout is consistent with itself no matter how badly it diverged from the version already deployed. The suite is not wrong. It is answering a different question.
The check that finds storage damage is the one that upgrades a proxy which already holds state, and then reads that state back. In this experiment it caught nine of the eleven defects. The two it missed were caught by checks that nobody writes either.
The setup
The subject is a small UUPS vault: a VaultV1 implementation behind an OpenZeppelin ERC1967Proxy, with deposits, a withdrawal fee, a pause switch, a mapping of balances, and an array of depositors.
// contracts/VaultV1.sol
contract VaultV1 is Initializable, OwnableUpgradeable, UUPSUpgradeable {
address public treasury;
uint64 public feeBps;
bool public paused;
uint256 public totalDeposits;
uint256 public legacyEpoch;
mapping(address => uint256) public balanceOf;
address[] public depositors;
VaultV2 is the correct upgrade: every V1 variable stays exactly where it was, two new ones are appended, and the new one that needs a value gets it from a reinitializer(2) function called in the same transaction as the upgrade.
The eleven defective versions are generated from VaultV2.sol by a script that applies one named edit each and throws if the edit's search string does not match exactly once. That matters more than it sounds: it means every row of the matrix has identical vault logic by construction, so a check that turns red is reacting to the declared defect and nothing else. Ten of the eleven are one or two edits. The eleventh, a refactor to ERC-7201 namespaced storage, is written by hand, because moving storage into a namespace is not a local edit.
The six checks are:
| Check | What it does | |
|---|---|---|
| G1 | Storage-layout diff | forge inspect ... storageLayout for the candidate, compared with V1 slot by slot |
| G2 | OpenZeppelin validator | npx @openzeppelin/upgrades-core validate out/build-info --contract <name> --requireReference |
| G3 | The existing unit suite | Deploy this version fresh behind a new proxy and run the vault's own tests |
| G4 | Upgrade rehearsal | Upgrade a proxy already holding state, then read every pre-upgrade value back |
| G5 | Implementation lock | The implementation contract itself cannot be initialized directly |
| G6 | Upgrade authorization | After the upgrade lands, a stranger cannot upgrade again |
Versions: Foundry 1.5.1, solc 0.8.36 with the optimizer at 200 runs and the Cancun EVM target, @openzeppelin/contracts and @openzeppelin/contracts-upgradeable 5.6.1, @openzeppelin/upgrades-core 1.46.0, @openzeppelin/foundry-upgrades 0.4.2, forge-std 1.16.2, Node.js 22.17.0, on an Apple M3 Pro. Two consecutive runs produced byte-identical results.
Slot 0 is yours now
Before looking at what breaks, it is worth knowing where the variables are, because the answer changed with OpenZeppelin Contracts 5.
In version 4 and earlier, Initializable, OwnableUpgradeable and friends each put their own state into sequential slots ahead of yours, which is why every upgradeable tutorial from that era ends each base contract with a uint256[50] private __gap. Version 5 moved that state into ERC-7201 namespaces: Initializable stores its two fields at slot 0xf0c57e16840df040f15088dc2f81fe391c3923bec73e23a9662efc9c229c6a00, derived from keccak256(abi.encode(uint256(keccak256("openzeppelin.storage.Initializable")) - 1)) & ~bytes32(uint256(0xff)), and the library's own contracts carry no __gap at all. OpenZeppelin's upgradeable-contracts documentation is explicit about why: namespaced storage "allows us to freely add new state variables in the future without compromising the storage compatibility with existing deployments."
The practical consequence is that treasury above really is slot 0, offset 0. The compiler will tell you so.
Every slot number in this figure comes from forge inspect <contract> storageLayout, read out of the compiler rather than counted by hand.
Look at the D3 panel. The only edit is uint64 public feeBps becoming uint96 public feeBps. Twenty bytes of address plus twelve bytes of uint96 exactly fills the slot, so paused no longer fits beside them and moves to slot 1 — which is where totalDeposits was. One character of type width, and every variable in the contract except treasury is now reading someone else's value.
D2 is subtler still. It swaps the declaration order of treasury and feeBps. Both stay in slot 0, so a slot-counting review sees nothing move. The offsets swap: treasury goes from offset 0 to offset 8, and after the upgrade it reads the top twenty bytes of a word whose low eight bytes were a fee.
What the two static checks see
Both static checks catch every layout defect in the set. G1 — a hand-rolled diff of forge inspect output, maybe forty lines of script — flags all six, naming the exact movement: totalDeposits moved from slot 1+0 to slot 2+0. The OpenZeppelin validator flags the same six and names the cause instead of the symptom: Inserted `keeper`, Upgraded `feeBps` to an incompatible type, VaultV1: Deleted `legacyEpoch`.
Two of those six deserve attention because they are the ones a careful human reviewer misses.
D5 adds one base contract to the inheritance list. The vault's own source file is otherwise untouched; the storage lives in a separate file nobody opened during review. Its single uint256 public maxPerBlock takes slot 0, and the entire vault shifts down one slot. The validator's message points at the right file: contracts/RateLimited.sol:8: Inserted `maxPerBlock`.
D10 is the modernisation. Somebody reads the version 5 release notes, decides the project should use namespaced storage too, and moves all nine variables into an ERC-7201 struct. The external interface is unchanged and the contract is arguably better written. The validator reports seven deleted variables. The proxy, after such an upgrade, would read zero for every one of them, because the namespace root is a hash of a string and has nothing to do with slot 0.
The validator sees two things the layout diff does not. D8, which marks the version-2 initializer initializer instead of reinitializer(2), trips its initializer analysis: Missing initializer calls for one or more parent contracts: `OwnableUpgradeable`. D11, an implementation that does not inherit UUPSUpgradeable, trips a dedicated rule: Implementation is missing a public `upgradeTo(address)` or `upgradeToAndCall(address,bytes)` function.
The layout diff also has a blind spot the validator does not: it compares names, so a legitimate rename would read as a deletion. OpenZeppelin's storage rules allow renaming and say the value carries over.
The three defects no static check sees
Three of the eleven pass both static checks, and pass the unit suite, and would ship.
D6 removes _disableInitializers() from the constructor. The layout is perfect. The upgrade works. The vault behaves correctly forever. And the implementation contract sitting behind the proxy can be initialized by anyone who finds its address, which is public in the ERC-1967 implementation slot. Only G5 catches this, and G5 is five lines.
D7 appends a new variable and no initializer for it. minDeposit is meant to be 0.1 ether. On a fresh deployment the deploy script sets it. On the upgrade there is nothing to call, so the variable stays at its zero default and the minimum silently does not exist. Nothing is corrupted; the contract is simply misconfigured in a way that no static tool can know about, because only the deployment intent says what the value should be. Only G4 catches it, and only because the rehearsal asserts the new configuration as well as the old state.
D9 removes onlyOwner from _authorizeUpgrade. This is the defect a checklist would catch and every static check in this experiment missed. The validator's documented rules cover constructors, initializers, selfdestruct, delegatecall, immutable and assigned state variables, and storage, but nothing checks that the upgrade hook is access-controlled. Only G6 catches it.
That is the finding that reframes the rest. The static checks cover exactly one class of defect — the layout class — completely. Outside that class they caught two of five.
The rehearsal
G4 is the check worth building first. It is an ordinary Foundry test that does what the deployment will do.
// test/UpgradeMatrix.t.sol
/// G4. The rehearsal: upgrade the proxy that already holds state, then read
/// back every value that existed before the upgrade.
function gateRehearsal(uint256 i) external {
VaultV2 vault = _deployV1AndSeed();
// deployCode issues a CREATE, which would consume the prank.
address impl = deployCode(_fqn(i));
vm.prank(OWNER);
IUUPS(address(vault)).upgradeToAndCall(impl, _upgradeData(i));
require(vault.treasury() == TREASURY, "treasury moved");
require(vault.feeBps() == FEE_BPS, "feeBps moved");
require(vault.paused() == false, "paused moved");
require(vault.totalDeposits() == 3 ether, "totalDeposits moved");
require(vault.balanceOf(ALICE) == 1 ether, "alice balance moved");
require(vault.balanceOf(BOB) == 2 ether, "bob balance moved");
require(vault.depositorCount() == 2, "depositors length moved");
require(vault.depositors(0) == ALICE, "depositors[0] moved");
require(vault.legacyEpoch() == 1, "legacyEpoch moved");
require(vault.minDeposit() == MIN_DEPOSIT, "minDeposit not configured");
vm.prank(ALICE);
vault.withdraw(1 ether);
require(vault.totalWithdrawn() == 1 ether, "totalWithdrawn");
require(vault.totalDeposits() == 2 ether, "totalDeposits after upgrade withdraw");
}
Four things about it are deliberate.
It seeds state before upgrading. Two depositors, a non-zero total, a populated mapping and a populated array. A rehearsal against an empty proxy proves almost nothing, because zero reads the same from every slot.
It asserts every variable, not a sample. Of the nine defect rows this check turns red, the assertion that fires first varies: totalDeposits moved for D1 and D3, treasury moved for D2, D5 and D10, alice balance moved for D4. A control version of this test that asserts only the two variables the release notes mention, and skips the withdrawal at the end, passes five of those nine: D1, D2, D3, D5 and D10. A shifted layout is perfectly self-consistent for a variable the new code both writes and reads, which is exactly the trap the fresh-deploy suite falls into.
It includes the array and the mapping. Their contents do not move when a slot shifts — a mapping's values live at hashes of their keys and the base slot — which is exactly why they are the assertions that break most loudly. In D4, deleting one unused legacyEpoch variable moves balanceOf from slot 3 to slot 2, and every recorded balance becomes zero.
It asserts the new configuration too. Without the minDeposit line, D7 is invisible.
The other thing this test does is exercise the vault after the upgrade. Reading state back proves the layout survived; a withdrawal proves the new code can write to it.
Authorization runs in the old implementation
G6 has a structural constraint that is easy to get wrong.
In UUPS, upgradeToAndCall lives in the implementation, so the _authorizeUpgrade hook that runs during an upgrade belongs to the implementation that is currently installed, not the one being installed. A test that deploys a V1 proxy and checks that a stranger cannot upgrade it to the new implementation passes — because V1's hook is intact. Written that way, the check is green for all twelve versions, D9 included. The defect only becomes reachable after D9 is the live implementation.
So the check has to upgrade first, as the owner, and only then try the unauthorized upgrade. Written that way, D9 turns red with a stranger upgraded the proxy.
Two rows return a third verdict here: blocked. D8 and D11 cannot reach the authorization check because their upgrade transaction never lands. That is not a pass, and scoring it as one would overstate the coverage of the gate.
D11's failure is interesting on its own. Upgrading a UUPS proxy to an implementation that is not UUPS used to be the classic irreversible mistake: the proxy would happily accept it and lose its only upgrade path. Under OpenZeppelin Contracts 5, UUPSUpgradeable._upgradeToAndCallUUPS asks the new implementation for its proxiableUUID before installing it, and reverts with ERC1967InvalidImplementation(address) when that call fails. That is a check nobody in the project wrote, enforced by the implementation you are upgrading away from.
The validator has its own traps
The OpenZeppelin validator is the strongest single tool here, and it has three rough edges worth knowing before you put it in continuous integration. All three were measured, not read.
It rejects the recommended constructor. OpenZeppelin's own guidance is to call _disableInitializers() in the implementation's constructor. Do exactly that, and the validator rejects the contract unless you also add @custom:oz-upgrades-unsafe-allow constructor above it. A control version of the correct upgrade, identical apart from that missing annotation, fails with Contract `VaultV2NoConstructorAnnotation` has a constructor. A second control with no constructor at all — that is, with the D6 defect and nothing to annotate — passes. A developer who takes the error at face value and deletes the constructor has made the contract less safe and the validator quieter.
Reinitializers are skipped by default. The CLI prints a note for every one it finds: "Reinitializers are not included in validations by default", and suggests @custom:oz-upgrades-validate-as-initializer. Adding that annotation to the correct upgrade makes it fail, with Missing initializer calls for one or more parent contracts: `OwnableUpgradeable` — because a version-2 reinitializer that sets one new field has no business calling __Ownable_init again. The annotation is worth using deliberately, not reflexively.
Stale build info produces a confusing error. The library's README says to run forge clean first. Running the validation against an out/build-info directory holding two build-info files fails all three of its tests with Found multiple contracts with name contracts/VaultV2.sol:VaultV2. and the hint This may be caused by old copies of build info files. After forge clean, the same three tests pass.
You can run the validator either way. The Foundry library turns it into one Solidity call, and it needs ffi = true because its Core.sol builds the argv npx, @openzeppelin/upgrades-core@^1.45.0, validate and runs exactly the command G2 runs.
// test-upgrades/PluginValidation.t.sol
contract PluginValidationTest is Test {
/// External so that a failing validation reverts one call deeper than the
/// test body, which is what vm.expectRevert and try/catch need.
function validate(string memory contractName) external {
Options memory opts;
Upgrades.validateUpgrade(contractName, opts);
}
function test_acceptsTheCorrectUpgrade() public {
this.validate("VaultV2.sol:VaultV2");
}
function test_rejectsTheInsertedVariable() public {
try this.validate("VaultV2Inserted.sol:VaultV2Inserted") {
revert("the library accepted an inserted variable");
} catch Error(string memory reason) {
assertTrue(_contains(reason, "Inserted `keeper`"), "validator named the inserted variable");
}
}
Those three tests agree with the command-line results exactly, including the third one, which asserts that the library accepts the uninitialized implementation.
Two of these defects have already happened
Both have a well-documented history, and the evidence for both is still readable on chain.
The Audius governance takeover in July 2022 was a storage collision, though not between two implementations. Audius used a custom proxy that stored the admin address in slot 0, where the implementation's Initializable flags also lived. Its post-mortem is precise about the coincidence that made it exploitable: "Because the last byte of the proxyAdmin address is 0xac, initialized was interpreted as a truthy value. Similarly, because the second byte of the proxyAdmin address is 0xab, initializing was also interpreted as a truthy value." The initializer modifier therefore never stopped anyone, and 18,564,497 AUDIO left the community treasury.
You can still read the slot:
cast storage 0x4DEcA517D6817B6510798b7328F2314d3003AbAC 0 \
--rpc-url https://ethereum-rpc.publicnode.com
# 0x0000000000000000000000004deca517d6817b6510798b7328f2314d3003abac
The address ends abac. Solidity packs bool initialized at byte 0 of the slot and bool initializing at byte 1, so those two bytes — 0xac and 0xab — are what the modifier read. Checked on 22 September 2026.
The Parity multi-sig library in November 2017 is the ancestor of the _disableInitializers() rule. It was a delegatecall library rather than a proxy implementation, but the mechanism is D6 exactly: the shared contract still required initialization, a stranger initialized it and then called its kill() function, and, in Parity's words, "this action blocked funds in 587 wallets holding a total amount of 513,774.16 Ether". eth_getCode for 0x863df6bfa4469f3ead0be8f9f2aae51c91a907b4 still returns 0x.
The pattern then repeated for proxies. OpenZeppelin's September 2021 advisory for UUPSUpgradeable, CVE-2021-41264, covered versions 4.1.0 up to 4.3.2: an attacker could initialize an implementation, appoint themselves upgrade admin, and DELEGATECALL into a contract with a SELFDESTRUCT in it. Before disclosing, the team initialized more than 150 uninitialized implementations across five chains themselves.
Here is what has changed since, measured on D6 rather than assumed. A stranger can still take the implementation: after calling initialize directly on it, owner() returns the stranger's address. But calling upgradeToAndCall on the implementation now reverts with UUPSUnauthorizedCallContext(), because version 5's onlyProxy modifier refuses to run it outside a proxy's delegate context. The takeover is real and the destruction is not.
That is not a reason to skip _disableInitializers(). It is a reason to keep the check that verifies it, since the thing standing between this defect and the 2021 outcome is a modifier in a dependency you might upgrade.
The matrix
Twelve versions, six checks, seventy-two verdicts.
| Version | The one change | G1 layout | G2 validator | G3 unit suite | G4 rehearsal | G5 impl lock | G6 authorization |
|---|---|---|---|---|---|---|---|
| C0 | none — the correct upgrade | pass | pass | pass | pass | pass | pass |
| D1 | a variable inserted, not appended | fail | fail | pass | fail | pass | pass |
| D2 | two variables swapped inside slot 0 | fail | fail | pass | fail | pass | pass |
| D3 | uint64 widened to uint96 |
fail | fail | pass | fail | pass | pass |
| D4 | an unused legacy variable deleted | fail | fail | pass | fail | pass | pass |
| D5 | a new base contract in the inheritance list | fail | fail | pass | fail | pass | pass |
| D6 | _disableInitializers() dropped |
pass | pass | pass | pass | fail | pass |
| D7 | a new variable with no initializer | pass | pass | pass | fail | pass | pass |
| D8 | initializer instead of reinitializer(2) |
pass | fail | pass | fail | pass | blocked |
| D9 | _authorizeUpgrade without onlyOwner |
pass | pass | pass | pass | pass | fail |
| D10 | storage refactored to an ERC-7201 namespace | fail | fail | pass | fail | pass | pass |
| D11 | the implementation is not UUPS | pass | fail | pass | fail | pass | blocked |
Read the columns rather than the rows. G3, the suite that already exists, is green in all twelve. G1 finds six defects, G2 finds eight, G4 finds nine, and G5 and G6 find one each — the one nothing else finds.
What this does not settle
One vault, one proxy pattern, one library version. Everything here is UUPS with OpenZeppelin Contracts 5.6.1; a transparent proxy moves the admin logic into the proxy and changes what G6 has to test, and a beacon proxy changes who can upgrade whom. Diamond storage is not covered at all.
Eleven defects is a set I chose. It leans toward the mistakes that are easy to make in a diff and does not include the ones that need a bad actor inside the process — a malicious implementation, a compromised deployment key, a governance timelock with the wrong delay. The set also contains no renames, which is the case where G1 gives a false positive and G2 does not.
Nothing here runs against a forked mainnet deployment with real state, real token balances and a real multisig executing the upgrade. That is the next rehearsal after this one, and it tests different things: the roles named in the configuration, the calldata the multisig will actually sign, and the behaviour of tokens the local fixture does not model. The production-readiness ladder I built earlier deliberately had no upgradeable contracts in it; this fills that gap for the contract, not for the deployment.
The results are also specific to these tool versions. The validator's initializer analysis — the rule that caught D8 — did not exist in the 2024 releases most tutorials were written against.
What I would keep
Four things, in the order I would add them.
Write the rehearsal first. It caught nine of eleven, needs no dependency beyond forge-std, and is the only check in the set that knows what the deployed proxy actually holds. Seed real state, assert every variable, and assert the new configuration as well.
Add the OpenZeppelin validator second, in continuous integration, with forge clean in front of it. It catches the same layout defects earlier and cheaper than a test run, names the cause rather than the symptom, and adds two rules a homemade diff will not have. Add @custom:oz-upgrades-unsafe-allow constructor to the implementation and leave the _disableInitializers() call in place.
Then write the two small tests that cover what neither of those sees: that the implementation contract rejects a direct initialize, and that after the upgrade lands, an unauthorized account cannot upgrade again. Together they are about a dozen lines and they are the only checks that caught two of the eleven defects.
Keep the storage-layout diff if you already have one, but do not let it be the whole story. It is the check that feels like it is about storage, and it is the one that misses the defects that are not.
The upgrade that breaks storage does not fail loudly. It deploys, it verifies, the tests are green, and the first symptom is a balance that reads zero. The only test that would have said so is the one that upgrades something.