Data Recovery Case File · NAS & Network Storage · The Filesystem Is Being Truthful

A ZFS Pool Reporting Corrupt Metadata

This enquiry came from a technical team and described the problem accurately, which makes it unusually workable. A ZFS pool on a server reporting corrupted metadata: "the problem first appeared after an office move; the physical device may have been misconfigured — drives misconnected and/or not connected — during boot-up after the move. We do not have the necessary in-house expertise available to examine the problem further right now. There are data in the pool that were written since the last backup." Two things are worth saying immediately. ZFS reporting corruption is the filesystem being honest rather than broken — it verifies everything and refuses to serve data it cannot vouch for, which is a feature. And the actions that would ordinarily be tried next are, on this filesystem, the ones most likely to make the situation permanent.

The decode: why ZFS behaves this way, and the commands not to run

Why the error is honesty rather than damage: ZFS checksums everything it stores, including its own metadata, and verifies those checksums on every read. Where a conventional filesystem may serve corrupted data without noticing, ZFS detects the mismatch and refuses. So a pool reporting corrupted metadata is telling you precisely what it has found, which is more information than most filesystems ever provide. It is also why the pool will not simply mount and let them copy things off: it will not present data it cannot vouch for.

What an office move probably did: their own hypothesis is sound. ZFS identifies member devices by labels written to them rather than by their physical position, which normally makes it robust against reordering. But a boot with devices missing, or with a controller presenting them differently, can lead to a pool being imported in a degraded or inconsistent state — and if writes then occurred against that state, the pool's view of itself and the devices' actual contents diverge. That produces exactly the report they are seeing, and it means the underlying data is very likely intact while the pool's coherence is not.

The commands not to run: and this is the practical core, because the obvious remedies are the dangerous ones. Do not clear the error state to make the pool importable — that discards the record of what is wrong. Do not force an import, which can commit an inconsistent view as authoritative. Do not scrub, which reads and rewrites across the entire pool and, on a pool whose metadata is already inconsistent, can propagate the damage. And do not resilver or replace devices, which rewrites data based on a view that may be wrong. Each of those is standard practice on a healthy pool and each risks converting a recoverable inconsistency into a permanent one here.

What should happen instead: every member device imaged individually and write-blocked, before anything else. All subsequent work — examining the pool's transaction history, identifying a consistent earlier state, and attempting import — happens on the copies, where a wrong hypothesis costs nothing and can be retried. ZFS keeps a history of transaction groups, and rolling back to the last coherent one is frequently the route in; but that is an experiment, and experiments belong on images.

The honest framing: pools in this condition are often recoverable and occasionally are not. What determines it more than anything is whether corrective commands were run before imaging.

On the bench

The pool was left unmounted and unaltered, and no corrective command was issued — because on this filesystem the standard remedies are the destructive ones and the record of what is wrong is itself valuable. Every member device was imaged individually and write-blocked on the Atola TaskForce 2, with its physical position recorded. Pool state, labels and transaction history were then examined on the images, and import attempted offline from the copies only, including rollback to the last coherent transaction group where the history supported it. Recovered data was verified before delivery, and the findings documented for the team.

The outcome

Every member imaged before any corrective action, the pool examined and imported offline from copies, and the recovered data verified. Free assessment, one fixed written figure including VAT; where a drive has to be opened, 50% of parts and labour is payable upfront with the balance only on success — otherwise no recovery, no fee. The decode, for anyone with a pool reporting corruption: ZFS checksums everything and refuses to serve data it cannot verify, so the error is the filesystem being truthful rather than broken, and it means the underlying data is often intact while coherence is not; a move with devices missing or reordered at boot is a classic cause; and the ordinary remedies are the dangerous ones — do not clear the error, force an import, scrub, or resilver before every member has been imaged, because each can commit an inconsistent view as authoritative.

Our case files are drawn from genuine enquiries received by our laboratory over the past ten years, anonymised to protect client confidentiality. Each one describes the diagnostic and recovery procedure our engineers apply to that fault, using the equipment listed.