A logistics office near Braehead ran its world — two virtual machines, bookings and accounts — on a Cisco rack server. When a disk in the RAID 5 failed, a replacement went into the wrong bay, a rebuild was accepted against a stale member, and twenty minutes later the VMware datastore was offline with both VMs inside it.
Their IT contractor made the call that mattered: no second rebuild, no “force online”, power off and send the lot. The full set of disks arrived labelled by bay — including the failed original and the mis-slotted replacement — which is exactly right. A RAID job wants every member, in any state, because even a failed disk carries metadata that helps date the array’s history. What kills these recoveries is a second rebuild; this one never happened.
The disks now carried two conflicting stories: the original array’s geometry, and a partial rebuild that had recalculated parity against a member whose data was weeks stale. Where the rebuild had progressed, fresh parity had overwritten good parity; where it hadn’t, the original survived. The VMFS datastore — and the two VMDKs inside it — straddled both regions. The job was archaeology: establish which epoch each stripe belonged to, then read each accordingly.
Every member was imaged read-only, the originals then shelved. From the images, the pre-failure geometry was solved — order, stripe size, parity rotation — and the array reassembled virtually, steering around the rebuild-damaged bands using parity from the surviving epoch. The VMFS volume mounted from the reconstruction, both VMDKs were extracted whole, and the Windows file systems inside each were checked before anything was called recovered.
Both virtual machines went home on fresh storage and booted — bookings, accounts, mail archive, all present and verified against a file listing. The work sat in the from-£500 + VAT multi-disk band, quoted in writing after the free assessment, no surprises after. The lesson travels well beyond Cisco kit: after any failed rebuild, the enemy is the next rebuild. Stop at one, keep every disk, and the odds stay firmly on your side.
A failed or wrong-disk rebuild is recoverable far more often than it looks — if the array is left alone afterwards. Power down, keep every member including the failed ones, and send the set: from £500 + VAT, assessed free and quoted in writing first.