It wasn't ZFS perse... the software I was using wouldn't properly enable the drives configured as hot spares... that and a few errors (I didn't use ECC), and a couple bad drives, and everything (11TB of data) lost. All the really important stuff was well backed up, as was my audio library. I never did re-rip all my DVDs & BluRays again since that happened.
Good info, thanks. I'll admit, I'm pretty terrified of having to replace one of the drives in my array. I've replaced dozens or maybe even hundreds of drives in Sun, SGI and DEC storage arrays over the years, but I have a lot less confidence in my homebrew setup. Fortunately, I have regular complete backups of everything there, so even if I lost the whole array, all I'd really lose is the time involved in fixing it, but that's still painful.
I felt the same way, which was why I went Raid-Z2 (dual parity) with two hot spares configured... even had a spare controller next to the box. Still, lost it all.
The FreeNAS version I was using had the bug wrt hot spares... and I didn't use ECC... The errors also weren't clear which drive was failing... was 0 in some places 1 in others, and it turns out it was both. The LSI board I was using apparently had some buggy issues as well.
That combined with using Seagate 7200 rpm drives in a year that was particularly bad (an offline storage company mentioned them in a blog article iirc). just a bad chain of events. The drives when I pulled out and tested them individually after the entire array crashed, about 7 out of 12 had significant issues, and of the other 5, 2 more developed errors shortly after (I only used them as scratch/project drives).
Honestly, today, I'd probably get another Synology box, a bit more expensive, but been so much less hassle. I had upgraded my 4-drive (2010 plus model iirc) synology box to 4TB wd red drives, and been running that ever since. I don't think I'd ever do a homebrew NAS on a single machine ever again. If I had to do something in a company, would probably lean towards distributed file systems for that level of redundancy.
Were you not scrubbing at all? (Blaming FreeNAS rather than you if so - but if one's doing it by hand following the ZFS HOWTOs it's definitely something that's impressed on one to set up).
When I use a NAS software, that's meant to be a NAS, I expect it to do maintenance that's required. I don't expect to have to be an expert in ZFS. By the same note, I don't expect to have to know what the storage and growth configurations of AWS RDS instances either.
My Synology NAS device has been in service for around 6 years now... I upgraded the drives from 4x1tb to 4x4TB when the freenas server bit the dust... and it's still going strong... I don't have to think about it, it just sits there and runs.... A single-purpose software like this should do its' job. The end.
Agreed. I just think it's not really fair to say that ZFS lost your data, if the main cause was that FreeNAS uses/configures ZFS in a way directly contrary to the ZFS documentation.
I've lost my movie collection to a different issue, and I never got around to re-rip it again. But I have promised myself that if I have to rip it again, I will do it script based, such that backing the script up, re-ripping will be a matter of installing the same software, and running the script.
