Recovering Data from a Failed Synology NAS
by Ganesh T S on August 22, 2014 6:00 AM ESTConcluding Remarks
My Saturday plans went haywire, thanks to the DS414j going belly-up. However, I did end up proving that as long as the disks were functional, it is possible to easily recover data from a Synology RAID-5 volume by connecting the drives to a PC and using UFS Explorer. Users wanting more of a challenge can also use Ubuntu and mdadm for the same purpose. In my case, the data was in a SHR (Synology Hybrid RAID) volume with 1-disk redundancy, but the disks were all of the same size (making it RAID-5 effectively).
Lessons that I learned from my data recovery experience:
- Have access to a PC with multiple spare SATA slots, preferably hot-swap capable
- Back up data written to a NAS frequently (if possible, in real-time)
- Have access to a high capacity DAS (with more free space than the largest NAS volume that you may have to recover)
- Avoid encrypting shared folders and/or volumes, if possible
- Prefer straightforward RAID-x volumes compared to customized (note: customized need not necessarily mean proprietary) RAID implementations and/or automatic RAID level management (such as Synology's SHR / Seagate's SimplyRAID / Netgear X-RAID2)
- In critical environments, run two NAS units in high availability (HA) mode
Things I would like from the NAS vendors' side (Synology already ticks most of these):
- Don't use proprietary RAID / hardware RAID for consumer NAS units
- Instead of (or, in addition to) supplying backup software, provide licensed versions of data recovery software such as UFS Explorer (or, supply one developed internally for Windows / Mac / Linux)
- Provide official documentation for recovering data using PCs in case of NAS hardware failure (using either commercial software such as UFS Explorer or open source ones like TestDisk)
Synology alone is not to blame for this situation. If QNAP's QSync had worked properly, I could have simply tried to reinitialize the NAS instead of going through the data recovery process. That said, for the same purpose, QNAP's QSync worked much better than Synology's Cloud Station (which was the primary reason our configuration utilized a share set up on the DS414j as the target folder location for QNAP's QSync). In any case, I would like to stress that this anecdotal sample point in no way reflects the reliability of Synology's NAS units. I used to run a DS211+ 24x7 without issues for 3 years before retiring it. More recently, our Synology DS1812+ has been running 24x7 for the last one year as a syslog server. The DS414j which failed on me has been in operation for less than two months. I put it down to the 'infant mortality' component in the reliability engineering 'bathtub curve'. Synology provides a 2-year warranty on the DS414j, and any end-users affected by such hardware issues are definitely protected. One just needs to make sure that the data on the NAS is backed up frequently.
Facebook
Twitter
RSS
55 Comments
View All Comments
deeceefar2 - Friday, August 22, 2014 - link
If instead of using one qnap and one synology they were both the same brand, you wouldn't have had an issue. You could have just popped the drives immediately into the other nas, and sent the synology back for refurbishing. They way we did it was 2 Qnaps, one at the office, and one at my house. When we had a failure of the main Qnap we sent it in for repairs, and brought the one from home in. You have them doing remote replication, and then using dropbox sync we had one version in the cloud that was synced to individual workstations. So workstations doing video editing could do that much faster locally and then that would get synced to the main drive and then to the remote version at the same time.ruidc - Friday, August 22, 2014 - link
we had a Thecus that died and were told that we could simply plug the drives into the shipped replacement unit. When we did so, it initialized the array. Now, I'd go UFS every time instead (having used it successfully on a single drive to get the contents of an XFS drive that would not mount on another occasion). But I did not have a spare machine capable of connecting all the drives. Luckily nothing of importance was lost.imaheadcase - Friday, August 22, 2014 - link
Ganesh T S any plans to do a custom NAS buying guide like the one done in 2011? Lots of custom options out now for that.matt_v - Friday, August 22, 2014 - link
This article really takes me back to my own experience with NAS data recovery. After a firmware upgrade in 2012, my QNAP completely lost its encrypted RAID 6, and claimed it had 6 unassigned drives. After much Googling and careful experimenting with nothing but a CentOS VM on my notebook, I was able to extract all the files with all Unicode filenames intact (VERY important in a tri-lingual family).- SSH into the NAS as root and create a new mountpoint that's not referenced in any possibly corrupted config (I used /share/md1_data, where the default is md0_data)
- Assemble and mount the mdadm volume using the same console commands Ganesh used
- go into the web GUI and unlock the volume that "magically" appears in the Encrypted File System section
- Open WinSCP and log into the QNAP as root
- Copy out the contents of /share/md1_data to a backup volume of your choice (I used a NexStar HX4R with a 4x4TB RAID 5+1)
After successfully extracting all the files from the array, I completely nuked the QNAP configuration, built a new array from scratch, and copied the files back. These days, the Nexstar acts as a backup repository using TrueCrypt and a Windows batch script. Ugly, but functional, and the QNAP hasn't had a single config panic since.
Oyster - Friday, August 22, 2014 - link
"If QNAP's QSync had worked properly, I could have simply tried to reinitialize the NAS instead of going through the data recovery process."It seems you're blaming QSync for the failure as well... didn't you say the Synology circuit board died? How do you expect any external applications to "talk" with the Synology unit? Can you share your thoughts on why/how you expected QSync to function in this scenario?
This is no different than having OneDrive on two machines, and then blaming OneDrive for not syncing when one of the machines die on you!?!?
ganeshts - Friday, August 22, 2014 - link
The fact that the circuit board died is orthogonal to QSync's working.The data was in the hard drives of the DS414j for more than 12 hours before the 414j died. The CIFS share on the unit was used to upload the data, so there was actually no problem accessing the CIFS share at that time and for some time thereafter too.
The CIFS share was mapped as the 'QSync' folder on a separate Windows 8 PC (actually a VM on the TS451). QSync was installed on that PC. QSync's design intent or the way it presents itself to users is that it does real time bidirectional sync. It should have backed up the new data in the QSync PC folder (i.e, the DS414j share) to the TS451, but it didn't do it.
I had personally seen the backup taking place for the other data that I had uploaded earlier - to either the TS451 QSync folder or the DS414j share - so the concept does work. I actually noted this in my coverage of the VM applications earlier this week - the backup / sync apps don't quite deliver on the promise.
Oyster - Friday, August 22, 2014 - link
Thanks for the detailed clarification. Much appreciated.Two things I want to point out:
1) I was under the impression that QSync is simply for syncing folders. I'm surprised you're using it for full blown backups. Was this something QNAP suggested? I'm asking because I own a QNAP and would be good to know where QNAP is taking QSync.
2) I have backups setup on my QNAP NAS using the Backup Station app. I was always under the impression that Backup Station is the go-to app for maintaining proper backups on QNAP (it even provides rsync and remote NAS to NAS replication). This app has a notification feature which ties in with the notification settings in the Control Panel. I haven't had anything fail on me, but I tested the notification functionality using a test email, and it worked fine. I'd think had you utilized Backup Station, you would have been notified the moment things stopped working.
Just to point out, I'm in no way being defensive about the QNAP. I'm in full agreement with you that some of these utilities could use more work. Especially, something that allows us to read raw drives in a PC environment in the face of a failure.
ganeshts - Friday, August 22, 2014 - link
I am not sure how QSync is being understood by the users, but my impression after reading the feature list was that it could be used as an alternative to Dropbox, except that it used a 'private cloud'.Do I use Dropbox for folder syncing or backup? I would say, both. On my primary notebook, I work on some files and upload it to Dropbox. On my work PC, I could work on other files and upload them to the same Dropbox. In the end, I expect to be able to work with both files on both the notebook as well as the work PC. Extending this to QSync - I could put the files to 'sync/backup' through the QNAP QSync folder or upload it to some other path mapped as a QSync target along with the QSync program / application on a PC.
I believe backup and RTRR (real-time remote replication) are both uni-directional only. My intent was to achieve bidirectional sync / backup, which is possible only through 'Dropbox-style' implementations. If there are alternatives, I would love to hear about it.
Gigaplex - Saturday, August 23, 2014 - link
"I was under the impression that QSync is simply for syncing folders. I'm surprised you're using it for full blown backups."What is a backup? It is a copy of the data. What does syncing do? It copies data.
hrrmph - Friday, August 22, 2014 - link
I noticed AT's steady increase in NAS coverage and I wondered how long it would take to get to this point. Well, not too long it would seem.It just proves that once again, complexity kills. RAID, NAS, etc. aren't backup solutions, but rather are high capacity, high availability, and high performance solutions.
It's good to see that most people writing articles, and commenting today, already understand that a NAS isn't a 'safety' device. It is a fools errand to think of a NAS or RAID as providing any safety.
The value of a NAS as a high performance solution is questionable because a single SSD popped into a spare bay on a desktop system will outperform the NAS. Except when the NAS is populated with SSDs in a performance RAID configuration. Then you have the problem of getting a high enough bandwidth connection between the NAS and client. For performance, you are best sticking with a high performance desktop. If you insist on a laptop as your main machine, then connect it to the high performance desktop using Thunderbolt.
As for high availability, you are either a business and know how to do this yourself (and have a competent IT department to implement it), or you are a consumer. A consumer can just buy high availability as a service (such as from Amazon services). Or the consumer is a tinkerer, and doesn't care about efficiency, or cost effectiveness. Which brings us back to AT's series of articles on NAS devices.
If you are like me, and aren't ready to relinquish everything to the cloud, or a dodgy proprietary NAS scheme, or an even dodgier RAID setup, an alternative is to just build a low-power PC fitted with a 16-port HBA card and an appropriate chassis with racks. The hardest part these days is finding a case that is appropriate for a bunch of front loading racks to hold all of the quick swap drives. But, it is nonetheless one of the most viable ways to improve capacity and safety without going to the cloud.
As SSD prices slowly descend, this even becomes a viable performance option, with non-RAID drive setups capable of supplanting a bunch of spinning disks in a performance RAID setup.