workflow/backups & failed drive question

[laural]

Hi Brian,

Thank you for all the knowledge that you share here, I'm learning so much browsing the forums. I wanted to ask a few questions about HDDs and get your feedback on my storage/backup setup.

1) I have a Synology disk station that I use as NAS to keep my "master/working" photos (dng/psds). It has two 2TB drives set up in RAID 1 (actually, upon setup it suggested it's own Synology Hybrid RAID which the manual describes as "SHR is an automated RAID management system that makes storage volume deployment easier than traditional RAID systems. Unlike Classic RAID, SHR divides each drive volume into smaller chunks and creates additional redundant storage. By using SHR, you can use the unavailable 4.5 TB volume (in the example) as smaller usable chunks, therefore maximizing storage capacity of each drive." I said OK but I assume since I initially used drives of equal size it didn't make a difference. Do you know anything about this proprietary RAID? Should I switch to classic RAID 1 at the next opportunity?).

Anyway, a while ago I noticed one drive was reporting vague i/o errors but it supposedly was successful at self-repairing. Last night the unit got knocked a bit and the power button got hit. I wasn't connected or accessing any files but it was on and running (I usually leave it turned on and plugged into a UPS unit). So, that one disk is failing now and it is suggesting I replace it. I was going to buy an identical drive, although it is not in stock for 2-4 weeks. If I wait that long, aside from having no redundancy until then, is there any added risk of still moving files onto the volume (one working drive)? I'm worried somehow the other disk may be fragile, if that makes sense (a quick test said it was healthy but I’ve scheduled an extended test to run overnight). I'm inclined not to "rock the boat" until I get it replaced but I was *just* about to clear space on my Macbook which only has 10GB free (which I learned here is unwise to have that full, lol) and iPhone which only has free space on the order of MBs. The NAS volume is 75% full; I do have major culling to do but I'm afraid to do that right now. How much can these kind of drives (it's a WD "Red" NAS Harddrive FWIW) be safely filled? As I write this I wonder if I should get a larger capacity drive (4tb) while I'm at it...

2) I was also wondering if the way I access and change the files is OK. I have a Macbook and go to Connect to server and connect via AFP. If I want to access just one photo I usually open the file directly from ACR, leaving the dng file on the server, let it write the changes when I click open, and then I save my psd locally until I’m done working on it then move it over to the server. Sometimes though if I have a large folder of photos I haven’t culled (or if I’m going away) I drag the whole folder over to my laptop, generate 1:1 previews for culling, then edits to the dng and psds, and then drag the whole folder back over. I don’t do anything else on my computer while it’s moving the folder but I’m wondering if it’s “safe” to do it that way? I started doing that because trying to cull with 1:1 previews seemed very slow when they were on the server, and at the time I was doing edits in LR and I would always get error messages when LR would try to write to the metadata (I’ve since turned off that option and just backup the catalog instead, but I mostly use ACR/PS now anyway and have never gotten that error).

3) As backups I use two cloud services, one that saves jpgs + video files, and Amazon Cloud that I save dngs, psds, and videos. (Two is probably overkill but I started with the first and just kept it since it was cheap after Amazon Cloud started accepting raw files). I upload manually and then don’t touch the files or let my computer sync anything, so that if something on my NAS becomes corrupt or deleted that it doesn’t propagate. My other backup is a random assortment of HDDs that I currently keep inside the house with my computer but know I should probably move them off site somehow. I’m wondering if it’s safe to keep HDDs in the garage in a safe, or if winter weather would be too cold for the drives? And how reliable cloud services are?

Thank you in advance for any insight/advice/suggestions. Sorry this post turned out long-winded, hopefully my questions make sense

 

Signed,

An OCD hobbyist photographer/digital pack rat

[Brian]

Oh Boy!! A juicy question!!

Here is the thing with RAID 0, it's FAST and allows you to maximize space. Two 4TB drives will give you one big 8TB drive, etc. The downside to RAID 0, is that if one drive fails, the whole thing goes Kablooey. RAID 1 is different, as one drive is automatically mirrored to a 2nd hard drive. It's great for redundancy, but the downside is if a file gets corrupted or deleted on one HD, it automatically becomes corrupted / deleted on the other drive. I'm sure you are aware of this and I'm not insulting your intelligence. LOL! I do have a point. That point is...you are somewhere in between RAID0 and RAID1. Which probably works just fine...when it works. If something were to crash and you sent out your HD for extreme data recovery, it might be harder to get things back on a proprietary system. In short, you are at the mercy of the proprietary software working perfectly, which may not be the case 100% of the time.  The old-school tech in me doesn't trust manufacture's software and would always recommend a traditional RAID1 for file storage. If someone was editing video and needed a large HD to store the cache file(s), then RAID0 is the way to go for that scenario.

So should you switch to a true RAID1? That's your call. Me personally...I would. That said, a traditional RAID works best if you have identical HDs, with identical firmware. That HD seems to be on back-order for about 4 weeks...and I wouldn't hold my breath either. I'm sure doing a data-dump on the working drive would be fine, but as you stated, no redundancy AND if both drives are of equal age and usage, you might be playing a game of roulette. Power surges and other power related issues kill hard drives, AND UPS batteries. I would test the UPS now and see how long it lasts, but use something like a lamp or clock-radio to test things. Simply un-plug the UPS and see how long it runs. If it dies right away or within a minute, it's time for a new UPS battery. Also, things wear out and you might want to invest in a new UPS. The trick with a UPS is to balance the load with the Volt Amp Rating on the box. A hard drive RAID unit and maybe a cable/dsl modem / router, you could probably get away with a 450VA model. Maybe 350VA. If you have something like a 1000VA, that's overkill and if you have too little draw on the UPS, it kills the battery. If you have too much draw on the UPS, it kills the battery. You are after the Goldie-Locks Zone, right in the middle.

If it were me, I wouldn't push things. In fact, I would invest in something like a 8TB EHD, backup the NAS via a Data-Dump to a folder called "Synology EHD Unit." Have a backup for the backup, even though you have Cloud Service. Then do a data-dump from your MacBook and iPhone to their own folders. I just saw this deal on Amazon, it's a way to backup your iPhone. You simply hook up the SD Reader to the Lightning Port, then use a SD card to transfer your files. Easy-Peasy.

I'm a big fan of Western Digital Caviar BLACK Hard Drives and really don't like the WD Red Drives. I've seen too many people bitching about the high-failure rates. I get that they are meant to run 24/7 and be in a NAS environment, but something about them just turns me off. (I'm sure it's people bitching about them, asking WD why they fail so much and having things fall of deaf ears...)

Any way, as far as hard drive capacity, a good rule-of-thumb is to NOT go over 75% full. Whether it's an internal HD, EHD or NAS. Especially if you have a Mac...performance takes a big hit once you cross 65% full, and gets much worse when you go over 75% full. If your MBP is 95% or more full, you are in danger of file corruption and are in un-charted territory. Things might run fine for awhile, and then...Kablooey...it all goes to hell. OS, files, software, etc. So try to keep at least 25% free at all times, preferably more.

Bottom Line what I would do: I'd replace the 2TB WD Red Drives with at least 4TB identical hard drives or go larger, maybe a couple of 6TB EHDs. I'd purchase a 8TB EHD to ultimately backup the NAS, which would then be kept off-site. (The Data Dumps from the NAS, MBP and iPhone is just to get you through.) Cloud Service is fine, just as long as it doesn't delete things when you delete stuff locally on your computer. You'd be surprised on just how many Cloud Services do this, you upload all your stuff, then delete things on your local HD, only to find out that after 30 days, the files on the Cloud will be deleted as well. (To save them space, how nice...) So always read the fine-print on the terms and conditions. There might be some sort of sync-clause that grants them permission to delete your stuff after a period of time, after it's deleted locally on your HD.

However you access your files and if it works for you, it's fine. If you have the .psd files, there is NO NEED to keep JPEGS. If you ever had to re-create a JPEG, just do it from the .psd. 1:1 Previews across the network is going to be slow. Period. So having a Local Drive manage things while editing is a good thing. You could even setup a RAID0 for this purpose, as I've stated above, RAID0 is FAST and would be perfect to be used as a "Working Drive," then have the files moved to a RAID1 for storage. I would definitely invest in a set of CAT 6 Ethernet Cables on all devices and make sure your switch / computer is at 1GB (or faster at this point.) Cat 5e will "do" 1GB, but CAT 6 Ethernet is meant for Gigabit Ethernet. Of course, there are many other bottlenecks that you will have to deal with, like working on a friggin' laptop with a full HD, and the speed of the NAS itself and how it transmits data.  Yes, leave your computer alone when transferring large amounts of data, you are fine in that dept. They only thing that concerns me is if each file size is 4GB or larger. 99.999999% of the time, this isn't an issue, but for those who edit video, it could be. I use CAT 6 Ethernet Cables from Monoprice.com.

Oh, you want to store your HDs in a consistent temp / humidity environment. NOT THE GARAGE. Even though they aren't being used, there are soldered components that expand and contract during hot and cold periods. That HD that hasn't been turned on in three years which sat in the garage may not fire up if you try it. In fact, one of your assignments is to go through ALL of those drives in the garage and see if they work. You might have a nasty surprise. Or you might get lucky. It could go either way.

Hopefully you can follow my answer. I kinda rambled on. If I think of anything else, I will add to this thread, but I suspect you are gonna be busy these next few weeks.

[laural]

Thank you so much for your reply!

Oh man, I wish I hadn't chosen the hybrid RAID now. I thought it was truly RAID1 but then with some proprietary extras. I'm going to go with your suggestion and go with RAID1 if I can. Having the same firmware on them didn't occur to me but they are the same drives, and same age.

I never thought of testing the UPS, that they could have that short a lifespan was not even on my radar. The one I have now is an APC UPS 550VA BE550G, I have the Synology NAS and a Fios combined router/modem plugged into the back up+surge outlets, and in the surge only we have our TV, XBOX, cable box, and a DVD player. Do I leave the latter surge-only out when figuring if it's too much?

I think what I'm leaning towards doing immediately is getting two new 4TB each drives for the NAS unit so I can increase the storage now (and resolve to go back and cull what's on there now... ). I've looked up WD Black Cavaiar on Amazon and wondering if I have the ones you mean, would this one work as NAS? I ask because it says Desktop/PC and I do tend to leave it on. Though I could easily make it a habit of powering it down. I found the Red ones in stock on B&H but I will go for the Black Caviar if they are in fact OK in NAS.

I'm taking stock of my HDDs (I hadn't moved them into the garage yet but was thinking about it, so glad I asked!) and have enough room to dump a bunch to clear up space on my laptop and Phone to get me through until replacement hard drives come for the NAS unit.

I'm going to re-read the fine print for Amazon Cloud Drive. When I subscribed the terms said raw files were okay, as well as video, as long as they were personal files and not from a business/clients. I don't have anything syncing and do everything manually and everything is still there that I've moved off my Macbook to NAS. When I'm re-checking backups I'll compare the # of files as a quick reference and Amazon has never deleted anything so far. I randomly download files to make sure they're uploaded properly and everything looks okay. But still seems too good to be true or else I must be the kind of customer they're losing money on because right now I have 650GB in dng photos and 105GB in video uploaded. Though my NAS says it's got 1.35TB used. So I must be missing something or my PSDs (which I don't upload to Amazon) are taking up a lot of space. Hmmm.

One more question... I started an extended test on both drives ~48H ago and the drive I thought was fine passed, but the other has been stuck at 90% complete. I assume I shouldn't shut down the whole unit (through DiskStation, not the power button I mean) but I can't find a way to cancel the test or even any command line feature to xkill it or something. I'm trying to figure out if the drives can be hot swapped, but I don't think so, and have never done anything like that before anyway. Can you recommend a course of action?

Thank you again!

Laura

PS - I checked out your Instagram and see you are from Central PA, I'm from near Lewisburg

 

[Brian]

Yep that's the drive I'm talking about. Here is the newer model of the WD Caviar Black Drive. The primary difference between the Red and the Black is the Red is meant for 24/7 operation and to be in RAID/NAS environments but has higher failure rates. The WD Caviar Black has better performance (7200RPM Drives vs 5400RPM), generates a little more heat, and is louder. (Duh! It's a 7200RPM drive.) Are you planning on running your NAS 24/7? If so, I don't want to give you bad advice. Even though I HIGHLY recommend the WD Caviar Black Drives, I'm the type that would shut off his NAS when not using the computer. So maybe a RED drive is in your future? This is a tough call. I really like those WD Black Drives. Yes, both kinds will work in a NAS.

Here is a WD NAS that has WD Caviar Black Drives and is on my short-list. Notice the really big fan on the back? That's to compensate for the heat produced by the faster drives. (Yes, there IS A DIFFERENCE between a 5400RPM drive and a 7200RPM drive.)

For the UPS, for what is on the battery/surge-side, 550VA is fine. I have a 550VA UPS and cable modem / switch mounted on a backboard in my basement. I went a little crazy and ran Cat 6 wires all through my house, I hate WiFi. LOL!! Don't worry about the stuff that is plugged into the surge-only side for the rating stuff. Oh, UPS batteries last about 5 years on the average, and you could find a local replacement, Batteries + Bulbs carry them, or an equivalent battery store (Interstate Batteries Store come to mind.) That said, I've found that the Genuine OEM APC Batteries tend to last longer than generics, but cost more. APC and Amazon sell them.

9 hours ago, laural said:

So I must be missing something or my PSDs (which I don't upload to Amazon) are taking up a lot of space. Hmmm.

 

Or the type of filesystem that Amazon is using is different, making the file sizes different. Just like Microsoft says 1024MB is 1GB and the HD manufactures say 1000MB is 1GB. Amazon could also be doing a lossless compression type of thing, which is very plausible.

 

9 hours ago, laural said:

One more question... I started an extended test on both drives ~48H ago and the drive I thought was fine passed, but the other has been stuck at 90% complete. I assume I shouldn't shut down the whole unit (through DiskStation, not the power button I mean) but I can't find a way to cancel the test or even any command line feature to xkill it or something. I'm trying to figure out if the drives can be hot swapped, but I don't think so, and have never done anything like that before anyway. Can you recommend a course of action?

Let it run for another 24 hours. If it's still stuck and it's the drive that is dying...I would pull the power. LOL!! If there is no way to stop the test, you have no choice. It's already broke. As far as hot-swap, unless the unit specifically stated it on the box, believe me...they will make a big deal about being hot-swap, I wouldn't risk it. Shut it down and swap out the drives.

I've driven through Lewisburg, seems like a nice town. I live just outside of Harrisburg.

PS: Using RAID1 will cut your usable space in 1/2. A 8TB NAS unit will become a 4TB when using RAID1. So you might want to look at a 6TB drive or 8TB drive.

[laural]

Gotcha! Thank you for the clarification. I don't suppose it is wise to mix a red and black drive together in RAID1? Haha. I'm going to shop around and may just let what is in stock and my wallet make the decision, but I definitely could make the change to just shutting everything down when I'm not using it because it's only for storage right now.

9 hours ago, Brian said:

Or the type of filesystem that Amazon is using is different, making the file sizes different. Just like Microsoft says 1024MB is 1GB and the HD manufactures say 1000MB is 1GB. Amazon could also be doing a lossless compression type of thing, which is very plausible.

Had no idea, very good to know! You likely saved me from pulling out my hair trying to account for the discrepancies

Thanks for all the good info and help! Just sent some "beers" your & Damien's way. Not a beer person myself but heard the ABC brews from that area aren't bad

Edited February 16, 2017 by laural

[Brian]

Thank YOU!!