Photo Storage and Backup

[Marie Monforte]

I just bought a 27"iMac i5 with the 2TB Fusion Drive. (I also upgraded the RAM in-store to 16GB).  

I'm in the process of cleaning up the file structures of all our family photos and trying to figure out my ideal photo storage and backup system.  I would LOVE to have all our personal family photos, plus that current years' client files, in ONE place (not 6 different 1TB drives).  I would like this whole system to duplicate itself onto two drives (and backup to the Cloud as well of course).  Then, at the end of each year I plan to remove the Client files only and transfer them onto a 2TB external drive, which will then go up on a shelf for the next 3 years or however long I decide to keep it.  But I want to be able to leave my personal files on there year after year so I can always easily go back and scroll through them, or use them for projects, or whatever.  I also want to leave my computer virtually empty except for business operations files, and then pull only the session I'm currently working on onto the desktop, and then save the images onto the big hard drive when I'm done. So, I started looking into 12TB (2x6TB) options like those offered here: http://www.bhphotovideo.com/c/search?Ntt=WD+12TB+USB+3.0&N=0&InitialSearch=yes&sts=ma&Top+Nav-Search= 

My husband searched actual transfer speeds and found that there doesn't seem to be much of a real-life difference between the USB 3.0 and the Thunderbolt, so I was leaning towards the USB 3.0 since it's several hundred $ cheaper.  (see article in Macworld for reference: http://www.macworld.com/article/2039427/how-fast-is-usb-3-0-really-.html)  

But one of the B&H reviews on the 12TB model had this to say, (which kind of freaked me out):  "1 star: cannot disable encryption...compact, high capacity RAID 1 capable usb3 storage. 
Warning: Cannot disable the encryption... if the controller PCB fails, your data cannot be decrypted even if the drives are still good"

So it seems there are drives out there that offer a "disable encryption" option - trouble is, this reviewer failed to mention which. And I'm not sure I should even worry about that?

Long question short - What is your recommendation for storage given my file structure wishes?

Please be specific - I am not tech savvy. 

Thank you so very much!

Marie

 

 

[Brian]

I'm going to address the three sides of your question in different comment boxes.   

When using a mechanical (traditional) HD, that's where the bottleneck is with transferring data. So it's not a Thunderbolt problem, per se. Thunderbolt is still meant for 10Gbps /second (or 20Gbps with Thunderbolt II,) while USB 3.0 caps out at 5Gbps. 

Of course, real-world speeds are usually slower, but again...it all depends on what drive is being used. If you are using a Thunderbolt SSD drive, read/write speeds will be WAY faster than a SSD on a USB 3.0 port. I personally still like Thunderbolt for a main backup drive for hosting photos.

El Capitan has an interesting quirk, on some Macs, it likes spinning USB EHDs down and doesn't always wake them up. This is kinda important for things like a Time Machine drive. Even if you tell the power settings to never spin down hard drives.  No, there isn't a fix. It's an El Capitan quirk. Bonus: Like the Black Boxes issue in PS, it's random. It doesn't affect all Macs. 

 

Part 2 and 3 will be answered after I have my coffee.  

[Brian]

Part 2:

OK, there are a few things to keep in mind. These unit's HDs are configured as a RAID. RAID stands for Redundant Array of Independent Disks.

Huh?

What that means is two or more Hard Drives act either in unison or towards the same goal. There are different Levels of RAID:

• RAID0 - Two Hard Drives act as one large Hard Drive. So if you put two 6GB HDs and configure them as a RAID 0, you'll have a 12GB HD. The reason is that the data is spread between both hard drives.
  • Pros: RAID0 is FAST. It's great for cache files when you are editing video, as it just soaks up the data so effortlessly.
  • Cons: If one HD fails, the whole thing goes ka-plooey. I wouldn't recommend RAID0 for ANY type of long-term storage. You might as well keep using your 6 1TB EHDs.
• RAID1 - One hard drive is mirrored automatically onto a second hard drive. So if one hard drive fails, you can replace the failed HD and the data will mirror itself back automatically, or from a prompt of software.
  • Pros: The simplest form of redundancy. One hard drive is automatically copied over to the other in real time.
  • Cons: Storage Capacity is cut in half. If you have two 6TB HDs in a RAID1 configuration, you'll have 6TB available, not 12TB. The other con is there is no discrimination of the data from one hard drive to the other. So if you have corrupted data on one HD, that corrupted file is automatically copied over in real time.
• RAID5 - Three or more Hard Drives act in unison. If one HD fails, the other two pickup the slack and keep going. Your data is accessible. If two hard drives fail in a RAID5, the whole RAID Array is blown away, in which you lose all your data.
  • Pros: RAID5 is like the combination of RAID0 & RAID1. It's faster than RAID1 and you get the benefit of the data being spread over multiple HDs, so if one fails, it's not a big issue. Many of my Servers that I configure have a RAID5 for storage of their critical data and database files.
  • Cons: Cost. It's expensive to implement a RAID5 Array. You will need 3 identical HDs to start with, plus have a hardware controller or software package support RAID5, which is also not cheap.

To whom it may concern: Yes, I'm well aware that there are more RAID levels out there, but I'm keeping things simple with just these three.

With any RAID Array, the data is controlled by either a Software Package or Hardware Controller. It's better to get a Hardware Based RAID Controller. As with any software, if any data corruption happens, i.e. if that code-book gets corrupted, your data is GONE. Of course your data isn't available if the controller dies on you, but out of the two of them, a Hardware Based RAID is the best option in my humble opinion. Always go with a Hardware Based Device if you have a choice.

To give you an example, I had a customer of mine who had a server with a RAID5. One of the drives was down and there was a second that was about to fail. Since this was a big and fancy server, with hot-swap Hard Drives (meaning I didn't have to power down the server, just unlock and remove the HDs,) I pulled the failed drive, and installed the new one. The controller said, "Thanks! I will take care of this for you!" A few hours later, the new drive was implemented into the Array. I then pulled the 2nd flaky HD and let it rebuild. (I also did the 3rd one, just in case that was going to die, since they were all installed at the same time.) So after a few hours, the RAID5 Array was stable and the only thing my customer noticed was a slight performance slow-down while the RAID rebuilt itself, which is totally expected.

 

[Brian]

Now for your selections. The manufacturers are finally releasing new NAS systems with larger HDs. This Western Digital Unit caught my eye. I've had very good luck with the WD Caviar Black Line of HDs. It's what I usually recommend when it comes to Western Digital's products. I highly recommend that you reconfigure that HD as a RAID1, of course, you will have only 6TB available if you configure it that way. I think 8TB hard drives are the largest at this point, but I haven't really checked lately, they could be bigger, but I'm thinking 8TB is the largest as I type this.

QNAP's products are highly regarded in the industry. This QNAP TS-431 unit allows you to have the ability to create a RAID5. If you use all 4 hard drives as a RAID 5, after formatting, you should get close to 10TB total storage capcity, give or take. In addition, this unit has an Gigabit Ethernet Port, so all you have to do is hook it up to your router / switch and it acts like it's own server. If you configure it as a RAID5 with three HDs and one hot-spare with 4TB HDs, you'll get about 7TB or so of usable space, give or take.

Your dream of having 12TB of storage space is still a bit early, unless you want to use RAID0. (Again, not recommended.)  The hard drives aren't that big...yet. Still, 6TB is 6TB!! You will just need to cull and hack your images. Saving everything isn't a good idea anyway. Those images that will never see the light of day will continue to just sit there, taking up space.

[Brian]

So now we come to the part of even more redundancy and cloud storage. With 6TB or more of Data, you will be uploading forever. It's best to find a Cloud Vendor that accepts Hard Drives being sent to them with your data. You could send two or three 4TB HDs, one for client data, one one for personal data and one for misc. stuff (or overflow from one of the other two hard drives.) Investigate the Cloud Vendor thoroughly. Often they will delete files after a month if they aren't available on your computer. Not all Cloud vendors are the same, so read the fine print. It's not a "Set it and forget it" type of thing. Don't ever assume that it's okay to delete any file on your HD to free up space, because it's in "The Cloud." It may not be.

As you can see, the prices are starting to add up. Hopefully you'll have about $2000+ to spend on this project, because this stuff isn't cheap. You'll probably want two of the NAS systems, one as a primary, the other as a backup / secondary. You would copy NAS #1 to NAS #2 on a weekly / monthly basis. Then upload the files from either NAS #1 or NAS #2 (Probably NAS #2) to the Cloud. Of course, after you do your initial data-dump. Most ISPs have data caps and I'd hate for you to get penalized for going over 250GB of data.

You might not want to purchase a second NAS, and that's fine. Just purchase a bunch of 4TB to 6TB hard drives to copy stuff to. Then backup your NAS to the Cloud. I know that you are trying to get away from having a bunch of HDs floating around, but the truth is to do that, it's gonna cost money. So before you do anything, have a realistic budget set. If you did go with the QNAP, I think you can configure two hard Drives as RAID0, then mirror those HDs to the other two. Check the manual.

[Brian]

As for which one to buy....

That WD is probably the easiest one to use for the "average" user. I'm sure there are setup wizards that will configure things for you. The downside is that this unit is a glorified external HD and your Mac will need to be powered on in order to access the data.

The QNAP though...it allows more flexibility. If you wanted to create a home media server, the Ethernet Ports are a bonus as it will act as it's own device. Streaming in house isn't going to be anywhere near 100Mb / second, unless you are streaming 4K HD over your network. If you didn't want to use the Ethernet ports, it does have USB 3.0 ports to use, as well as a eSATA.

It all boils down to budget.

[Marie Monforte]

Thank you so much for such a well thought-through and informative response! Unfortunately, my budget is definitely under $1000, so I guess I'm going to have to come up with an alternative plan.  If you would care to weigh in on what might be a good option for me given my budget constraints, I'm all ears.  It sounds like manually managing several 2-4TB hard drives is going to be more realistic?

 

[Marie Monforte]

OK wait, I just reread this (for the 3rd time - told you I was slow, haha) and it seems that IF I'm ok with just 6TB per drive, that I should just get the one WD unit you linked...right?  I think I can then back that up to the cloud?  And still be reasonably ensured my data will be fine as long as I'm under the 6TB?  Again, sorry...I need things spelled out real simple-like, haha

[Brian]

Yep. Buy this and be happy:

http://www.bhphotovideo.com/c/product/1148572-REG/wd_wdbdtb0120jsl_nesn_my_book_pro_12tb.html
 

Set it up as a RAID1.

[Brian]

@Marie Monforte: Use the USB 3.0 port OR try the Thunderbolt port. It has two Thunderbolt Ports and one USB 3.0 port.

See which one is faster and let me know.

[Brian]

Quote

 

One last thing... Other than file transfer speed, is there another significant reason I should spend $899 on the one you linked, versus $449 on this one: http://www.bhphotovideo.com/c/product/1053138-REG/wd_wdblwe0120jch_nesn_12tb_my_book_duo.html

I understand it's only the Red Level and not Black Caviar - just not sure if I could still get away with it, or its just too darn risky.  (Or maybe its because of the data encryption thing?)

 

Personally, I DO NOT like Western Digital RED drives. Yes, they are supposed to be meant for server environments, but I have seen way too many problems being reported with them in the Geek Groups.

Why did I recommend the QNAP device even though it had WD Red Drives? Because I hate Seagate Drives even more than WD Red Drives. It was the lesser of two evils in that case.

[Brian]

What HDs does Brian buy for his computers? Western Digital Caviar Black HDs. That's why I recommended that particular WD NAS.

[Marie Monforte]

THANK YOU Brian!!! 

[Marie Monforte]

Ok, sorry to open this up again...but I do have one more question.  

Maybe you answered this in a way that I didn't interpret correctly, but I am wondering about the data encryption.  One review of the WD My Duo 3.0 USB indicated there is not option to disable the encryption so that if the PCB connector fails I cannot read my files.  I cannot find any information about whether or not the Black Tier Thunderbolt version has a "disable encryption" option or not?  And is this a big deal?

[Brian]

I didn't tell you to buy that WD Duo. Save up and get the one that I linked to.

But it Right...Buy it Once

All RAIDs have their own way of doing things and encrypting data if that option is turned on or even available. So that's why I recommend using a RAID that is configured and controlled by Hardware, and not software. 

Yes, if the data is encrypted and something happens to that code-book, you are SOL. 

[Brian]

So that WD Duo comes with HDs that I don't like, and software that encrypts your data and can't be turned off....

Hmm....are you nuts? I would avoid that one like the Plague!! It should be off the table at this point. I know it's less than $500. I get it. I'm broke myself. 

When I recommend things in Ask Brian, I have your best interests in mind. I receive no compensation from B&H for recommending one brand over another. Often I will link to the more expensive option. Not because I have expensive tastes, well I do but that's not the point, the truth is I hate-hate-hate wasting money on this sh*t!! It doesn't do anyone any good by buying something that is "almost as good as" and then blowing up in 3 months. 

So please save up your money. Resist the urge to have instant gratification. 

Buy it Right...Buy it Once. 

[Marie Monforte]

Apparently I wasn't clear. I want to buy the "Black Level" Thunderbolt one you recommended, I just want to make sure, before I do, that it won't automatically encrypt my data,  like the "Red" Duo USB version does, but instead will let me disable the encryption.

I am not able to find anything about data encryption in the description on B&H so I thought maybe you knew about it. 

[Brian]

The Black WD NAS doesn't seem to encrypt. At least from what I've read. It might be a newer model as well. 

[Brian]

One more than thing, and this is what you actually might be asking:

With a RAID setup in a NAS device like this, if the NAS itself were to fail, then you would need to get a replacement NAS in order to access the hard drives. This is true for all NAS units, regardless of brand.

These HDs can't simply be removed and hooked up to another computer. They don't work that way. If you want to have that option, then you are looking at purchasing two or three individual EHDs and not a NAS. 

[Brian]

NAS = Network Attached Storage.