Backup workflow

[Eran]

Hi , till today all my raws and psds are backuped up weekly to external USB storage.

My raw file are huge (50mb seamless compressed ) and the psds are sometime 2-3gb (I also do composites ).

I purchased a qnap Nas with 8Tb storage and also have backblaze b2 account .

I have 3 HDs on my PC:

500gb M2 for Windows and programs .

4tb 7200rpm sata for raws and psds

1TB ssd as scratch disk for Photoshop .

 

What would be the best way to backup /archive my raws and psds ? Should I also convert the raws to compressed dng (reducing the dynamic range but saving a lot of space also)

 

Should I "sync" files to NAS and from there backup to backblaze or backup the HD to Nas and backblaze (the later is keeping me safe from ransomware but makes it terrible to access specific files if needed )

 

Any workflow will be welcomed ?

[Damien Symonds]

I hope you don't mind, I've moved this post into Brian's area, since he's the expert in these matters.

All I can suggest is to run through all the worst-case scenarios in your mind:

• Fire or theft
• Internal hard drive failure
• NAS failure
• Backblaze goes out of business or corrupts your files

All of those seem very unlikely, but they're all possible, so you need to be confident your system is robust enough to withstand any of them on any given day.

30 minutes ago, Eran said:

Should I also convert the raws to compressed dng

Absolutely not, because of Backblaze.  https://www.damiensymonds.net/2012/02/to-dng-or-not-to-dng.html

[Brian]

Backblaze is not 100% bullet-proof. In fact...

IF YOU DELETE YOUR FILES OFF YOUR HARD DRIVE, BACKBLAZE ALSO DELETES THOSE FILES OFF THEIR SERVERS!!

That's right, BackBlaze likes to have a "Mirrored" image of your hard drive's data files. So if you save a bunch of stuff to your HD and then run out of room, once you start deleting files, sooner or later BackBlaze's servers will also delete those same files on their servers' HD, without letting you know, because they were no longer on your HD. It's a way for them to save on storage and for them not to be dumping grounds.  

Now what's the "Best" way depends on your budget. If it's small, then culling and deleting old stuff that will never see the light of day with the remaining data stored on a EHD. How? I'm old school, copy and paste. You could also use Microsoft's Sync Toy to do it automatically. I would not convert your Raw files to DNG. Cull your RAW files if you have to, converting things to DNG is converting them to a proprietary format. Adobe may not be around forever. If you don't have physical prints of your important photographs, your descendants may not have access if Adobe goes out of business. Don't laugh, Sears has been around for a long time, so was Kodak. Now Sears is on the verge of going out of business forever and Kodak is no more. So from my perspective, I'd rather have true camera OEM negative files to work with rather than a converted file that uses a company's proprietary format.

Now if you have a healthy budget, and I'm talking about $2000 or more, then you can get a nice NAS to backup stuff. Synology makes great NAS units and they have software that automatically backs stuff up from your computer to the NAS. (Basically what you are wanting to do.) Of course, you could start out smaller and then add hard drives later to increase space but there is a trick to doing that. The good news is you only have 4TB of data, that's not a lot by today's standards. You could invest in a NAS that has 12TB of capacity but depending on how the 12TB hard drives are configured will determine the final capacity. Clear as Mud?

What I mean is if you have two 6TB HDs and make them a RAID0, you will have 12TB total (don't do this). If you configure the two hard drives as a RAID1, you will have 6TB mirrored. If you have three 6tb HDs in a RAID5, you will have 12TB of storage with 6TB reserved for protection. Personally, I've been looking at the Synology DS1019+ NAS Unit. The Synology DS918+ is also excellent and is highly recommended. What's the difference? Another slot for a HD, faster CPU and more RAM comes with the 1019+. Basically it's the 2019 model. Western Digital Gold Hard Drives is what I'd use for the NAS, you do not want to go cheap on your NAS HDs. For you 4TB drives are the smallest I would go, but would probably look at 8TB drives (three or more) and those don't come cheap @ $340 per drive. So with three 8tb HDs, plus the NAS, without a m.2 drive for caching, you are at $1670 before tax and shipping.

Now Damien has made some very good points and for his scenarios the only way to combat them is to utilize the 3-2-1 Backup Rule. Which means:

1. Have at least three independent copies of your data.
2. Store the copies on two different types of media.
3. Keep one backup copy offsite.

Of course, you could have to copies on two separate external hard drives or other media, this is not a hard-and-fast rule. Basically you don't want to lock yourself into one format. For example, WD Green external hard drives are JUNK. I love the higher-end Western Digital products but the cheap drives at the Big-Box-Stores I avoid like the plague. What happens if you have your data backed up on two different green hard drives? And one or both fail because they suck? Now you are stuck and a data recovery place becomes involved. Fortunately, there are cloud-based products like BackBlaze...but as I've pointed out above, they aren't fool-proof. That said, you could have one EHD and a Cloud Service and that would full-fill point #2. For point #3, you still need another EHD that is a mirror of your HD for point #2, and it kept off-site. This way if you have a fire or flood at your main location, and BackBlaze's servers go off-line, you still have a good backup at another location.

So as you can see, either way you choose...it's gonna cost money. So how is your budget?

[Barry685]

Amazon Prime also includes unlimited photo storage including raw's (but for some reason not Psd's) you can use either:

https://www.goodsync.com/?gclid=Cj0KCQjwyoHlBRCNARIsAFjKJ6CS93HkL3Bl0uhroHc4q_JWS9NbettOpZHtSz39Zom9oasL4ik2WMUaAkQPEALw_wcB

or

https://www.2brightsparks.com/syncback/sbpro.html

To automate the backups. The only thing you have to change is to make sure you filter the backups to only include photos.

[Eran]

8 minutes ago, Barry685 said:

Amazon Prime also includes unlimited photo storage including raw's (but for some reason not Psd's) you can use either:

https://www.goodsync.com/?gclid=Cj0KCQjwyoHlBRCNARIsAFjKJ6CS93HkL3Bl0uhroHc4q_JWS9NbettOpZHtSz39Zom9oasL4ik2WMUaAkQPEALw_wcB

or

https://www.2brightsparks.com/syncback/sbpro.html

To automate the backups. The only thing you have to change is to make sure you filter the backups to only include photos.

Cool , can you trust Amazon as backup service? Also , do you happen to have NAS or you backup directly from PC ?

[Barry685]

I backup to an internal drive. and an external drive, Amazon Prime, and from one particular pc,  backblaze as well. I have no reason not to trust Amazon any less than Backblaze. Also with Amazon Prime you can backup as many Pc's as you want. I currently have about 11tb stored on Amazon. 

I do also have a 2 bay Qnap NAS with some image files on it, but it's primary function is for a  Plex Server ?

[Eran]

43 minutes ago, Barry685 said:

I backup to an internal drive. and an external drive, Amazon Prime, and from one particular pc,  backblaze as well. I have no reason not to trust Amazon any less than Backblaze. Also with Amazon Prime you can backup as many Pc's as you want. I currently have about 11tb stored on Amazon. 

I do also have a 2 bay Qnap NAS with some image files on it, but it's primary function is for a  Plex Server ?

Oh , 
I have amazon prime / qnap (2 bays X 8TB each) /  8 TB Western digital elements.
I backup from my PC to my qnap (using freefilesync but Its probably better purchasing a good backup app) and from the QNAP to the 8TB HD & Backblaze...
Backblaze upload is very slow.
Btw - When you backup to Amazon , you backup the raws + xmp (they barely take space).

Did you purchase goodsync license or you use the free version with the limitations ? 

Regarding the plex server , is it because you stream to PS4/Xbox ? 
 

[Barry685]

I don't have xmp's  ( I use Capture 1 not ACR) If I did have xmp 's I would not upload them because they are not photos (even though they dont take up much space). 

I don't use any of the above programs, but I have tested them and I know they work.   I use this for free: 

https://www.cloudberrylab.com/explorer.aspx

But its completely manual, so right after a session I simply upload the folder that contains the raw's.

 

Im also on a pc and am not sure of MAC availability on those programs.

[Eran]

40 minutes ago, Barry685 said:

I don't have xmp's  ( I use Capture 1 not ACR) If I did have xmp 's I would not upload them because they are not photos (even though they dont take up much space). 

I don't use any of the above programs, but I have tested them and I know they work.   I use this for free: 

https://www.cloudberrylab.com/explorer.aspx

But its completely manual, so right after a session I simply upload the folder that contains the raw's.

 

Im also on a pc and am not sure of MAC availability on those programs.

Do you use C1 because you shoot tethered a lot or because you use Sony/Fuji cameras and its much better as raw processor for them ? 
(I purchased C1 v. 12 lately but didn't use it much yet because of Damien's tutoring... required using ACR)

[Barry685]

I have been using C1 since v3 so I'm just used to it. I am taking the raw and skin classes from Damien and I am trying to apply his Raw processing technique to C1.

[Eran]

26 minutes ago, Barry685 said:

I have been using C1 since v3 so I'm just used to it. I am taking the raw and skin classes from Damien and I am trying to apply his Raw processing technique to C1.

Let me know how it works for you , Most annoying thing with the diff. raw processors is that WB presets isn't the same (which is so weird) but Daylight in ACR and daylight in C1 show different results...