Do I really need a RAID drive, and why? And is my iMac failing?

[mistyprochaska]

To start at the beginning, my 2011 iMac has been behaving strangely. Slow, frequently unresponsive, constant rainbow wheel, tons of problems with LR being really slow, LR displays the wrong preview for the photo I am working on (I basically just use Chrome and LR on the computer, so that is where I notice more problems). It takes forever to start up. Last night, little program icons just started disappearing from the strip on the bottom- PS, then iTunes, and so on. The computer locked up then, and I shut it down by pushing the button, and then it was ok. SO my husband and I started talking again about if it is time to replace it, and in looking at everything, we realized that the 3TB hard drive where I store all of my photos is almost totally full. At the rate I shoot, I probably have about a week left before it is full, unless I start culling like crazy. Which I need to do too. 

So I started reading here last night. My husband has wanted me to get a RAID system forever. And I looked at the RAID system that Brain recommended elsewhere here, http://www.bhphotovideo.com/c/product/1148572-REG/wd_wdbdtb0120jsl_nesn_my_book_pro_12tb.html

There are some things I don't understand. Normally, I replace my EHD every one to two years, or if it starts making odd noises, to try to catch it before it fails. Do I need to replace these internal hard drives like that too, or are they longer lasting/more reliable? I can't really tell for sure, but I think they are $300 each. If the advantage is that they mirror each other, couldn't I get two less expensive EHDs and do that manually?I hate all of this, because last time I changed out my EHD to a new one, it was a terrible mess as far as LR and getting it back up and running, I made some sort of mistake, and ended up on the phone with Adobe Customer Service in India for a day. And I cried a lot, but then didn't take notes on how I did the move to avoid the pain this time. I am just dreading this process, but I know I have to do it. As an aside, we also have CrashPlan. 

The other thing is, does it sound like I need a new iMac, and if so, does it make sense to get it now, at the same times as the new EHD? Or should I run CleanMyMac or Onyx on my current machine, add more RAM from Crucial, or take it in to get looked at? I am currently a hobbyist but I shoot A LOT and plan to go into business of some sort in the next year or two. I have a six and seven year old that will get my current iMac when I get a new one. Is 2011 old for an iMac? 

Any advice or clarification you could provide would be SO appreciated. As much as I would like to ignore this situation and do nothing, I am pretty sure it isn't going to work out for me much longer. 

 

 

[Brian]

How full is your internal Mac HD? If it's almost completely full, that is a major problem.  Macs get cranky when your HD is about 75% full and you will notice a performance hit. They get weird and start corrupting themselves about the 95% mark. When you hit 99% full, you are asking and begging your Mac to give you problems. 

Hard Drives are NOT dumping grounds. You can't expect to fill them up. You seriously need to do some major culling. Be ruthless. You do not need 75 Raw Image files of your Lunch from 4 years ago. 

Yes, internal HDs go bad, just like externals. It will run you between $250-$300 to have Apple do it, which they should. I would cull like crazy and free up as much space as you can, then run a Time Machine Backup without any external drives hooked up, except for the TM drive. 

Now, there is the old cliché, Garbage In - Garbage Out. Which means that if your data is somehow corrupted and gets backed up, when you go to restore said corrupted data, you'll go into a complete circle. So I'm hoping the data files are intact, and it's just your programs that need to be re-installed. 

Now for RAID. That stands for Redundant Array of Independant Disks.  Some people refer to the I as "inexpensive," which was the original meaning. Anyway, the keyword in the term RAID is Redundant. Which means if something fails, there is another component waiting to pick up the slack of the failed component or just take over immediately. 

With RAID 1 or RAID 5, if a single HD fails, the others will pick up the slack and keep going until you replace the failed drive. They have the benefit of redundancy, but still aren't to be treated as dumping grounds. 

So I would start killing files...like now. You want 25% free. So on a 1tb drive, no more than 750GB taking up space. This also includes Externals. The reason is the Mac OS treats an external drive the same as the internal drive. Remember, this is a OS that is based on UNIX (pronounced You-Nix) and not Windows. HDs are "Mounted" and all treated the same. 

After you do that, then we will talk on what to buy. 

[mistyprochaska]

The internal hard drive on my Mac is 2TB, and there is 896.18 GB free. 

The external hard drive where I store nothing but photos and back-ups of my LR catalog is 3TB, and there is 194.69 GB free. 

I am guilty of not culling. I have exactly photos of my lunch from four years ago in there. It is all true. Every year I tell myself I am going to cull for 15 minutes a day! Or my only project for the year is culling and keywording! And no. I have never, ever come close to actually filling a hard drive though.  I wouldn't say that I am an idiot or that I have terrible computer skills and don't know what I am doing- I am just not culling, and keeping too many photos. Last night, I used Onyx to clean up, repair scripts, etc., and that made no difference in performance. 

I think that there was a miscommunication in the language I used when I asked my question. Regarding the RAID and EHD and internal hard drive part. In looking at the RAID you recommended, I was trying to figure out if the hard drive used with the RAIDS, which I called internal, are better quality or will need to be replaced periodically. I was having trouble finding them by themselves, and since I am not familiar with the RAID system, I don't know what they are "called". I don't replace my iMac's internal hard drive ever and don't plan on it. I am just trying to make sure it makes more sense to buy a RAID vs. individual EHDs, given that I have Crash Plan. And I am trying to figure out if it is time to replace my iMac or not. And I have figured out that I need to cull, cull. cull. And I will start doing so as fast as as I can. But I am pretty sure I also need to order a new hard drive and maybe a new computer. Don't yell at me. I hope that makes sense, I have to take my son to gymnastics, then I will be back. 

[Brian]

All RAID does for you is give you the peace of mind that if one HD fails, there is another one that allows access to your files. A RAID is nothing more than separate hard drives that are acting in unison for a common purpose. 

>> With the exception of RAID 0. If one of those HD fails in a RAID 0, the whole thing goes ka-plooey. <<

You need RAID 1 or RAID 5 for the redundancy that I'm taking about. (For the technical crowd, yes...I know there are other levels of RAID, but I'm keeping things simple for the majority.) 

With your main drive and 896 GB free, you are fine there. Using Onyx isn't going to fix your problems. Onyx is more of a icing on the cake. It is possible that your internal HD has bad sectors or your OS is corrupted in some way. 

For a performance boost, clear off your Mac Desktop. Seriously. The more crap you have on the desktop, the slower a Mac runs. Got 1300 photos of the "Smith-Jones" wedding? The Mac OS treats them as 1300 open files. 

You do not need any fancy software to speed up a Mac. Just clean off the desktop. Seriously. That said, a software program that I do recommend and use weekly is CleanMyMac from MacPaw.com. It helps keep things like cache files and other crap that gets installed in check. 

As far as the RAID HDs, they usually aren't your typical HDs you get for $79 from the local box store. They tend to be a little higher end, but it could go either way. Chances are, you'll have a better chance of getting higher quality drives if you spend more. The ones I link to, I would purchase myself. 

 

[Brian]

All HDs will wear out eventually. There is mechanical and electronic parts contained within them. Hard drives are the number one thing that I replace in computers. Followed by power supplies. 

Keep in mind, you get what you pay for. If you are buying the low-end economy or "Green" HDs on a consistent basis, then your failure rate will be higher than a higher end HD. I personally would never recommend a EHD from a big-box store, especially ones that are meant for important things, like client photos. 

Personally, I like G-Drives. I own this one: 

http://www.bhphotovideo.com/c/product/1013785-REG

[mistyprochaska]

So is there an advantage in going with the WD RAID you posted and using it as RAID1, vs. having two seperate hard drives and replicating the data on one to the other regularly? Through software or manually? And is it overkill to have a RAID system along with CrashPlan? 

I will clean off my desktop. There are ROES links there for several companies, and quite a few exported Jpegs in folders. I will also do some research on ways to optimize LR so it runs faster. That is the biggest issue.

And I assume you think Drobo is very overpriced?  

[Brian]

I'd kill those exported JPEGs first. ROES links aren't going to slow things down that much, it's all the other crap. 

RAID1 is "Automatic Drive Mirroring." Meaning things get written to HD A and then are automatically duplicated to HD B. That's why the "12TB" drive becomes a 6TB drive. You have two 6TB Drives in that unit. I think that WD has Caviar Black Drives, which is what I recommend. 

As far as overkill? When it comes to backups, there is never overkill. I had a customer who needed to replace his worn out tapes for his server. Those tapes were expensive. So instead of replacing the whole set, he tried to save money and only purchase a couple. Which never made it into rotation. 

Guess what happened?

His server crashed one day. His previous night's backup didn't restore properly, due to a crappy worn-out tape, so we had to go to a full backup that was two years old to get him running. So instead of spending $1000 on a new set of tapes, he now has to hire someone to input 50,000+ records of transactions and other data. 

So if you are shooting professionally, you owe it to yourself and your clients to have your sh*t together. Lawyers aren't cheap.  

Oh, creating a separate catalog for each gig helps keep LR nice and speedy. It's when you have 50,000 photos on one catalog do things slow down to a crawl. Like HDs, LR isn't a dumping ground. 

Drobo is overpriced just like everyone else. 

[mistyprochaska]

Well, I don't yet shoot professionally- I just decided to move towards that after being a serious hobbyist since 2011. And for me, lawyers are cheap (I am married to one. )

I do have 142,093 photo in a single LR catalog. I have asked people about this, and lots of people assure me that this is ok. Yep, it is one massive single catalog. I will move all of the jpegs though. 

I just have to make a decision I guess, regarding what level of security I want. I do understand the differences between RAID1 and RAID0. I have been researching for the last 24 hours. I have been told that one big EHD and Crashplan is enough, to buy a Drobo because it is easy, and then I am looking at the option you have presented here, the WD RAID drive. I have no idea about the need to replace my iMac or not. I thought Onyx and CleanMyMac performed many of the same functions. 

[Brian]

142,093 photos. In a single catalog? 

::: FACEPALM :::

You have been given bad advice.

Those people that said that its ok to have that many in a LR catalog should find the closest tree and apologize to it for stealing the oxygen it produces. 

Just because you can do something, doesn't mean you should. 10,000 photos, fine. One hundred forty-two thousand? Yeah...LR will come to a screeching halt. 

[mistyprochaska]

So how would you propose I divide them? Right now they are organized in folders by year and month. I have about 1800 photos per month (on average). And need to cull, as we have covered. I could try one catalog per year, I guess?  

I have read and talked to a lot of people, and some agree it is a bad idea, but many people say it doesn't make any difference. 

And if you were in my situation, would you do the RAID drive, or the single G Drive you linked plus Crashplan?

 

[Brian]

I'd export the whole catalog and start fresh. A catalog that large is extremely dangerous, and WILL kill performance.

Onyx doesn't do a whole lot in terms of performance. What it does is nice, but it's not extraordinary. If you haven't upgraded to 16GB RAM by now, you should. That will give you a performance boost. Especially if you are at 4GB. CleanMyMac does more than Onyx, but it really keeps the crap in cache files in check. For performance these three things are what you are after:

1. Clean Desktop

2. 16GB RAM or More

3. HD that has at least 25% free or more.

Those three things will give you performance. CleanMyMac helps with #3. 

The problem is, it's not your file organization. It's the LR catalog. They way you are doing things is great if you used Adobe Bridge. But that's a whole other topic. 

You might want to get into the habit of creating catalogs based on theme or job. For example, the "Smith-Jones" wedding gets its own catalog. The Fall Mini 2016 Sessions get their own catalog, the Spring Mommy & Me sessions get their own catalog etc. Store the catalog with either the Raw files or edited .psd files. Kill the JPEGs. You can always recreate JPEGs from the edited .psd files. JPEGS do nothing but eat space after exporting / uploading a job. 

Whatever you do, don't go above 10,000 in a single catalog. 

[Brian]

It depends on you personally if you want a RAID 1 or not. RAID 0 is great if you edit video and need a spot for cache files. Otherwise, go RAID 1 or RAID 5.  

The G-Drive is just like you are doing things now. The one I linked to contains a higher-end HD. The WD RAID is also a good choice and having things automatically duplicated gives peace of mind if something goes wrong. I can't make this decision for you. It's a personal thing. The RAID drive will also show up as one HD, not two.

Personally, I think the next EHD will be a RAID unit of some sort. The G-Drive will eventually become my Time Machine Drive. 

[Brian]

"So is there an advantage in going with the WD RAID you posted and using it as RAID1, vs. having two seperate hard drives and replicating the data on one to the other regularly?"

Advantage is things get copied from one HD to another in real time, rather than weekly or monthly, etc. 

"What about Crashplan or something similar? Is that overkill?"

No. Having a off-site backup is part of a thorough backup plan. What happens in case of Earthquakes, Fire or Floods? Sometimes floods aren't caused by rivers and streams, they are caused by a failed sump-pump or broken pipe in the ceiling. 

[Brian]

@mistyprochaska: I'm truly sorry. I was a bit of a Major Ass earlier today.

I really shouldn't answer questions on my phone. Especially when battling a Server that was being stubborn which was causing me to be very grumpy. So let's start over!

I never did answer your question about your 2011 Mac properly. I'm running a 2009 iMac myself, and it works fine. I also am still using a D300s with the same lenses. So it's still around the 2010-era for me. I do have 16GB of RAM installed, which has helped. So if you have a camera like me that's a bit dated, you should be able to keep running your Mac until it dies. It does sound like you are doing everything "correctly," CleanMyMac, Onyx, etc. The only thing I can assume is you have a full mac desktop, along with a LR catalog that has way too many photos...those things KILL performance. As I've said earlier, adding RAM from Crucial is a good thing and you should be able to take it up to 16GB of RAM. So if you are like me, using the same camera from a few years ago, you can keep going. That said, if you do upgrade to something that has 24MP or larger, then that might cause issues. For example, if you upgrade from a Nikon D90 to a D810. Your Mac will not like that change. Neither will your externals.

Now, here is the fun part. You'll get mixed answers in this dept. The rule of thumb that I've come up with is if your 2011 iMac has a built-in DVD SuperDrive, you can go up to 16GB. If it doesn't have a DVD Drive and it's one of the "newer" styles (from 2012-present) you can go up to 32GB, ONLY IF you have the 27" model. The 21.5" models go up to 16GB, even though Crucial themselves might state otherwise. This info I've come across from Apple's literature over the years.

This Western Digital MyBook Pro comes with "Caviar Black" Hard Drives. I've owned these hard drives for YEARS. The run fast, run cool, and are extremely reliable. They are the ONLY internal HD I will recommend or install into my own equipment. In fact, I've made my own externals (Like my Time Machine HD) that contains a WD Caviar Black 1TB drive. Bought it in 2009. Works fine in 2016.

Unlike a lot of the "off-the-shelf" EHDs, these hard drives are kinda like the "Canon 5D Mark III" of the world. They aren't the low-end "Green" Drives which tend to drop like flies when you need them. Often, you will find the lower-end "Eco-Friendly" or "Green" Drives in many externals. You will have to seek out the "Pro-Grade" or "High-Grade" Hard Drives. The reason is the "Green/Eco-Friendly" HDs are cheap to produce, combine them with a crappy external case, and you can sell a EHD with a very high profit-margin. People eat those drives up. "$79 for a 2TB EHD!! Great!! I have 6 sessions that I need to download!!!" Uh-huh. Que the panicked photographer six months from now who can't get to their photos.

Let's talk about RAID...again. Thanks for reading over the other thread. For the majority of the people on this website, RAID 1 is fine. You have the benefit of instant real-time duplication of your files from one to the other. That's the nice part about RAID 1. The downside to it, is if your file is somehow corrupted on Drive A that corrupted file is automatically corrupted on Drive B. So in reality, think of RAID 1 as your typical external that you are used to, with the added benefit of having things automatically duplicated in real time. That small chance of something going horribly wrong in between your duplication session is non-existent. For example, having a EHD fail on a Monday before your weekly duplication on a Thursday. With RAID 1, stuff is copied in real-time.

How does this work with Cloud Backup? In a word: Speed. Once you replace the failed HD, the working HD in a RAID 1 automatically copies stuff over. That said, this depends on the RAID Device and the software. Sometimes you have to tell it to start the rebuild. Other times it does things on its own. You'll have to read the manual on this one, I'm afraid.

Now I'm sure you are wondering why I said speed. After all, people have fast internet connections these days. Why not simply download the file in a few seconds? I would say that's true. You have to remember that RAID has been around for a long time. The Cloud didn't really start until 2011. So you have older-based technology, which is proven. Keep in mind that Cloud Backups tend to vary on how long they keep things on their servers. Some will delete files monthly, others will delete files if you delete them locally on your HD. So it's one of those things, read the fine print. Not all Cloud Storage Vendors treat data the same and I'd hate for you to get a nasty surprise.

As for that WD External RAID, it looks really interesting. I like the fact that it has WD Caviar Black Drives and they are hot swap hard drive caddies. What that means is you can pull the HD while the thing is powered on, replace the failed HD and have it rebuild the RAID. Most of the Servers that I build use this technology. It's awesome.

I think you also had a question about Drobo. Drobo is Drobo. It was one of the first NAS (Network Attached Storage) systems geared towards the average user. Their devices are extremely easy to setup. I haven't looked at their models recently, the reason is it's a software-based RAID, and "Their Version of RAID," rather than an industry standard system. That's why you can mix and match different hard drives in a Drobo. A 1TB Drive and a couple of 2TB drives with a few 500GB HDs can be installed in a Drobo and it creates a "RAID." Traditionally, you had to have ALL of the HDs in a RAID be the same, or have the same capacity across the board. No mix-and-matching. So for that feature, you are paying a premium. That said, you end up paying the same amount in the end if you went with multiple high-quality HDs that were the same type/capacity/model number for stability. In addition, Drobos used to have proprietary power supplies, so if it died out of warranty, you would have to pony up the $$$ to get access to your data. There was a big situation years ago that Scott Kelby went through that really caused an uproar. I'm not sure if Drobo changed things or not. For me, it wouldn't be a big deal. If a Drobo used a standard 3-prong computer cord, it still would have a proprietary power supply in the damn thing. Just like some computers.

So, hopefully I answered your questions. Any more, send 'em my way. I'd be happy to help.

[mistyprochaska]

It was strange today, because I remember talking to you on FB about something years ago, and I had remembered that you were really nice. So I was surprised to feel like you were talking down to me today, and I guess just the kind of response I got from you in general was surprising. I appreciate your clarification and apology. 

I researched the issue of having a large number of files in a single LR catalog, and almost everyone I talked to said it shouldn't be a problem, and that to divide all of my personal work into smaller catalogs would just be confusing and difficult to search. Many people talked about having more photos than I do in a single catalog without issue. So for now, I am going to leave it. When I start doing client work later this year, I will create a new catalog for each job/client. And if I can't solve the performance issues, I will revisit the idea of creating multiple small catalogs. 

Camerawise, I have two Fuji bodies, and I had a 5D3 until last October, when I switched to a Nikon D750 as my main camera. And I always shoot RAW, so yes, big files. I am not sure what to do about the EHD, still. I would like at least 6TB if I were to get a single EHD for photo storage, and that is hard to find- you are looking at RAID drives when you look for that size. I don't know if I need the RAID1 mirroring situation happening because I do think I am happy with or confident in Crashplan as a backup. I do understand when and what CrashPlan deletes and under what circumstances. I am not opposed to a RAID drive either though, I am just not sure if it is necessary given the additional cost. I don't need immediate duplication- I looked at CrashPlan tonight and the photos I took this evening were already available to download while I was still editing them. For any situation I can forsee, that is fast enough for me. 

I am also a little confused on the response you gave regarding the WD RAID drive I mentioned- it was your recommendation in another thread to look at that specific set-up. But above you said it looks interesting like you are not familiar with it. I guess I just wasn't clear on if that is the one you would recommend above all others. I don't think I am interested in Drobo based on all of the reading I have done about it. 

I think I am maxed out on RAM in my current 2011 iMac. I am going to take it in and have them do some diagnostics and then try to make a decision regarding if it makes sense to just get a new computer, or try to get this one running optimally somehow. My six year old and seven year old will need a computer soon, and giving them this one is a consideration for sure. It would work really well for word processing and other basic needs like that. I do have to figure out the EHD thing as soon as possible, as I am very nearly out of space. 

Anyway, I think I am on my way to figuring it out. I do appreciate your follow up response, and thanks for all of the information you have provided. Misty

[Brian]

I don't have an unlimited budget to try out these devices.  I just go by my own personal experience and what I know. That WD unit that was listed in the other thread was when I first discovered it. Looking at it again did increase my interest. Especially since I saw a future model coming with 8TB drives. 

RAID is just one option out of many. It depends mainly on your overall backup strategy. Most people don't have RAID and are fine. 

[Brian]

A D750 does have a 24MP sensor. So it might be time to upgrade to a new Mac. If you do, please get a 27" iMac and skip the 21.5" models, they are a waste of money and can't be upgraded. Apple really wants you to "go big or go home."