Note: This tutorial was recently updated to include OS-X Lion and Lightroom 4.
Recently I had dinner with an entrepreneur who had an exciting new idea for a software service start-up. He talked about how hard it was for photographers to manage their images. I was not surprised, thinking about my own quest for file organization and image management, his idea seemed important. His product is specifically for Lightroom users and it’s very clever. It’s also ground breaking and could save many hours and much heartache if a hard drive crashes. No. it’s not file back up but it does do that. It’s designed to make Lightroom much more useful and make file management a snap. It’s not quite released yet, I will let you know when it is and I will share with you how to get it. But that’s just what made me think about writing this blog post after I came home from dinner last night.
I Used to Suck at File Management
One surprising element of our conversation was the idea that most photographers who use Lightroom don’t really understand the file management aspect of the software and how critical it is to being a digital photographer. I didn’t either when I first bought Lightroom, but I do now and that was no easy journey.
So here’s what I came up with: Now remember, my system is for ME and may not suit you. Besides file management, this article does cover some basics on file structure and Lightroom features.
Before I tell you about my system, I will tell you about how I photograph. I am a Lens Traveler, it’s true. I don’t take out my camera until I am going on a trip. I can’t, I don’t have time while I am working. But many of my trips are spent joyfully romping the world and making photographs. Knowing I will be in remote places, I needed a system that suited my style of usage. I also wanted a system that would expand as I had more time to make more trips. I believe this is the final solution to file management and I am grateful that I have it.
Another element of my work flow is that I have a desktop (MacPro) and a MacBook. The desktop is my repository and main “catalog” while my MacBook is for use in the field. The question I get from others on trips with me is “how do you manage to get your images (with edits) back into your desktop and still preserve your tags, edits and selections?” I will show you how.
I used to dump my images in a directory all labeled with the date. That’s it. That was my file management. So in the past, I would think “Now when did I take that photo?” and if I had two minutes to have labeled that directory “Ireland 2006” that was a bonus. I had not tagged a single image and of course could not have known if an image had been edited or not. So with such a “perfect system,” why would I want to get all complicated and spend hours learning how to use Lightroom? Because now – ALL my editing, filing, tagging, searching and even a direct connection to my LensTraveler18.com web site is automatically handled. I would go as far as saying that without Lightroom I wouldn’t be able to handle the quality and volume of images I come back from a trip with.
Importing Images and Keeping Them Sorted
If you don’t shoot RAW files then you are leaving behind the opportunity to get the most from each photo.
If you need to know what a RAW file is, think of it as the photographic negative. These days, a “finished print” may be a jpg or an actual print. Either way, shooting RAW is the only way to go if you are serious about making great photographs.
I suggest you set up Lightroom up to convert all your newly imported images as .DMG files. This is Adobe’s universal raw image format. It’s universal and it’s timeless. Imagine that 10 year old file from a now-obsolete camera could possibly cause you to loose access to your image, no thanks! As a bonus, there is no “side car” file and the new lossless DMG file is about 40% smaller than the original raw file. So it loads faster and it’s timeless. No brainer, just do it. You can access this setting in the “Preferences” selection under “Lightroom” from the top menu bar (on a mac) and you will see “File Handling” where you will select “dng” and while you are there, make sure your Jpg preview is set to “Medium.” Lightroom does the conversion for you when you import images. And DMG files are readable by every other software product I have ever encountered. Even Picasa reads DMG files. Of course all Adobe products do to.
While in “Preferences” Go to the “general” tab and choose “When starting up use this catalog” now select: “Prompt me when starting Lightroom” which is important later as you will see. Also while in “Preferences” go to the “External Editing” tab up top and select your color space as “ProPhoto RGB.” NEVER use sRGB unless you are interested in limiting your color space severely.
Now we can close “Preferences” and continue.
A Slightly Bigger Picture
So lets review: Shoot only raw files, open Lightroom, create a new catalog, set your import preferences to DNG file format and sit back and watch your computer work. After your images are in Lightroom, you can edit and modify them as needed. From Lightroom, you can do 5 basic functions; Library Management, Development, Slide Shows, Print and Web Site Creation. We won’t deal with the slide shows, print or web sites but library management is the key to all making everything work and if you don’t have your library management perfect, you will only suffer later, I promise. Now where does everything go….
Catalogs & Importing
This next part is the hard part, but it’s worth the trouble. I want to explain this as clearly as I can.
Here’s what took me a long time to finally get right. Remember I mentioned to you that I take my MacBook with me when I photograph? It all starts there.
When I arrive on location on the night of my first day, I create a new catalog on my MacBook named with my location and year: “Ireland 2006” and then import my images from that day to that catalog. When importing that day’s images, I use custom text and key words. I like using the day number (Day 1, Day 2, etc) and general location for file name, which you specify when starting your import.One other nuance, I like to use the file number that comes from the camera so “day one-4567” is not the same as “day one-3874” from an earlier trip. Since tagging is universal with each batch you import, I try to keep it simple. I might have as my tag for that day “County Claire” and then as I scan through the images, I can tag more directly each image as I need.
Now I create a “Collection Set” named with the destination I am in, like “Ireland 2006” the collection set name perseveres in my main (home) library but the catalog name does not. Ultimately I will erase the MacBook’s catalog and library since I don’t need it anymore.
I then create a sub-collection for “day one” and I drop my picks in there. I do the same thing every day and this starts my sorting process. As I build HDR images from the raw photo imports, I drop those finished HDR’s into the day they were shot. Finally I have another sub-collection called finalists. And yes, I can probably avoid this next step but I create another sub-collection called “gallery” where I sort through the finalists and drop those into my gallery collection.
Meanwhile, as I travel and throughout my trip, I can edit files, tag, modify images to my hearts content and preserve all those changes in my main catalog and not lose one bit of work. I review my images upon downloading, I quickly view them and press “P” for picks as I see the ones I like.
Docking with The Mother Ship
When I get home, I take my MacBook and open Lightroom, then choose “Export as Catalog” which consolidates all my files into one bundle and puts them in a package Lightroom can import, and export those using a small USB drive. Then, on my Mac Pro, I import them using the command “import catalog” as shown below.
That’s exactly how I am set up to move files from my MacBook to my Mac Pro when I travel.
Two observations: As my catalog increased in size it slowed down considerably. I found several blog posts referring to this phenomenon. I have about 15,000 which is considered a medium size library, but it was still slow. I found out that previews are what slows down the catalog loading times and affect perceived performance. So putting the catalog/previews on a much faster drive and the image library on a separate drive made a lot of sense.
Some people run their entire image library on external drives. I wasn’t going that route, I decided on splitting my drive resources with an SSD as my boot/applications drive. I found this SSD on sale for $249. Its probably not the fastest possible SSD drive but it’s darn fast and I am so glad I made the decision to upgrade. I will also use two conventional server-grade 1 TB drives each for Photos and Music. I have a large Music collection from idle days spent ripping CD’s and my live recordings of jazz/rock/blues bands but that’s another story.
So you can see logically, this is all simple to understand; 3 drives, each with a dedicated purpose. Yet, it did take some months of part-time tinkering to get it all to work. I started with a clean O/S install on the SSD and it booted so fast I thought it was haunted. Then I started to reload my apps, one at a time. Some were quick and easy, others took a long time to reinstall.
I knew from reading warnings that I couldn’t actually copy my whole library from my old 2 TB drive to the new “Photos” drive so I did some digging and experimentation.
This next point is critical; You can’t move anything on the hard drive without doing it from within Lightroom or you will have difficulty linking the catalog and the library contents together. So with that in mind, I booted from the old 2 tb hard drive and exported catalogs one at a time to the photo drive, then imported them one at a time into the new installation of lightroom after rebooting from the SSD. This was tedious but it preserved all my settings, changes, edits, etc.
Another pain in the rear I hadn’t anticipated was that all my add-on’s had to be re-installed. One at a time, all the presets, all the add-on’s everything I had painstakingly set up had to be redone. Same with all my external links to my facebook page and my SmugMug site. But now it’s done and I am fully operational again.
Wow, so all that sounds kind of complicated, right? I mean THREE drives that all must be backed up and a full recovery plan on what to do if one crashes. Let me explain how I back up. And before I do, I hope that someone reading this will tell me what they do so I can compare.
I use a three tier strategy for back up.
Tier One: – SSD – Boot Disk is backed up with Time Machine. My thinking is that if the SSD drive crashed I can have a new one in 24 hours (thank you Amazon.com) and reload the OS, then restore from time machine, down time about one day.
Tier Two: – I use a 2 TB USB drive from IO Safe, which is a fireproof, waterproof vault drive which is partitioned to 3 drives. Each night SuperDuper incrementally duplicates each of those three drives and makes exact duplicates of each.
Tier Three: – And then finally I use www.CrashPlan.com to upload all my files to the cloud for $50 a year unlimited storage.
Now there’s one other backup strategy that I employ and it’s free but it’s only for a small amount of data. I use SugarSync to keep all my business files synchronized between my MacBook and Mac Pro. That’s just to make life a little easier and it’s totally free.
My Paranoid Plan for a MacBook Crash
While traveling to Bhutan (a small country between Bangladesh and India) my buddy Ron’s MacBook crashed. It happened on Day One. Wow, that really is a blow. I realized then that I couldn’t risk it happening to me. So here’s what I do. I use CarbonCopyCloner to clone my drive to an identical drive in a small USB case that I carry with me in my briefcase, along with two screwdrivers. Every night before I go to sleep, I set up CCC to clone my MacBook drive.
Those screwdrivers and the USB drive will allow me to swap drives in about 10 minutes. I also carry a OSX Leopard DVD which allows me to use disk utility on my drive if it needs to be repaired by the OS. So far, I haven’t needed this, my MacBook has been solid but none the less, I feel a lot better having it with me. And it works great. Note: With Lion, there’s a hidden boot drive on your hard disk but if you are not near the internet or of your whole drive is fried, that hidden drive won’t be accessible either.
Maybe I should have stuck with 3 trays of chemistry and a print washer…. Not sure. But I DO love Lightroom and I really appreciate what it can do. Now that I know I can totally preserve my photography and never lose anything to a mishap or a drive failure, I wouldn’t ever go back to doing it the old way.
If you have a different/interesting backup strategy of your own, please tell me all about it, and show me your gallery! I love to see how others have solved this problem.