Digital Asset Management
Digital Asset Management is essentially the acts of looking after and finding your images. One needs to be aware that there is a LOT more to this than might meet the eye. Have a little think about how much the medium of photography has changed over the last 30 years. Do you really think that we will be taking pictures and storing them in the same way in 2050? Think of the Vinyl record, the Cassette Tape and the CD-Rom. Think of the floppy disk! Now think about your RAW, NEF or DNG. If you aren't concerned about their longevity you should be!
Issues in Photographic Digital Asset Management
Digital photography is growing rapidly. If you are anything like me you might be capturing images on a phone one minute and then a DSLR the next minute. You may even capture a video or two as well.
Your images might be in a number of different digital formats, captured at different times and places. Depending on what type of photographer you are there may be a need to catalogue and categorise these images in any number of different ways and for any number of different purposes.
Moving on from image capture, we also encounter all kinds of derivative uses of images. We might have resized versions for web use, proofed versions for print, originals to store, black and white or other copies with different treatments. We may sell some images while yet more may just be for the consumption of friends and family.
Some images may be business critical, while others are simply critical to our state of well-being and our memories.
All of these possibilities point to the vital need for a new level of organisation and control. Without it, we will not be able to organise our digital assets nor make effective use of them.
Digital Asset Management (DAM) relates to all the choices we make around the issues outlined above. It also incorporates our choice of software, file format, backup, storage and workflow.
In all of this we can define three main goals:
- Don’t damage or lose your precious images
- Find images when needed
- Save time and be productive
We could go on and define a few sub-goals too:
- Make sure you can use the images far into the future
- Make them look the way you want them to
If we break-out the goals above, we can isolate some important points.
It is very easy to damage images. Consider an edit in a simple program like Microsoft Paint. Once edited, we cannot go back to the way the image was captured. We can only go back if we save the new image as a version of the original.
At its simplest level, this is an example of a ‘non-destructive workflow’ (NDW). The problem is that we are now left with two copies of the same image taking up twice as much space.
It’s a very inefficient way to work and could quickly involve a requirement for massive amounts of storage space as well as the creation of a labyrinthine folder structure!
There simply has to be a better way right?
Well, you guessed it, there is!
Instead we could keep the original safe in an archive folder and make adjustments in an image editor. Instead of saving these adjustments into the original image we save them as instructions in the editor. Whenever we open that image, the editor parses the instructions it has recorded and applies them to a view or representation of the original image.
The original is never touched, the editing program just has to store the set of instructions instead. And get this, the editor could store multiple sets of instructions for the same image showing you all the different versions and treatments without ever touching your original.
This is the method applied by most NDW image editors. Examples of this software include the most popular, Adobe Lightroom - you have probably heard of it!
The second part of the goal set out above; “don’t lose your precious images”, is also partially solved with a program like Adobe Lightroom (LR) because it also acts as a catalogue of your files. It remembers where they are located on your hard drives and it can also record additional data. It can store an image rating, keywords, title, description, captions etc… alongside the information you have created during the editing process.
LR includes powerful capabilities that allow you to edit, store and search through thousands of original and derivative images. In other words it is both a Non-destructive image editor and Digital Asset Management application.
Of course, LR is not the only software that does this and there are some very good alternatives. On1 Photo RAW is worth a look.
Organising Folders and Files
Once we start to use good DAM software we do not need to worry as much about where images are stored, nor do we need to create filenames that describe the image. Instead, we can use the DAM software tools to search images based on the metadata stored in the images themselves.
However, there is one major problem with this approach. We still need to back up our files and sometimes, in very large collections, the backup media is not big enough to hold all of our images.
This is where an organised folder structure and filename convention come into play. We should be trying to store our images in volumes that will fit on our working drives and onto backup drives or other backup media like CD-Roms.
A folder structure will allow us to group ‘packets’ of files in roughly the right sizes for storage. Additionally, it will help us to find them later.
Typically, I shoot locations not events. I might revisit a nature reserve many times. In my case, I name the shoot folder after the location and add a date string to it.
A lot of people recommend a folder naming convention with a date first and a location or event second. For location shooters, this doesn’t make a lot of sense because we probably want to see our folders grouped by locations and not dates.
For event photographers, a date followed by an event is probably a good way to go, because the events will all be unrelated to each other.
Here are some examples:
Location based: Rietvlei_171023 (indicates a shoot at Rietvlei nature reserve on the 23rd October 2017).
Using this notation will show all the Rietvlei images next to each other in Windows Explorer, and it will also organise them by year, then month and finally day.
Event based: 171023_Harris
Using this name, you would see all the events in October 2017 next to each other in Windows Explorer. If these were wedding weekends, then it should be easy to find the right one.
File names present a similar opportunity to organise your images. However, there is one thing that you really need to be aware of. You know that number that gets applied to a filename inside the camera? With Canon’s it is a 4-digit number.
Watch out! If you shoot a lot of images, more than 10,000, you will start to reuse the same image codes and overwrite files.
This is mainly an issue where you either shoot more than 10,000 images in a shoot, or, more likely, where you shoot over a period of days but save the images to a hard drive. Using a drive with a built in SD card reader is extremely dangerous because you cannot see the images getting overwritten. I lost a lot of images like this.
I now prefer to specify my own filename when importing files. In my case, I add my name at the beginning, then an indicator of which camera the image comes from, then the 4-digit image ID, and finally the short date (YY/MM/DD).
Here’s an example: goodlet_5D1234_171023
I use my own name, in case the image has been sent to a third-party; they will know whom it came from. It is also almost certainly likely to be a unique file name in my overall collection of images. Unless I shoot more than 10,000 images on one body in a day. Unlikely!
It is easy to specify a file name ‘mask’ like this when using DAM software for imports.
Ingestion Workflows & Backups
Ingestion is another word for importing and refers to the process of bringing the original files onto a computer for further processing.
Ingestion is a really important step because it can achieve so much more than a straight copy from a card to a PC. With a good ingestion application you can copy the files to more than one location, check the integrity of the files, change the file name, add keywords, add IPTC metadata, complete titles, captions and other fields using variables.
One tool that does this really well is Photo Mechanic. Photo Mechanic is designed to be the best ingestion tool on the market. It is lightning fast and packs in a bunch of additional tools to make the process efficient.
Photo Mechanic is beloved of sports and photo-journalists and anyone who needs to blast through the ingestion, key-wording, rating and uploading parts of the workflow as fast as possible.
It may be overkill for many other photographers, but personally I love it. It really speeds things up.
The first step in my workflow is to ingest the images to two locations using Photo Mechanic. At the same time that this is going on, variables are used to populate image captions, copyright and creator information. Location and event information is added as well, all automatically.
A note on drives
A lot of photographers make use of many external drives. With faster connections this is viable but I don't see the point really. It's messy and unnecessary, especially if you design or specify your primary editing machine correctly (see how here) you can include swappable drives in internal bays that take advantage of fast internal connections and various caching technologies that are not available on external drives.
The location of the copies is as follows:
Virgin Backup Unit (VBU)
The virgin backup unit is a store for all the images taken at the shoot. The drive is internal to the computer to take advantage of those faster drive connections. It is never touched and simply filled with images. In my case, the drive is 1 terabyte and when it gets full it is removed and replaced before being taken to a secure off-site location.
This VBU is the backup of last resort.
Temporary Import Unit (TIU)
The TIU is simply another separate smallish physical drive that holds a copy of the shoot. Its job is to hold an untouched copy until the project is completed and archived. When the project is complete, this copy is deleted. It is important that these are separate physical drives in case there is a drive failure at any point. There is little point, for example, in creating two folders on the same physical drive.
The next stage is to copy the images onto the Working Drive (WD). The WD is for editing and completing the project. In my case it is a small but very fast Solid State Drive. Its job is to allow fast manipulation of the images and to allow fast creation of rendered previews.
Once the project is complete, the images are moved using the DAM software to a safe Archive located on yet another physical drive. This one is very large and houses all the completed projects. In my case, it is still an internal drive and takes advantage of the fast internal PC connections.
During the work on the project, it is automatically backed up in two additional places: The Internal Backup Drive and to online cloud storage.
These drives are mirrored. This means they reflect any changes to the archive and working folders automatically.
Offsite Backup Drive
Finally, another backup unit is swapped once a week and kept off site. The time delay means that I could lose the most recent shoot but it also means that If I accidentally delete something, the deletion won't be replicated everywhere! The cloud backup is another safeguard against this as it also supports versioning.
Where you have enough storage space it is a good idea to set up versioning in the backups (most backup software can accomplish this) but it becomes difficult once space becomes a problem.
Versioning protects you from replicating corruptions, errors and viruses that might be transferred to your mirrored drives because they are slices in time that do not get modified by any subsequent changes to the files. They are, however, very costly in terms of space, requiring the ability to store at least 4 and possibly 5 extra sets of data (last year, six months ago, last month, last week and yesterday).
I think versioning could be usefully applied to your most important finished files without taking up too much space.
So, at any one time I would have the following copies available:
- Virgin backup Unit (Discrete Internal Drive) 1TB HDD -> Not backed up - archived offsite when full
- Temporary Import Unit (Discrete Internal Drive) 1TB HDD -> Copied to 3 & deleted when complete
- Working Drive (Discrete Internal Drive) 500GB SSD -> Mirrored to 5,6 & 7
- Archive Drive (Discrete Internal Drive) 4TB HDD -> Mirrored to 5,6 & 7
- Backup Drive (Discrete Internal Drive) 6TB HDD -> Mirrored to 6 once every 2 weeks
- Offsite Backup Drive (External Drive) 6TB HDD
- Cloud Backup (Crashplan) - Unlimited but slow
I may even still have two copies on the SD and CF cards from my camera if I haven't shot much that week.
You can see I take backups really seriously and storage is a major component of my system.
However, in the workflow below I do not backup the Virgin Backup Unit. This means that there is only ever one viable copy of my rejects. Depending on your business or use case, you may need to consider backups for the rejects too.
Rejects are a thorny topic for me because some may be rejected based on current editing technology. They may actually be good subjects. Keeping rejects like this might mean that sometime in the future they could be developed and make great pictures. Also, our tastes and abilities change over time, we may have rejected images in the past that we could now take further.
Another issue for my personal use case, is that I have a lot of panoramic component images. I have to be very careful that I do not base the rejection on subject matter as they may be background or bits of a scene and difficult to identify as part of a panorama. It's easy to make mistakes but in my case, I don't feel I can justify the costs of more storage for backing up rejects.
Digital Photography Workflow
NAS and DAS Systems
NAS stands for Network Attached Storage and you will see a lot of photographers use and recommend this. NAS is a way of making your entire (or very large parts) of your portfolio available anywhere. It is available on your wifi connection, on the internet (because it is a small server) and, crucially, to other photographers or users.
This functionality might be important to you. However, NAS does not = Backup. Think of it as a very large network and internet enabled hard Drive. It can protect against defective drives if set up that way but that is not a feature that is unique to it.
In my case NAS doesn't add much because I work alone and would need a second set of identical disks to back it up. Additionally, I feel that separating your catalogue onto smaller disks offers safety. Because the disks are separate and smaller, you will only lose what they contain and not everything you have.
A more flexible and cost effective solution for the lone shooter is DAS. DAS stands for Direct Attached Storage. DAS is basically any storage device attached to your computer, internal or external. It is very flexible, fast, can make use of any drive and can also provide assurance when set up as a RAID. The only major difference between NAS and DAS is that NAS is network and internet enabled and DAS is only available to the computer it is currently connected to.
For me, DAS makes more sense.
Metadata and Keywords
Metadata is data about data. In our case, it is things like the exif information of the image, the date and time of capture, the file type, camera make, lens model etc...
DAM software uses metadata to help us organise and search for images. The good news is that we don't need to add this data, it is built into the image already.
Keywords, on the other hand, along with International Press Telecommunications Council (IPTC) fields are not pre-populated. IPTC fields hold information about the creator, owner, contact details, copyright, location, event, title, caption etc...
It's probably worth separating out IPTC fields specifically, as many of these values will always be the same: I create my images and own them etc... so the fields can be pre-populated and applied during ingest (using a Photo Mechanic Variable).
Keywords can also be applied on ingest, particularly where they are appropriate for all the images in the shoot. However, most of the time, keywords need to be applied to either small groups of images or individual images. This can be very time consuming.
I have five pieces of advice when it comes to key-wording:
- Only keyword keepers - don't waste time key-wording images you will delete (see the workflow diagram above)
- Use keyword hierarchies to automatically apply additional keywords and synonyms
- Use enough keywords to define and locate the picture but don't waste time adding too many keywords. If you do need to add keywords, for example for stock, you can do it to the final selected image instead of wasting time on all the shoot images.
- Do not delete the keywords from exported files - I think Lightroom defaults to this. Keeping the keywords in the exported files means you can use a second catalogue to manage these files (if you keep them on disk).
- Make use of "non-exporting" keywords. DAM software can tag a keyword so it does not export. This, I find is a fantastic way to organise images for workflows. For example, I have a ''component pano" keyword, that does not export but collects all the images for the same panorama in one smart collection.
You can download and try out my Keyword Hierarchy for free. It has about 7,000 items currently and is specifically designed for South African nature photographers. If you are from elsewhere in the world, use the structure to help inform your own keywording hierarchy, it is easy to make changes or additions!
Image & Colour Rating Systems
People use Image Rating Systems in all kinds of ways, sometimes they are also used in workflows (for instance, successive pass workflows).
Colour Rating & Smart Collections
I like to keep ratings really simple. I start with colours. Yellow, Red and Green. I use the colours in conjunction with Smart Collections to move my images through a workflow automatically.
Remember, I have already selected to import only the very best images in Photo Mechanic. Out of +/-1,200 images on any given day shooting wildlife, I would probably have imported less than 100 into Lightroom.
- Yellow = a "Keeper" - I select shots that I may want to develop in a quick first pass through the shoot. I have a Smart Collection that automatically groups yellow/keepers.
- Red = a "Select" - I look through the Keepers Smart Collection and choose a "Select". For me, this usually means the best image in a small set. For example, at a Lilac Breasted Roller sighting, I may have taken 20 shots initially. I probably tagged 5 and imported them to Lightroom. 3 might have been marked as "Keepers" and of these final three, one might be marked as a "Select". The select is an image I want to develop. Marking it Red/Select will automatically move it to the next Smart Collection in the workflow. All the other remaining Yellow images that I don't select are marked with the "~archive" non-exporting keyword and automatically move out of the Keeper Smart Collection and into the Archive Collection.
- Green = Completed - A green image is complete and developed. At this point I also use a star rating.
My star rating system is very very simple.
- 0 stars means I don't export the image.
- 1 star, use for web only. E.g. Facebook
- 2 stars, use in blog posts and higher quality content on my website
- 3 stars, portfolio quality
I don't use 4 and 5 stars. It's always good to have room to grow your rating system as you improve as a photographer. The last thing you want to have to do is to go back into years of archived images and change all the "fantastic" 5 star early images you took back to 2 or 3 star rated ones!
The DAM Book is required reading for anyone who is serious about looking after their digital assets. Peter Krogh has put together a truly helpful and encyclopaedic work on the subject.
DAM stands for 'Digital Asset Management' and Peter takes one through the entire 'digital ecosystem' in this seminal and comprehensive work. It is the ONLY book for serious photographers.