How I handle lots of large files from an event.

Joined
Apr 30, 2005
Messages
3,795
Location
Massachusetts
Real Name
David
And it's REALLY annoying when you're talking event work. I shot one event last year where I took just under 2000 images, 750 of them were from my D800 with the other 1250 from my D750. Bugs me when people say "why are you complaining, disks are cheap". I'd guess your machine is probably more beastly than the one I'm using, mines a few years old but still runs like a top. But I'm at about 32TB on disk, I'm real glad disks are cheap.

Hi David. Iam wondering how you work with these laege amount of files? Do you pp them and save the files as jpeg and then transport theme ro your complete work hard driving? Do you then remove the original raw files from your C drive or do you keep the originals on your workong Main C drive. Do you have a single hard drive for each year as storage and replace it with a new one every year.
Regards.
Mike

Hi David. Iam wondering how you work with these laege amount of files? Do you pp them and save the files as jpeg and then transport theme ro your complete work hard driving? Do you then remove the original raw files from your C drive or do you keep the originals on your workong Main C drive. Do you have a single hard drive for each year as storage and replace it with a new one every year.
Regards.
Mike
Well first, remember that I have requirements since this is a business and not just a serious hobby for me.

Drives:
C: OS / applications / etc - SSD
D: Images / Documents / etc - 2x2TB Western Digital SE enterprise hard drives in RAID 1
W: Online internal with other stuff - I have an application that syncs the files.
Z: Attached storage device that is on only during backup and if I need to access an archived image.
O: I have a hot swap bay in my computer and a stack of 2TB drives on a shelf, so bare drives not connected to any electrical source. This is mirrored with another set of disks in a safe deposit box and I rotate the set.

The RAID 1 on D: with the sync onto W: is probably a little OSD on my part.

Processing? I cull images in Photo Mechanic and then processt (normally 500-800 per event) in Lightroom. I keep the RAW files and a Lightroom catalog for each event. The final images (jpegs) are uploaded to Smugmug, I have a professional account with unlimited storage.
 
Last edited by a moderator:
Joined
Jun 14, 2008
Messages
5,336
Location
Ireland
Real Name
(Mike) Michael Skerritt.
I have a Samsung 960 m.2 1tb as my C os drive. I also have a Samsung 256 Gb m.2 as my scratch drive. I then have Samsung 850 1Tb ssd as backup of me main C OS drive. I then have tree 4Tb internal storage devices. I then have four 4Tb drive in a Raid 10 as backup of the compleate machine. I also have a Nass with four 4Tb drives in it that I back up all my finisher jpeg files to.
Cheers.
Mike
 
Joined
Apr 30, 2005
Messages
3,795
Location
Massachusetts
Real Name
David
I have a Samsung 960 m.2 1tb as my C os drive. I also have a Samsung 256 Gb m.2 as my scratch drive. I then have Samsung 850 1Tb ssd as backup of me main C OS drive. I then have tree 4Tb internal storage devices. I then have four 4Tb drive in a Raid 10 as backup of the compleate machine. I also have a Nass with four 4Tb drives in it that I back up all my finisher jpeg files to.
Cheers.
Mike
I should add that O: and Z: are full and complete copies of EVERYTHING. While D: is just current and active jobs, whether I'm processing the event, waiting of images to be selected for an album, etc.
 
Joined
Jun 14, 2008
Messages
5,336
Location
Ireland
Real Name
(Mike) Michael Skerritt.
I like the hot swap bay and bare drives. Full internal SATA drive speed and I don't have to keep track of a bunch of power adapters.

I am currently looking at this one as an external enclosure. It seems to get very good reviews with the exception of the unit not turning off automatically when your turned off your machine. As was said in one review this could easel be rectified with a firmware update.
TerraMaster D5-300 USB3.0(5Gbps) Type C 5-Bay External Hard Drive Enclosure Support RAID 0/5 Hard Disk RAID Storage ( Up to 40TB, Diskless)
 
Last edited:
Joined
Apr 30, 2005
Messages
3,795
Location
Massachusetts
Real Name
David
(Moved out of another thread)
Well first, remember that I have requirements since this is a business and not just a serious hobby for me.

Drives:
C: OS / applications / etc - SSD
D: Images / Documents / etc - 2x2TB Western Digital SE enterprise hard drives in RAID 1 -
W: Online internal with other stuff - I have an application that syncs the files.
Z: Attached storage device that is on only during backup and if I need to access an archived image.
O: I have a hot swap bay in my computer and a stack of 2TB drives on a shelf, so bare drives not connected to any electrical source. This is mirrored with another set of disks in a safe deposit box and I rotate the set.

D: is just current and active jobs, whether I'm processing the event, waiting of images to be selected for an album, etc.
O: and Z: are full and complete copies of EVERYTHING.
The RAID 1 on D: with the sync onto W: is probably a little OSD on my part.

Processing? I cull images in Photo Mechanic and then processt (normally 500-800 per event) in Lightroom. I keep the RAW files and a Lightroom catalog for each event. The final images (jpegs) are uploaded to Smugmug, I have a professional account with unlimited storage.

Also I prefer hot swap bays for external backups. Full internal SATA drive speed and I don't have to keep track of a bunch of power adapters.
 

Commodorefirst

Admin/Moderator
Administrator
Joined
May 1, 2005
Messages
23,561
Location
Missouri
Great thread and info, Thanks to David and Mike! Merged several posts from another file, so feel free to edit any of your posts for clarity, need anything, holler.
 
Joined
Jun 14, 2008
Messages
5,336
Location
Ireland
Real Name
(Mike) Michael Skerritt.
(Moved out of another thread)
Well first, remember that I have requirements since this is a business and not just a serious hobby for me.

Drives:
C: OS / applications / etc - SSD
D: Images / Documents / etc - 2x2TB Western Digital SE enterprise hard drives in RAID 1 -
W: Online internal with other stuff - I have an application that syncs the files.
Z: Attached storage device that is on only during backup and if I need to access an archived image.
O: I have a hot swap bay in my computer and a stack of 2TB drives on a shelf, so bare drives not connected to any electrical source. This is mirrored with another set of disks in a safe deposit box and I rotate the set.

D: is just current and active jobs, whether I'm processing the event, waiting of images to be selected for an album, etc.
O: and Z: are full and complete copies of EVERYTHING.
The RAID 1 on D: with the sync onto W: is probably a little OSD on my part.

Processing? I cull images in Photo Mechanic and then processt (normally 500-800 per event) in Lightroom. I keep the RAW files and a Lightroom catalog for each event. The final images (jpegs) are uploaded to Smugmug, I have a professional account with unlimited storage.

Also I prefer hot swap bays for external backups. Full internal SATA drive speed and I don't have to keep track of a bunch of power adapters.

This is a very detailed description of your setup and description of how it works for you. I like your thinking like me you have your OS and applications on the fasted drive you can get right now. I have them on an Samsung 960 Pro 1Tb m.2 drive. I haven't seen you mention a cash drive! I have a Samsung 256Gb 950 M.2 assigned as my cash drive this rely speeds thing along especially for PS/LR etc. I like your thinking also with the two 2Tb drives in raid 1 this should give you some additional speed. I use my Samsung 1Tb 850 Pro SSD in the same way as your D drive IE. current working jobs drive and its fast.
I find it good to see how others manage a large file structure. I also think that you have this well sorted. A well structured approach to this ever increasing problem of larger and larger file from today's high Megapixel cameras.
Cheers.
Mike
PS. I should mention that this is a new PC that I have built especially for my Photography work. I have only completed the build in the last month or so and I am still not completely happy with my current setup for back up. The PC itself is extremely fast and well fit for purpose.
Mike
 
Last edited:

Growltiger

Administrator
Administrator
Joined
Apr 26, 2008
Messages
13,436
Location
Up in the hills, Gloucestershire, UK
I have a fear of programs that do two way sync. They have to either depend on a database of what they have done before, or on file date/times to decide which way to sync each file. There are potential disastrous scenarios where the old files get copied over the new ones. This is why I use I use Bart, a very simple program that only copies in one direction.
 
Joined
Jun 14, 2008
Messages
5,336
Location
Ireland
Real Name
(Mike) Michael Skerritt.
I have a fear of programs that do two way sync. They have to either depend on a database of what they have done before, or on file date/times to decide which way to sync each file. There are potential disastrous scenarios where the old files get copied over the new ones. This is why I use I use Bart, a very simple program that only copies in one direction.

Hi Richard. Have you got a like to this program please. I would love to see more about it .
Regards.
Mike
 

Commodorefirst

Admin/Moderator
Administrator
Joined
May 1, 2005
Messages
23,561
Location
Missouri
I will add what I do at events, for onsite backup, I usually have dual card readers (1 extra for backup) and a laptop and 2 external HD, (one not used, just spare) since my cameras are older, I am only getting 18-20mb size 14bit compressed lossless raw images, and usually download 8-16gb per card depending on cards. I run a minimum of two bodies, and have 3 cards for each body labeled by camera lens or camera and card 1,2,3 etc.

As I am shooting, I download the cards to the laptop, and also duplcate the image folders on the external hard drive, 512gb to 1tb firewire or thunderbolt. My mantra at the scene is no card is formatted to be used again n camera until the images have been backed up on both the laptop, and the external HD.

I can easily handle 10,000 images in an 8 hour shooting day without to much stress.

My current laptop and desktops, external drives etc, aren't ready for the crushing file sizes of the 810 or 500, hence why I am reticent to upgrade.

Home, setup, is a simple dual HD cpu, 1tb size, and a rotation of 3 external HD also 1 tb which one is used as an auto backed up daily if needed, one weekly, and one off site which is brought home every few weeks and rotated in to the flow.

Big important shoots, I used to go get the out of house hd, just to be safe, and update all three before culling, and immediately take one off site.

Photo Mechanic to cull, and several different programs for editing, sometimes batch editing, but usually single. Keep maybe 10%, after the top 10% remain all keeper raws get backed up everywhere and only then, are the remaining 90 % culled raw images deleted.

All raw images renamed by yymmdd name of event

170328elitedance01.nef

I would keep same process but my gear would all have to change for storage and workflow speed with the large image file sizes.
 
Last edited:
Joined
Apr 30, 2005
Messages
3,795
Location
Massachusetts
Real Name
David
I have a fear of programs that do two way sync. They have to either depend on a database of what they have done before, or on file date/times to decide which way to sync each file. There are potential disastrous scenarios where the old files get copied over the new ones. This is why I use I use Bart, a very simple program that only copies in one direction.
I have it set to one way, mirror. Yes I wouldn't want a two way sync either.

I will add what I do at events, for onsite backup, I usually have dual card readers (1 extra for backup) and a laptop and 2 external HD, (one not used, just spare) since my cameras are older, I am only getting 18-20mb size 14bit compressed lossless raw images, and usually download 8-16gb per card depending on cards. I run a minimum of two bodies, and have 3 cards for each body labeled by camera lens or camera and card 1,2,3 etc.....
I tend to use 32gb and 64gb cards, both bodies have dual cards. I don't take the cards out of the bodies on site. I only do that at home. Kind of a mix that goes back to the thought of bent pins in a CF cards reader being the biggest issue and easiest to loose images. If I'm in an area I might be concerned I take one card out of each body and put it in my pocket.
 
Joined
Apr 30, 2005
Messages
3,795
Location
Massachusetts
Real Name
David
For the D750 it's loseless and I use 12bit during the reception, 20-28mb files.
For the D800 when I max it out to 14bit uncompressed they can be 72-78mb each. But I normally keep it a loseless. I use 14bit for the rehearsal and formals, 45-55mb files. Then 12bit for the reception, 30-40mb files.
 
Joined
Jun 14, 2008
Messages
5,336
Location
Ireland
Real Name
(Mike) Michael Skerritt.
For the D750 it's loseless and I use 12bit during the reception, 20-28mb files.
For the D800 when I max it out to 14bit uncompressed they can be 72-78mb each. But I normally keep it a loseless. I use 14bit for the rehearsal and formals, 45-55mb files. Then 12bit for the reception, 30-40mb files.
This ie the reason I built the new PC with such a fast processor the i7-6850k and the M.2 drives. As I said earlier I have seven spinning 4Tb hard drive in it.
With four of them in a raid 10 configuration as backup Of the entire PC with the AseUs backup software. But I am wondering if it would be a better decisions for me to have an external four or five bay raid set up for my files? Like the one I posted above eirler. I have onlp picked up the D500 a couple of weeks ago nd am holding off for the D810's replacement to be released. If the replacement doesn't come out soon I am thinking that I will just get the D810!

Mike
 
Last edited:
Links on this page may be to our affiliates. Sales through affiliate links may benefit this site.
Nikon Cafe is a fan site and not associated with Nikon Corporation.
Forum post reactions by Twemoji: https://github.com/twitter/twemoji
Forum GIFs powered by GIPHY: https://giphy.com/
Copyright © Amin Forums, LLC
Top Bottom