Copying very large folders really slow.

Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
I'm having this problem with any Windows computer I use. The old work computer is an i5 with 8 gig of RAM, the new laptop is less than a year old with an SSD and 8 gig of RAM. My desktop is in i7 8700 with 16 gig of RAM and it's the same with all of them.
We have a folder of "workshop manuals" at work, 460 gig worth and many sub-folders for different makes and models. I'm trying to make backup copies on several 1TB drives but it takes very long. On the old workshop computer copying from the internal D: drive it took two days.
On my 'new' desktop I have copied it from an old USB2 drive onto my internal drive which took about 6 hours the first time. Since then it takes really long to copy to an external drive.
The new drives I bought are USB3 drives - the first issue I had is that it told me I needed more space to copy 460gig to a 1TB drive and i saw they were ExFat and changed them to NTFS.
Someone who works with computers suggested that FreeFileSync was the best way to copy the files over and I have installed it.
I start it copying and it tells me I have 1 hour and 15 minutes left for the 7.2 million files. Over time it gets really slow. After a few hours now it is sitting at 1.5 kb/s and 1350 days left.
It seems to be mostly related to the "cars" folder where we have the most files. If I copy the other folder over "Trucks", "Earthmoving" etc. with less files they can drag and drop in a few minutes. The "cars" folder with all the sub folders really slow things down and when I cancel the transfer and use FreeFileSync it looks promising to begin with but then gets really slow again.
To give an idea of a "basic" delete I just told it to delete the folder with all its files from one internal hard drive. It took windows 10 half an hour to add up all the files
and tell me it's too big to move to the recycle bin and then one and a half hours to permanently delete it.
Am I stuck with the slow copy times or is there a better way?

Subscribe to see EXIF info for this image (if available)
 
Joined
Jul 8, 2019
Messages
573
Location
SF Bay Area, California, USA
Thoughts:
  • A DEEP directory structure could be a reason.
  • If "CARS" has a LOT of files and is SLOW, distribute the files.
    • Make a CARS1, CARS2 and CARS3 directories, then distribute the files to them. Obviously you have to backup, before making a significant change like that.
  • Many small files copy MUCH slower than a few large files. I suspect it has to do with the overhead of writing the file data to the index and file system.
  • Just because the interface is USB3 does not mean the drive is capable of SUSTAINED data transfer at that speed. It is an old trick to put a cheap slow drive into a box with a fast interface, and to advertise it as a FAST drive.
  • Did you plug the drive into a USB3 port on the computer, and not a USB2 port?
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #3
Thoughts:
  • A DEEP directory structure could be a reason.
  • If "CARS" has a LOT of files and is SLOW, distribute the files.
    • Make a CARS1, CARS2 and CARS3 directories, then distribute the files to them. Obviously you have to backup, before making a significant change like that.
  • Many small files copy MUCH slower than a few large files. I suspect it has to do with the overhead of writing the file data to the index and file system.
  • Just because the interface is USB3 does not mean the drive is capable of SUSTAINED data transfer at that speed. It is an old trick to put a cheap slow drive into a box with a fast interface, and to advertise it as a FAST drive.
  • Did you plug the drive into a USB3 port on the computer, and not a USB2 port?
It's plugged into a USB3 drive.
The "cars" folder is divided into all the major brands of cars but then of course the various models in each folder. The "Holden" folder has several models in it and about 1 million files. At the moment I'm doing a comparison. the desktop is running FreeFileSync and at the moment says it has 150 days left as it has slowed to300kb/s. On the laptop however I'm copying from a drive on one usb port to the other 'drag and drop' one vehicle brand at a time and it is maintaining 'normal' speeds. I may have to resort to that - copy all the 'smaller' folders then go to the 'cars' folder and open it up and copy one model at a time.
 
Joined
Mar 20, 2017
Messages
1,484
Location
Central Ohio
Real Name
Andrew
I've had to do this before. The best way I found is to manually copy the files over small chunks at a time. I'll load up enough to run for 30-60 minutes at a time....then let it run. come back later and repeat. I've done that when I had to back up my 1TB drive to a new 4TB drive when I was running out of image space.

Did that over the course of 2 days in my free time and not an issue.
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #6
I've had to do this before. The best way I found is to manually copy the files over small chunks at a time. I'll load up enough to run for 30-60 minutes at a time....then let it run. come back later and repeat. I've done that when I had to back up my 1TB drive to a new 4TB drive when I was running out of image space.

Did that over the course of 2 days in my free time and not an issue.
Thanks, that's what I've resorted to. I just woke up this morning and checked the FreeFilesynch operation and it has 75 days left.
It's amazing, you'd think after all these years they would have a program that could do this properly.
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #8
Something else I notice each time I go into task manager is that Java Platform SE binary [32 bit] is using 50 cpu
and "power usage" is at "very high". This happens after each computer restart. End task stops it and the fans slow down but it's not related[?] to the copy speeds.
 
Joined
Mar 25, 2011
Messages
2,695
Location
London
I
Something else I notice each time I go into task manager is that Java Platform SE binary [32 bit] is using 50 cpu
and "power usage" is at "very high". This happens after each computer restart. End task stops it and the fans slow down but it's not related[?] to the copy speeds.
I removed java from my laptops.
Not a problem.
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #15
I tried that.
But the zipping times got ridiculous for me.
I wonder how long it would take to unzip? Realistically we are selling copies for NZ$600 a go - if this goes on for a while I wouldn't mind it taking a week to zip the lot on the work computer if it means much faster copy times after that.
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #17
So, you're asking if changing one time consuming task into three time consuming tasks is a good way to save time?
Obviously not. If I could spend some time making it easy to copy in the future then instead of having to spend an entire day dragging and dropping I could perhaps get it to copy in one drag and drop - so rather than 100 copies taking 100 days they could be done in an hour each - do you understand that?
 
Joined
May 21, 2019
Messages
224
Location
Albuquerque, NM USA
Real Name
Don Roy
You're stuck with two sides to the problem. On one side the task of drag-drop, though manual, is easy if it completed in a few mins. On the other side is the device slowness dragging out that 10 min job into an all day project. Having the mental part of the project take all day was the main issue for me, not that the actual data transfer takes that long. I fixed the problem by getting real backup software that I could automate, Acronis True Image. I have defined routines to do daily auto-backups, and other routines needing a click or two to kick off unscheduled backups. They work in the background so don't need monitoring.
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #19
You're stuck with two sides to the problem. On one side the task of drag-drop, though manual, is easy if it completed in a few mins. On the other side is the device slowness dragging out that 10 min job into an all day project. Having the mental part of the project take all day was the main issue for me, not that the actual data transfer takes that long. I fixed the problem by getting real backup software that I could automate, Acronis True Image. I have defined routines to do daily auto-backups, and other routines needing a click or two to kick off unscheduled backups. They work in the background so don't need monitoring.
The thing is I'm not backing files up - I'm copying 460 gig with 7.2 million files to a new drive.
 
Joined
Sep 17, 2006
Messages
12,448
Location
Hamilton , New Zealand
  • Thread Starter Thread Starter
  • #20
I have found that if I drag and drop one folder it may slow to 30kb/s and say it will take 9 hours to complete - but if I start dragging and dropping a few more folders it doesn't interfere with that and they might sit at 10 or 20 Mb/s so I can have a few going at a time - the slow ones are probably working with multiple small files.
 

Latest posts

Links on this page may be to our affiliates. Sales through affiliate links may benefit this site.
Nikon Cafe is a fan site and not associated with Nikon Corporation.
Forum post reactions by Twemoji: https://github.com/twitter/twemoji
Copyright © 2005-2019 Amin Forums, LLC
Top Bottom