Estimated "remaining time" is off significantly due to scanning time with...
A free file archiver for extremely high compression
Brought to you by:
ipavlov
Menu
▾
▴
1 Attachments
#2270 Estimated "remaining time" is off significantly due to scanning time with very many files
7-zip 19.00 on Windows x64. I'm compressing a directory with > 32M files. The "scanning" stage of this, before it even started compressing, took ~12-13 hours! In the screenshot the Elapsed Time is ~21.5 hours, but only ~8 hours of that was compression and it's 8% done, so I would expect the remaining time to be ~90-100 hours, but 7-zip shows 235 hours. (It's going down gradually over time.) I suspect this is because it's extrapolating from the entire "Elapsed time" so far, i.e. 21.5 hours / 8%, which gives quite a misleading estimate in this - admittedly very edge - case.
That case is unusual.
Is it SSD or HDD for that slow scanning?
I'm not sure, since it's a virtual machine and I have no access to the physical machine, but I suspect it's probably some kind of network storage, rather than a real local disk.
Note that 32 M files can create big problem at the end of process, if you have no enough RAM.
7-Zip stores UTF-16 file paths in memory several times.
So you must have many GBs of RAM to support such big lists.
And if physical RAM is small and if virtual memory with SWAP will be used, it will be too slow for PAGE SWAPPING.
And it can fail with "not enouh RAM" message at the end of process, when 7-Zip will create final metadata stream with all file names and another properties.
Last edit: Igor Pavlov 2020-09-19
You can track Virtual and Physical memory usage of 7zg.exe in Task Manager.
Well, it did fail near the end, at 98%, with "The system cannot allocate the required amount of memory". This happened during the night, so I don't know the exact amount of RAM it was using then, but it was up to ~30 GB before, even though the "Create archive" dialog estimated something like 2.5GB for compression (I chose Ultra, LZMA2). Trying again now with Maximum, will let you know in 4 days or so. :)
"Fast/Maximum" level will not solve that problem. 7-Zip shows only memory size required for data compression. But your problem is memory for file names that cannot be reduced now in 7-Zip with options.
As solution you can create 3-4 archives instead of one big archive. I suppose 10 million files per archive probably will be OK for your RAM size.
Select some folders and compress them to archive_1 . Then compress another folders to archive_2 and archive_3.
Last edit: Igor Pavlov 2020-09-22
Ah OK, thanks for saving me 4 days. :) I'll try again with Ultra, but with an even bigger pagefile. (Yes, it's very slow, but it's very slow even before using swap space.)
Maybe it can work with big page file, but note that main problem time will be at the end after 98-99%. At that stage it allocates additional big memory block with filenames to store in archive.
Well, it worked! However, the file count in the dialog at the end is slightly lower than the total number of files compressed: 32466178/32466766. I opened the resulting .7z file and 7-zip confirms that there are 32466766 in it, so that seems like another (minor) issue with the progress dialog.
Call the "Test" command for archive. I hope it will be OK and faster than creation operation.
You can track memory consumption for "Test" command in Task Manager. 7-Zip loads all filename paths in archive to memory at starting of Test operation.
Oh, yes, I did that and the test passes and reports the correct number of files (32466766). It takes about 2 minutes just to open the file, peaking at ~12.3 GB RAM used. The test itself takes about 15 minutes on my machine (not the same machine where the archive was created).