#764 Archiving performance of seperate files 3x slower since v8.3.0

[Gildum]

I have been using v8.0 the whole time so I wanted to check out the newest version 9.4 and how it performs. These are my test results:

Test: archivie multiple files seperately at once

• peazip portable WIN64 versions used (from github)
• test files: 107 game roms, 8 to 16KB per file
• archiving game roms as seperate zips is my main use case, that's why I do this test
• peazip option used: add to seperate archive
• everything else is default

Results v9.4

• 100-100ms per file (as displayed by peazip UI)
• 40.4 seconds total

Results v8.0

• 22-33ms per file (as displayed by peazip UI)
• 19.4.4 seconds total

Comparison v9.4 vs v8.0

• per file compression speed is ~3 times slower
• total task completion time however is "just" ~2 times slower

System Info

• OS: Windows 7 x64bit
• CPU: Intel Core i5-3230M
• 8GB RAM (1600MT/s)

Peazip version 8.3.0 - regression start

• I went back and downloaded a bunch of other peazip versions between 8.0 and 9.4 to see where the performance decrease first appeared
• I managed to poinpoint v8.3.0
• I notcied that v.8.3.0 is the first version with an additional "pea.exe" included
• before, "peazip.exe" was the only exe, so the issue might be related to this somehow

[Gildum]

I can't seem to edit my original post, so here is are some post edits:

• v9.4 per file time: 100-110ms
• v8.0 total time: 19.4 seconds
• my laptop has an 2.5" HDD

[Giorgio Tani]

Thank you very much for reporting the issue.

Release 8.0.0 uses 7z 19.00 for ZIP format, while release 9.4.0 uses 7z 23.01.

Running the test on compression on multiple small files shows the new version introduces more overhead per file (while it is faster on a single large file), even if in my case the difference was lower with 28 seconds vs 36 seconds, test was repeated 3 times for an average time and first run was discarded to avoid disk caching effect (i7 W10 64 bit on ssd).

I'll analyze the issue better in order to understand it the overheads are introduced by PeaZip scheduler, or if 7z 23.01 is inherently slower on smaller files.

EDIT: I can confirm the scheduler in recent releases is slower and it is the cause of the overhead introduced per single file, which becomes evident in the test case of many small files.
I'll work for improving it.

[Giorgio Tani]

After some testing, I've verified the memory management in the task scheduler of current version is slower than in older version - because it was optimized for speeding up different types of tasks (pre-allocates more memory to parse content of large files, which takes some milliseconds more multiplied for each single task).
I'm improving the scheduler for the next version in order to be better balanced and provide good performances to this type of tasks.

[Admin Korkman]

Is extracting many small files affected by the same issue? 7zips contextual menu item "Extract to \" is blazing fast (blink of an eye for a batch of 10), while PeaZip is busy refreshing the GUI and needs about 1 second per archive. Measured on RAM drive, Win11 (Temporary storage NVMe SSE), 5900X., PeaZip 9.4.0

Edit: about .5 seconds per archive when extracting from PeaZip GUI with "Extract then move to destination" unchecked.

[Giorgio Tani]

Improving the internal task launcher scheduler will improve also the speed for extraction of many small archives, but in this case there also other sub tasks taking additional time at each iteration - test if the archive needs a password, and (as correctly noted) moving data to destination option.

The root limitation in performances working on many small files at once (archiving or extraction) is that due the architecture of PeaZip each task is launched as a separate process (which can be exported as a CLI script), and this costs some time depending on the host operating system.

[Giorgio Tani]

Release 9.5.0 changes how memory is allocated by the task scheduler, and it is faster on multiple small tasks, about twice as fast as 9.4.0 in most test on archiving and extracting multiple separate small files - but still not as fast as 8.0 so I'll keep testing possible performances improvements for future updates.