#89 Large CPU and memory usage when syncing large number of files (delete them from target)

[Anonymous]

Originally created by: daniel.l...@gmail.com

What steps will reproduce the problem?
1. Delete a large number of files in the source directory of a pair
2. Sync
3. See large consumption of CPU and memory usage

What is the expected output? What do you see instead?
Files are deleted but computer remains responsive

What version of the product are you using? On what operating system?
1.1.4.241, Vista 32bit SP2

Please provide any additional information below.
When executing the loop in FolderSync.cpp in the function CFolderSync::SyncFolder() that logs "counterpart of file %s does not exist in src folder, delete file" a large number of CPU (threads) and memory resources are consumed and overall PC becomes unresponsive. See attached image for example of CPU (threads) and memory usage.

I noticed that the flag FOF_NORECURSION is not used and wonder if all sub-directories are scanned when a file is deleted, making the delete operation longer. The MSDN information on the SHFileOperation  function states that "File deletion is recursive unless you set the FOF_NORECURSION flag in lpFileOp".

[Anonymous]

Originally posted by: tortoisesvn

Not sure why you get this, but on my machine CryptSync uses only very little CPU time (more stressing the harddrive than the CPU) and uses no more than 20MB of RAM.

Test done with a folder of 5000 files in it, deleting 4000 of them.
Once deleting while CryptSync is running (not mirror mode) and once starting CryptSync after the deletion, sync pair set to mirror mode.
Both tests were fine.

Try reducing the number of lines in the log file.
Also: since the deleting is done using the windows explorer API and the deleted files are moved to the trash bin, emptying the trash bin will help here a *lot*, especially on pre-Win8 systems.
And of course explorer plugins that don't behave properly could also be the reason for the huge amount of memory use you're seeing.
The memory use of CryptSync itself should not exceed 50MB in most setups:
default for number of lines in the log file is 10000, even if every line in the log file is 500 chars long, that would only use up 10000*500*2=10MB.
All the other memory that's used is only used temporary and freed as soon as it's not needed anymore.

[Anonymous]

Originally posted by: daniel.l...@gmail.com

I have 5000 files in 269 folders, files of various sizes.

The log file is only 2 MB. I exited CryptSync and renamed it to see if that
would help but it did not. I disabled all Explorer add ons and had the same
behaviour.

Before the sync starts CryptSync has 4 threads and uses 22.7 MB of memory.
Once the mass delete in the target folder is initiated I see is a very large
number of thread being started (> 400).  Once the deletions are done the
number of threads goes back to 4 and memory goes back down to
approx. the same amount.

If your code is not creating the threads perhaps the SHFileOperation
function is doing this? I searched the Web for information about this
function and saw that the FOF_NORECURSION should be used to prevent
recursion, perhaps this is something to explore. I also found
https://github.com/vmg/clar/pull/26 which indicates a method to prevent COM
being brought up / down on each call (I don't understand this, just bringing
it to your attention).  If SHFileOperation creates a new thread and works
asynchronously then the loop will create multiple threads.

If I empty the Recycle Bin the number of thread starts
low but increases quickly to 100 (see attached graph) and ramps down back to
5 (see
attached graph) when deletes are done. The CPU utilization goes to 90% for
the CryptSync process.

http://msdn.microsoft.com/en-us/library/windows/desktop/bb762164(v=vs.85).aspx
states that SHFileOperation has been replaced in Windows Vista by
IFileOperation. Since the minimum OS level for CryptSync is now
IFileOperation them perhaps IFileOperation would bring better results.

Hope this helps

[Anonymous]

Originally posted by: ilog...@gmail.com

I have relate issue when sync one PDF file 40 MB. 7zip run with 400 MB memory and 100% CPU. I don't need high compress.

[Anonymous]

I have the same issue. 7Zip console uses constantly 10-15% of CPU, even when nothing is being synced. I had to disable the program from running automatically.