#917 Copying very large files or huge amounts of small files may make pcmanfm lock up

[Andreas E]

Hi, since I assumed previously that it's another classic PEBKAC case, I had been reluctant to post about this behavior before. But now I got some good replies even from (usually very knowledgeable) stackexchange community - for this please see here:

http://unix.stackexchange.com/questions/92472/out-of-memory-error-while-copying-large-files-with-pcmanfm

Hence, I do no longer think I'm making this up. In particular, mind this part of a user's post:

"A very quick look around the pcmanfm's source in libfm seems to show that the program may try to copy the file contents into its internal clipboard which is obviously not the way to do it."

UH-OH. And once that clipboard can no longer breathe, these lock-ups will happen.

Again, in a nutshell:

• Try to raw-copy huge files like several *.VOBs across partitions using PCManFM.
• Or, try to copy an insane amount of small files (a 10 GB audio collection on MP3 will do fine) in one go from one partition to another.

Expected:

• Nothing spectacular should happen.

Actual:

• Though not 100% reliable to say, it might happen that the copying process STALLS in the middle on one lone file, and then nothing will happen anymore.
  -- Plus, the system is locked up so severely that not even a shutdown -r now from console will work.
  -- Accessing a root console (CTRL-F1 to -4) works, but it will not make it to the shell prompt.

Assumption:
Behavior might not be the same if you copy ext3 to ext4 or extX to extY with X equal to Y.
I often have to copy NTFS-NTFS or NTFS-ext_something or ext_something-NTFS.
So it might be cross-related to FUSE backend.

And the way out? Power-cycle the machine. No other option.

SERVER USERS: Way too critical to copy huge amounts of files currently with PCManFM. For instance, think of system images. Absolutely stay to the console and good old 'cp' to be on the safe shore. Works in 100% of cases.

Bug category: ** CRITICAL **
(Despite that, I only raised Priority by +1.)

[Lonely Stranger]

Thank you very much for reporting this but unfortunately, I have to disappoint you - only clipboard operation that pcmanfm does itself is copying paths to clipboard (just a text like "/home/user/file-to-copy"), and only way libfm provides files to clipboard is their paths (as stated in general specification for "text/uri-list"), it does not support full-file or data copying at all.

As for file copying operations, PCManFM uses LibFM, and LibFM uses only GLib function g_file_copy() for it so if it hangs, pcmanfm cannot do anything with it as it never uses any own file copying functions. I'm sorry.

[Andreas E]

Thanks anyway for your reply. Well, then probably g_file_copy() is at fault here indeed.

Rats.

Hrm. To debug this further, I'd be in need of a very verbose pcmanfm build with lots of debugging noise. (just the thing one would profoundly HATE with KDE applications starting with K, this is what I do require here ;))

Any chance I could get hold of one?

[Lonely Stranger]

I'm sorry for very late answer. When I debug such things I literally just compile libfm using '--enable-debug' and '--enable-demo' switches to configure script, probably even add few extra debug lines (like g_debug("starting copy file '%s'",fm_file_info_get_disp_name(fi));) into relevant code area, then start

G_MESSAGES_DEBUG=all G_DEBUG=fatal-criticals src/libfm-demo

in the libfm sources root. The demo app is practically stripped-down version of pcmanfm (no tabs, no desktop, no per-folder config, etc.) so it does almost everything pcmanfm can do but even if it crashes it will not affect your current desktop handled with pcmanfm.

Thank you very much.

[Lonely Stranger]

After reading your description I remembered a thing that some virtual file system emulators (like few ones that mc uses) absorb whole file into memory from the source when doing copy, and then release it to the destination. Just to illustrate that PCManFM has no problems with files copying I've copied a 14 GB file (Blu-Ray disk image) from NFS partition to local one (your mentioned case of other-filesystem-to-Ext). The resources usage of PCManFM (which also handles my desktop, of course) in middle of that process:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 4018 andrej     20   0  687M 40944 20788 S  3.0  1.0  0:18.01 pcmanfm --desktop

so you can see neither memory nor CPU is used much. I would blame FUSE for your troubles, the most probably it is what eats your memory.

[Andreas E]

Yes, me too! The culprit must be FUSE. However, when I cp these insanely large files without using PCManFM, things work perfectly.
They would be copied using FUSE as well, wouldn't they?

BTW, thanks for not forgetting about this issue from last year. :)

[Lonely Stranger]

How exactly do you use cp for these files? Are they mounted somewhere? And what is a difference when you copy them using pcmanfm? Are you in exactly the same folder or in some other one like virtual mounted folder? If they are different then can you test from exactly the same folder where you execute cp? That may change everything as well, you know. Thank you.

[Andreas E]

Problem still persists...
I'm aware that you can't do much from your side, but WTH - your support is excellent. That problem does fascinate you technically in some way yourself, right? :) That'd not surprise me; you may even come across complaints from kernel's side. Alas, tail -f /var/log/messages, though incredibly handy, has become a thing of the past in the 'Buntu world. This bugs me because sometimes splitting of messages into categories (kern, auth etc.) gets clumsy. OK, dmesg works, but never "live", lacking the ability to add messages in real-time that might give you a clue of where to look for the issue.

Anyways...
Finally I could convince myself to formulate my problem for the NTFS-3G guys @ tuxera.com.
Despite a rather poor response rate, frequency of users who read it has been amazingly high. Take a look at my latest post. Screenshot shows PcManFM while copying stuff, getting stuck pretty early - and a snapshot of top at that very instant. It also contains a very detailed description of the issue.

jpa assumed that PcManFM (plus some more libfm-based file managers) would open too many file handles at the same time. Yes, perhaps that's the reason why my (admittedly low!) physical memory gets depleted in no time. And libfm visibly despises my swap, just look...;)

http://tuxera.com/forum/viewtopic.php?f=2&t=31007