#554 FD leak in appledouble = v2

[Arnaud Bergeron]

With appledouble = v2 in afp.conf there is an fd leak in certain circumstances while using the Finder. This can lead to an afpd process using 11000+ file descriptors given enough time.

System details:

Server: FreeBSD 9.2 (netatalk 3.1.0)
Client: Mac OS X 10.9.2

How to reproduce:

1. Mount a share and browse to it in the Finder.
2. Make sure the Finder window mode is set to "columns"
3. Click on an image file (this may work with other types, but I haven't tried).

Attached is a debug log where I click around in a folder with images and the process leaks around 300 fd.

[Arnaud Bergeron]

This (may) be addressed by https://github.com/Netatalk/Netatalk/pull/21

[Daniel Hazelbaker]

I am running into this not because I have set appledouble = v2, but because I am re-sharing an NFSv3 share. Since it doesn't support EA it defaults back to appledouble v2. Based on the information you dug up, I did some more digging. It appears that at the heart of the matter is that ad_open opens the HF (metadata) twice.

If the AppleDouble version is 2 and they have requested that the RF (resource fork) be opened, it also flags the HF to be opened as well and opens it (ad_open.c line #1863-1867).

Then, back in afp_openfork() we AGAIN open the HF (fork.c line #381-387). Our reference count is now 2.

During the FP_CLOSEFORK command, ad_flush() is only closing the HF one time instead of twice to match the corresponding double open during the FP_OPENFORK command. The attached patch adds code to mirror the ad_open() code (ad_flush.c new line #432-441). If we are closing the RF and the AppleDouble version is 2, also close the HF. A quick test in the Finder reveals this to be working.

Test performed before patch:
1) On client in Finder in Column View (so it shows a preview), click on a word document.
1a) Server shows AppleDouble file opened.
2) On client hit the space bar to quickview the file.
2b) Server now shows 2 AppleDouble files opened (same file opened twice).
3) On client, double click word document to open it. The file will open "blank" because the HF is open and locked (guess, but seems accurate, why Word doesn't throw an error is unknown).
3a) Server continues to show 2 AppleDouble files opened (same file opened twice).

Test performed after patch:
1) On client in Finder in Column View (so it shows a preview), click on a word document.
1a) Server shows NO AppleDouble files open (logs show that it was opened and then closed).
2) On client hit the space bar to quickview the file.
2b) Server shows NO AppleDouble files open (logs again show that it was opened and then closed).
3) On client, double click word document to open it. The file will open "blank" because the HF is open and locked (guess, but seems accurate, why Word doesn't throw an error is unknown).
3a) Server now shows 1 AppleDouble files opened (as is expected).
4) On client, close the file.
4a) Server now shows 0 AppleDouble files opened (as expected).

[Ralph Böhme]

Fix integrated for 3.1.2 in [a5f965283b1646cdcc8a8006a3f3bef0c3e572f4].

I didn't test it, just did a formal review and it looked right. If it breaks anything, user complaints will be delivered to your address. :)

Thanks!