Re: [Netatalk-admins] Performance issues in OS X Finder, 10.8 and 10.9

[Ralph B. <rb...@ne...>]

Hi Chris,

sorry, I don't have much time at the moment for community support. Netatalk has a directory cache, it's just not used everywhere, so in some places the code falls back to getcwd() for a few things. I guess this could be fixed in a few days work. But then the cache has a limit of the maximum number of entries (same applies to Samba), so if you really have millions of folders and the client scans them recursively, a cache won't help.

And of course there are network filesystem change notification mechanisms for AFP and SMB.

Cheerio!
-r

Am 12.08.2014 um 12:06 schrieb Chris Ferebee <cf...@fe...>:

> Hi Ralph,
> 
> I’ve got some more input on our netatalk performance problems on SmartOS/Illumos with the OS X 10.9 Finder. Aside from the discussion we’ve been having on SmartOS-discuss
> 
> 	<http://www.mail-archive.com/sma...@li.../msg00989.html>
> 
> which points to the high cost of getcwd() on Illumos as a major bottleneck, I got a very interesting response from Jon Gotow of St. Clair Software, developer of DefaultFolder. I'm attaching his email below with his permission.
> 
> In particular, he mentions changes in the Finder as of 10.9.0 that resulted in extreme slowdowns due to the way the Finder accesses xattrs for tags and labels. This was improved somewhat (for local volumes) in 10.9.2.
> 
> He also says that the Finder relies on FSEvents for local volumes to updates its (small) cache of xattr data as needed, whereas there appears to be no FSEvents mechanism for network volumes.
> 
> This got me thinking, in relation to your new Spotlight work. Is there a facility in any of Apple's network filesystem stacks (AFP, SMB2) for FSEvents or an equivalent, or is that on the roadmap or visible on the horizon? Perhaps if there were, it would be possible to implement it (or even spoof it) in order to throttle the Finder's incessant thrashing through fsgetpath(), getxattr(), and getattrlist().
> 
> Best,
> Chris
> 
> ===== forwarded from private communication: =====
> 
>> Von: Jon Gotow <...>
>> Betreff: Re: Mavericks Finder killing us with fsgetpath(), getxattr() and getattrlist() calls
>> Datum: 11. August 2014 22:27:17 MESZ
>> An: Chris Ferebee <cf...@fe...>
>> 
>> 
>> On Aug 8, 2014, at 1:25 PM, Chris Ferebee <cf...@fe...> wrote:
>> 
>>> Interestingly, when no Finder windows are open, the Finder still loops continuously making these calls, but they never hit the server. Apparently they are cached locally on the Mac at some level. However, as soon as the window is open, the calls go through to the server over AFP.
>> 
>> Right - and the Finder doesn't cache data from servers, so it keeps hitting the server(s) over and over and over.
>> 
>> Here's my take on what I _think_ is happening, though I haven't followed up with debugging to determine whether I'm right:
>> 
>> In Mavericks, Apple added tagging to the filesystem. Tags are stored in extended attributes, which are accessed separately from other filesystem data via the xattr API. At the same time, they made the incredibly stupid change of moving the Finder's label data into the tagging subsystem. The label color used to be stored as a direct HFS attribute on files and folders - now it's stored in the xattr entry for the file. What this did in Mavericks is doubled the number of filesystem calls that the Finder has to make, since it has to know the label color for every file and folder it's displaying (as opposed to the tag information, which is generally less frequently accessed). On top of that, the xattr api seems to be relatively inefficient - this resulted in horrible Finder slowdowns in the first couple releases of Mavericks.
>> 
>> How this relates to Open and Save dialogs: in Lion and Mountain Lion, Apple rewrote significant portions of the Cocoa NSSavePanel implementation so that it now uses FinderKit, the framework that's used by the Finder for displaying files and folders (Open and Save panels previously had a completely separate implementation for displaying files and folders in table views). The upshot of this has been that when the Finder's performance was trashed in Mavericks 10.9.0, the Open and Save dialogs suffered too.
>> 
>>> Analyzing the server side with DTrace shows that afpd is spending most of its time in getcwd(), a syscall that returns the path for the current working directory. Apparently, this syscall is quite expensive on Solaris, and nobody has bothered to optimize it too much because the expectation is that it will never be called in a hot path.
>>> 
>>> I suspect that this may explain in part why performance is better on the Synology NAS, which uses Linux. I’ve seen hints that getcwd() is much faster on Linux.
>>> 
>>> Samba, BTW, appears to have an internal cache designed to minimize calls to getcwd(), but netatalk does not.
>> 
>> Ouch. That does corroborate what I've seen from customers complaining about performance in 10.9.0.
>> 
>>> In summary, I suspect that I may have hit on a particularly pathological instance of this issue:
>>> 
>>> - Solaris with its expensive getcwd()
>>> - very large number of files and directories on the server, perhaps overwhelming any internal cache OS X may have?
>> 
>> I believe that the Finder's cache for server volumes is very thin. For local volumes, the Finder relies on filesystem notifications to selectively invalidate and update its cached information. There are no equivalent notifications for server volumes (that I'm aware of) so I suspect it does more frequent updates and caches only information that's accessed very frequently. If the server doesn't do adequate caching on its end, this results in poor (and sometimes very poor) performance. 
>> 
>>> - netatalk with its lack of a getcwd() cache mechanism.
>>> 
>>> However, it looks like this is a more general issue.
>>> 
>>> 1. From what I have heard from other sites, it happens with SMB shares, though less severely so.
>>> 
>>> 2. As mentioned on the Apple Discussion Forum where I saw your comments, it seems to happen with local drives.
>> 
>> The problem on local volumes was mitigated somehow in 10.9.2 - though the Finder is still painfully slow when compared to Snow Leopard.
>> 
>>> 3. I recently spoke with a support engineer at a HELIOS reseller, and HELIOS EtherShare (the premier enterprise-grade AFP server on the market) suffers from the same issues. Apparently, there are new tricks in the upcoming release of EtherShare that work around the problem by spoofing the Finder in some way.
>> 
>> It'd be very interesting to know what they did.
>> 
>>> More relevant to your line of work, the exact same issues affect the Open/Save dialogs on my client’s Macs. Either because they tie into the Finder, or just because the Finder is saturating the AFP connection with spurious requests to a degree that all I/O to the server slows to a crawl.
>> 
>> It's because the Open/Save dialogs use exactly the same code that the Finder does, so they're just as susceptible to these problems.
>> 
>>> If you want more nitty-gritty info, I can point you to my discussions with the netatalk and SmartOS kernel engineers on the mailing lists. :-)
>> 
>> That'd be great. I don't know that there's a whole lot that can be done from the client side - it seems the information has to be more aggressively cached on the server side, along with cleaning up the Solaris implementation of getcwd() so it's not so inefficient. Or perhaps using some alternate call - the Solaris filesystem isn't slow, so perhaps there's some other way to get the current directory besides calling getcwd()? 
>> 
>> - Jon
>> 

-- 
Ralph Böhme <rb...@ne...>
Netatalk Developer | Support | Services
NetAFP joins SerNet: <http://www.netafp.com/sernet-expands-apple-support-netatalk-and-samba-merge-1230/>