Thanks both for the report. It could indeed be related to the new APFS file system.
Can you give me an indication of the slow down? How long does it now take to scan your drive, and how many files and folders does it contain?
Btw, it may be a while before I can look into it. I only have one Mac, which I do not want to upgrade just yet. Furthermore, I would like to verify that after any improvements, assuming that is possible, performance is still acceptable on older versions of Mac OS. I do want to get this fixed though.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I came here with the same observation. I'm running 10.13 on the newest Macbook Pro with an SSD and it feels like it is crawling, taking 15+ minutes to scan my 1TB disk with about 200,000 folders.
I thought the new AFPS kept track of things like the size of all files in a folder. Which could be useful in doing an initial scan.
I still love GrandPerspective and just bought it on the App Store.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can confirm on 10.13.2 - in the middle of a full disk scan on a 10.13.2 SSD (that I eventually had to cancel), it was doing approx 1-2 folders per second. I'll try and post some better stats later today. Any logging that can be pulled or enabled?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Attached scan of 579GB on a 15" 2016 Macbook Pro with a 1TB SSD took 23 mins. Oddly, the initial pass I did looked like it was going to take hours. Feel free to contact me if you need more data - happy to help.
Thanks for the further info. I hope to soon be able to investigate. My new MacMini with SSD is on its way.
You can get more logging if you invoke it from Terminal as follows: GrandPerspective.app/Contents/MacOS/GrandPerspective -logAll YES
This will give log message for each directory that is scanned, together with a timestamp. No need to share the entire log here, but I would be interested to where the time is spent. Does the scan of each directory take about the same time, or are there specific directories that are slow to be scanned?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Stefan for providing me with the detailed log of the (slow) scan on your system via DropBox. I compared it with the (still fast) scan on my system (still on OS X 10.11).
On my system GrandPerspective scans on average 1950 folders per second, whereas on your system only 373. As both systems have SSDs, the performance difference is significant.
Analysing the data further, it seems the difference is that the fixed cost to scan a folder seems to have increased. On your system, the scan of each folder takes approximately the same time, with a few folders that take longer to scan (presumably because of the number of files they contain). In contrast, on my system the majority of folders take neglibible time. The scan of more than 68% of the folders on my system takes no time at all. Presumably these are the folders with only few files.
Attached is a plot that illustrates the difference.
I have yet to investigate it further, and then see what, if anything, can be done about this, but this gives a few pointers.
Fast directory sizing allows Apple File System to quickly compute the total space used by a directory hierarchy, and update it as the hierarchy evolves.
Fast directory sizing works by precomputing the size of directory as content is added and removed. Therefore, it is most appropriate for directories that contain many files and have relatively little churn. For example, a user’s Documents folder is a good candidate for fast directory sizing, whereas the /tmp directory would not.
The file system can enable fast directory sizing on empty directories. You cannot enable Fast Directory Sizing on directories containing files or other directories directly; you must instead first create a new directory, enable fast directory sizing on it, and then move the contents of the existing directory to the new directory.
"
Interesting. This machine recently had a full restore from a Carbon Copy Cloner backup. I wonder whether FDS would be enabled as part of that restore.
Also: https://forums.developer.apple.com/thread/79901
Hmm. I'm pretty much at the limit of my knowledge depth here.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A progress update: Initial analysis on a system with OS X 10.13 did not reveal a small, easily fixable cause for the performance regression.
GrandPerspective is an old app though, and the API it uses to scan directory contents is meanwhile deprecated. Therefore I decided to switch to the new file enumeration API. This is a big change, as it fundamentally changes the scanning algorithm (from breadth-first to depth first iteration, and not using recursion anymore). The initial results are promising though. Performance is significantly improved.
Note, it will take some more time before the changes are finalized, thoroughly tested, and ready for usage.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
Exactly the same situation... late 2013 MBP 15" with SSD. Maybe it's related to the new APFS file system?
Thanks both for the report. It could indeed be related to the new APFS file system.
Can you give me an indication of the slow down? How long does it now take to scan your drive, and how many files and folders does it contain?
Btw, it may be a while before I can look into it. I only have one Mac, which I do not want to upgrade just yet. Furthermore, I would like to verify that after any improvements, assuming that is possible, performance is still acceptable on older versions of Mac OS. I do want to get this fixed though.
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
MBP 2011
Files 470000
Folders 110000
Scan 05:40
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
Thanks for providing the extra info. That does seem a bit slow indeed.
For comparison some stats from scanning my MacMini 2011 with SSD:
Files 890000
Folders 200000
Scan 1:26
So nearly twice as much more data to scan, yet more than three times as fast.
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
I came here with the same observation. I'm running 10.13 on the newest Macbook Pro with an SSD and it feels like it is crawling, taking 15+ minutes to scan my 1TB disk with about 200,000 folders.
I thought the new AFPS kept track of things like the size of all files in a folder. Which could be useful in doing an initial scan.
I still love GrandPerspective and just bought it on the App Store.
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
Same problem here -- scan takes half hour instead of minutes previously. Software is not usable like this.
Thanks for the reports. I soon hope to have a access to a Mac with MacOS 10.12, after which I can start looking into it.
View and moderate all "bugs Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Bugs"
I can confirm on 10.13.2 - in the middle of a full disk scan on a 10.13.2 SSD (that I eventually had to cancel), it was doing approx 1-2 folders per second. I'll try and post some better stats later today. Any logging that can be pulled or enabled?
Attached scan of 579GB on a 15" 2016 Macbook Pro with a 1TB SSD took 23 mins. Oddly, the initial pass I did looked like it was going to take hours. Feel free to contact me if you need more data - happy to help.
Hi Stefan and Anonymous,
Thanks for the further info. I hope to soon be able to investigate. My new MacMini with SSD is on its way.
You can get more logging if you invoke it from Terminal as follows: GrandPerspective.app/Contents/MacOS/GrandPerspective -logAll YES
This will give log message for each directory that is scanned, together with a timestamp. No need to share the entire log here, but I would be interested to where the time is spent. Does the scan of each directory take about the same time, or are there specific directories that are slow to be scanned?
Thanks Stefan for providing me with the detailed log of the (slow) scan on your system via DropBox. I compared it with the (still fast) scan on my system (still on OS X 10.11).
On my system GrandPerspective scans on average 1950 folders per second, whereas on your system only 373. As both systems have SSDs, the performance difference is significant.
Analysing the data further, it seems the difference is that the fixed cost to scan a folder seems to have increased. On your system, the scan of each folder takes approximately the same time, with a few folders that take longer to scan (presumably because of the number of files they contain). In contrast, on my system the majority of folders take neglibible time. The scan of more than 68% of the folders on my system takes no time at all. Presumably these are the folders with only few files.
Attached is a plot that illustrates the difference.
I have yet to investigate it further, and then see what, if anything, can be done about this, but this gives a few pointers.
Given that fast directory sizing was one of the big features touted for APFS, this does seem a little perverse.
https://www.google.com/search?client=safari&rls=en&ei=s5ZSWqiADuaMjwSFta2gCg&q=apfs+folder+sizing+&oq=apfs+folder+sizing+&gs_l=psy-ab.3..33i160k1l2.5394.6153.0.8061.7.7.0.0.0.0.152.597.0j5.5.0....0...1.1.64.psy-ab..2.4.444...0.0.VpE0QVtRHtU
"Fast Directory Sizing
Fast directory sizing allows Apple File System to quickly compute the total space used by a directory hierarchy, and update it as the hierarchy evolves.
Fast directory sizing works by precomputing the size of directory as content is added and removed. Therefore, it is most appropriate for directories that contain many files and have relatively little churn. For example, a user’s Documents folder is a good candidate for fast directory sizing, whereas the /tmp directory would not.
The file system can enable fast directory sizing on empty directories. You cannot enable Fast Directory Sizing on directories containing files or other directories directly; you must instead first create a new directory, enable fast directory sizing on it, and then move the contents of the existing directory to the new directory.
"
Interesting. This machine recently had a full restore from a Carbon Copy Cloner backup. I wonder whether FDS would be enabled as part of that restore.
Also:
https://forums.developer.apple.com/thread/79901
Hmm. I'm pretty much at the limit of my knowledge depth here.
A progress update: Initial analysis on a system with OS X 10.13 did not reveal a small, easily fixable cause for the performance regression.
GrandPerspective is an old app though, and the API it uses to scan directory contents is meanwhile deprecated. Therefore I decided to switch to the new file enumeration API. This is a big change, as it fundamentally changes the scanning algorithm (from breadth-first to depth first iteration, and not using recursion anymore). The initial results are promising though. Performance is significantly improved.
Note, it will take some more time before the changes are finalized, thoroughly tested, and ready for usage.
The fix is complete and will be part of the upcoming release. Keeping this item open until then.
Fixed in version 2.0.0, which has just been released
Fantastic work! Thanks so much for the significant speed improvement.