#130 Scan of 1.52 TB server volume stops after 147 GB

[James SF DeLaHunt]

I used GrandPerspective to scan a large-ish file server volume. GrandPerspective's scan stops after just 149 GB from one top-level subdirectory of the volume.

The upshot of this is that I can't use GrandPerspective to understand the overall space usage on this volume. Maybe I will get something useful done if I scan subsets of the volume one at a time.

How to reproduce:

1. Run GrandPerspective 3.8.0
2. File… Scan Folder… select a volume, which has (per macOS Finder) 1.52 TB of content and 1.72 TB free, mounted via SMB from file server
3. press OK button
4. GrandPerspective shows a window giving scan progress, taking about 15 minutes
5. GrandPerspective displays a scan view of the server volume
6. View… Zoom… Reset zoom (so, no zoom effects)
7. View… Selection Focus… Reset focus (so, no focus effects)
8. View… Display Focus… Reset focus (so, no focus effects)

Expected behaviour:
The scan view includes all 1.52 TB of files, and perhaps the 1.72 TB of free space, on the volume.

Observed behaviour:
The scan view includes only 147 GB, limited to partial contents of just one of the top-level directories of the volume. The rest of the volume contents are not visible.

This is with no file selected, and no focus or zoom effects.

The stats which the Info tab of GrandPerspective's control panel window reports are:

• File sizes: Physical (Binary)
• Filter: none
• Volume size: 2.94 TB
• Size of scanned files: 147 GB
• Miscellaneous used space: 1.24 TB
• Initial free space: 1.54 TB
• Freed space: 0
• Number of scanned files: 52547
• Number of deleted files: 0

The Focus tab reports:

• Folder in view: (path name of server volume)
• Selected folder: (path name of server volume)
  Both of those are as expected, and I interpret them as confirming that selection, zoom, and focus are not affecting the results.

Discussion:

Observed with GrandPerspective 3.8.0, on macOS Sonoma 14.8.9, on a mac with Apple Silicon CPU.

[James SF DeLaHunt]

I tried this on a subdirectory of the same server volume. Again GrandPerspective failed to read in more than about 10% of the directory. The directory contains 499 GB of content, as measured by du -sh. The scan window in GrandPerspective showed only 42.1 GB.

For what it's worth, du on the server-mounted directory yielded many du: cannot access errors. Also du perceives a circular directory structure. This is because the underlying file system has symlinks between directories which do form cycles, and the SMB software presents those symlinks as actual directories, leading to apparent infinitely-deep and circular directory structures. Maybe GrandPerspective is silently failing when it encounters these.

[Erwin Bonsma]

GrandPerspective does not follow symlinks, as that does not help to get a good view of actual disk usage. However, every symlinked folder should be accessible also without following a symlink, so that does not explain why the app only finds a small subset of the actual files.

A possible reasons why GrandPerspective cannot find all files are lack of permissions. This could be because they are owned by a different user, or they are system files that require specific entitlements to be read (e.g. Data Vaults).

What you can do, is use the new logging to find out why certain directories are being skipped while scanning. See https://grandperspectiv.sourceforge.net/HelpDocumentation/HowToViewDiagnosticLogs.html (this log feature is newly added in version 3.8.0).

With help of the logs you can hopefully establish why certain large folders/files that you know exist, are not included in the scan. Please share the results if you think there is anything that can be improved.

Consider first viewing the logs without enabling full logging (i.e. do not provide the -logAll YES option) as this generates fewer messages, so you are less likely to overlook important/relevant log messages. However, enable full logging if you think that may be helpful.

[James SF DeLaHunt]

GrandPerspective does not follow symlinks, as that does not help to get a good view of actual disk usage. However, every symlinked folder should be accessible also without following a symlink, so that does not explain why the app only finds a small subset of the actual files.

I agree that the "symlink" detail does not explain the main problem.

However, I want to clarify the symlink detail. I believe that GrandPerspective does not see any symlinks when it scans the file server volume. The client mounts a Samba share of the file server volume. The symlinks are only in the server's underlying file system. The Samba Samba software reads these symlinks, and presents them to the client as if they were directories which are identical to the symlink's target. It does not present symlinks to the client.

Therefore, in this test, GrandPerspective is not reading symlinks. It is reading simulated directories (and maybe files) constructed by the Samba server software.

[James SF DeLaHunt]

Thank you for the pointer to the logging capability of GrandPerspective 3.8.0. Very helpful!

I used Howard Oakley's Ulbow app (https://eclecticlight.co/consolation-t2m2-and-log-utilities/) instead of the macOS log command, but it gives the same result. The log messages mark the path names as <private>, even when running GrandPerspective with --args -logAll YES, so I followed Oakley's instructions at https://eclecticlight.co/2023/03/08/removing-privacy-censorship-from-the-log/ to reveal them.

The log messages seem to say that GrandPerspective thinks that the missing directories do not exist. There are log messages of the form:

13:38:49.674201 net.sf.grandperspectiv Error reading directory /Volumes/VolName/dirname/tmp: No such file or directory

Interestingly, when I ask GrandPerspective to scan /Volumes/VolName/dirname/tmp directly, it is able to read that directory. In some cases it fails to read subdirectories of the scanned directory.

It will take me a little more work to look at the rejected directories and figure out what they have in common that might cause GrandPerspective to reject them.

Here is the redacted log from running GrandPerspective on /Volumes/VolName/dirname:

13:33:04.623625 net.sf.grandperspectiv trusted log enabled = 0
13:33:05.727700 net.sf.grandperspectiv Restored access to: /
13:33:05.727701 net.sf.grandperspectiv If this is not your root volume, you should reset the bookmark using: defaults delete net.sourceforge.grandperspectiv rootVolumeBookmark
13:33:05.727701 net.sf.grandperspectiv Obtained access to root volume
13:33:05.760741 net.sf.grandperspectiv FDA permission check using /Users/myuser/Library/Safari succeeded
13:33:05.760741 net.sf.grandperspectiv Verified FDA permissions
13:33:12.713182 net.sf.grandperspectiv fastPackageCheckEnabled = 1
13:33:12.713232 net.sf.grandperspectiv VolumeURLKey: url = file:///Volumes/VolName/dirname/, volumeRoot = file:///Volumes/VolName/
13:33:12.715194 net.sf.grandperspectiv Volume format = SMB (OS X)
13:33:12.715200 net.sf.grandperspectiv fstypename = smbfs
13:33:12.715200 net.sf.grandperspectiv ignoreHardLinksForDirectories = 0
13:33:12.715298 net.sf.grandperspectiv Scanning volume /Volumes/VolName [VolName], starting at dirname
... [85 "Error reading directory" messages for various directories elided] ...
13:38:49.674201 net.sf.grandperspectiv Error reading directory /Volumes/VolName/dirname/tmp: No such file or directory
... [elided] ...
13:38:49.674672 net.sf.grandperspectiv Done scanning: 5015 folders scanned (0 skipped) in 336.96s.

[Erwin Bonsma]

Thanks for sharing the extra detail.

Just to check, when invoking GrandPerspective with --args -logAll YES do you not even see the path names in the Scanning <path> messages? If so, that is not supposed to happen. This is what the expected behavior is:

Scanning volume <private> [<private>], starting at <private>
scanTreeForDirectory <private>
Scanning /Users/erwinbonsma/temp
Scanning /Users/erwinbonsma/temp/GrandPerspective-3_7_0
Scanning /Users/erwinbonsma/temp/GrandPerspective-3_7_0/src
Scanning /Users/erwinbonsma/temp/GrandPerspective-3_7_0/src/de.lproj
Scanning /Users/erwinbonsma/temp/GrandPerspective-3_7_0/src/tree
Scanning /Users/erwinbonsma/temp/GrandPerspective-3_7_0/src/mapping

So most messages are still censored, but the most detailed ones not. However, if you want to view fully uncensored logs, then you need to indeed install a profile as you did.

Once you manage to identify why GrandPerspective cannot read certain content, please let me know. Is it due to permissions, or does the path it tries to read differ from the actual path name, or something else? As long as I cannot reproduce it locally, it is difficult to fix it.

[James SF DeLaHunt]

I just performed an experiment:

Run Ulbow app to collect logs. Set Predicate to '(subsystem="net.sf.grandperspectiv")'. Checked the menu option "Get Debug" (which refers to debug-level messages). I did not install the module which removes censorship from log messages.

I then ran GrandPerspective 3.8.0 using this command:
open /Applications/GrandPerspective.app --args -logAll YES
The GrandPerspective app opened.

I scanned the directory Documents/Pr/2026 on my local computer which is the origin for the files on the file server. Ulbow revealed the following log messages:

13:35:44.141602 Info 13337846 18508 net.sf.grandperspectiv fastPackageCheckEnabled = 1 
13:35:44.141716 Info 13337846 18508 net.sf.grandperspectiv VolumeURLKey: url = <private>, volumeRoot = <private> 
13:35:44.142433 Info 13337846 18508 net.sf.grandperspectiv Volume format = <private> 
13:35:44.142459 Info 13337846 18508 net.sf.grandperspectiv fstypename = <private> 
13:35:44.142460 Info 13337846 18508 net.sf.grandperspectiv ignoreHardLinksForDirectories = 1 
13:35:44.142595 Info 13337846 18508 net.sf.grandperspectiv Scanning volume <private> [<private>], starting at <private> 
13:35:44.438131 Default 13337846 18508 net.sf.grandperspectiv Done scanning: 5352 folders scanned (0 skipped) in 0.30s. 

Then I scanned the directory /Volumes/VolName/dirname/Projects/2026 on my file server. This is backup of most but not all of the files in Documents/Pr/2026. Ulbow revealed the following log messages:

13:35:50.400835 Info 13337846 18508 net.sf.grandperspectiv fastPackageCheckEnabled = 1 
13:35:50.408964 Info 13337846 18508 net.sf.grandperspectiv VolumeURLKey: url = <private>, volumeRoot = <private> 
13:35:50.409637 Info 13337846 18508 net.sf.grandperspectiv Volume format = <private> 
13:35:50.409652 Info 13337846 18508 net.sf.grandperspectiv fstypename = <private> 
13:35:50.409652 Info 13337846 18508 net.sf.grandperspectiv ignoreHardLinksForDirectories = 0 
13:35:50.409880 Info 13337846 18508 net.sf.grandperspectiv Scanning volume <private> [<private>], starting at <private> 
13:36:23.911631 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.912283 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.912883 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.913389 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.914056 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.914587 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.915187 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.915683 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.916242 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.917442 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.917953 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.918450 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.918939 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.919446 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.919948 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.920443 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.920942 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.921455 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.922979 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.923597 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.923925 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.924263 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.924605 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.924905 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.925198 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.925491 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.925784 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.926062 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.926339 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.926629 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.926906 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.927187 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.927469 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.927749 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.928035 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.928316 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.928603 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.928919 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.929224 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.929532 Info 13337846 18508 net.sf.grandperspectiv Error reading directory <private>: No such file or directory 
13:36:23.930052 Default 13337846 18508 net.sf.grandperspectiv Done scanning: 1908 folders scanned (0 skipped) in 33.53s. 
13:36:41.611870 Info 13339896 18508 net.sf.grandperspectiv Thread terminated. 
13:36:41.611886 Info 13339897 18508 net.sf.grandperspectiv Thread terminated. 
13:36:41.611908 Info 13339896 18508 net.sf.grandperspectiv ScanTreeRoot-dealloc 

Note that GrandPerspective scanned many more folders on the local directory, and printed zero error messages about "No such file or directory".

Also, it seems there is a second issue: you expect that Scanning <path> messages will have path information when GrandPerspective has --args -logAll YES, but I observe the messages have the path information censored.

[Erwin Bonsma]

Also, it seems there is a second issue: you expect that Scanning <path> messages will have path information when GrandPerspective has --args -logAll YES, but I observe the messages have the path information censored.</path>

That is strange. The "Scanning <path>" messages are logged with a public modifier to a trusted logger that is disabled by default. That should prevent sensoring, and does so on my system. Maybe this is a limitation of your version of macOS (14.8), which is fairly old.</path>

Anyway, I will also the "Error reading directory <path>" messages to the trusted logger, as those are generally useful as well for diagnostics.</path>

[Erwin Bonsma]

Note that GrandPerspective scanned many more folders on the local directory, and printed zero error messages about "No such file or directory".

What do the paths look like in the error messages? Do they look correct? Are they accessible from the terminal, for example using ls -al <path>?

I suspect the issue you are experiencing is specific to the Samba server software that is used. GrandPerspective's scanning logic uses the generic FTS API. There's one thing that may still be worth a try (but that requires a dedicated debug build). GrandPerspective scans with the FTS_XDEV option set, which prevents it from descending into directories that have a different device number than the file from which the descent began. Maybe disabling this option has an impact.