#5122 Strange System Information > Local disk space report

[J.M. Rivera]

Hi, I has been experiencing a very strange issue with Wembin "System Information > Local disk space", after few hours the "Local disk space" start reporting an extremelly inflated local disk space data;

After trying to find cause for days with no luck, I decided to report this problem with images and system disk usage right after and before the issue, only solution I've found was just rebooting the server for now, unfortunately the issue start right after few hours.

System report during normal state:

Normal State Data txt

System report during wrong state:

Wrong State Data txt

Testing system info:
Webmin: 1.880, 1.881 (default install)
OS: Vanilla FreeBSD 11.1 RootOnZFS
MB: Supermicro X8SI6-F
CPU: Xeon X3450
RAM: 16GB DDR3 ECC RDIMMs.
OS Disk: 1x 250GB, GPT/ZFS Stripe
Data Disk: 6x 320GB, GPT/ZFS RAIDZ2

Regards

[Ilia Rostovtsev]

The stats in the theme rely on mount::local_disk_space. This really odd issue, I have never seen it happening.

Jamie, what do you think could be the cause of the issue?

[J.M. Rivera]

Hi, thanks for the prompt response, I forgot to add that restarting Webmin, switching between themes has no effect, so I guess is more likely a core issue.

However, I re-installed and setup latest FreeBSD 11.1 | Webmin on my system, I will update on any finds.

EDIT: Unfortunately right after the fresh setup, the strange disk usage started to raise rapidly again just after one hour:

Latest System Data txt

Regards

[Jamie Cameron]

What does the df -h command output on this system?

[J.M. Rivera]

Hi Jamie, each picture have a link below with all relevant information during each system state for convenience, however, I will give you the requested info.

df -h
Filesystem                    Size    Used   Avail Capacity  Mounted on
zroot/ROOT/initial-install    222G    1.1G    221G     0%    /
devfs                         1.0K    1.0K      0B   100%    /dev
storage                       176G    192K    176G     0%    /mnt/storage
storage/mydata                534G    358G    176G    67%    /mnt/storage/mydata
storage/mymedia               778G    602G    176G    77%    /mnt/storage/mymedia
storage/sysdata               181G    4.5G    176G     3%    /mnt/storage/sysdata
storage/tmp                   176G    176K    176G     0%    /mnt/storage/tmp
zroot/tmp                     221G    176K    221G     0%    /tmp
zroot/usr/home                221G     88K    221G     0%    /usr/home
zroot/usr/ports               222G    661M    221G     0%    /usr/ports
zroot/usr/src                 221G     88K    221G     0%    /usr/src
zroot/var/audit               221G     88K    221G     0%    /var/audit
zroot/var/crash               221G     88K    221G     0%    /var/crash
zroot/var/log                 221G    420K    221G     0%    /var/log
zroot/var/mail                221G     96K    221G     0%    /var/mail
zroot/var/tmp                 221G     88K    221G     0%    /var/tmp
zroot                         221G     88K    221G     0%    /zroot
/mnt/storage/mymedia/Media    778G    602G    176G    77%    /mnt/shared/Media

Regards

[Jamie Cameron]

Oh, you're running ZFS .. that makes calculating free local disk space a lot harder!

What does the zpool list -P and zpool list -p commands output on your system?

[J.M. Rivera]

Heres is zpool list -p

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  1906965479424  1563515768832  343449710592         -    39%     81  1.00x  ONLINE  -
zroot    246960619520  1916407808  245044211712         -     0%      0  1.00x  ONLINE  -

Regards

[Jamie Cameron]

If you run the mount command, is the type of those filesystems shown as zfs ?

[J.M. Rivera]

Yes, all my filesystems in the server are ZFS, ecxept for a shared directory which is a Read-Only NULLFS, heres the output for "mount":

# mount
zroot/ROOT/initial-install on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
storage on /mnt/storage (zfs, local, noatime, nfsv4acls)
storage/mydata on /mnt/storage/mydata (zfs, local, noatime, nfsv4acls)
storage/mymedia on /mnt/storage/mymedia (zfs, local, noatime, nfsv4acls)
storage/sysdata on /mnt/storage/sysdata (zfs, local, noatime, nfsv4acls)
storage/tmp on /mnt/storage/tmp (zfs, local, noatime, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
/mnt/storage/mymedia/Media on /mnt/shared/Media (nullfs, local, read-only)

Regards

[J.M. Rivera]

Hello just an update on this, I think I found the cause and actually what I thought, seems that Webmin free disk space code is counting and summing out every ZFS snapshots(if they set as visible) as additional files/file systems, even if the snapshots started at ZERO disk usage, and is easily reproduced on VM as well on my hardware.

The more spanpshots I create(obiously they ZERO disk usage since no files changed), the more the Local disk space usage in Webmin increases.

Definitely the snapshots(.zfs) should be excluded except the space portion they using wen files changes, but not the wole snapshot at all.

How to reproduce this issue
Enable ZFS snapshot visibility with:

zfs set snapdir=visible tank/dataset

Trow few large files inside the created dataset.
Star creating a lot of snapshots of that dataset as much as you can with:

zfs snapshot tank/dataset@random-numbers

Then look at how the space usage start increasing on Webmin status page, also clean browser cache several times as seems Webmin takes some time to prossess this info.

Regards.

[Jamie Cameron]

Ok, that could explain it. Are these snapshots shown when you run df -h ?

[J.M. Rivera]

Hi, df -h command does not show the zfs snapshots, indeed it would be a really bad idea if it will(theres too many), though I left them hidden for now as I can cd to them regardless if visible or not, since that I haven't noticed that odd disk space I was experiencing before.

Regards

[Jamie Cameron]

Do they show up in /etc/mtab (or the equivalent on FreeBSD) though?

[J.M. Rivera]

No they aren't shown in /etc/fstab nor any other file that I know.
Maybe I will try reproduce this on ZFS on Linux too, I'm really curious about this.

[Jamie Cameron]

Is there a command to determine which directories are mounted on ZFS snapshots and which are not?

[J.M. Rivera]

Hi, mounted zfs snapshots will be populated and displayed with the standard mount system command.

However I will post an deeper test on this issue for some clarificastions.(note that my zfs pool/datasets all has compression LZ4 enabled)

Testing on: FreeBSD 11.1 P10

================================================
Single dataset testing with 'du' command

Report for directory "/mnt/storage/mymedia"(with zfs set snapdir=hidden storage/mymedia):

cd /mnt/storage/mymedia
du -h -A
Results = 603G (Seems OK)
================================================
Report for directory "/mnt/storage/mymedia/.zfs/snapshot/auto-2018-04-12_20.00.01--30d" (one of the 44 currently available snapshots):
cd /mnt/storage/mymedia/.zfs/snapshot/auto-2018-04-12_20.00.01--30d
du -h -A
Results = 603G (See below)

zfs list storage/mymedia@auto-2018-04-12_20.00.01--30d
NAME USED AVAIL REFER MOUNTPOINT
storage/mymedia@auto-2018-04-12_20.00.01--30d 16.0K - 602G -

Is actually using only 16K but 'du' refers to 603G, then this 'refer' usage is being
sumed out by 'du' for the every snapshot it found.
================================================
Report for directory "/mnt/storage/mymedia/.zfs" (theres 44 snapshots there):
cd /mnt/storage/mymedia/.zfs
du -h -A
Results = 26T (Seems 603x44=26532)
================================================
Report for directory "/mnt/storage/mymedia" (with zfs set snapdir=visible storage/mymedia):
cd /mnt/storage/mymedia
du -h -A
Results = 27T (Seems all was summed)
================================================
Testing on whole zfs pool with 'du' command
With ==> snapdir=hidden on all datasets

Report for directory "/mnt/storage"(theres 3 datasets in there, all with snapshots but hidden):

cd /mnt/storage
du -h -A
Results = 980G (Seems OK)
================================================
Testing on whole zfs pool with 'du' command
With ==> snapdir=visible on all datasets

Report for directory "/mnt/storage"(theres 3 datasets in there, all with snapshots visible):

cd /mnt/storage
du -h -A
Results = 43T (Seems WTF)
================================================
A posible Workaround
Testing with 'du -h -A -I .zfs' (-I mask switch will be used to exclude the '.zfs' dirs)

Report for whole directory "/mnt":

cd /mnt
du -h -A -I .zfs
Results = 1.4T (Shared nullfs mountpoit was traversed)

My real disk usage is:

du -h -A -I .zfs -I shared (Excluded 'shared' a mountpoint for media directory)
Results = 977G (Bingo)
================================================
For now Webmin is wrongly reporting on the Dashboard:

Local disk space 42.31 TB used / 22.33 TB free / 64.64 TB total
Seems 'du' command should use '-I .zfs' if working out with ZFS overall.
================================================
Unfortunately mounts with special file systems are traversed and counts as additional space such as NULLFS, making 'du' command more trivial, however the posible solution/workaround is to use -I switch with du to ignore .zfz and NULLFS mountpoints, I dunno how this will work uder Perl.

I think Webmin should calculate space as individual zpool basis gathering information from the available zfs commands sush as 'zpool list' for acurate results.

Regards

EDITED: 05-29-2018

[J.M. Rivera]

I found on CPAN the Filesys-ZFS-1.5 > Filesys::ZFS I think that perl module could be handy for working with ZFS overall Regards

[Jamie Cameron]

Yeah, maybe looking at mounted filesystems is a mistake - does zpool list show total disk space available and free in ZFS?

[J.M. Rivera]

Hi, yes, zpool list and zpool list -p gives per pool disk raw data usage statistics(raid parity usage included), however Oracle have some writings about this topic HERE.

Regards

[Jamie Cameron]

On the other hand, that would include disk space that hasn't been made available to the system yet, so wouldn't be accurate to include either..

[J.M. Rivera]

Right, maybe gathering/including information right from zpool list should suffice for each imported/available zfs pools on the system, exported and/or unformatted/unallocated drives should be ignored indeed.

Heres the sample output of zpool list on my FreeBSD testing system, zpool list already provides human readable information, while zpool list -p provides parsable and exact number values: