#460 Xigmanas access to dirs and files slows down with time; smbd needs to be restarted frequently

[Carlo Tognetti]

I've got a NAS created with Nas4free, regularly upgraded thought the versions to 12.3.0.4
It has been working fine (fast and reliably) for many of years.

Starting on September 2022 it has begun to become slower and slower as the time goes on.
Just restarting smbd it works again fast like a charm for a while, then begins to slow down again: after some hours it requires minutes to list a directory content via smb and also via local "ls". Even transferring small files via SMB becomes too slow.

Harware and OS details:

Version: 12.3.0.4 - Artisia
Platform: x64-embedded on Intel(R) Pentium(R) CPU G3220 @ 3.00GHz
System: LENOVO ThinkServer TS140
System BIOS: LENOVO Versione: FBKT82AUS 04/02/2014
CPU clock: 3000MHz
Memory utilization: 11.31GiB Alloc, 10.76GiB used, 569.78MiB available
Average Load: 0.27 0.38 0.34
Used pool: Z-pool-01 Total: 12TB | Used: 8.46TB | Free: 3.53TB | Status: DEGRADED

Boot from USB, no swap, 4 HDD WD Red 3TB (ada3 offline is a WD Res 4 TB), 3 network cards (2 in lagg0, 1 for a share on a separate network).

The content of the smb shares is about 5.000.000 files.

top:

last pid: 62171;  load averages:  0.30,  0.45,  0.39                                                                                                                                                                                    up 18+23:36:10  18:51:09
33 processes:  1 running, 32 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 24K Active, 126M Inact, 61M Laundry, 11G Wired, 64M Buf, 444M Free
ARC: 7718M Total, 1345M MFU, 5906M MRU, 8224K Anon, 66M Header, 393M Other
     6061M Compressed, 7774M Uncompressed, 1.28:1 Ratio

zpool status

pool: Z-pool-01
state: DEGRADED
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0 in 0 days 17:39:51 with 0 errors on Sat Feb  4 22:01:10 2023
config:

NAME                        STATE     READ WRITE CKSUM
Z-pool-01                   DEGRADED     0     0     0
raidz2-0                    DEGRADED     0     0     0
ada0                        ONLINE       0     0     0
ada1                        ONLINE       0     0     0
ada2                        ONLINE       0     0     0
replacing-3                 UNAVAIL      0     0     0
18392465009040593064        OFFLINE      0     0     0  was /dev/ada3/old
10328513456421092355        UNAVAIL      0     0     0  was /dev/ada3/old
16851209080113461305        UNAVAIL      0     0     0  was /dev/ada3

errors: No known data errors

The degaded status of the pool should not impact for 2 reasons:

1. ZFS shoul not suffer of that
2. restarting smbd the system works very fast for some hours

The smbd configuration is attached (from Diagnostic > Information > testparm) together with ZFS details.