#132 CIFS share: unlink()/getdent() are missing files

9.1.0.1
open
nobody
1
2014-10-09
2013-10-01
Vall
No

While running bonnie++ (v.1.96, but evidences seem to point that the problem is not particular to that version or even to bonnie++) on a RHEL6 client against a CIFS share on a NAS4Free server, the command aborts with the following error:

Bonnie: drastic I/O error (rmdir): Directory not empty

Re-running the command under strace and examining strace's file shows us that:

1) Of the many files that were created during bonnie++'s previous stage ("Create files in sequential order"), many are reported as ENOENT when the program tries to unlink() them, for example:

30898 write(2, "Create files in sequential order...", 35) = 35
[...]
30898 open("000000000atXXeTev7O", O_WRONLY|O_CREAT|O_EXCL, 0600) = 3
30898 close(3)                          = 0
[...]
30898 write(2, "done.\nStat files in sequential order...", 39) = 39
[...]
30898 stat("000000000atXXeTev7O", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
[...]
30898 write(2, "done.\nDelete files in sequential order...", 41) = 41
[...]
30898 unlink("000000000atXXeTev7O")     = -1 ENOENT (No such file or directory)

2) This makes rmdir() correctly return an error when bonnie++ tries removing the directory:

30898 rmdir("./Bonnie.30898")           = -1 ENOENT (No such file or directory)

3) Which them produces the above error message:

30898 write(2, "Bonnie: drastic I/O error (rmdir): Directory not empty\n", 55) = 55

4) Confirming the above, an "ls" shows the file is there:

ls -l ./Bonnie.30898/000000000atXXeTev7O
    -rw-------. 1 1000 root 0 Sep 30 18:51 ./Bonnie.30898/000000000atXXeTev7O

Further details:

a) Command used to mount the CIFS share on the NAS4Free client:

mount -t cifs -o username=nas4freeuser //nas4freeserver/nas4freecifsshare /mnt2

b) Command used to run the bonnie++ benchmark:

strace -f -s 128 -o /tmp/bonnie++_nas4free_cifs.txt bonnie++ -u 0 -d /mnt2

c) Complete output of the above bonnie++ command:

Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...Bonnie: drastic I/O error (rmdir): Directory not empty
Cleaning up test directory after error.

d) share section on nas4freeserver:/var/etc/smb.conf:
[nas4freecifsshare]
comment = CIFS filesystem
path = /pool01/uzfs
writeable = yes
printable = no
veto files = /.snap/.sujournal/
hide dot files = yes
guest ok = yes
inherit permissions = yes
vfs objects = shadow_copy2
shadow:format = auto-%Y%m%d-%H%M%S
shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:localtime = yes

e) /pool01/uzfs is a ZFS filesystem created under a ZFS pool.

f) Running the same bonnie++ command on the same client against the same server but on a NFS share produces no errors.

g) NAS4Free server OS and Samba versions:

uname -a
    nas4freeserver 9.1-RELEASE-p5 FreeBSD 9.1-RELEASE-p5 #0 r254466M: Sat Aug 17 22:54:54 CEST 2013     root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64  amd64

smbd --version
    Version 3.6.18

h) Linux RHEL6 server OS and Samba client versions:

uname -a

    Linux linuxclient 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 17:06:37 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

mount.cifs --version
    mount.cifs version: 4.8.1

smbclient --version
    Version 3.6.9-151.el6

i) Looked for this bug in NAS4Free, in FreeBSD and in Samba bug tracking systems and hfound nothing similar, so I'm creating a bug report in NAS4Free first.

If any more details are needed, please do not hesitate to contact me.

Discussion

  • Vall
    Vall
    2014-03-19

    For the record, the problem continues precisely as above with the newest version of NAS4Free: just installed version 9.2.0.1.943, and the problem repeats in exactly the same way.

     
    Last edit: Vall 2014-03-19