Commit [6a2d06] Maximize Restore History

Implement asynchronous readdir

Accessing directories with many (several thousand) files over sshfs is
slow, because most SFTP server implementations only send a fixed amount
of entries per READDIR command (e.g. OpenSSH SFTP: 100 entries). This
patch implements sending several READDIR commands in parallel, in order
to speed up directory listing in these cases.

An option (sync_readdir) is also added so that users can easily switch
on the old behaviour.

The performance improvement is astonishing. Accessing a directory with
30k files in from a remote server that has a RTT of 15ms via OpenSSH
SFTP:

Synchronous readdir:
$ ./sshfs -o sync_readdir host:/tmp /mnt/temp
$ time "ls" -1 /mnt/temp/test | wc -l
30000
"ls" -1 /mnt/temp/test 0.07s user 0.01s system 1% cpu 6.928 total

Asynchronous readdir:
$ ./sshfs host:/tmp /mnt/temp
$ time "ls" -1 /mnt/temp/test | wc -l
30000
"ls" -1 /mnt/temp/test 0.07s user 0.01s system 12% cpu 0.605 total

Accessing a directory with 100k files shows even more dramatic
improvement:

Synchronous readdir:
$ ./sshfs -o sync_readdir host:/tmp /mnt/temp
$ time "ls" -1 /mnt/temp/test2 | wc -l
100000
"ls" -1 /mnt/temp/test2 0.67s user 1.22s system 0% cpu 3:31.56 total

Asynchronous readdir:
$ ./sshfs host:/tmp /mnt/temp
$ time "ls" -1 /mnt/temp/test2 | wc -l
100000
"ls" -1 /mnt/temp/test2 0.20s user 0.03s system 14% cpu 1.631 total

This can easily be reproduced by creating a directory on a server and
touching a lot of files in it:
$ mkdir /tmp/test
$ cd /tmp/test
$ for i in $(seq 1 30000); do touch $i; done

Signed-off-by: Alexander Neumann <alexander@bumpern.de>

Alexander Neumann Alexander Neumann 2013-12-22

Miklos Szeredi Miklos Szeredi 2014-01-08

changed ChangeLog
changed sshfs.1
changed sshfs.c
ChangeLog Diff Switch to side-by-side view
Loading...
sshfs.1 Diff Switch to side-by-side view
Loading...
sshfs.c Diff Switch to side-by-side view
Loading...