Re: [fuse-devel] Baffled issue to list large number files

[Miklos S. <mi...@sz...>]

On Wed, 2 Jun 2010, hce wrote:
> On Tue, Jun 1, 2010 at 10:24 PM, Miklos Szeredi <mi...@sz...> wrote:
> > On Tue, 1 Jun 2010, hce wrote:
> >> I've looked at readdir for both fusexmp_fh and fusexmp, it seems the
> >> relevant difference is the offset of the fuse_fill_dir_t filler where
> >> fusexmp sets it to 0, and fusexmp_fh sets it to telldir(dp). Since I
> >> am running a Fuse storage application, I am not using DIR *dp but
> >> providing file names, file attributes and 0 offset to fuse_fill_dir_t
> >> filler, not clear if the 0 offset would be a problem or what kind of
> >> value should be used for offset.
> >
> > You can make up any kind of offset, it has to have the property of
> > being able to restart the readdir from this offset.
> >
> > Using sequence numbers is the simplest method (i.e. 1 for the first
> > entry, 2 for the second, etc...)  The hard part is making this work
> > even if entries are removed or added to the directory.
> 
> I tried to use sequence numbers, it did not work, 0 is fine. Anyway,
> the offset doesn't seem the cause for this issue. You mentioned
> "That's the fuse filesystem's fault", could you please elaborate more.

See the documentation for the readdir() callback:

	/** Read directory
	 *
	 * This supersedes the old getdir() interface.  New applications
	 * should use this.
	 *
	 * The filesystem may choose between two modes of operation:
	 *
	 * 1) The readdir implementation ignores the offset parameter, and
	 * passes zero to the filler function's offset.  The filler
	 * function will not return '1' (unless an error happens), so the
	 * whole directory is read in a single readdir operation.  This
	 * works just like the old getdir() method.
	 *
	 * 2) The readdir implementation keeps track of the offsets of the
	 * directory entries.  It uses the offset parameter and always
	 * passes non-zero offset to the filler function.  When the buffer
	 * is full (or an error happens) the filler function will return
	 * '1'.
	 *

Currently your filesystem is doing method (1).  In this case the
directory is read once and buffered while the directory is open.  This
usually works fine, but NFS export will make it pretty unusable.  The
reason is: for each readdir NFS will effectively do:

 opendir()
 seekdir()
 readdir()
 closedir()

It means that for each readdir() the whole directory with all the
hundreds of thousands of entries needs to be re-buffered for just a
few entries which will be supplied by the readdir.  Then the whole
buffer is thrown away.

Fuse could try some clever caching of the buffered directory contents,
but that's not trivial.

> Any suggestions if I could try anyting to resolve this problem (patch
> and recompile the fuse library ....)?
> 

Method (2) should work.  It's more difficult to get right than method
(1), but it should still be far easier than trying to add caching to
libfuse.

Thanks,
Miklos