Re: [ntfs-3g-devel] Release candidate for ntfs-3g with file ownership and permissions

[Szabolcs S. <sz...@nt...>]

Hi,

Good start for 2008! :-) This year seems to be quite exciting on the NTFS 
field!

Sorry I didn't reply earlier. I've been on vacation most of the time in the 
last one month.

On Mon, 14 Jan 2008, Jean-Pierre ANDRE wrote:

> This is to announce a candidate release of an extension to ntfs-3g for file
> ownership and protections (ntfs-3g-1.1120SC.1, based on the standard version
> ntfs-3g-1.1120).
> 
> Details and dowload are available on
> http://pagesperso-orange.fr/b.andre/security.html
> 
> The only change since the latest beta-test version (ntfs-3g-1.1120SB.5, Dec
> 14th) is the use of an LRU cache to get the inode numbers 

Cool, I'll check it out. 

Btw, could you please move the /NTFS-3G directory to a hidden /.NTFS-3G? 
I think we agreed on this sometime ago when we discussed that OS X also 
needs to store some files in /.NTFS-3G.

> and the deactivation of cacheing of file attributes by FUSE. This 
> decreases significantly the CPU usage by ntfs-3g (the impact on CPU usage 
> by fuse is unknown) and fixes a hard link problem caused by the 
> unawareness of hard linked files by FUSE.

Hmmm, if I remember correctly then disabling FUSE attribute caching doesn't 
solve all the problems (I'll check for the known issues). To avoid those 
problems Miklos suggested the usage of the low-level FUSE interface and 
this is indeed planned for a while for much better performance and less 
code.
 
> For instance the following sequence (wrong in standard ntfs-3g when run in a
> shell script) now displays the correct result.
> 
> rm -f ntfs/link*
> echo linked file > ntfs/link0
> ln ntfs/link0 ntfs/link1
> ln ntfs/link1 ntfs/link2
> ls -l ntfs/link*
> echo more >> ntfs/link1
> # the sizes displayed are not the same
> ls -l ntfs/link*

Great. Some backup software use hard link tricks and I have seen some 
problem reports but didn't get more detail.

> I have benchmarked this version on recompiling ntfs-3g from the tarball
> (gzip, tar, configure, make), doing about 4000 file or directory openings
> (among which 900 were file creations). The relative CPU usage by successive 
> versions of ntfs-3g are as follows :
> 
> standard ntfs-3g no security           100% (reference)
> ntfs-3g plus security no caching       130%
> ntfs-3g plus security cached           108%
> same plus inode number cacheing         45%
> same without fuse attribute cacheing    47%
> 
> An equivalent ratio is met on the number of physical reads requested by
> ntfs-3g (242661, instead of 534743 for the reference version), but the real
> impact on data access times is probably much less because of the cacheing
> done by the kernel. Moreover the cacheing has no effect on writings, so there
> should be no impact on disk throughput when copying large files.

Thank you for doing this!

I have also made some benchmarks. User mapping wasn't used, so I could 
compare the speed improvements itself better with the standard version.

Bonnie++ result:
  File creation: 10% improvement
  File stat:     50% decrease     <-- disabled fuse cache, quite probably
  File removal:  20% improvement
  
Concurrent write (tiotest):  20% improvement

Unpacking kernel (23,000+ files): 50% improvement
Running 'find' on kernel:         20% improvement
Removing kernel tree:             40% improvement

Quite good results! Moving to the low-level interface should also improve
at least further 20-50% in general on the performance.

Btw, I think we don't need path_normalize(). FUSE should do it already. 
Afair, paths must always start with '/' and the trailing slashes seems to 
be always removed as well.

Regards,
	    Szaka