[Pfspd-users@sf.net] New file alloc method on local NTFS file system

[Bram R. <bra...@nx...>]

Hi all,

Just checked in into the development repository: a new file alloc method.
I also simplified the SetFilePointer operations.

Finally, I intend to create a new cpfspd release in a few days from now.
On my todo list:
- allow speed to pre-allocate the file (check for write speed advantage)

Regards, Bram.


>From the Readme file:

- Internal simplification of cpfspd_fio.c: routine fio_fp_set_filepointer()
  is substituted by direct calls of SetFilePointerEx().
- Fast file allocation on local windows NTFS file systems as described
  a new section "File allocation on harddisk" in this readme file.
  This works only for users with administrator rights. Further, on cygwin
  at least version 1.7.x is required. On old cygwin versions or accounts
  without administrator rights, cpfspd silently falls back to the old
  behavior. As a side effect, this release requires at least OS version
  Windows Server 2003 or Windows XP - older windows versions are not
  supported anymore.


File allocation on harddisk
---------------------------

When executing p_write_header(), not only the header is written
to disk, but also all frames (as indicated in the header) are
allocated. This is done by opening the file, seeking to the desired
file size position and setting the end-of-file mark. This is convenient,
since problems with file size (e.g. due to quota limitations)
appear at program start, not after hours of computations...

On unix systems, sparse files are supported by default: only data that
is actually written is physically allocated on disk. Hence, this
"file allocation" executes very fast since only the end-of-file mark
is set, but no actual data is written (apart from the header itself).

Windows systems with a local NTFS file system, however, behave entirely
different. By default, 'holes' in a file are automatically filled with
zero data. Hence, the standard strategy of file allocation results in
the complete file being filled with zeros. This may take a considerable
amount of time. We experimented with sparse files in NTFS, but that
leads to considerable file fragmentation (a 10G file was normally
stored in a few dozen of fragments; after setting the file sparse
this increased to 3000 a 4000 fragments). This is caused by the NTFS
allocation policy to locate the smallest free area for an allocation
request. In a sparse file, each extension of the file is a new
allocation request. An alternative solution was implemented, using
a special windows API function call SetFileValidData(). This just
informs the file system that all data up to the end can be considered
written (hence, skipping the need for zero-filling). This may involve
security issues, and therefore the function requires an elevated
priveledge that is only granted for accounts with administrator rights.
The security risk is that this function may expose data that already
resides on the disk (originating from deleted files). This risk can be
alleviated by assuring that the application actually writes all frames
announced in the header. Finally, we found that the function fails
in a cygwin version 1.5.25-15 environment; this is resolved in version
1.7.1.

In summary, for fast file allocation on NTFS file systems:
- execute with administrator rights
- when using a cygwin exec, install cygwin version 1.7.x
- write all frames (and all components) announced in the header

In case the special priveledge is not granted, the cpfspd library
silently falls back to the original behavior of zero-filling the
file.


Tot ziens / Kind regards, Bram
--
Bram Riemens
Senior Principal, Central R&D / Research
NXP Semiconductors

High Tech Campus 32 (floor 1, office 138), 5656 AE Eindhoven, The Netherlands
Tel: +31 40 27 25910; Fax: +31 40 27 29648
Email: bra...@nx...

The information contained in this message is confidential and may be legally privileged. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, or reproduction is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

Unless otherwise recorded in a written agreement, all sales transactions by NXP Semiconductors are subject to our general terms and conditions of commercial sale. These are published at www.nxp.com/profile/terms/index.html<http://www.nxp.com/profile/terms/index.html>