From: Bram R. <bra...@nx...> - 2010-01-25 12:56:20
|
Hi all, Just checked in into the development repository: a new file alloc method. I also simplified the SetFilePointer operations. Finally, I intend to create a new cpfspd release in a few days from now. On my todo list: - allow speed to pre-allocate the file (check for write speed advantage) Regards, Bram. >From the Readme file: - Internal simplification of cpfspd_fio.c: routine fio_fp_set_filepointer() is substituted by direct calls of SetFilePointerEx(). - Fast file allocation on local windows NTFS file systems as described a new section "File allocation on harddisk" in this readme file. This works only for users with administrator rights. Further, on cygwin at least version 1.7.x is required. On old cygwin versions or accounts without administrator rights, cpfspd silently falls back to the old behavior. As a side effect, this release requires at least OS version Windows Server 2003 or Windows XP - older windows versions are not supported anymore. File allocation on harddisk --------------------------- When executing p_write_header(), not only the header is written to disk, but also all frames (as indicated in the header) are allocated. This is done by opening the file, seeking to the desired file size position and setting the end-of-file mark. This is convenient, since problems with file size (e.g. due to quota limitations) appear at program start, not after hours of computations... On unix systems, sparse files are supported by default: only data that is actually written is physically allocated on disk. Hence, this "file allocation" executes very fast since only the end-of-file mark is set, but no actual data is written (apart from the header itself). Windows systems with a local NTFS file system, however, behave entirely different. By default, 'holes' in a file are automatically filled with zero data. Hence, the standard strategy of file allocation results in the complete file being filled with zeros. This may take a considerable amount of time. We experimented with sparse files in NTFS, but that leads to considerable file fragmentation (a 10G file was normally stored in a few dozen of fragments; after setting the file sparse this increased to 3000 a 4000 fragments). This is caused by the NTFS allocation policy to locate the smallest free area for an allocation request. In a sparse file, each extension of the file is a new allocation request. An alternative solution was implemented, using a special windows API function call SetFileValidData(). This just informs the file system that all data up to the end can be considered written (hence, skipping the need for zero-filling). This may involve security issues, and therefore the function requires an elevated priveledge that is only granted for accounts with administrator rights. The security risk is that this function may expose data that already resides on the disk (originating from deleted files). This risk can be alleviated by assuring that the application actually writes all frames announced in the header. Finally, we found that the function fails in a cygwin version 1.5.25-15 environment; this is resolved in version 1.7.1. In summary, for fast file allocation on NTFS file systems: - execute with administrator rights - when using a cygwin exec, install cygwin version 1.7.x - write all frames (and all components) announced in the header In case the special priveledge is not granted, the cpfspd library silently falls back to the original behavior of zero-filling the file. Tot ziens / Kind regards, Bram -- Bram Riemens Senior Principal, Central R&D / Research NXP Semiconductors High Tech Campus 32 (floor 1, office 138), 5656 AE Eindhoven, The Netherlands Tel: +31 40 27 25910; Fax: +31 40 27 29648 Email: bra...@nx... The information contained in this message is confidential and may be legally privileged. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, or reproduction is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message. Unless otherwise recorded in a written agreement, all sales transactions by NXP Semiconductors are subject to our general terms and conditions of commercial sale. These are published at www.nxp.com/profile/terms/index.html<http://www.nxp.com/profile/terms/index.html> |