linux-ntfs-dev — Development discussions and bug reports.

Re: [Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Anton A. <ai...@ca...>]

At 00:59 26/05/2001, Jeff Garzik wrote:
>Anton Altaparmakov wrote:
> > NTFS 1.1.15 - ChangeLog
> > =======================
> > - Support for more than 128kiB sized runlists (using vmalloc_32() instead
> > of kmalloc()).
>
>If you are running into kmalloc size limit, please consider some 
>alternative method of allocation.

Why? I was told that's what vmalloc is for. And anyway, Windows NT/2k 
operate on a 256kB sized memory allocation basis and NTFS has been designed 
with this in mind. Breaking away from that is a PITA. I was very happy to 
learn about vmalloc as it solved a lot of my headaches with regards to 
structures being >128kb.

btw. Maybe you, or someone else, can explain me whether vmalloc() will 
always work or whether vmalloc_32() is required? I am thinking of highmem 
pages which vmalloc could return. Would I have to make sure the pages were 
accessible by calling some function or is this done transparently?

>Can you map it into the page cache, as Al Viro has done in recent patches?

If someone would document how the page cache actually works from a kernel 
point of view, perhaps...<hint, hint> Sorry, but I haven't had the time to 
go through all the related source to understand it properly. I have an idea 
of how it works conceptually (well, kind of...) but not a deep enough 
understanding to start implementing anything that goes anywhere near the 
page cache.

I would love to move inode metadata to the page cache. That would simplify 
so much in the NTFS driver and would result in an incredible speed boost 
due to getting rid of all the silly copying of data back and forth inside 
the driver as we could just pass pointers around instead (to locked pages 
or other form of internal locking).

>Can you break your allocation into an array of pages, obtained via 
>get_free_page?

No thank you. I would rather have it contiguous or at least logically 
contiguous. I don't care if the pages are physically contiguous as long as 
I can see them as one piece... I mean, ok, I could break up run lists, as 
they are arrays of fixed size structs and the boundary cases would be 
predictable (but still would slow down the code with boundary checks) but 
this doesn't solve the generic problem later on as more things get 
implemented where we encounter variable width structures. It is much 
cleaner and easier to implement using a logically contiguous memory 
allocation IMHO. It's a shame Linux kernel doesn't support this kind of 
stuff (last time I read somewhere about this it said it doesn't anyway. I 
would be happy to be told wrong here).

>If runlists are on-disk structures, can you look at bh->b_data instead of 
>keeping them in memory?

They are not. They are derived from compressed on-disk structures (which 
are re-compressed when updating the inode). Runlists of such large size are 
only needed for A) huge files/directories, B) extremely fragmented 
files/directories, or C) a combination of A and B. (Remembering that 
metadata is stored as files, so the same applies for the metadata itself.)

Best regards,

Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Re: ANN: NTFS new release available (1.1.15)

[Jeff G. <jg...@ma...>]

Anton Altaparmakov wrote:
> NTFS 1.1.15 - ChangeLog
> =======================
> - Support for more than 128kiB sized runlists (using vmalloc_32() instead
> of kmalloc()).

If you are running into kmalloc size limit, please consider some
alternative method of allocation.
Can you map it into the page cache, as Al Viro has done in recent
patches?
Can you break your allocation into an array of pages, obtained via
get_free_page?
If runlists are on-disk structures, can you look at bh->b_data instead
of keeping them in memory?

-- 
Jeff Garzik      | Disbelief, that's why you fail.
Building 1024    |
MandrakeSoft     |

[Linux-NTFS-Dev] ANN: NTFS new release available (1.1.15)

[Anton A. <ai...@ca...>]

Announcing a new NTFS patch (1.1.15). It is generated against kernel 
2.4.4-ac16 but also applies cleanly to -ac17. Quite probably it will apply 
fine to any 2.4.4 kernel (at least 2.4.4-pre5 I think).

How to get the patch
====================

- Wait for one of the next -ac series kernel patches (patch is submitted to 
Alan Cox).
- For the impatient: patch is available (for a while only, for some value 
of while) for download as bzip2 or gzip compressed file from:
         http://www-stu.christs.cam.ac.uk/~aia21/ntfs/
(To apply patch, uncompress it, cd into your kernel directory and type:
         patch -p1 < 
/path/to/patch/ntfs-kernel-2.4.4-ac16-to-ntfs-1.1.15.diff.gz
Then you will need to recompile your kernel and reboot into it.)

NTFS 1.1.15 - ChangeLog
=======================

- New mount option show_sys_files=<bool> to show all system files as normal 
files.
- Support for files and in general any attributes up to the full 2TiB size 
supported by the NTFS filesystem. Note we only support up to 32-bits worth 
of inodes/clusters at this point.
- Support for more than 128kiB sized runlists (using vmalloc_32() instead 
of kmalloc()).
- Fixed races in allocation of clusters and mft records.
- Fixed major bugs in attribute handling / searching / collation.
- Fixed major bugs in compressing a run list into a mapping pairs array.
- Fixed major bugs in inode allocation. Especially file create and mkdir.
- Fixed memory leaks.
- Fixed major bug in inode layout assignment of sequence numbers.
- Lots of other bug fixes I can't think of right now...
- Fixed NULL bug found by the Stanford checker in ntfs_dupuni2map().
- Convert large stack variable to dynamically allocated one in 
ntfs_get_free_cluster_count() (found by Stanford checker).
- New versioning scheme as I was too confused by the date based one... 
(versions prior to this are internal only, never been released to public, 
so you haven't missed the numbers before 1.1.15).

Known bugs and (mis-)features
=============================

- Do not use the driver for writing as it corrupts the file system. If you 
do use it, get the Linux-NTFS tools and use the ntfsfix utility after 
dismounting a partition you wrote to.

- Use "ls -l" instead of just "ls", otherwise the last entry in the 
directory is not displayed for some directories. (?!?)

- Use the show_sys_files mount option which should make things work 
generally better. (It results in both the short and long file names being 
shown as well as the sytem files.)

- Special characters are not treated correctly. For example if the file 
name contains an apostrophe you will not be able to open it.

- Writing of extension records is not supported properly.

Please send bug reports/comments/feed back/abuse to the Linux-NTFS 
development list at sourceforge: lin...@li...

For daring people, write support now works ok for simple files and 
directories. For example it is relatively safe to create a directory inside 
a not too big directory, and create a few relatively small files in it. 
umount, run ntfsfix, reboot in Windows, chkdsk will run and (hopefully) not 
detect any problems at all!

Thank you for your attention.

Best regards,

         Anton


-- 
   "Nothing succeeds like success." - Alexandre Dumas
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sf.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] RFC: Searching for attributes (& CVS compiles again)

[Anton A. <ai...@cu...>]

As of today the CVS finally compiles again and as far as I can tell, it
works fine, too. At least my partitions survived the testing of ntfsfix.

There still is no attribute list support. But searching for attributes has
become a lot cleaner (IMHO) and uses a search context so you can easily
iterate over all/several attibutes. - Due to lack of attribute list
support spurious error messages are being generated when an attribute list
is encountered. - Not intending to fix the error messages as I want to get
attribute list support working, instead. (-8

Currently include/attrib_RE.h and libntfs/attrib_RE.c contain what the
driver does to search for attributes while taking care of existing
attribute lists. It's ugly and very complex but if you are interested have
a look. I want to do it differently while preserving the
essential collation rules (which is why I bothered RE-ing what the driver
does in the first place).

Perhaps to have one function for searching an attribute list, just like
find_attr() (find_attr() searches an mft record ignoring an attribute
list). The new function would work the same way, except it would search
the attribute list instead of the mft record, and if none is present, it
would call find_attr(). If an attribute list is present, it would be
searched, and if the found attribute is part of the base mft record,
find_attr() would be called. If the attribute is in an extension mft
record, the mft record will be read, and find_attr() will be called on it.

This sounds nice and easy until we consider how to implement the search
context.

Problems are for example: Assume we have called the new function once,
which resulted in an extensin mft record being read and an attribute being
returned.

First problem: what happens with the old (base) mft record stored in the
search context? It needs to be preserved somehow, as does the attribute
list attribute pointer to the found attribute, otherwise we don't know
where to continue the search the next time the function is called. Also,
on second call, the function needs to be able to tell that we stopped
inside an attribute list and continue from there.

Now the function is called a second time. Second problem: what do
we do with the old extension mft record? We don't want to leak memory so
we can't just drop it. On the other hand, we don't know whether the user
doesn't still hold references to the attribute within the mft record, so
we can't just free() it. It would seem we need a linked list of all loaded mft
records perhaps? Or just policy?, e.g. when calling the function you need
to have finished with the found attribute and hold no references to it any
more?

Eventually these problems will go away to a big part as we will just be
locking/unlocking the mft records which will be present in the page cache
(or at least so for the future ntfs driver), but we need a simple,
managable solution for user space now.

If anyone has any ideas/suggestions/comments on this I sure would like to
hear them - or rather as it would be, read them. (-;

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / WWW: http://sourceforge.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Update to the NTFS Documentation

[Anton A. <ai...@ca...>]

 >Thanks for your comments and especially for the aural link!
s/aural/URL/

Stupid spellchecker...

         Anton


-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sourceforge.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Update to the NTFS Documentation

[Anton A. <ai...@ca...>]

At 20:42 31/03/2001, Bo Branten wrote:
>The book Windows NT/2000 Native API Reference by Gary Nebbett contains one
>chapter that describes the physical format of NTFS. I compared that to
>the documentation at http://www.flatcap.org/ntfs/index.html and the
>following is corection/new information.

We know about the book. I, at least, own a copy and assimilated all the 
information from the book into the Sourceforge Linux-NTFS project (see 
Sourceforge CVS: linux-NTFS/include/attrib.h , mft.h , bootsect.h , 
ntfs_rec.h).

The documentation you refer to is based on an old set of documents and is 
hence tremendously out of date but it is being worked on to make it up to 
date. This will of course take some time...

>The homepage for his book is:
>http://www.newriders.com/books/title.cfm?isbn=1578701996
>where it also is possible to download the .h file that I quoted below.

Thanks for your comments and especially for the aural link! I didn't know 
about that one.

Best regards,

         Anton


-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sourceforge.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Update to the NTFS Documentation

[Bo B. <bo...@in...>]

Hello,

The book Windows NT/2000 Native API Reference by Gary Nebbett contains one
chapter that describes the physical format of NTFS. I compared that to
the documentation at http://www.flatcap.org/ntfs/index.html and the
following is corection/new information.

The homepage for his book is:
http://www.newriders.com/books/title.cfm?isbn=1578701996
where it also is possible to download the .h file that I quoted below.


	Bo Branten


> Attribute type 0xF0
    is PropertySet
    (intended to support Native Structured Storage (NSS), a
     feature that was removed from NTFS 3.0 during beta testing)



> Standard Attribute Header
> Non-Resident, No Name
> 0x22 2 Compression Engine (b)
    is 1 byte
> 0x24 4 0x0000 Unused (c)
    is 5 byte

> b) This field has a value of 0x04 when the attribute is compressed
    logarithm to the base two of the number of clusters in a
    compression unit



> Attribute - $STANDARD_INFORMATION (0x10)
> 0x08 8 Last modification time 
> 0x10 8 Last modification time for FILE record 
    the opposite order

typedef struct {
    ULONGLONG CreationTime; 
    ULONGLONG ChangeTime;
    ULONGLONG LastWriteTime; 
    ULONGLONG LastAccessTime; 
    ULONG FileAttributes; 
    ULONG AlignmentOrReservedOrUnknown[3];
    ULONG QuotaId;                        // NTFS 3.0 only
    ULONG SecurityId;                     // NTFS 3.0 only
    ULONGLONG QuotaCharge;                // NTFS 3.0 only
    USN Usn;                              // NTFS 3.0 only
} STANDARD_INFORMATION, *PSTANDARD_INFORMATION;



> Attribute - $FILE_NAME (0x30)
> 0x08 8 Last modification time 
> 0x10 8 Last modification time for FILE record 
    the opposite order

> 0x38 8 Flags, e.g. Directory, compressed, hidden 
    is 4 byte followed of 4 byte reserved



> Attribute - $OBJECT_ID (0x40)

typedef struct {
    GUID ObjectId;
    union {
        struct {
            GUID BirthVolumeId;
            GUID BirthObjectId;
            GUID DomainId;
        } ;
        UCHAR ExtendedInfo[48];
    };
} OBJECTID_ATTRIBUTE, *POBJECTID_ATTRIBUTE;



> Attribute - $INDEX_ROOT (0x90)
> 0x04 4 Always 00 00 00 01? 
    CollationRule to sort index entrys

> 0x08 4 Size of index buffers ? 
    yes

> 0x0C 4 Number of clusters per index buffer ? 
    yes

0x10 4 Always 00 00 00 10 ? 
    yes since INDEX_ROOT is followed by DIRECTORY_INDEX:

typedef struct {
    ATTRIBUTE_TYPE Type;
    ULONG CollationRule;
    ULONG BytesPerIndexBlock;
    ULONG ClustersPerIndexBlock;
    DIRECTORY_INDEX DirectoryIndex;
} INDEX_ROOT, *PINDEX_ROOT;

typedef struct {
    ULONG EntriesOffset;
    ULONG IndexBlockLength;
    ULONG AllocatedSize;
    ULONG Flags;         // 0x00 = Small directory, 0x01 = Large directory
} DIRECTORY_INDEX, *PDIRECTORY_INDEX;

DIRECTORY_INDEX is followed by DIRECTORY_ENTRY

typedef struct {
    ULONGLONG FileReferenceNumber;
    USHORT Length;
    USHORT AttributeLength;
    ULONG Flags;           // 0x01 = Has trailing VCN, 0x02 = Last entry
    // FILENAME_ATTRIBUTE Name;
    // ULONGLONG Vcn;      // VCN in IndexAllocation of earlier entries
} DIRECTORY_ENTRY, *PDIRECTORY_ENTRY;



> Attribute - $INDEX_ALLOCATION (0xA0)

typedef struct {
    NTFS_RECORD_HEADER Ntfs;
    ULONGLONG IndexBlockVcn;
    DIRECTORY_INDEX DirectoryIndex;
} INDEX_BLOCK_HEADER, *PINDEX_BLOCK_HEADER;

followed by DIRECTORY_INDEX and DIRECTORY_ENTRY se above



> Attribute - $REPARSE_POINT (0xC0)

typedef struct {
    ULONG ReparseTag;
    USHORT ReparseDataLength;
    USHORT Reserved;
    UCHAR ReparseData[1];
} REPARSE_POINT, *PREPARSE_POINT;



> Attribute - $EA_INFORMATION (0xD0)

typedef struct {
    ULONG EaLength;
    ULONG EaQueryLength;
} EA_INFORMATION, *PEA_INFORMATION;



> Attribute - $EA (0xE0)

typedef struct {
    ULONG NextEntryOffset;
    UCHAR Flags;
    UCHAR EaNameLength;
    USHORT EaValueLength;
    CHAR EaName[1];
    // UCHAR EaData[];
} EA_ATTRIBUTE, *PEA_ATTRIBUTE;



Attribute - $LOGGED_UTILITY_STREAM (0x100)

An attribute logged utility stream contains whateber data the creator of the
attribute choses but operations on the attribute is logged to the NTFS
logfile just like the NTFS metadata changes, it is used by the encrypting
file system EFS.



> NTFS - Files
> 11 $Extend A directory containing $ObjId, $Quota, $Reparse 

    and $UsnJrnl

> File - $Boot (7)

typedef struct {
    UCHAR Jump[3];
    UCHAR Format[8];
    USHORT BytesPerSector;
    UCHAR SectorsPerCluster;
    USHORT BootSectors;
    UCHAR Mbz1;
    USHORT Mbz2;
    USHORT Reserved1;
    UCHAR MediaType;
    USHORT Mbz3;
    USHORT SectorsPerTrack;
    USHORT NumberOfHeads;
    ULONG PartitionOffset;
    ULONG Reserved2[2];
    ULONGLONG TotalSectors;
    ULONGLONG MftStartLcn;
    ULONGLONG Mft2StartLcn;
    ULONG ClustersPerFileRecord;
    ULONG ClustersPerIndexBlock;
    ULONGLONG VolumeSerialNumber;
    UCHAR Code[0x1AE];
    USHORT BootSignature;
} BOOT_BLOCK, *PBOOT_BLOCK;

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Martin v. L. <lo...@in...>]

> From: Anton Altaparmakov <ai...@ca...>
> Cc: lin...@ti...,
>         lin...@li...
> 1 message + 1 copy = 2

Yes, cross-posting is a problem. An expert internet user has ways of
dealing with that, of course: e.g. a small procmail recipe can easily
filter out duplicates (based on the message id).

> As for your comment about experts: merging i offered means that
> all subscribres will be automatically included into the new list.
> and after this all obsolete list (better say frozen for now)
> would send warning/description:t "this list is for current ntfs driver/
> linux-NTFS-0.0.2/etc, developers/users, use new NTFS-Layout (or
> whatever you like more) for disk layout/records discussion".

I'm not sure that all subscribers to my list would appreciate being
auto-subscribed to some other list; that would be a violation of
netiquette, IMO.

As I see it in the moment, the only people who care to run NTFS
mailing lists are the Linux people. it might be a good thing if that
changed - if you are not willing to host another list, and nobody else
steps forward, then nothing will change.

Regards,
Martin

[Linux-NTFS-Dev] Re: [Linux-ntfs] Re: Announcing NTFS Documentation

[Martin v. L. <lo...@in...>]

> What I do on all my SourceForge projects is to have a web-site in a CVS
> module "website", and then have a cron job to update the "live" server
> from the CVS repository. Works quite well, and all registered developers
> can update the web when required.

I had tried to setup such a scheme on SF as well. It works, but I
would not call it "work well". I run the update job every six hours,
which is quite infrequent when you want to see your changes. Running
it more frequently would result in the pages getting touched more
frequently (even if they are unchanged), which is also undesirable.

I also attempted to initiate an update to the pages on every
commit. Did you attempt such a thing? I failed, essentially because
the web server is on a different machine than the CVS server.

Regards,
Martin

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Michael N. F. <mi...@id...>]

On Mon, 26 Mar 2001, Martin von Loewis wrote:

> > As for me it would be better to start new (to select active members)
> > not Linux specific mailing list with understandable name for
> > concept, e.g. NTFS-Layout to collect/exchange information. Possible
> > it will attract people who writes not for Linux (like me). I dont
> > know if it is very hard to start new mailing list.
> 
> If you are not scared off by the "linux" in its name, I'd recommend to
> use the existing lists. It is likely that you'll reach more experts
> here than you will with a fresh list (not that all experts will
> comment all the time, or even identify themselves).

I'm not afraid of penguins and tiny devils with forks and MSDOS/Windows
centaurs too but i see people in many other linux related mailing lists
who asks what is "mount" command and why they cannot see their lovely 
VFAT disk in KDE, or what does Oops in xxx.o mean. And this is purpose
of such mailing lists (at least name claims it). I just trying to hint
that if there is any hope that development of linux NTFS driver will be
started (again) all this will happen to all of these mailing lists too.
BTW. I have friends who work at Windows drivers and will never try to
subscribe here because they are not interested in discussed code. Let me
quote a little (copy-n-paste):
"About Linux-ntfs  
This list is about the NTFS driver for Linux. It is intended primarily as 
a forum for diskussing the driver. You can download the latest public
release from 
http://www.informatik.hu-berlin.de/~loewis/ntfs or 
ftp://ftp.informatik.hu-berlin.de/pub/local/linux/ntfs"

then: existing lists have own rules of managment. Nobody can guarantee
that inportant message will be in the one selected list. It means that
everybody is subscribed everywhere:

This morning example:

From: Anton Altaparmakov <ai...@ca...>
Cc: lin...@ti...,
        lin...@li...
1 message + 1 copy = 2
...
From: Richard Russon <fl...@fl...>
To: Michael "N." Filippov <mi...@id...>
Cc: NTFS for Linux <lin...@ti...>,
        lin...@li...
1 message + 2 copies = 3
...
From: Martin von Loewis <lo...@in...>
To: ai...@cu...
CC: mi...@id..., lin...@ti...,
        lin...@li...
2 messages + 4 copies (2 copies of each) = 6
...
From: "Edward S. Marshall" <es...@lo...>
To: Anton Altaparmakov <ai...@cu...>
Cc: "Michael N. Filippov" <mi...@id...>,
        NTFS for Linux <lin...@ti...>,
        lin...@li...
1 message + 2 copies = 3
...

Sum: 5 messages + 9 copies = (about) 64% useless copies

As for your comment about experts: merging i offered means that
all subscribres will be automatically included into the new list.
and after this all obsolete list (better say frozen for now)
would send warning/description:t "this list is for current ntfs driver/
linux-NTFS-0.0.2/etc, developers/users, use new NTFS-Layout (or
whatever you like more) for disk layout/records discussion".

Possible i'm wrong. But even now i have received emails from people who
wants to look at XP file records, about MS Diskeditor and so on, 
It means that current mailing lists structure at least doesnt fit for
information exchange. 
There is big issue about privacy too, something like fake nicks in this
new list instead of real names/emails, for you know what ... 
But i'm not a specialist in here and it would be better if somebody else
would continue/correct.

Sincerely,
Michael

[Linux-NTFS-Dev] Re: [Linux-ntfs] Re: Announcing NTFS Documentation

[Henrik N. <hn...@he...>]

Richard Russon wrote:

> > >  * Suggestions
> > Here is one: How about moving the docs onto Sourceforge?
> > We can place them under CVS control for easy development and
> > put them up, say in http://linux-ntfs.sourceforge.net/ntfs/
> 
> I had a trial play on SourceForge.  The documentation system
> doesn't seem to allow sub-directories.  Furthermore it eats
> any colour scheme you have.  I'll put them under CVS control
> when I've finished attrib.h

What I do on all my SourceForge projects is to have a web-site in a CVS
module "website", and then have a cron job to update the "live" server
from the CVS repository. Works quite well, and all registered developers
can update the web when required.

No, I do not use the "Document manager" tool, with the big drawback that
there is no "discussion forums" tied to each document.. planning on
integrating some other annotation tool into the websites when I get the
time...

--
Henrik Nordstrom

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Edward S. M. <es...@lo...>]

On Mon, Mar 26, 2001 at 03:21:16PM +0100, Anton Altaparmakov wrote:
> At the moment, there are only 13 email addresses registered on the
> Sourceforge list, so if you want to reach lots of people you are better
> off sending your message to @tiger address. I don't mind merging them but
> I am not sure everyone on @tiger is interested in the content of
> @sourceforge, i.e. development of linux-ntfs project (which at the moment
> is admittedly close to zero, but that might change in the future...).

Traffic on this list is quite low as well; there doesn't seem to be much need
for two lists, at least not for the time being. Perhaps multiple lists could
be created when there's enough development going on to warrant it?

-- 
Edward S. Marshall <es...@lo...>                http://www.nyx.net/~emarshal/
-------------------------------------------------------------------------------
[                  Felix qui potuit rerum cognoscere causas.                  ]

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Martin v. L. <lo...@in...>]

> As for me it would be better to start new (to select active members)
> not Linux specific mailing list with understandable name for
> concept, e.g. NTFS-Layout to collect/exchange information. Possible
> it will attract people who writes not for Linux (like me). I dont
> know if it is very hard to start new mailing list.

If you are not scared off by the "linux" in its name, I'd recommend to
use the existing lists. It is likely that you'll reach more experts
here than you will with a fresh list (not that all experts will
comment all the time, or even identify themselves).

Regards,
Martin

Re: [Linux-ntfs] Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Martin v. L. <lo...@in...>]

> At the moment, there are only 13 email addresses registered on the
> Sourceforge list, so if you want to reach lots of people you are better
> off sending your message to @tiger address. I don't mind merging them but
> I am not sure everyone on @tiger is interested in the content of
> @sourceforge, i.e. development of linux-ntfs project (which at the moment
> is admittedly close to zero, but that might change in the future...).
> 
> What do other people think?

I intended the linux-ntfs@tiger list always for broader discussion on
NTFS, and it appears that people are using it for that purpose as
well. I would not have any problems with postings that are only
concerned with the SF project; it is not that there are dozens of
messages every day.

Note that the list on tiger is a closed, mailman-managed list - only
subscribers can post, so there should be no spam. It also means that I
have to approve messages from non-subscribers, which I do if they are
NTFS-related.

Regards,
Martin

Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Michael N. F. <mi...@id...>]

On Mon, 26 Mar 2001, Anton Altaparmakov wrote:

> On Mon, 26 Mar 2001, Michael N. Filippov wrote:
> > 1. I have a lot of questions and several answers (i hope) and not sure
> > which mailing list is more populated. I see that almost every active
> > member is registered here (http://tiger.informatik.hu-berlin.de/cgi-bin/mailman/listinfo/linux-ntfs)
> > and on sourceforge (http://sourceforge.net/projects/linux-ntfs) and have
> > to send copy. Why not to merge them. All - go to sourceforge :) ! (for
> > example.). It would be fine i think.
> 
> At the moment, there are only 13 email addresses registered on the
> Sourceforge list, so if you want to reach lots of people you are better
> off sending your message to @tiger address. I don't mind merging them but
> I am not sure everyone on @tiger is interested in the content of
> @sourceforge, i.e. development of linux-ntfs project (which at the moment
> is admittedly close to zero, but that might change in the future...).
> 
> What do other people think?

As for me it would be better to start new (to select active
members) not Linux specific mailing list with understandable name for
concept, e.g. NTFS-Layout to collect/exchange information. Possible it
will attract people who writes not for Linux (like me). I dont know if it
is very hard to start new mailing list.

> 
> > 2. I installed Windows Whistler Server (XP) to look at
> > NTFS 3.1 (as written in its $Volume file). The first change i see is
> > that in file record record_header->update_sequence_header.offset is 0x30
> > instead of typical 0x2A. These 6 bytes for every file/directory on
> > disk (i copied one directory without system/hidden etc files) have such
> > layout: first 4 bytes - mft_index, 2 bytes - zero. It means that only 32
> > bits are reserved instead of 48. Any guesses ? 
> 
> When you say mft_index, do you mean that this is the index number of the
> mft record in which mft_index is stored? (quite a bizzarre thing to store
> as it is completely redundant, but it does make sense for recovery
> purposes). Could you just confirm I understood you correctly?

exactly. Even more info: in extended file records this number is mft_index
too (not index of the base file record). It means that it is not index of
some attribute (for example $SECURITY_DESCRIPTOR, as we could expect). It
is just 32 bit index of _this_ record. (about 32 bit, possible i'm wrong -
i dont have 2 Tb drive - see below)

> 
> If you have a kernel debugger why not check out how it is accessed?

I dont know this trick, can you explain me this new (for me of course:)
technology. All i have: my tiny program with built it hex editor.

> 
> I don't have any version of XP but my guess would be that it still is
> 48-bits for the index. Don't forget it is stored in little endian, so that
> the first byte is the least significant byte. So effectively, the first
> four bytes are the lower 32-bits of the mft record number. The next two
> bytes, which you note as zero, are the high 16-bits of the mft record
> number (zero for you as your volume doesn't have 4Gi of files on it) and
> the final two bytes are of course the sequence_number which is also
> present in the mft_record at offset 0x10. Could you confirm that the two
> sequence numbers match up? 

Once again about 32 bits, or they have to combine 6 bytes to use 4 bytes
in little endian and next 2 bytes as high part. Let me quote my prog :) :

---> file 1234
File Record Header {
  Record Header {
    Magic:	'FILE'
    Update sequence offset: 30, count: 3      <-  note#1: 30 instead of 2e
    Log Sequence Number:	489c08
  }
  Sequence Number:	6e45
  Reference Count:	1
  Attributes Offset:	38
  Flags:		1 in_use
  Bytes In Use:		158
  Bytes Allocated:	400
  Base File Reference:	MFT index: 0, sequence number: 0
  Next Attribute Instance:	4
}
$STANDARD_INFORMATION resident
$FILE_NAME resident
$DATA nonresident

---> show inode 1234
46 49 4c 45 30 00 03 00 08 9c 48 00 00 00 00 00 FILE0.....H.....
45 6e 01 00 38 00 01 00 58 01 00 00 00 04 00 00 En..8...X.......
00 00 00 00 00 00 00 00 04 00 00 00 34 12 00 00 ............4...
                           ^^ ^^ ^^ ^^ ^^ ^^ ^^ <- note#2

6 bytes: value = 0x00001234000000.
and first part 4 bytes = mft_index (0x1234). Possible last 2 bytes are
high part, possible not (some new flags for example). They are 0 for all
records. Or course plain 48 would be more consistent but MS has its own
way.

> If they do, and I understood you correctly
> above, we now have full specs on at least one of the new features in NTFS 
> 3.1. (-:
> 
> Best regards,
> 
> 	Anton
> -- 
> Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
> Linux NTFS maintainer / WWW: http://sourceforge.net/projects/linux-ntfs/
> ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/
> 
> 

[Linux-NTFS-Dev] [OT] Web Design

[Richard R. <nt...@fl...>]

As for the web design, I've tried to keep it fairly simple.
The look of the pages depends on you browser's ability to
read CSS (cascading style sheets).

The beauty of CSS is the ability to restyle lots of
pages in one central file.

I tend to use mozilla (0.8) when renders everything
perfectly.  I imagine that Netscape 6 (based on
mozilla, would be similar (any comments?)).

Opera renders the pages very well, too.

Internet Explorer cuts a few corners, but looks good.

Netscape 4.76, however looks dreadful.  It doesn't know
about CSS and it can't scale fonts.

Lynx does a reasonable job.  If any Lynx users find
pages that aren't clear, let me know.

The access keys are part of HTML4, though I'm not sure
when they were introduced.  They are programmable
accelerators.  You can tie a link to, say, Alt-R.
Mozilla supports them and IE almost supports them.

The pages seem to work best in a long narrow window.
Perfect for viewing next to your hex dump / code.

FlatCap (Rich)
nt...@fl...

[Linux-NTFS-Dev] Re: [Linux-ntfs] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Richard R. <fl...@fl...>]

On 26 Mar 2001 20:12:53 +0700, Michael N. Filippov wrote:

> not sure which mailing list is more populated.

I recently asked Anton the same question.  Tiger has about 550 people
and SourceForge about 10 people.

> Why not merge them?

Hmm...  Good question.  Martin?  Anton?

> 2. I installed Windows Whistler Server (XP) to look at NTFS 3.1
> (as written in its $Volume file). 

Interesting.

> The first change i see is
> that in file record record_header->update_sequence_header.offset is
0x30
> instead of typical 0x2A. These 6 bytes for every file/directory on
> disk (i copied one directory without system/hidden etc files) have
such
> layout: first 4 bytes - mft_index, 2 bytes - zero. It means that only
32
> bits are reserved instead of 48. Any guesses ? 

It sounds like a file reference.  The first 6 bytes are the MFT record
number
(effectively the inode number) and the last 2 bytes are a sequence
number.
NTFS uses 64 bit numbers, but only allows 2^48 files.

If you have more information, Michael, let us know.

Cheers,
    FlatCap (Rich).
    nt...@fl...

[Linux-NTFS-Dev] Re: Announcing NTFS Documentation

[Richard R. <web...@fl...>]

On 26 Mar 2001 11:35:47 +0100, Anton Altaparmakov wrote:

> Great job! A lot more readable, indeed. The hovering links are a touch
> confusing but probably better than having almost every word
underlined, 
> disturbing the reading flow.

Thanks.  If I'm going to be looking at these documents a lot, I don't
want every other word to be blue and underlined.  When ALL the links
are in place, you should be able to click on almost anything.

> > There's still quite a lot of work to do on the docs.
> > There's still a lot we don't understand about NTFS.
> But we understand a lot more compared to before!

Absolutely.  But most of the information was in people's heads
or buried in source code.

> I think if we manage to transfer all the info from
> linux-ntfs/include/attrib.h on Sourceforge into your new docs
[Note: Anton has taken every SDK, API in existance and distilled
them into attrib.h, well worth a read]

Yes.   I've got to to go through attrib.h very carefully.  That
will take quite a while.  I've also got to re-read my 'decode' files.
Hundreds of Kb of annotated hex dumps (dating back to mid '96).

> we would have an _almost_ complete set of NTFS specifications.
> I would imagine the most complete set in existence. o-:

Shh...  Don't tell anyone.

> >  * Suggestions
> Here is one: How about moving the docs onto Sourceforge?
> We can place them under CVS control for easy development and
> put them up, say in http://linux-ntfs.sourceforge.net/ntfs/

I had a trial play on SourceForge.  The documentation system
doesn't seem to allow sub-directories.  Furthermore it eats
any colour scheme you have.  I'll put them under CVS control
when I've finished attrib.h

Cheers,
    FlatCap (Rich)
    nt...@fl...

Re: [Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Anton A. <ai...@cu...>]

On Mon, 26 Mar 2001, Michael N. Filippov wrote:
> 1. I have a lot of questions and several answers (i hope) and not sure
> which mailing list is more populated. I see that almost every active
> member is registered here (http://tiger.informatik.hu-berlin.de/cgi-bin/mailman/listinfo/linux-ntfs)
> and on sourceforge (http://sourceforge.net/projects/linux-ntfs) and have
> to send copy. Why not to merge them. All - go to sourceforge :) ! (for
> example.). It would be fine i think.

At the moment, there are only 13 email addresses registered on the
Sourceforge list, so if you want to reach lots of people you are better
off sending your message to @tiger address. I don't mind merging them but
I am not sure everyone on @tiger is interested in the content of
@sourceforge, i.e. development of linux-ntfs project (which at the moment
is admittedly close to zero, but that might change in the future...).

What do other people think?

> 2. I installed Windows Whistler Server (XP) to look at
> NTFS 3.1 (as written in its $Volume file). The first change i see is
> that in file record record_header->update_sequence_header.offset is 0x30
> instead of typical 0x2A. These 6 bytes for every file/directory on
> disk (i copied one directory without system/hidden etc files) have such
> layout: first 4 bytes - mft_index, 2 bytes - zero. It means that only 32
> bits are reserved instead of 48. Any guesses ? 

When you say mft_index, do you mean that this is the index number of the
mft record in which mft_index is stored? (quite a bizzarre thing to store
as it is completely redundant, but it does make sense for recovery
purposes). Could you just confirm I understood you correctly?

If you have a kernel debugger why not check out how it is accessed?

I don't have any version of XP but my guess would be that it still is
48-bits for the index. Don't forget it is stored in little endian, so that
the first byte is the least significant byte. So effectively, the first
four bytes are the lower 32-bits of the mft record number. The next two
bytes, which you note as zero, are the high 16-bits of the mft record
number (zero for you as your volume doesn't have 4Gi of files on it) and
the final two bytes are of course the sequence_number which is also
present in the mft_record at offset 0x10. Could you confirm that the two
sequence numbers match up? If they do, and I understood you correctly
above, we now have full specs on at least one of the new features in NTFS 
3.1. (-:

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / WWW: http://sourceforge.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] 1. which mailing list is the right one ? 2. NTFS 3.1 File Record

[Michael N. F. <mi...@id...>]

1. I have a lot of questions and several answers (i hope) and not sure
which mailing list is more populated. I see that almost every active
member is registered here (http://tiger.informatik.hu-berlin.de/cgi-bin/mailman/listinfo/linux-ntfs)
and on sourceforge (http://sourceforge.net/projects/linux-ntfs) and have
to send copy. Why not to merge them. All - go to sourceforge :) ! (for
example.). It would be fine i think.

Then less important. 

2. I installed Windows Whistler Server (XP) to look at
NTFS 3.1 (as written in its $Volume file). The first change i see is
that in file record record_header->update_sequence_header.offset is 0x30
instead of typical 0x2A. These 6 bytes for every file/directory on
disk (i copied one directory without system/hidden etc files) have such
layout: first 4 bytes - mft_index, 2 bytes - zero. It means that only 32
bits are reserved instead of 48. Any guesses ? 

Sincerely,
Michael

Re: [Linux-NTFS-Dev] Announcing NTFS Documentation

[Anton A. <ai...@ca...>]

At 07:32 26/03/2001, Richard Russon wrote:
>http://www.flatcap.org/ntfs        The Docs Online
>http://www.flatcap.org/download    Take a copy home
>
>Please have a browse.
>
>I've rewritten the NTFS Documentation.
>Why?  Well, though the original docs were technically
>accurate, they were a bit disorganised and hard to read.

Great job! A lot more readable, indeed. The hovering links are a touch 
confusing but probably better than having almost every word underlined, 
disturbing the reading flow.

>There's still quite a lot of work to do on the docs.
>There's still a lot we don't understand about NTFS.
>But now the documentation is easier to read, it's
>easier to see where the holes are.

But we don't understand a lot less compared to before! I think if we manage 
to transfer all the info from linux-ntfs/include/attrib.h on Sourceforge 
into your new docs we would have an _almost_ complete set of NTFS 
specifications. I would imagine the most complete set in existence. o-:

>This is where I need your help (and when you all look
>away :-)  If you have any:
>
>  * Suggestions

Here is one: How about moving the docs onto Sourceforge? We can place them 
under CVS control for easy development and put them up, say in:
         http://linux-ntfs.sourceforge.net/ntfs/

We just need to add some kind of short main page in 
linux-ntfs.sourceforge.net/ to point to the docs but that's a quick job.

What do you think?

Best regards,

         Anton


-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://sourceforge.net/projects/linux-ntfs/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Announcing NTFS Documentation

[Richard R. <nt...@fl...>]

Hi Everyone,

http://www.flatcap.org/ntfs        The Docs Online
http://www.flatcap.org/download    Take a copy home

Please have a browse.

I've rewritten the NTFS Documentation.
Why?  Well, though the original docs were technically
accurate, they were a bit disorganised and hard to read.

Not only have I tidied up the old docs, but I've added
new information, too.  

It's all very technical, but it's easy to navigate
around and there are lots of easy-to-read tables.

There's still a quite a lot of work to do on the docs.
There's still a lot we don't understand about NTFS.
But now the documentation is easier to read, it's
easier to see where the holes are.

This is where I need your help (and when you all look
away :-)  If you have any:

 * Questions
 * Suggestions
 * Additions
 * Corrections

then please email the list (and me).

Also, if English is not your first language and you have
difficulty understanding anything, please let me know and
I'll happily rewrite anything.

Cheers,
    FlatCap (Richard Russon)
    web...@fl...

[Linux-NTFS-Dev] New WinXP file system breaks disk utilities

[Richard R. <nt...@fl...>]

There was an article in The Register today.

http://www.theregister.co.uk/content/4/17783.html

Apparently NTFS has changed once again.

NT   NTFS 1.2
2K   NTFS 3.0
XP   NTFS 5.1

Where did these guys learn to count?

FlatCap
nt...@fl...

[Linux-NTFS-Dev] fwd: project status

[Richard R. <nt...@fl...>]

> Anon wrote:
> What's going on with this project - looks kinda dormant...

This project has just started and needs a bit of kick-start.
It sounds like a lot of work to rewrite everything, but a lot of the
original code was taken directly from test programs we wrote.

> Are the rpms and source here the same as the current linux ntfs
> drivers or is this different?

No, at the moment there are two streams.  There's the original kernel
source which is static, except for bug fixes.  Then there's the NTFS
project on SourceForge.

> I am not trying to be impertinent but why do you want to make new
> drivers?  I have looked at the old drivers - they appear to me to be
> overly complicated and unclear. I have tried to follow the function
> calls and they get so deep that I lose track of the original purpose
> of the call.

Ha!  You've answered your own question.  The original driver WAS
complicated.  Martin did a great job just starting the project off.
There was no documentation for NTFS when we started.  It was a complete
nightmare.  Anyway, the code evolved and never got reviewed.

Writing the code to /read/ NTFS was relatively simple, but as the write
code was started, the complexity of the code made things difficult.
It's still a much needed project, it's just hard work.

> ps: I hope you didn't write it....

[grins] linux/fs/ntfs/fs.c (line 6): Copyright (C) 1996 Richard Russon

> I don't understand why the writer didn't use structure pointers to map
> the on disk structure.  Instead he used all these 'get' functions to
> get each value.  Like in the standard data attribute for filesize he
> uses something similar to
>       filesize= getu32( x );
> When it would have been much clearer to define
> struct stand_data {...
>                     unsigned long filesize;
> }
> And then the code could have just used d.filesize  or d->filesize
> which is alot cleaner and clearer.

OK, this is a portability issue.  The GETU32 macro (and its friends)
convert between disk (little-endian) format and memory (cpu) format.
On the i386 architecture, the conversion is a null op.  On an Alpha
however, the byte order has to be reversed.  The other reason for all
those macros is that field lengths vary between compilers.  Rather than
using short, long, etc we use ntfs_u16, ntfs_u32, etc.

> I applaud the guy who wrote it... it does work but, I think the code's
> complex.  It works but its difficult to follow, modify and maintain.

Anton's idea is to rewrite all the user tools from the ground up, with
all our current knowledge, and THEN try to rewrite the driver.

Cheers,
    FlatCap (Rich)
    nt...@fl...

[Linux-NTFS-Dev] NTFS bugs raised on Linux-Kernel mailing list

[Richard R. <nt...@fl...>]

In case you missed it some guys in Stanford have been busy with some gcc
plugins.  They checked the entire kernel source tree for improper use of
pointers, interrupts and large variables declared on the stack.

The results that apply to us are:

> ... appear to allocate stack variables >= 1024 bytes

> 2.4.1/fs/ntfs/super.c:335:ntfs_get_free_cluster_count:
> ERROR:VAR: suspicious sized variable: 'bits' = 2048 bytes

> This checker warns when the pointer returned by a "plausibly" failing
> routine is not checked before being used.

> ---------------------------------------------------------
> [BUG] ntfs_find_attr. the return value of ntfs_find_attr is checked at the
> first callsite, but not at the second callsite.
> 2.4.1/fs/ntfs/dir.c:854:ntfs_getdir_unsorted: 
> ERROR:NULL:831:854: Using unknown ptr "attr" illegally!
> set by 'ntfs_find_attr':831
> 
> Start --->
>         attr=ntfs_find_attr(ino,vol->at_bitmap,I30);
>         if(!attr){
>                 /* directory does not have index allocation */
>                 *p_high=0xFFFFFFFF;
>                 *p_low=0;
> 
>         ... DELETED 15 lines ...
> 
>                 return EIO;
>         }
>         attr=ntfs_find_attr(ino,vol->at_index_allocation,I30);
>         while(1){
> Error --->
>                 if(*p_high*vol->clustersize > attr->size){
>                         /* no more index records */
> ---------------------------------------------------------
> [BUG] ntfs_find_attr
> 2.4.1/fs/ntfs/inode.c:130:ntfs_extend_mft:
> ERROR:NULL:129:130: Using unknown ptr "bmp" illegally!
> set by 'ntfs_find_attr':129
> 
> Start --->
>         bmp=ntfs_find_attr(vol->mft_ino,vol->at_bitmap,0);
> Error --->
>         if(bmp->size*8<rcount){ /* less bits than MFT records */
>                 ntfs_u8 buf[1];
> ---------------------------------------------------------
> [BUG] ntfs_find_attr can return NULL
> 2.4.1/fs/ntfs/inode.c:104:ntfs_extend_mft:
> ERROR:NULL:102:104: Using unknown ptr "mdata" illegally!
> set by 'ntfs_find_attr':102
> 
> Start --->
>         mdata=ntfs_find_attr(vol->mft_ino,vol->at_data,0);
>         /* first check whether there is uninitialized space */
> Error --->
>         if(mdata->allocated<mdata->size+vol->mft_recordsize){
>                 size=ntfs_get_free_cluster_count(vol->bitmap)*vol->clustersize;
> ---------------------------------------------------------
> [BUG] ntfs_find_attr can return NULL
> 2.4.1/fs/ntfs/inode.c:1077:ntfs_new_inode:
> ERROR:NULL:1076:1077: Using unknown ptr "data" illegally!
> set by 'ntfs_find_attr':1076
> 
> Start --->
>         data=ntfs_find_attr(vol->mft_ino,vol->at_data,0);
> Error --->
>         length=data->size/vol->mft_recordsize;
> 
> ---------------------------------------------------------

flatcap.