linux-ntfs-dev — Development discussions and bug reports.

Re: [Linux-NTFS-Dev] NTFSResize (1.9.0) won't shrink as much as expected

[Anton A. <ai...@ca...>]

On Thu, 2004-05-06 at 11:07, Szakacsits Szabolcs wrote:
> On Wed, 5 May 2004, Dave Pacheco wrote:
> 
> > I apologize if this is a common question, but I could not quite find the
> > answer in the documentation.
> 
> No, it's not common and I'm sorry but I can't tell what's the problem if
> you don't send the 'ntfsresize --info partition' output.
> 
> The known limitations are documented in the ntfsresize manual page:
> 
>    There are some very rarely met limitations at present: filesystems hav-
>    ing  bad sectors, highly fragmented Master File Table (MFT), relocation
>    of the first MFT extent and resizing in the middle of some metadata in
>    some  cases aren't supported yet. These cases are detected and resizing
>    is refused, restricted to a safe size or the closest safe size is dis-
>    played.
> 
> Note, XP can defragment the MFT so in theory you can't have the above
> highly fragmented MFT problem.
>  
> > I am using ntfsresize through qtparted[1] using the latest
> > SystemRescueCD (www.sysreccd.org), which claims to have ntfsprogs 1.9.0.
> 
> ntfsresize from command line prints the version.
> 
> > When trying to resize, it was first reported that I could only free up
> > about 1-2 MB. But clearly, there should be at least several GB of free
> > space available. If I read the documentation correctly, this version of
> > ntfsresize should be able to shrink the filesystem, even without
> > defragmenting first. However, just to check, I tried defragmenting the
> > drive with the Windows XP defragger. Ntfsresize was then able to free up
> > 90MB. This is still not nearly enough for my purposes (installing a
> > Debian system).
> 
> It sounds you don't have the latest systemrescuecd ...
> 
> > Is this a known issue? Is there any known workaround? Or am I doing
> > something completely wrong...? Any help would be appreciated. Thanks in
> > advance.
> 
> Pretty lot of people uses it day by day without any problems. Yours would
> be the first such complain since release of version 1.9.

It is not.  Someone else reported the same on the IRC #ntfs channel and
I just told him to ignore the --info output and just resize to a smaller
size and it worked fine.  I meant to email you about it but I was too
busy at the time and then forgot...  )-:

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ &
http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] NTFSResize (1.9.0) won't shrink as much as expected

[Szakacsits S. <sz...@si...>]

On Wed, 5 May 2004, Dave Pacheco wrote:

> I apologize if this is a common question, but I could not quite find the
> answer in the documentation.

No, it's not common and I'm sorry but I can't tell what's the problem if
you don't send the 'ntfsresize --info partition' output.

The known limitations are documented in the ntfsresize manual page:

   There are some very rarely met limitations at present: filesystems hav-
   ing  bad sectors, highly fragmented Master File Table (MFT), relocation
   of the first MFT extent and resizing in the middle of some metadata in
   some  cases aren't supported yet. These cases are detected and resizing
   is refused, restricted to a safe size or the closest safe size is dis-
   played.

Note, XP can defragment the MFT so in theory you can't have the above
highly fragmented MFT problem.
 
> I am using ntfsresize through qtparted[1] using the latest
> SystemRescueCD (www.sysreccd.org), which claims to have ntfsprogs 1.9.0.

ntfsresize from command line prints the version.

> When trying to resize, it was first reported that I could only free up
> about 1-2 MB. But clearly, there should be at least several GB of free
> space available. If I read the documentation correctly, this version of
> ntfsresize should be able to shrink the filesystem, even without
> defragmenting first. However, just to check, I tried defragmenting the
> drive with the Windows XP defragger. Ntfsresize was then able to free up
> 90MB. This is still not nearly enough for my purposes (installing a
> Debian system).

It sounds you don't have the latest systemrescuecd ...

> Is this a known issue? Is there any known workaround? Or am I doing
> something completely wrong...? Any help would be appreciated. Thanks in
> advance.

Pretty lot of people uses it day by day without any problems. Yours would
be the first such complain since release of version 1.9.

Cheers,
	Szaka

Re: [Linux-NTFS-Dev] NTFSResize (1.9.0) won't shrink as much as expected

[Anton A. <ai...@ca...>]

Hi,

On Thu, 2004-05-06 at 00:55, Dave Pacheco wrote:
> I apologize if this is a common question, but I could not quite find the
> answer in the documentation.
> 
> I am using ntfsresize through qtparted[1] using the latest
> SystemRescueCD (www.sysreccd.org), which claims to have ntfsprogs 1.9.0.
> I'm trying to use it on a drive that has only 1 partition, which is
> NTFS, and approximately 37GB in size. Approx. 17GB are being used.
> 
> When trying to resize, it was first reported that I could only free up
> about 1-2 MB. But clearly, there should be at least several GB of free
> space available. If I read the documentation correctly, this version of
> ntfsresize should be able to shrink the filesystem, even without
> defragmenting first. However, just to check, I tried defragmenting the
> drive with the Windows XP defragger. Ntfsresize was then able to free up
> 90MB. This is still not nearly enough for my purposes (installing a
> Debian system).
> 
> Is this a known issue? Is there any known workaround? Or am I doing
> something completely wrong...? Any help would be appreciated. Thanks in
> advance.

Yes, I am not sure whether it is a bug or a feature (never got round to
asking Szaka who wrote ntfsresize).  Basically you can ignore what
ntfsresize reports and just try to shrink to a lower size.  If you are
using ~17GB you should be able to resize down to ~18GB at least I would
think.  Just try it with the --no-action switch and if it succeeds do it
again without the --no-action.  Basically the --info option lies to you
so you shouldn't rely on its output.  (It doesn't take into
consideration the fact that ntfsresize can move data around.)

Hope this helps.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ &
http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] NTFSResize (1.9.0) won't shrink as much as expected

[Dave P. <Dav...@br...>]

Hi there,

I apologize if this is a common question, but I could not quite find the
answer in the documentation.

I am using ntfsresize through qtparted[1] using the latest
SystemRescueCD (www.sysreccd.org), which claims to have ntfsprogs 1.9.0.
I'm trying to use it on a drive that has only 1 partition, which is
NTFS, and approximately 37GB in size. Approx. 17GB are being used.

When trying to resize, it was first reported that I could only free up
about 1-2 MB. But clearly, there should be at least several GB of free
space available. If I read the documentation correctly, this version of
ntfsresize should be able to shrink the filesystem, even without
defragmenting first. However, just to check, I tried defragmenting the
drive with the Windows XP defragger. Ntfsresize was then able to free up
90MB. This is still not nearly enough for my purposes (installing a
Debian system).

Is this a known issue? Is there any known workaround? Or am I doing
something completely wrong...? Any help would be appreciated. Thanks in
advance.

Regards,
Dave Pacheco

[1]: although I've been using ntfsresize through qtparted, I did verify
what it told me by running "ntfsresize --info --no-action" myself, and
got the same results.

Re: [Linux-NTFS-Dev] Re: [BUG] 2.6.5 ntfs

[Szakacsits S. <sz...@si...>]

On Wed, 5 May 2004, Anton Altaparmakov wrote:
> On Wed, 2004-05-05 at 13:32, Marcin Gibu³a wrote:
> > > If you run "chkdsk /f" from windows on this partition, does it detect
> > > any errors?
> > It doesn't.
> Ok.

Note, chkdsk /f fixes things sometimes totally silently so usually it's
worth to make sure it indeed wasn't a fixed corruption (without spending
significantly more time for investigation) by quickly reproducing the
problem afterwards.

	Szaka

[Linux-NTFS-Dev] Re: [BUG] 2.6.5 ntfs

[Anton A. <ai...@ca...>]

On Wed, 2004-05-05 at 13:32, Marcin Gibu=B3a wrote:
> > not be zero any more.  Have you ever run memtest on your machine?
>=20
> I'll try to do this.
>=20
> > If you run "chkdsk /f" from windows on this partition, does it detect
> > any errors?
>=20
> It doesn't.

Ok.

> > dd if=3D~/mymftdump of=3D~/inode0x78a bs=3D1024 count=3D1 skip=3D1930
> >
> > And then email me the file ~/inode0x78a (this is only 1kiB in size).
>=20
> Here it goes.

Thanks.  Analyzing this it is a highly fragmented file so it the $DATA
attribute is spread over three more inodes.  Could you email them to me,
too please?  They are 0x790 (=3D 1936), 0x791 (=3D 1937), and 0x792 (1938).

You can get them like this (assuming you are using Bash as your shell):

for i in 1936 1937 1938; do
	dd if=3D~/mymftdump of=3D~/inode$i bs=3D1024 count=3D1 skip=3D$i
done

Once I have those I can uncompress the mapping pairs arrays in the data
extents and ask you to capture the compressed data for me.  It will be a
simple script simillar to the above...

Thanks a lot for your help.

Best regards,

	Anton
--=20
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ &
http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: [BUG] 2.6.5 ntfs

[Anton A. <ai...@ca...>]

On Wed, 2004-05-05 at 13:47, Szakacsits Szabolcs wrote:
> On Wed, 5 May 2004, Anton Altaparmakov wrote:
> 
> > > NTFS-fs error: ntfs_decompress(): Failed. Returning -EOVERFLOW.
> 
> I'm aware at least ntfsprogs mapping pairs decompression doesn't check
> upper boundary and this can cause problems on corrupted NTFS. This might
> or might not have anything to do with the above.

Would you care to point out what code in particular is unbounded?  All
the code is checking for overflows as far as I am aware and will give
you an io error on any corrupt ntfs...

> > > NTFS-fs error (device hde4): ntfs_read_compressed_block(): ntfs_decompress() 
> > > failed in inode 0x78a with error code 75. Skipping this compression block.
> > > 
> > > But no oops ... 
> > > If you want I can run whatever is necessary.
> > 
> > If you run "chkdsk /f" from windows on this partition, does it detect
> > any errors?
> 
> It would be nice to get the NTFS metadata first (please see below).  
> However I also suspect, it's again an NTFS corruption that the Windows
> driver tolerates/handles better ...

Too late and it is not.  chkdsk detected no errors.

> > Assuming chkdsk doesn't detect and fix any errors, this would definitely
> > be worth investigating.  I don't think it has anything to do with the
> > oops but I would very much like a copy of this inode because it might
> > mean our decompression code has a bug in it and I want to check this
> > out.  To create a copy, I will assume you have the latest ntfsprogs
> > installed, then use ntfscat to dump your $MFT like this:
> > 
> > ntfscat -i 0 /dev/hde4 > ~/mymftdump
> 
> What's wrong with the below instead?
> 
> 	ntfsclone --metadata --output ntfsmeta.img /dev/hde4
> 	tar -cjSf ntfsmeta.img.tar.bz2 ntfsmeta.img
> 
> It has everything needed, zeros all the unused space for best compression,
> wipes private resident user data and it's even mountable. The compressed
> NTFS metadata is usually max 1-5 MB (of course please send it off-list).

It is still too big for people with modems and gives the same (ok, a bit
smaller) as the ntfscat method...  Otherwise nothing wrong with it of
course.  I now have the inode in question btw.  Note the ntfsclone still
doesn't give you the compressed data stream of the inode you want so it
is no more useful than the approach I described...

> If the ntfsclone consistency check wouldn't pass I have a patch that
> disables it (if time allows I'm just working on the refactoring of the
> relevant ntfsprogs utils for this purpose and having an ntfsck).
> 
> >   [Error: Formatting error: Non-hexadecimal character in QP encoding]
> 
> Apparently Sourceforge still doesn't like quoted-printable 
> Content-Transfer-Encoding ...

)-:  Not much I can do about it.  AFAICS Evolution cannot send email in
any other way...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ &
http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: [BUG] 2.6.5 ntfs

[Szakacsits S. <sz...@si...>]

On Wed, 5 May 2004, Anton Altaparmakov wrote:

> > NTFS-fs error: ntfs_decompress(): Failed. Returning -EOVERFLOW.

I'm aware at least ntfsprogs mapping pairs decompression doesn't check
upper boundary and this can cause problems on corrupted NTFS. This might
or might not have anything to do with the above.

> > NTFS-fs error (device hde4): ntfs_read_compressed_block(): ntfs_decompress() 
> > failed in inode 0x78a with error code 75. Skipping this compression block.
> > 
> > But no oops ... 
> > If you want I can run whatever is necessary.
> 
> If you run "chkdsk /f" from windows on this partition, does it detect
> any errors?

It would be nice to get the NTFS metadata first (please see below).  
However I also suspect, it's again an NTFS corruption that the Windows
driver tolerates/handles better ...

> Assuming chkdsk doesn't detect and fix any errors, this would definitely
> be worth investigating.  I don't think it has anything to do with the
> oops but I would very much like a copy of this inode because it might
> mean our decompression code has a bug in it and I want to check this
> out.  To create a copy, I will assume you have the latest ntfsprogs
> installed, then use ntfscat to dump your $MFT like this:
> 
> ntfscat -i 0 /dev/hde4 > ~/mymftdump

What's wrong with the below instead?

	ntfsclone --metadata --output ntfsmeta.img /dev/hde4
	tar -cjSf ntfsmeta.img.tar.bz2 ntfsmeta.img

It has everything needed, zeros all the unused space for best compression,
wipes private resident user data and it's even mountable. The compressed
NTFS metadata is usually max 1-5 MB (of course please send it off-list).

If the ntfsclone consistency check wouldn't pass I have a patch that
disables it (if time allows I'm just working on the refactoring of the
relevant ntfsprogs utils for this purpose and having an ntfsck).

>   [Error: Formatting error: Non-hexadecimal character in QP encoding]

Apparently Sourceforge still doesn't like quoted-printable 
Content-Transfer-Encoding ...

	Szaka

[Linux-NTFS-Dev] Re: [BUG] 2.6.5 ntfs

[Anton A. <ai...@ca...>]

On Tue, 2004-05-04 at 18:57, Marcin Gibu=B3a wrote:
> > Could you do the following inside the configured kernel source director=
y
> > corresponding to the compiled kernel that produced this oops:
> >
> > make fs/ntfs/attrib.s
>=20
> The kernel was compiled with gcc version 3.4.0 20040414 (prerelease), and=
 I=20
> have gcc 3.4 now, so I'm not sure if the results are the same... so, I=20
> attached both new attrib.s and dissassembled attrib.o from previous=20
> compilation (with this older gcc).

Thanks.  Using the attrib.o.asm you sent me, I can see where the oops
happened.  It is on line 566 in fs/ntfs/attrib.c:

for (dend =3D di; drl[dend].length; dend++)
	;

The oops happened when drl[dend].length is accessed and from the
register dump in the oops I can see that this came after a dend++ was
executed, i.e. not on the first access to drl[dend].length.

Also, ecx is e0b12010 which is is 0x10 bytes into a page boundary
(assuming your system uses 4kib pages which is likely to be the case).

Further, from eax and edx it would imply that the last drl[dend].length
was equal to 0xff807362ff746957 which is a non-sense value.

The only way this could happen is if drl was not terminated (i.e.
drl[last element].length was not set to zero) and hence the for loop
just kept going accessing random memory contents until it hit the end of
the page and overflowed into the next one which caused the oops.

The run list code was very heavily tested at the time it was written and
it hasn't been touched since, so I doubt very much that this was caused
in the run list code of ntfs.  Doubly so as you say that you cannot
reproduce the bug so the driver would have needed to manage to do two
different things with the same data at two different times which seems
extremely unlikely to me...

If I am correct and there is no bug in the run list code of ntfs, the
only possible explanation is that either other kernel code caused memory
corruption (i.e. a bug in some other kernel code trampled over ntfs
allocated memory) or what is much more likely is that you have flakey
memory and a flipped bit caused the length of the last element of drl to
not be zero any more.  Have you ever run memtest on your machine?

> > Also could you tell me which Windows version the ntfs partition was
> > created with and which windows version last accessed the files?
>=20
> It was created with Windows 2000 or Windows XP, I really don't remember w=
hich=20
> was that, is there some utility to check this?
>=20
> Windows XP last accessed this partition.
>=20
> > Finally is this bug reproducible when you access a particular file or i=
s
> > it a once off event?  If a particular file, then would you mind running
> > a utility that I can email you (source or binary or both whichever you
> > prefer) so I can capture the metadata for the inode of the file?
>=20
> I was trying to reproduce this bug with cat * > /dev/null but the only th=
ing=20
> I've triggered was this warning:
>=20
> NTFS-fs error: ntfs_decompress(): Failed. Returning -EOVERFLOW.
> NTFS-fs error (device hde4): ntfs_read_compressed_block(): ntfs_decompres=
s()=20
> failed in inode 0x78a with error code 75. Skipping this compression block=
.
>=20
> But no oops ...=20
> If you want I can run whatever is necessary.

If you run "chkdsk /f" from windows on this partition, does it detect
any errors?

Assuming chkdsk doesn't detect and fix any errors, this would definitely
be worth investigating.  I don't think it has anything to do with the
oops but I would very much like a copy of this inode because it might
mean our decompression code has a bug in it and I want to check this
out.  To create a copy, I will assume you have the latest ntfsprogs
installed, then use ntfscat to dump your $MFT like this:

ntfscat -i 0 /dev/hde4 > ~/mymftdump

This will create a rather large file.  You can compress it with bzip -9
and if this is small enough, please email it to me (my email will accept
up to 10Mib attachments) or make it available for me to download on the
internet.  If this is not workable for you then proceed as follows to
copy the specific inode 0x78a (=3D 1930 in decimal) out of the file:

dd if=3D~/mymftdump of=3D~/inode0x78a bs=3D1024 count=3D1 skip=3D1930

And then email me the file ~/inode0x78a (this is only 1kiB in size).

Once this is done I will email you with more instructions on how to get
the rest of the data I need (i.e. the compressed data of the file which
is causing the overflow).

I am afraid I am too busy at work at the moment to write a utility to
get the data automatically...  Your help is very much appreciated.

Best regards,

	Anton
--=20
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ &
http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] NTFS 2.1.6b for Linux 2.4.26

[Pawel K. <pk...@be...>]

Hi,

There's updated NTFS patch for Linux 2.4.26 kernel. It contains no code
changes just documentation updates:
 - kernel help for CONFIG_NTFS_RW describes what this options enables
right now
 - CONFIG_NTFS_RW is not marked as dangerous (because it's not)

See the sourceforge for downloads:
http://sourceforge.net/project/showfiles.php?group_id=13956&package_id=21892&release_id=235452

Anton,
Last time I did't send you a bit with removing 'DANGEROUS' note. Please
take the change from fs/Config.in to the bitkeeper repository.

pkot
-- 
mailto:pk...@be...
http://www.gnokii.org/

Re: [Linux-NTFS-Dev] Re: Ntfs patch for 2.4.26

[Pawel K. <pk...@be...>]

On Fri, 30 Apr 2004, Szakacsits Szabolcs wrote:

> On Tue, 27 Apr 2004, [ISO-8859-1] François Dupoux wrote:
>
> Sorry for the slow answer, I had a computer free vacation.
>
> > I would like to prepare a 2.4.26 kernel for SystemRescueCd-0.2.13. Do
> > you think it's possible to have an NTFS patch against 2.4.26 ?
>
> I don't know the status of the backported driver for 2.4.26. Pawel, Anton?

I hope this weekend - I didn't try 2.4.26 yet at all. I asked Anton about
2.1.7 driver changes, but they are 2.6.x kernel specific, so I'll just
make 2.1.6b version (with Configure.help updated).

pkot
-- 
mailto:pk...@be...
http://www.gnokii.org/

[Linux-NTFS-Dev] Re: Ntfs patch for 2.4.26

[Szakacsits S. <sz...@si...>]

Hello,

On Tue, 27 Apr 2004, [ISO-8859-1] François Dupoux wrote:

Sorry for the slow answer, I had a computer free vacation.

> I would like to prepare a 2.4.26 kernel for SystemRescueCd-0.2.13. Do 
> you think it's possible to have an NTFS patch against 2.4.26 ?

I don't know the status of the backported driver for 2.4.26. Pawel, Anton?

	Szaka

[Linux-NTFS-Dev] Re: NTFS null dereference x2

[Anton A. <ai...@ca...>]

On Sat, 17 Apr 2004, Szakacsits Szabolcs wrote:
> Dave Jones <da...@re...> wrote:
> > if vol is NULL, everything falls apart..
>
> AFAIS, neither vol nor vol->sb can be NULL below. The !vol check, that
> fooled you or an automatic checker, is bogus and probably it slipped
> through the user space library, thanks.
>
> Please note, by the patch you would introduce a real bug when you
> dereference the now uninitialized sb to assign a value to block_size.

Thanks.  While at present vol nor vol->sb can be NULL, I have changed the
code in my tree to do the assignments after the checks for completeness
sake.  I of course also moved the assignment that the patch ignored as
well, otherwise as Szaka said, it would all go horribly wrong...

Best regards,

	Anton

> > --- linux-2.6.5/fs/ntfs/attrib.c~     2004-04-16 22:45:53.000000000 +0100
> > +++ linux-2.6.5/fs/ntfs/attrib.c      2004-04-16 22:46:47.000000000 +0100
> > @@ -1235,16 +1235,19 @@
> >       u8 *al_end = al + initialized_size;
> >       run_list_element *rl;
> >       struct buffer_head *bh;
> > -     struct super_block *sb = vol->sb;
> > +     struct super_block *sb;
> >       unsigned long block_size = sb->s_blocksize;
> >       unsigned long block, max_block;
> >       int err = 0;
> > -     unsigned char block_size_bits = sb->s_blocksize_bits;
> > +     unsigned char block_size_bits;
> >
> >       ntfs_debug("Entering.");
> >       if (!vol || !run_list || !al || size <= 0 || initialized_size < 0 ||
> >                       initialized_size > size)
> >               return -EINVAL;
> > +     sb = vol->sb;
> > +     block_size_bits = sb->s_blocksize_bits;
> > +
> >       if (!initialized_size) {
> >               memset(al, 0, size);
> >               return 0;

-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Cluster used from MFT

[Anton A. <ai...@ca...>]

Hi,

On Wed, 21 Apr 2004 fa...@pa... wrote:
> I know that you are busy but in order to finish my project I need one little
> hint. What I am gonna finish is to expand the utility ntfsfix that check for
> the consistency of the disk. I have creat a code that store the current
> $Bitmap in an array and now I am creating my own bitmap exploring the disk.
>
> For the first task I have used and changed the code you have develop in
> order to check if one cluster is in use or not ( utils_cluster_in_use ). So
> I have stored the $Bitmap in an array.
>
> In order to build my own Bitmap I found in the code of ntfscluster.c the
> function dump_file where for each node give you the number and the lenght of
> the clusters used from the "non_resident" attribute. I loop from the first
> MFT to the last (vol->nr_mft_record) and for each cluster used I marked the
> corresponded bit to one.
>
> Now the problem that as newbie I cannot resolve is that I don't know in
> which cluster is stored each MFT record. So when I access to an MFT I cannot
> mark the bit of the cluster where that MFT reside because I don't know.

That is very easy.  The $MFT (i.e. the file with mft record number 0) has
a $DATA attribute (non-resident).  This attribute contains all the MFT
records themselves.  So the mft record for $MFT itself is at offset zero
in this attribute.  Then the mft record 1 ($MFTMirr) is at offset 1024 in
the attribute.  The n-th mft record is at offset n * 1024 (== n *
vol->mft_record_size).

> I was wondering that this number could be the number of MFT multiplied for
> the size of the MFT divided per the cluster lenght.
>
> I catch this information reading this code in mft.c :
>
> int ntfs_mft_records_read(const ntfs_volume *vol, const MFT_REF mref,
> 		const s64 count, MFT_RECORD *b)
> {
> 	s64 br;
> 	VCN m;
>
> 	Dprintf("%s(): Entering for inode 0x%llx.\n", __FUNCTION__, MREF(mref));
> 	if (!vol || !vol->mft_na || !b || count < 0) {
> 		errno = EINVAL;
> 		return -1;
> 	}
> 	m = MREF(mref);
> 	if (m + count > vol->nr_mft_records) {
> 		errno = ESPIPE;
> 		return -1;
> 	}
> 	br = ntfs_attr_mst_pread(vol->mft_na, m << vol->mft_record_size_bits,
> 			count, vol->mft_record_size, b);
> ........

With what I explained above you should now understand this code.  It reads
from the $MFT/$DATA attribute at the offset m * vol->mft_record_size to
get the m-th mft record.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Cluster used from MFT

[Szakacsits S. <sz...@si...>]

Hello,

On Wed, 21 Apr 2004 fa...@pa... wrote:

> I know that you are busy but in order to finish my project I need one little
> hint. What I am gonna finish is to expand the utility ntfsfix that check for
> the consistency of the disk. 

But the purpose of ntfsfix is different: it marks NTFS for consistency
check and resets the journal file.

> I have creat a code that store the current $Bitmap in an array and now
> I am creating my own bitmap exploring the disk.

Sounds like what ntfsresize (and ntfsclone) does as the first step, for
two years.

I just recently started to convert the relevant parts of the code to be a
function (something like check_allocation_bitmap()). It's in the BK
repository and so far only in ntfsresize.c. What you need seems to be is
basically these parts from the ntfsresize code:

	ntfs_fsck_t fsck;
	memset(&fsck, 0, sizeof(fsck));
	setup_lcn_bitmap(&fsck.lcn_bitmap);
	build_allocation_bitmap(&fsck);
	compare_bitmaps(&fsck.lcn_bitmap);

This works since the start but to be reusable it still needs some days of
work.

I'm going to a vacation tomorrow and perhaps during that you could 
finnish the conversation, clean up or get the idea how to continue 
your implementation.

Cheers,
	Szaka

[Linux-NTFS-Dev] Cluster used from MFT

[<fa...@pa...>]

Hi Anton and all list,
I know that you are busy but in order to finish my project I need one little
hint. What I am gonna finish is to expand the utility ntfsfix that check for
the consistency of the disk. I have creat a code that store the current
$Bitmap in an array and now I am creating my own bitmap exploring the disk.

For the first task I have used and changed the code you have develop in
order to check if one cluster is in use or not ( utils_cluster_in_use ). So
I have stored the $Bitmap in an array.

In order to build my own Bitmap I found in the code of ntfscluster.c the
function dump_file where for each node give you the number and the lenght of
the clusters used from the "non_resident" attribute. I loop from the first
MFT to the last (vol->nr_mft_record) and for each cluster used I marked the
corresponded bit to one.

Now the problem that as newbie I cannot resolve is that I don't know in
which cluster is stored each MFT record. So when I access to an MFT I cannot
mark the bit of the cluster where that MFT reside because I don't know.

I was wondering that this number could be the number of MFT multiplied for
the size of the MFT divided per the cluster lenght.

I catch this information reading this code in mft.c :

int ntfs_mft_records_read(const ntfs_volume *vol, const MFT_REF mref,
		const s64 count, MFT_RECORD *b)
{
	s64 br;
	VCN m;

	Dprintf("%s(): Entering for inode 0x%llx.\n", __FUNCTION__, MREF(mref));
	if (!vol || !vol->mft_na || !b || count < 0) {
		errno = EINVAL;
		return -1;
	}
	m = MREF(mref);
	if (m + count > vol->nr_mft_records) {
		errno = ESPIPE;
		return -1;
	}
	br = ntfs_attr_mst_pread(vol->mft_na, m << vol->mft_record_size_bits,
			count, vol->mft_record_size, b);
........

I know that my project is not very big, but for me was a challenge to
understand the code and to figure-out how it works.
If you have any concerns or warnings about my concepts understing be free to
express them.
I hope to finish and send you my code in a couple of weeks, thank you to all
of you for your support.

Fabiano

Re: [Linux-NTFS-Dev] [PATCH] typo

[Anton A. <ai...@ca...>]

Hi,

On Mon, 19 Apr 2004, Aleksey Gorelov wrote:
>   I've been looking through ntfs implementation and noticed a 'typo'
> among sanity checks in mst.c. Here is small patch to fix it.

Ah, that is not a typo.  If you read the function comment it explains
this.  This is the relevant part of the comment:

NOTE: We consider the absence / invalidity of an update sequence array to
mean that the structure is not protected at all and hence doesn't need to
be fixed up. Thus, we return success and not failure in this case. This is
in contrast to pre_write_mst_fixup(), see below.

And hence the return 0; is not a typo it is intended.

Thanks for looking at the code.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] [PATCH] typo

[Aleksey G. <Ale...@Ph...>]

Hi,

  I've been looking through ntfs implementation and noticed a 'typo'
among sanity checks in mst.c. Here is small patch to fix it.

Aleks.

[Linux-NTFS-Dev] Re: NTFS null dereference x2

[Szakacsits S. <sz...@si...>]

Dave Jones <da...@re...> wrote:

> if vol is NULL, everything falls apart..

AFAIS, neither vol nor vol->sb can be NULL below. The !vol check, that
fooled you or an automatic checker, is bogus and probably it slipped
through the user space library, thanks.

Please note, by the patch you would introduce a real bug when you
dereference the now uninitialized sb to assign a value to block_size.

	Szaka

> --- linux-2.6.5/fs/ntfs/attrib.c~     2004-04-16 22:45:53.000000000 +0100
> +++ linux-2.6.5/fs/ntfs/attrib.c      2004-04-16 22:46:47.000000000 +0100
> @@ -1235,16 +1235,19 @@
>       u8 *al_end = al + initialized_size;
>       run_list_element *rl;
>       struct buffer_head *bh;
> -     struct super_block *sb = vol->sb;
> +     struct super_block *sb;
>       unsigned long block_size = sb->s_blocksize;
>       unsigned long block, max_block;
>       int err = 0;
> -     unsigned char block_size_bits = sb->s_blocksize_bits;
> +     unsigned char block_size_bits;
>
>       ntfs_debug("Entering.");
>       if (!vol || !run_list || !al || size <= 0 || initialized_size < 0 ||
>                       initialized_size > size)
>               return -EINVAL;
> +     sb = vol->sb;
> +     block_size_bits = sb->s_blocksize_bits;
> +
>       if (!initialized_size) {
>               memset(al, 0, size);
>               return 0;

Re: Accessing odd last partition sector (was: [Linux-NTFS-Dev] mkntfs dirty volume marking)

[Szakacsits S. <sz...@si...>]

On Sat, 10 Apr 2004 vi...@pa... wrote:
> > > > 
> > > > > > TODO.ntfsprogs conatins the following TODO item under mkntfs:
> > > > > >  - We don't know what the real last sector is, thus we mark the volume
> > > > > > dirty and the subsequent chkdsk (which will happen on reboot into
> > > > > > Windows automatically) recreates the backup boot sector if the Linux
> > > > > > kernel lied to us about the number of sectors.
> > > > 
> > > > The ioctl BLKGETSIZE64 will tell you the size (in bytes) of a
> > > > block device.

Unless kernel 2.4.1[567] used (they return the size in sectors) or no
ioctl conflict with the unofficial but used BLKSETLASTSECT (the issue was
also summarized at http://lwn.net/2001/0906/kernel.php3). The last one
could (did?) corrupt NTFS (NTFS keeps metadata there) when one tried to
get the device size by BLKGETSIZE64 ...

Unfortunately not many softwares get BLKGETSIZE64 right, but at least the
latest fdisk (2.12a) and e2fsprogs (1.36-WIP) are ok AFAIS, although they
workaround these issues differently.

> > > So will lseek() to SEEK_END, actually (both 2.4 and 2.6).
> > > And yes, last sector _is_ accessible for dd(1) et.al.

I checked these for 2.4.25 and 2.6.5. In 2.4 the last odd sector is
visible only by BLKGETSIZE64 and BLKGETSIZE, otherwise it can't be
accessed, as we agreed later on.

But 2.6 is ok in all cases, size is seen correctly by BLKGETSIZE64,
BLKGETSIZE and accessible by lseek.

I don't know how intrusive, risky would be the backport but I suspect 
it's not worth. 

Thank you for clarifying the issue,

	Szaka

Re: [Linux-NTFS-Dev] Re: $LogFile state

[Szakacsits S. <sz...@si...>]

On Wed, 14 Apr 2004, Anton Altaparmakov wrote:

  [ ... great analyzis ... ]

> I think the significance of flag bit 2 is pretty clear now.  Now we just
> need to give the flag a name.  (-:  Perhaps RESTART_VOLUME_IS_CLEAN?

Whatever :) The hardest part is done :)
 
> > > We can always provide a force option for ignoring the fact that
> > > we think the journal is dirty.  Without the force we just work
> > > read-only.
> >
> > I even don't like the "force" option in these cases:
> >
    [...]
> > Solution is very easy. No such option, no such problems. If one doesn't
> > like it, he can modify the available, free source code and shot himself in
> > the foot. Such fun definitely deserves some extra work ;)
> 
> Not sure I agree, e.g. we should at least have a "force_but_then_readonly"

With that I agree. 

> option because we may be overzealous with our checks and we may be
> flagging perfectly clean ntfs as dirty and user might want to mount the
> volume anyway.  Also perhaps the user wants to recover data from a damaged
> volume.  In that case cannot expect a clean fs.

Yes, I think other sane filesystems do the same. If it's dirty then only
fsck, journal reply or [forced] read-only mount are possible.

	Szaka

[Linux-NTFS-Dev] PRM file for Red Hat 7.2 version 2.4.7-10

[Dr. M. N. i. <mn...@ph...>]

Dear Sir,
I am looking for prm file for Red Hat Linux release 7.2 version 2.4.7-10
Could you please let its location from where I can down load it. I need it
to read ntfs data file in my linux OS machine.
with regards,
Nazrul
---------------------------
(Dr. Md. Nazrul islam)
Department of Physics
Bangladesh University of Engineering & Technology
Dhaka-1000, BANGLADESH.
Fax:+880-2-8613046, +880-2-8613026
Tel:+880-2-9665613(w), 9665650 to 80 (ext. 7612 Res.; 7310 Off.)
http://www.buet.ac.bd/phy/naz.htm

Re: [Linux-NTFS-Dev] Re: $LogFile state

[Szakacsits S. <sz...@si...>]

On Wed, 14 Apr 2004, Anton Altaparmakov wrote:
> On Sat, 10 Apr 2004, Szakacsits Szabolcs wrote:
> > On Fri, 9 Apr 2004, Anton Altaparmakov wrote:
> > > btw.  There is another difference between NTFS 3.1 LFS 1.1 and NTFS < 3.1
> > > LFS 1.1 in that the client_array_offset in the first case is 0x40 and in
> > > the second case 0x30 so there might be additional fields in use by NTFS
> > > 3.1 LFS 1.1 despite the LFS version being the same as the old one...
> >
> > Yes, I also think so there are additional things there for XP/W2K3.
> 
> But these are always zero AFAICS so not sure why they added the extra 16
> bytes...

Same here.
 
> > Interesting mutant is the W2K with the NTFS 3.1 driver. Seemingly it also
> > behaves like what's written in logfile.h (so not the NTFS version but the
> > OS seems to matter).
> 
> Very odd.  But perhaps not that odd.  Do I assume correctly that this is
> the XP driver copied to w2k but that the volume was formatted by w2k?  

I don't think so. I guess W2K service pack, so W2K is able to read 3.1
volumes.

> If so the LogFile format would be preserved so this is why it stays
> like old driver.  If you erase the logfile I would guess that the
> newly created one will be like the XP one.
> 
> > I have lots of mostly metadata only images but unfortunately I skipped
> > logfile dumping totally for ntfsclone -metadata because its size is too
> > huge. But saving the first 8 KB could have been very useful now :(
> >
> > Maybe I add this feature too, better later than never.
> 
> That would be cool.

I checked it in. The first 8 KiB saved or one cluster if the cluster size
is greater than 8 KiB.

Later,
	Szaka

Re: [Linux-NTFS-Dev] streams

[Anton A. <ai...@ca...>]

On Sun, 11 Apr 2004, - wrote:
> i'm just wondering if this has any relevance since on
> the site, there's mentioning that
> MountMgrRemoteDatabase is not too well known about. I
> did something that's not usually done, that is to
> linkd my c:\documents and settings under a non-native
> XP bootup using WinInternals. I am a debian linux
> however and that's why I'd like to see how this can be
> helpful:->
>
> I accidently played around with linkd and
> sysinternals.com's streams program, from the dosbox i
> got on my c:\
>
> C:\>streams *
>
> Streams v1.5 - Enumerate alternate NTFS data streams
> Copyright (C) 1999-2003 Mark Russinovich
> Sysinternals - www.sysinternals.com
>
> C:\a:
>                :a:$DATA 5
> C:\Documents and Settings:
>    :$MountMgrRemoteDatabase:$DATA       124
> C:\temp:
>                :a:$DATA 5
> C:\tmp:
>                :a:$DATA 5
>
> C:\>
>
>
> i have picked up a slight understanding what this may
> represent. can these :letter:$DATA 5 mean  reparse
> mountpoints with a length of 5?

I don't know the tool you used but it looks to me like you created named
streams called "a" in your directories/files c:\temp, c:\tmp and c:\a.

> i read this article from online,
> http://www.winntmag.com/Articles/ArticleID/8303/pg/1/1.html
>
> i'd like to know if the ntfs is not really dead seeing
> that the last release was like 2 years ago and i was
> just chatting in a chatroom about this patch tester
> saying it isn't.. does anyone really care about
> finishing up this driver or something?

What do you mean ntfs is dead?  Microsoft haven't killed it AFAIK.  And we
are still actively developing Linux-NTFS.  And what do you mean the last
release was 2 years ago?  The last release of what?  The last ntfsprogs
release was only a few weeks ago...  http://linux-ntfs.sourceforge.net/

> i also am wondering if this is any useful
> echo "123" >foobar:strmname  and more
> <foobar:strmname, i accidently bumped into these cool
> redirections after seeing the author of stream's
> download page on sysinternals.

That is all old hat I am afraid but thanks anyway.

> i really do hope the project isn't dead, i find the
> project remarkable in getting windows users to
> transide over linux like myself.. i run on mswindows
> and linux 50 vs 50% of the time..

Not dead at all.  I think you might have been looking at the wrong web
pages...  Have a look at http://linux-ntfs.sourceforge.net/

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: $LogFile state

[Anton A. <ai...@ca...>]

On Sun, 11 Apr 2004, Szakacsits Szabolcs wrote:
> On Fri, 9 Apr 2004, Anton Altaparmakov wrote:
> > True.  Although the LFS drivers are quite happy to recognise any LFS
> > version including strange ones likes 0.-1...
>
> Hmmm, this also suggests the value isn't really used and LFS is tightened
> with the current driver (not even NTFS version).

Yes it is rather, especially since in both 2k and XP they are in the same
file ntfs.sys!  The only thing the version matters for is during OS
upgrades or when you move a disk from one windows to another so the driver
knows how to deal with the logfile.

Something unrelated I noticed:  In XP efs.sys no longer exists either and
it is part of ntfs.sys, too.

> > > True from pedantry point of view but I doubt even Microsoft cares. Show me
> > > an example when LFS is used outside of NTFS.
> >
> > I don't know any.  I don't think anything other than NTFS uses LFS.
>
> My suspicion is that that LFS was overdesigned, then it was realised it
> doesn't work so great as it was expected (see tons of inconsistent NTFS)
> so its use got limited and this "volume clean" bit was added (if it indeed
> means that).

Quite possible.

> If the volume is marked clean, no need to try to recover thus there is
> less chance to screw up the filesystem.
>
> > I have to apologise at this point:  I must have somehow managed to muddle
> > my logfile copies.  I took a new copy just now and it is just like you say
> > and nothing like my old copy.  So I must have been working with w2k rather
> > than xp logfile all along.  )-:
>
> Yes, I also can not see this bit set on any w2k NTFS images I have.
>
> > I can confirm the same as you see.
>
> Great news! :)
>
> > This makes everything even more difficult.
>
> Yes and no.
>
> Special case handling, yes.
>
> But for now, we could ignore the minority unknown cases (Google Zeitgeist:
> XP 46%, W2K 18%, NT4 3%). Anyway, the time already working hard to solve
> the W2K and earlier cases :-)
>
> The project is huge and we are badly lacking development resources. We can
> not solve everybody's problem at once. If we ever want to get anywhere, we
> must cut corner cases.
>
> > We will need to parse the NTFS version and then deal with the logfile
> > depending on what we see there.  So on NTFS 3.1 with LFS 1.1, we will
> > compare the restart pages (_not_ binary compare but logical compare)
>
> Plus we must extend the logical compare inasmuch as you noticed NTFS 3.1
> has more fields.
>
> > and if identical and flags = 2 we can say clean and dirty otherwise.
>
> Well, I hope so it's true :-)
>
> But I'd still prefer if somebody would check what's going on in the
> restart pages during NTFS use from Windows. Unfortunately Captive NTFS
> isn't ok because it doesn't journal continuously, that's one of the
> reasons people lose things when Captive crashes:
>
>   http://www.jankratochvil.net/project/captive/doc/Details.html.pl#logfile
>
> I don't have the needed softwares to do this and disassembling would take
> too long compared to just sniffing.

Not true at all.  Sniffing is inaccurate, non-exhaustive and can be
misleading and is not faster than disassembly with the right tools and
skills...  (-;  I ran the XPSP1 ntfs.sys through my somewhat out of date
IDA Pro 4.5.0.762 (http://www.datarescue.com/idabase/) and here is the
info:

btw. If anyone reading this wants to donate me an upgrade to the latest
4.6SP1 this would be greatly appreaciated!

The restart area flag bit 2 can indeed be taken to mean "clean umount of
ntfs volume".  It was not necessarily intended to mean this by MS but it
in effect means just that for all intents and purposes.  The flag is
affected in three places:

When a volume is mounted the code path is as follows:

NtfsMountVolume() -> NtfsStartLogFile() -> LfsOpenLogFile() ->
LfsRestartLogFile() -> LfsUpdateLfcbFromRestart()

1) LfsUpdateLfcbFromRestart() tests the flag bit 2 and if it is set, it
takes the $LogFile size passed in from the caller and uses this to set the
logfile size in the Lfcb.  If the flag bit 2 is zero, it checks the
logfile size specified in the restart area against the one passed in from
the caller and uses the smaller one.  I interpret this as that it means it
assumes that if the flag bit 2 is set, the logfile is consistent and there
is no need to check the file size specified in the restart area, it can be
assumed to be accurate.  LfsUpdateLfcbFromRestart() later returns to the
caller.

2) Immediately after this happens, LfsRestartLogFile() sets the flag bit 2
to zero unconditionally and before it returns to the caller it writes out
the restart areas to disk, thus we now have the bit set to zero.

Note at this point no log replay or anything like this has happened at
all.  This is the very first thing that happens so it cannot be that we
will see bit 2 set with something having been written to the volume that
would change the "clean shutdown" state.

When a volume is umounted the code path is as follows:

NtfsDismountVolume() -> NtfsPerformDismountOnVcb() -> NtfsStopLogFile() ->
LfsCloseLogFile() -> LfsWriteLfsRestart()

3) LfsCloseLogFile() sets bit 2 just before writing out the restart areas
to disk.  Note this it the very last i/o on the logfile and i/o
completion is waited for (LfsWriteLfsRestart() is called with the
WaitForIo flag set to true), thus we know we have the bit 2 set on disk at
this point.  Immediately after this happens, the Lfcb is deallocated and
all associated log file in memory structures are thrown away and
LfsCloseLogFile() returns to the caller.

I think the significance of flag bit 2 is pretty clear now.  Now we just
need to give the flag a name.  (-:  Perhaps RESTART_VOLUME_IS_CLEAN?

> > And on NTFS 1.x, 2.x, and 3.0 with LFS 1.1 we need to check the
> > restart pages and if identical and the criteria I describe in
> > ntfs/logfile.h are met (i.e. the logfile is shut, no clients are
> > registered in use) it is clean
>
> If it's true, sounds ok.
>
> I can't investigate these at all either, I have only such metadata images
> and I know nothing about in what state they were imaged.
>
> But they must be clean because ntfsresize's consistency check passes
> otherwise all dirty cases I've ever seen failed the check.
>
> Well, actually ntfsclone won't clone if the consistency check doesn't
> pass. I'll enable cloning inconsistent NTFS for --metadata, when it's
> possible.
>
> > and assumed dirty otherwise (although it could still be clean in that
> > case but that is too complicated for us to figure out at this point as
> > it involves parsing the whole logfile and checking that all ops are
> > checkpointed).  Anyway, it is better to be erring on the safe side
> > IMO.
>
> I agree.
>
> > We can always provide a force option for ignoring the fact that
> > we think the journal is dirty.  Without the force we just work
> > read-only.
>
> I even don't like the "force" option in these cases:
>
>   - most people aren't really aware of the consequences. They hope it will
>     work but it won't. I'm not sure we want to debug these cases and
>     hear about the crappy NTFS driver that corrupts data (you can read
>     it anywhere anyway even if it isn't possible for years).
>
>   - if the option is there it will be used. Some distributor will just
>     configure it by default and people use it without knowing about it.
>
> Solution is very easy. No such option, no such problems. If one doesn't
> like it, he can modify the available, free source code and shot himself in
> the foot. Such fun definitely deserves some extra work ;)

Not sure I agree, e.g. we should at least have a "force_but_then_readonly"
option because we may be overzealous with our checks and we may be
flagging perfectly clean ntfs as dirty and user might want to mount the
volume anyway.  Also perhaps the user wants to recover data from a damaged
volume.  In that case cannot expect a clean fs.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/