linux-ntfs-dev — Development discussions and bug reports.

[Linux-NTFS-Dev] Testing NTFS

[Richard R. <nt...@fl...>]

Hi All,

Sorry for the slow reply.

Fabio:
> Well, it seems logical that the best test is really if windoze accepts
> our changes to the FS or not...

Absolutely.  We should be able to create some basic tools for testing
the read-only part in Linux, but we will have to ask Windows if a write
was correct.

Fabio:
> What tests do you guys normally perform?

Anton had a very badly fragmented partition which also had sparse and
compressed files on it.  He took an image of it and he uses that for
tesing.  The test is simply to md5sum every file on the disk and compare
that to a known good set.

The first set should test the most vital parts of the driver.  Does
the module insert/remove OK.  Try out some mount parameters.

Next I think we need to check that directories can be read correctly.
Can we see all the entries (special attention to the first and last
files).  In addition we should check that filenames in non-Western
character sets are handled reliably.

Next read a file.  We need to test every combination of:
  Resident file
  Non-resident file
  Sparse
  Compressed
  Encrypted
Each should be read from beginning and special section (e.g. sparse) and
end.  Then we need to repeat the tests using mmap.  (Encrypted files
will fail, atm).

Finally write to a file.  We need to test every combination of:
  Resident file
  Non-resident file
  Sparse
  Compressed
  Encrypted
carefully checking the end conditions.  Then we need to repeat the tests
using mmap.  Sparse, compressed and encrypted files will currently fail.

We should perform each of the write tests in each of NT, 2K and XP.

Andrew:
> Can switching virtual machines also be [easily] automated?
> The only method I can think of is TCP/IP communication saying
> things like: "move onto the next test now" and "verify that has
> no errors now", etc.  I guess it's not too hard, but a bit messy.

Szaka:
> favouritevm winbox
> while (prepare next test)
>         ssh winbox mount drive
>         ssh winbox execute test
>         wait for result or timeout/crash
>         if (timeout/crashed)
>                 restart favouritevm winbox
>         else
>                 ssh winbox umount drive

Hmm, this is starting to look promising.  I found an ssh server for
windows (yet to test it): http://www.networksimplicity.com/openssh

Andrew:
> simple test cases where we can prepare input and expected output
> after an operation.  The e2fsck unit tests work like this... take a
> look!

Szaka:
> One way could be "embrace, customise, extend" the ext2 test tools.

They're certainly a nice source of ideas, but we're limited by the
output of chkdsk.  We could create a set of tiny test images, as they do
and copy then over our test partition.

Andrew:
> windows might silently fail.
> windows might silently correct errors.

That's a real worry.

Andrew:
> diff(1)

This could work.  Matt Fanto started work on "ntfsinfo".  This needs
more work, but the idea was to dump ntfs structures in a human-readable
format.  We may not be able to get enough info from chkdsk, but the
ntfsinfo output and diff would be able to tell us everything that
changed.

Fabio:
> ... the ideal scenario is to have the unit testers test individual
> functions of the API.  I'm not sure this is feasible with this type of
> code.

Some functions would be good to add specific tests for.  "mount", and
"umount", obviously.

Szaka:
> The question is, I think, more like who will do it?

Hmm...  I'd prefer to not have to do this myself.

Most of the tools will only require a knowledge of unix programming,
"open file", "write to file", etc.  Each program will be very simple.

The other tools we'd need are to mount/unmount a volume under windows.
This shouldn't be too hard under 2K and XP.  I don't have any tools to
compile under Windows, so I'll have to delegate this task.

Cheers,
  FlatCap (Rich)
  nt...@fl...

WWW:  http://linux-ntfs.sf.net
IRC:  #ntfs on irc.freenode.net

Re: [Linux-NTFS-Dev] The Ntfs driver doesn't compile with gcc-3.1

[Anton A. <ai...@ca...>]

Yes we know. gcc-3.1 is broken. Thanks for your report though.

Best regards,

         Anton

At 19:32 13/10/02, Samuele Kaplun wrote:
>The Ntfs driver doesn't compile with gcc-3.1
>this are the errors:
>In file included from fs/ntfs/inode.h:29,
>                  from fs/ntfs/debug.h:30,
>                  from fs/ntfs/ntfs.h:40,
>                  from fs/ntfs/aops.c:30:
>fs/ntfs/layout.h:299: unnamed fields of type other than struct or union are
>not
>allowed
>fs/ntfs/layout.h:1449: unnamed fields of type other than struct or union are
>not
>
>allowed
>fs/ntfs/layout.h:1465: unnamed fields of type other than struct or union are
>not
>
>allowed
>fs/ntfs/layout.h:1714: unnamed fields of type other than struct or union are
>not
>
>allowed
>fs/ntfs/layout.h:1891: unnamed fields of type other than struct or union are
>not
>
>allowed
>fs/ntfs/layout.h:2051: unnamed fields of type other than struct or union are
>not
>
>allowed
>fs/ntfs/layout.h:2063: unnamed fields of type other than struct or union are
>not
>
>allowed
>make[2]: *** [fs/ntfs/aops.o] Error 1
>make[1]: *** [fs/ntfs] Error 2
>make: *** [fs] Error 2
>
>
>
>-------------------------------------------------------
>This sf.net email is sponsored by:ThinkGeek
>Welcome to geek heaven.
>http://thinkgeek.com/sf
>_______________________________________________
>Linux-NTFS-Dev mailing list
>Lin...@li...
>https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev

-- 
   "I haven't lost my mind... it's backed up on tape." - Peter da Silva
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] The Ntfs driver doesn't compile with gcc-3.1

[Samuele K. <ka...@in...>]

The Ntfs driver doesn't compile with gcc-3.1
this are the errors:
In file included from fs/ntfs/inode.h:29,
                 from fs/ntfs/debug.h:30,
                 from fs/ntfs/ntfs.h:40,
                 from fs/ntfs/aops.c:30:
fs/ntfs/layout.h:299: unnamed fields of type other than struct or union are=
=20
not=20
allowed
fs/ntfs/layout.h:1449: unnamed fields of type other than struct or union ar=
e=20
not=20

allowed
fs/ntfs/layout.h:1465: unnamed fields of type other than struct or union ar=
e=20
not=20

allowed
fs/ntfs/layout.h:1714: unnamed fields of type other than struct or union ar=
e=20
not=20

allowed
fs/ntfs/layout.h:1891: unnamed fields of type other than struct or union ar=
e=20
not=20

allowed
fs/ntfs/layout.h:2051: unnamed fields of type other than struct or union ar=
e=20
not=20

allowed
fs/ntfs/layout.h:2063: unnamed fields of type other than struct or union ar=
e=20
not=20

allowed
make[2]: *** [fs/ntfs/aops.o] Error 1
make[1]: *** [fs/ntfs] Error 2
make: *** [fs] Error 2

Re: [Linux-NTFS-Dev] Remote testing?

[MAIL <fl...@we...>]

OK, I get it.  Obvious... the VM is running on the same machine, and the
TCP/IP is between Linux and the VM.

Sorry for the stupid question. :-/

Fabio

> > On Wed, 9 Oct 2002, Andrew Clausen wrote:
> > > The only method I can think of is TCP/IP communication saying
> > > things like: "move onto the next test now" and "verify that has
> > > no errors now", etc.  I guess it's not too hard, but a bit messy.
> > 
> > Looks more efficent :)
> > 
> > favouritevm winbox
> > while (prepare next test)
> > 	ssh winbox mount drive
> > 	ssh winbox execute test
> > 	wait for result or timeout/crash
> > 	if (timeout/crashed)
> > 		restart favouritevm winbox
> > 	else
> > 		ssh winbox umount drive
> > 
> >     Szaka
> 
> Humm... but how can we do tests on a remote machine?  The tests must 
> be done by linux code and windoze code on the HD of the same 
> machine.  Therefore, there is no remote testing via TCP possible.  
> You need to reboot/re-VM the machine back and forth between linux 
> and windoze.
> 
> Fabio
> 
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Linux-NTFS-Dev mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linux-ntfs-dev

[Linux-NTFS-Dev] Remote testing?

[MAIL <fl...@we...>]

> On Wed, 9 Oct 2002, Andrew Clausen wrote:
> > The only method I can think of is TCP/IP communication saying
> > things like: "move onto the next test now" and "verify that has
> > no errors now", etc.  I guess it's not too hard, but a bit messy.
> 
> Looks more efficent :)
> 
> favouritevm winbox
> while (prepare next test)
> 	ssh winbox mount drive
> 	ssh winbox execute test
> 	wait for result or timeout/crash
> 	if (timeout/crashed)
> 		restart favouritevm winbox
> 	else
> 		ssh winbox umount drive
> 
>     Szaka

Humm... but how can we do tests on a remote machine?  The tests must be done
by linux code and windoze code on the HD of the same machine.  Therefore,
there is no remote testing via TCP possible.  You need to reboot/re-VM the
machine back and forth between linux and windoze.

Fabio

Re: [Linux-NTFS-Dev] Testing ntfs writes

[Szakacsits S. <sz...@si...>]

On Wed, 9 Oct 2002, Andrew Clausen wrote:

> Can switching virtual machines also be [easily] automated?

while (prepare next test)
	favouritevm
	wait for result or timeout/crash
	kill -9 `pidof favouritevm`

> The only method I can think of is TCP/IP communication saying
> things like: "move onto the next test now" and "verify that has
> no errors now", etc.  I guess it's not too hard, but a bit messy.

Looks more efficent :)

favouritevm winbox
while (prepare next test)
	ssh winbox mount drive
	ssh winbox execute test
	wait for result or timeout/crash
	if (timeout/crashed)
		restart favouritevm winbox
	else
		ssh winbox umount drive

    Szaka

Re: [Linux-NTFS-Dev] Testing ntfs writes

[Andrew C. <cl...@gn...>]

On Wed, Oct 09, 2002 at 07:26:01AM +0200, Szakacsits Szabolcs wrote:
> 
> On Wed, 9 Oct 2002, Andrew Clausen wrote:
> 
> > * rebooting into windows is slow.
> 
> So don't reboot ;) just boot it in your favourite virtual machines.

What if you need to switch contexts for 1000 tests?

Can switching virtual machines also be [easily] automated?

The only method I can think of is TCP/IP communication saying
things like: "move onto the next test now" and "verify that has
no errors now", etc.  I guess it's not too hard, but a bit messy.

> The question is, I think, more like who will do it?

Indeed ;)

Cheers,
Andrew

Re: [Linux-NTFS-Dev] Testing ntfs writes

[Szakacsits S. <sz...@si...>]

On Wed, 9 Oct 2002, Andrew Clausen wrote:

> * rebooting into windows is slow.

So don't reboot ;) just boot it in your favourite virtual machines.
Preferably Win using a native disk partition so you not only can check
"easily" the Win fixes/output but you can also trace how it's
happening through your favourite kernel/syscall hooks.

> Unit tests are more useful if they are easy to run.

As probably anybody, I also agree on the necessity of e2fsck style
unit tests. The question is, I think, more like who will do it? One
way could be "embrace, customise, extend" the ext2 test tools.

	Szaka

Re: [Linux-NTFS-Dev] Testing ntfs writes

[MAIL <fl...@we...>]

> On Wed, Oct 09, 2002 at 02:25:06AM -0300, MAIL wrote:
> > Well, it seems logical that the best test is really if windoze accepts our
> > changes to the FS or not... maybe we should write testers in windoze, and run
> > them from windoze.
> 
> While this is right in principle, it is still good to provide some
> other alternatives, because:
> * windows might silently fail.
> * windows might silently correct errors.
> * rebooting into windows is slow.  Unit tests are more useful if they
> are easy to run.

Yeah, good point.  Windoze testers would be slow to use and would have to be
complemented by linux user space testers anyway.

> Other alternatives:
> * ntfsfix
> * diff(1)
> * simple test cases where we can prepare input and expected output after
> an operation.  The e2fsck unit tests work like this... take a look!

OK, I'll mark to take a look at those unit tests.

One thing I didn't mention in my post is that the ideal scenario is to have
the unit testers test individual functions of the API.  I'm not sure this is
feasible with this type of code (do I need to say I never did kernel
development before?). Like, for each API function that writes to the NTFS, we
write a tester that prepares the test (mounts FS, etc), calls just that API
function, then checks the result.

Probably doesn't work with this type of code... I imagine most of the API
functions would corrupt the fs if called by themselves.  I'll start looking at
the API to get a feeling for that.

Fabio

Re: [Linux-NTFS-Dev] Testing ntfs writes

[Andrew C. <cl...@gn...>]

On Wed, Oct 09, 2002 at 02:25:06AM -0300, MAIL wrote:
> Well, it seems logical that the best test is really if windoze accepts our
> changes to the FS or not... maybe we should write testers in windoze, and run
> them from windoze.

While this is right in principle, it is still good to provide some
other alternatives, because:
* windows might silently fail.
* windows might silently correct errors.
* rebooting into windows is slow.  Unit tests are more useful if they
are easy to run.

Other alternatives:
* ntfsfix
* diff(1)
* simple test cases where we can prepare input and expected output after
an operation.  The e2fsck unit tests work like this... take a look!

Cheers,
Andrew

[Linux-NTFS-Dev] Testing ntfs writes

[MAIL <fl...@we...>]

web...@fl... wrote:
> Even though we've tested the driver quite a bit, we don't have any
> specific tests.  We really need a set of tests that we can run every
> time we make any changes to the code.  Proper regression testing.
>
> If you can think how to get us started that would be much appreciated.
> (Or if you know what questions I should be asking myself).

OK.

I am very fond of automated tests.  I had excellent experience with the xUnit
test framework in a big development I lead last year.  Not sure how to apply
those concepts here though...  For those who never heard of xUnit a rough idea
is to make code that tests your code, with specific unit tests.  So after you
do a change to your source you would run the automated tests to check if you
broke something that is already covered by the tests.

For the ntfs write part I believe We could do something like this:
- we know some basic rules the ntfs structures follow when they consistent
- we write functions that test for these rules, example:
  - do the two XYZ structures that exist in the fs have the same information
  - do all entries in the journal have valid transactions
- we write tests that invoke these functions and inform us if all is OK
- each time we implement more ntfs write functionality or change ntfs write
code we run the new code, change a file, then run our automated tests
- if our automated tests fail, well, our code additions/changes suck
- I know the ntfsfix has some "tests" inside it, but from what I saw they are
not organized as functions that could be called from an automated tester system.

The good thing about using automated testers is that it becomes quite fast to
test, and as we write new automated testers we get more tests without spending
much more time, because they are automated anyway.

Well, it seems logical that the best test is really if windoze accepts our
changes to the FS or not... maybe we should write testers in windoze, and run
them from windoze.

How about this:
- we create test code in windoze to open, read a file, then modify it
- if all operations succeed, in theory our ntfs write didn't break anything
- so, we run our ntfs write
- we do NOT run ntfsfix
- reboot in windoze
- run our windoze tester on the file we manipulated

Another idea, mixing Linux and windoze:
- we create a utility that makes a copy of the NTFS structures to disk files
- we run our ntfs write code, followed by ntfsfix to ask windoze to check the ntfs
- reboot in windoze, let it check and if necessary fix
- reboot in Linux
- run ou utility again, and have it intelligently compare the current
structures with the ones we dumped earlier.  By intelligently I mean with
functions that identify what changed in which structure and translates that to us.
- this idea might not work due to other changes that windoze might make to the
FS independent of the changes we did (like due to antivirus or other automatic
on boot crap)

Maybe some of this is what you guys are already doing.  Comments?

Fabio

Re: [Linux-NTFS-Dev] Interesting discovery

[Richard R. <nt...@fl...>]

Hi Tim,

> > > It might be worth advertising on the sourceforge project page that you can 
> > > read windows software raid'ed disks under linux
> > 
> > Yeah.  I'll add something to the FAQ when I get a second.
> 
> I can write you up a few paragraphs on this if you want

Thanks, that'd be appreciated.  A nice simple explaination of what's
possible with a few examples (you can crib from ntfs.txt).

The other bit we could do with, from ntfs.txt, is all the mount options.
But don't worry about that unless you really have too much free time.

Many thanks,
  FlatCap (Rich)
  nt...@fl...

Re: [Linux-NTFS-Dev] Interesting discovery

[Tim F. <ti...@ni...>]

> > If you had read Documentation/filesystems/ntfs.txt of the new ntfs driver 
> 
> By which Anton means, if you'd read a file that wasn't in your source
> tree and isn't referenced anywhere :-)

I have now read this file and it does indeed talk about this :)

> > Oh IC, I looked at the website and stuff and found the tool for accessing 
> > software raid stuff listed as "not yet started" and also googled a bit.
> 
> Ah, yes.  The tool I was thinking about hasn't been written yet.  What
> I'd like is something to read the LDM database and spit out a raidtab.
> We certainly have enough information to pull this off.

Ah right, I took that to mean that is was needed for even the most basic 
usage

> > It might be worth advertising on the sourceforge project page that you can 
> > read windows software raid'ed disks under linux
> 
> Yeah.  I'll add something to the FAQ when I get a second.

I can write you up a few paragraphs on this if you want

-- 
      Tim Fletcher                  .~.
                                    /V\       L   I   N   U   X   
    ti...@ni...         // \\  >Don't fear the penguin<
ti...@pa...   /(   )\
irc: Night-Shade on Openprojects   ^^-^^

"I love the way Microsoft follows standards.  In much the
same manner that fish follow migrating caribou."
                                            Paul Tomblin

Re: [Linux-NTFS-Dev] Interesting discovery

[Richard R. <ri...@fl...>]

Hi Tim,

> If you had read Documentation/filesystems/ntfs.txt of the new ntfs driver 

By which Anton means, if you'd read a file that wasn't in your source
tree and isn't referenced anywhere :-)

> Oh IC, I looked at the website and stuff and found the tool for accessing 
> software raid stuff listed as "not yet started" and also googled a bit.

Ah, yes.  The tool I was thinking about hasn't been written yet.  What
I'd like is something to read the LDM database and spit out a raidtab.
We certainly have enough information to pull this off.

> It might be worth advertising on the sourceforge project page that you can 
> read windows software raid'ed disks under linux

Yeah.  I'll add something to the FAQ when I get a second.

Cheers,
  FlatCap (Rich)
  nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.freenode.net

Re: [Linux-NTFS-Dev] Interesting discovery

[Tim F. <ti...@ni...>]

> If you had read Documentation/filesystems/ntfs.txt of the new ntfs driver 
> you would have found a detailed description of how to use the various 
> software raid types supported by windows on Linux... So you aren't telling 
> us anything new... But nice to know it worked fine for you, too. (-:

Oh IC, I looked at the website and stuff and found the tool for accessing 
software raid stuff listed as "not yet started" and also goggled a bit. It 
might be worth advertising on the sourceforge project page that you can 
read windows software raid'ed disks under linux, you might well have done 
I just missed it but I did make the effort to look :)

> Note that it is no problem to read the set under 2k again. You have 
> correctly specified not to use persistent super blocks which means that 
> nothing was written anywhere at all... And you don't even need to run 
> mkraid. You can just do a simple (as root): raid0run -a
> and this will start all devices specified in /etc/raidtab.

The reading under 2k is not really a great problem as one of the disks 
failed shortly afterwards but that is useful to know.

-- 
      Tim Fletcher                  .~.
                                    /V\       L   I   N   U   X   
    ti...@ni...         // \\  >Don't fear the penguin<
ti...@pa...   /(   )\
irc: Night-Shade on Openprojects   ^^-^^

"While preceding your entrance with a grenade is a good tactic in
   Quake, it can lead to problems if attempted at work." -- C Hacking

Re: [Linux-NTFS-Dev] Interesting discovery

[Anton A. <ai...@ca...>]

At 19:02 21/09/02, Tim Fletcher wrote:
>I am helping a friend move a fileserver from 2k to Linux and as such have
>been playing with LDM and NTFS. He has a 4 disk software RAID0 stripe set
>using LDM the data on it being "useful to have" so I could experiment with
>it and try to read it under linux. The LDM driver in 2.4.20-pre7-ac3 saw
>the partitions fine, I then setup a raidtab with a 4 disk raid set without
>persistent-superblock and a 64k stripe size. Once I had run mkraid I could
>mount the resulting md device with ntfs and read it without problems. and
>have copied off 130GB of data. I haven't tried to read the set under 2k
>again as I feel that I might have written rubbish over the database at the
>end of the disks (which I assume is really rather important)
>
>I have the raidtab if others are interested

Hi,

If you had read Documentation/filesystems/ntfs.txt of the new ntfs driver 
you would have found a detailed description of how to use the various 
software raid types supported by windows on Linux... So you aren't telling 
us anything new... But nice to know it worked fine for you, too. (-:

Note that it is no problem to read the set under 2k again. You have 
correctly specified not to use persistent super blocks which means that 
nothing was written anywhere at all... And you don't even need to run 
mkraid. You can just do a simple (as root): raid0run -a
and this will start all devices specified in /etc/raidtab.

Best regards,

         Anton


-- 
   "I haven't lost my mind... it's backed up on tape." - Peter da Silva
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Interesting discovery

[Tim F. <ti...@ni...>]

I am helping a friend move a fileserver from 2k to Linux and as such have
been playing with LDM and NTFS. He has a 4 disk software RAID0 stripe set
using LDM the data on it being "useful to have" so I could experiment with
it and try to read it under linux. The LDM driver in 2.4.20-pre7-ac3 saw
the partitions fine, I then setup a raidtab with a 4 disk raid set without
persistent-superblock and a 64k stripe size. Once I had run mkraid I could
mount the resulting md device with ntfs and read it without problems. and
have copied off 130GB of data. I haven't tried to read the set under 2k
again as I feel that I might have written rubbish over the database at the
end of the disks (which I assume is really rather important)

I have the raidtab if others are interested

-- 
      Tim Fletcher                  .~.
                                    /V\       L   I   N   U   X   
    ti...@ni...         // \\  >Don't fear the penguin<
ti...@pa...   /(   )\
irc: Night-Shade on Openprojects   ^^-^^

"Commercial OS vendors are, at the moment, all closed economies, and 
doomed to fall in their competition with open economies just as 
communism eventually fell."
                   -- H. Reiser, Unix OS developer

[Linux-NTFS-Dev] ANN: NTFS 2.1.0a for Linux 2.4.19 and 2.4.20-pre-BK

[Anton A. <ai...@ca...>]

The new NTFS driver 2.1.0(a) is now available for kernel 2.4.19. NTFS 
2.1.0(a) implements the first steps towards file overwrite support.

Full and incremental patches are available from the Linux NTFS download page:
         http://linux-ntfs.sf.net/downloads.html
and from the Sourceforge project page (also older patches here):
         http://sf.net/project/showfiles.php?group_id=13956&release_id=107961

If you use bitkeeper, you can get NTFS 2.1.0a by pulling from our bitkeeper 
repository (note this is based on Marcelo's current bitkeeper tree so it is 
at 2.4.20-pre5 at the moment and will move forward as Marcelo's repository 
moves forward):
         http://linux-ntfs.bkbits.net/ntfs-2.4

The current code is relatively well tested both for mmap(2) and write(2)
both using existing applications to randomly write to files and using
custom programs to do specialized writes to test boundary conditions.

Still the code has only been run on two machines, so people trying it,
please have backups! I am confident it won't eat your data, but I am not
willing to guarantee it! I have put in an appropriately very scary config
help message to scare off the casual user for the moment...

Features of NTFS 2.1.0(a)
=========================

It is now possible to write over existing files both with mmap(2)
and write(2).

It is now possible to setup a loopback on an NTFS file and then you have
full read/write access to the loopback device. You can create a Linux fs
on the loop device for example and mount it.

This has been a much requested feature because it allows installation of
Linux on an NTFS partition using the loopback trick, i.e. from windows one
creates a large file on NTFS, then one boots Linux (from installation CD,
rescue floppies or whatever) and as root does:

mount -t ntfs -o rw /dev/hda1 /mnt/ntfs
losetup /dev/loop0 /mnt/ntfs/some_dir/preprepared_large_file
mke2fs -j /dev/loop0
mount -t ext3 /dev/loop0 /mnt/new_root
mkdir old_root
<install Linux into /mnt/new_root>
umount /mnt/new_root
losetup -d /dev/loop0
umount /mnt/ntfs

 From now on, you can boot Linux and using a minimal ramdisk loaded via
floppy for example, one just needs to have something simillar to the
following done:

mount -t ntfs -o rw /dev/hda1 /mnt/ntfs
mount -t ext3 -o loop /ntfs/some_dir/preprepared_large_file /mnt/new_root
cd /mnt/new_root
pivot_root . old_root
exec chroot . sh <dev/console >dev/console 2>&1
umount /old-root

[Note you probably cannot umount /old-root but it doesn't matter. It doesn't
disturb anyone... You could always hide it inside root/old_root or something
so users don't see it.]

I haven't actually tried to install Linux in the above way but Richard
Russon (flatcap) tested the loopback/mke2fs/read-write stuff and it
worked fine for him.

Limitations of NTFS 2.1.0(a) overwrite abilities
================================================

- Resident files only written to in memory so far, i.e. writes to files
   smaller than 1kiB won't be permanent. Warnings to that effect are shown
   via printk().

- Filling in of holes/non-initialized areas is not supported yes.

- File resize/truncate not implemented and actively trapped and aborted.

Anyone who tries this new code please let me know how you get on...

Best regards,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] CVS -> BK migration

[Richard R. <nt...@fl...>]

Hi Szaka,

Nice to see you're still alive.

> Hmmm ... www.bitmover.com, from where the needed binaries could be
> downloaded, runs a remote exploitable Apache,
> 	http://www.cert.org/advisories/CA-2002-17.html

Thanks for letting us know.

> Even worse, unfortunately, the binaries (or their md5sums) aren't
> signed. How can one make sure the binaries aren't tampered and safe to
> use?

You can't, but that's something you'll have to take up with BK.
If cc'ing Larry hasn't done the trick, I'd recommend su...@bi...
(Assuming that hasn't been compromised, too ;-)


Cheers,
    FlatCap (Rich)
    nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net

Re: [Linux-NTFS-Dev] CVS -> BK migration

[Szakacsits S. <sz...@si...>]

On 22 Aug 2002, Richard Russon wrote:

> The deed is done.  I've migrated all the source code to bitkeeper

Hmmm ... www.bitmover.com, from where the needed binaries could be
downloaded, runs a remote exploitable Apache,
	http://www.cert.org/advisories/CA-2002-17.html

Even worse, unfortunately, the binaries (or their md5sums) aren't
signed. How can one make sure the binaries aren't tampered and safe to
use?

	Szaka

[Linux-NTFS-Dev] CVS -> BK migration

[Richard R. <nt...@fl...>]

Hi Anton,

The deed is done.  I've migrated all the source code to bitkeeper
and "hidden" the original CVS files, to prevent confusion.

Original CVS Repository      BK Repository

linux-ntfs                   bk://linux-ntfs.bkbits.net/ntfs-progs
dynamic-disk                 bk://linux-ldm.bkbits.net/ldm-progs

flatcap                      bk://linux-ntfs.bkbits.net/tng-support
orig-ntfs-tools              bk://linux-ntfs.bkbits.net/tng-support

ntfs-driver-2.4              bk://linux-ntfs.bkbits.net/ntfs-2.4
ntfs-driver-tng              bk://linux-ntfs.bkbits.net/ntfs-2.5-devel

htdocs                       Not moved, still on SF/CVS

orig-ntfs-tools-2.2          Unused archive, support dropped
orig-ntfs-tools-2.4          Unused archive, support dropped

Cheers,
    FlatCap (Rich)

Re: [Linux-NTFS-Dev] MFT's without $INDEX attributes

[Richard R. <nt...@fl...>]

Hi Maria,

> I am running some tests on an ntfs system and have run
> into an MFT without an index_root or index_allocation.

> the ntfs driver is able to find this information in
> another MFT,

It sounds like the MFT record is fragmented.

> from what i am able to figure out the
> second MFT is an entry in the $SECURITY attribute.

No.  That's something else altogether.

> -- How do i know that the MFT is extended
> and where to find the info about the second MFT.

Take the first record of the MFT (a FILE record[1]), remove
the fixup[2], and look at the attributes[3].

Look for 0x20 Attribute List[4].  Its presence means that some
of the attributes didn't fit into this FILE Record.  It lists
all the attributes for that MFT (except itself) and where they
can be found.

> My question is how to interpret the data in the security
> attribute?

This is vaguely described here[5].  Though you don't need it for
anything other than enforcing file / directory permissions.

[1]  http://linux-ntfs.sf.net/ntfs/concepts/file_record.html
[2]  http://linux-ntfs.sf.net/ntfs/concepts/fixup.html
[3]  http://linux-ntfs.sf.net/ntfs/attributes/index.html
[4]  http://linux-ntfs.sf.net/ntfs/attributes/attribute_list.html
[5]  http://linux-ntfs.sf.net/ntfs/attributes/security_descriptor.html

I hope this helps,
  FlatCap (Rich)
  nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net

[Linux-NTFS-Dev] MFT's without $INDEX attributes

[Maria S. <qe...@ya...>]

I am running some tests on an ntfs system and have run
into an MFT without an index_root or index_allocation.
 the ntfs driver is able to find this information in
another MFT, from what i am able to figure out the
second MFT is an entry in the $SECURITY attribute.  My
question is how to interpret the data in the security
attribute -- How do i know that the MFT is extended
and where to find the info about the second MFT.

Thanks
Maria Sehovich


__________________________________________________
Do You Yahoo!?
HotJobs - Search Thousands of New Jobs
http://www.hotjobs.com

[Linux-NTFS-Dev] Re: Invalidating pages

[Anton A. <ai...@ca...>]

At 14:38 12/08/02, Richard Russon wrote:
>Ages ago you split up my data runs code into two pieces.
>Check that the operation is legitimate, then actually perform the
>operation.
>
>That's fair enough since we can't recover if something goes wrong.

Yup.

>If we were to write to an MFT record can we bypass this.  If things
>go wrong can we just tell the cache that the page is invalid?
>Is this a legal thing to do?  Is it sensible?

Pages can be invalidated, yes. The easiest form of invalidation is 
"ClearPageUptodate(page); ClearPageDirty(page);". Note such invalidation 
must happen under PageLock protection. But it is bad to use this method. 
There are other methods to discard a page. Some discarding contents, some 
preserving them, some actually truncating the file. I am sure we will find 
a function that does just what you suggest.

Clearing PageUptodate() means that on next (read) access the page will be 
read back in via ->readpage. [We just need to take care of the buffers in 
the page, they will have to either be thrown away or at least marked 
non-uptodate, too. But this assumes we will have buffers which is not 
carved in stone because it causes a lot of complications...]

It is a legal thing to do in general. However it is not sensible for mft 
record pages for these reasons:

1) There are 4 or more records in a page. invalidating the page will throw 
away the other 3 or more records as well. What if they were perfectly 
valid, but dirty, and not written to disk yet? We would lose their changes. 
Oops...

2) The record we are writing to may have been written to previously but the 
contents may not have been written to disk yet. Example: User creates a 
file, then writes 1kiB, then writes another 1kiB. We edit the mft record 
three times, first marking it in use and cleaning it up for reuse, adding a 
new file name attribute, etc, then the second time we extend the run list 
of the attribute by 1kiB and the third time extend it by another 1kiB. If 
the first write succeeds but the second one fails, and we invalidate the 
page, we loose the changes from the first write. We may even lose the file 
creation step itself. That would mean we have an allocated mft record in 
$MFT/$Bitmap, we have clusters allocated in $Bitmap/$DATA for the attribute 
data, we have a directory entry for the file, but the mft record for the 
file is marked not in use (we never wrote the file creating/extension to 
disk after all and just threw away the page) so we end up with a lost mft 
record, a (few) lost clusters, and a stale directory entry. Oops...

The easiest way to recover a failed write from within the write functions, 
will be to roll back what we changed. Just think "journalling fs" without 
the journal. The write function needs to keep the old and changed state in 
some fashion (perhaps just have variables like: "BOOL size_is_changed" and 
"s64 old_size" and on error, we then do "if (size_is_changed) 
resize(old_size);", you get the idea). Or we could go fully journalling but 
I would rather not get involved with journalling yet... Non-journalled 
writes are difficult enough.

>(Obviously before starting we'd have to make sure the whole page is
>uptodate, not just the record).

For the $MFT/$DATA (and in fact for all MST protected attributes) we will 
need to either have the whole page uptodate or nothing at all. We may 
actually end up having to have completely separate codepaths for MST 
protected vs non-protected attributes. Certainly for writepage it looks 
like we are heading down distinct routes. The way we perform the i/o is 
just too different. We actually will need to violate a lot of VM/buffer 
layer semantics to get MST protection to work as I would like it to work. 
Should it turn out that this is impossible because the VM/buffer layer 
interact with us badly it will cause data corruption - we will notice and 
then we will have to do some kludges to get this all to work. But lets get 
normal attribute i/o working first, then worry about mst protected i/o...

I hope I will manage to get ntfs_write_block() hammered out tonight so some 
of the problems that occur will be more easy to explain.

It is all about page vs buffer state. Each page can be buffer less (e.g. 
resident attributes) in which case the state of the page (dirty, uptodate, 
etc) is definitive. However, when there are buffers, the buffers are 
definitive and the page state is only a hint to the overall state. Note the 
hint has rules; for example it is not legal to have dirty buffers against a 
clean page. But it is quite legal to have a dirty page where only some 
buffers are dirty - then only the dirty ones need to be written out. The 
rest ignored. [This occurs for small writes via generic_file_write() which 
uses ntfs_prepare/commit_write(). It cannot occur for mmap()ped writing as 
that only operates on pages as a whole.]

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Invalidating pages

[Richard R. <nt...@fl...>]

Hi Anton,

Ages ago you split up my data runs code into two pieces.
Check that the operation is legitimate, then actually perform the
operation.

That's fair enough since we can't recover if something goes wrong.

If we were to write to an MFT record can we bypass this.  If things
go wrong can we just tell the cache that the page is invalid?
Is this a legal thing to do?  Is it sensible?

(Obviously before starting we'd have to make sure the whole page is
uptodate, not just the record).

Cheers,
    FlatCap (Rich)
    nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net