linux-ntfs-dev — Development discussions and bug reports.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> Yes. In your case [and quite probably in many others]. But the partition
> can be much bigger than the NTFS volume size and chkdsk always creates the
> backup boot sector in the last 512 bytes of the partition. Believe me, it
> has nothing to do with cluster bitmap. Your case is pure accident.

Oh, bloody hell.  What happens if it's missing ?

> So probably you will also need a mechanism to put data relative from the
> volume end or handling this case as an exception (but in that case what
> will be the next exception? ;)

Glerk.

Ian.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Szakacsits S. <sz...@si...>]

On Thu, 10 Apr 2003, Ian Jackson wrote:
> Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> > Only comment now is that the backup boot sector, the last 512 bytes of the
> > partition, isn't strictly part of the NTFS. Nowhere NTFS refers to it. So
> > imaging based on exclusively cluster bitmap isn't enough.
>
> Yes, I noticed that.  The corresponding bit in the cluster bitmap is
> set (on the volumes I have to hand, anyway) but it's beyond the end
> according to the cluster count.

Yes. In your case [and quite probably in many others]. But the partition
can be much bigger than the NTFS volume size and chkdsk always creates the
backup boot sector in the last 512 bytes of the partition. Believe me, it
has nothing to do with cluster bitmap. Your case is pure accident.

So probably you will also need a mechanism to put data relative from the
volume end or handling this case as an exception (but in that case what
will be the next exception? ;)

	Szaka

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

Anton Altaparmakov writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> I agree that a simple image format is a good idea but I also think that
> being able to create a simple sparse file for debugging purposes
> especially a metadata only file is extremely valuable, too.

This can be done, of course, just by piping the output of the imager
into the restorer.  The few resulting extra memory copies will
probably pale into insigificance next to the overhead of disk reads
and writes.  Although, I'll try it and see.

>  If we can combine the two utilities that would probably be the best
> thing. My suggestion would be to create one utility rather than two
> different ones and that utility for example (just picking at random,
> feel free to use different) takes a first parameter "b" for create
> image/backup, "r" for restore image/backup and then a second
> optional parameter say "s" for working with a sparse file as the
> source ("r"/restore case) or destination ("b"/backup case). Much
> like the tar utility works in a way...

I think this is a bad idea because the image format isn't
ntfs-specific, so the restorer is really only in the ntfsprogs as a
convenient place to put it.  The imager and the restorer hardly share
any code at all (pretty much only the format magic number, so far).

> Yes, of course. But ideally I would like to see the two functionalities in
> one rather than having two separate imaging programs, one for sparse
> files and one for custom image format...

Certainly if a program is needed to make sparse files (rather than
just feeding the custom image stream into the restorer), it should be
the same program as the imager that generates a stream.

> >  - restoreimage should be able to create the image as a sparse file. I
> >    don't think this would be a big issue.
> 
> I think this is the wrong level. I really don't want to have to create an
> image to be able to create a sparse file from that, that would take twice
> the space on disk and twice the time.

You don't have to store the image stream anywhere (indeed, you'd
probably prefer not to for the reasons you say).

>  Not worth it IMHO. Much better to have the imaging program do it
> immediately on image creation.

Is the additional complexity in the imager worth it to avoid the need
to run two commands (or have a shell alias) ?

>  It would also be really nice if on restore the imaging program can
> take the sparse file and use that to restore the data to the
> partition.

Copying a sparse file to a partition isn't a restore, it's an
image-and-restore.  So in my model you use the imager and restorer,
just as before.

Ian.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> Only comment now is that the backup boot sector, the last 512 bytes of the
> partition, isn't strictly part of the NTFS. Nowhere NTFS refers to it. So
> imaging based on exclusively cluster bitmap isn't enough.

Yes, I noticed that.  The corresponding bit in the cluster bitmap is
set (on the volumes I have to hand, anyway) but it's beyond the end
according to the cluster count.

> For me it came free: in the same session both consistency check and imaging
> are done (except stdout case). So I guess you just use the cluster bitmap.

That's right.

> > If there's a library function to do it, excellent.  Otherwise I'll punt
> > on that for now.
> 
> OK, I'll make, actually it's also in progress as part of making the
> inode/attrib/cluster walking code shared with many callback points for
> ntfsresize/ntfsmeta/ntfsck/ntfsdefrag/my_ntfsimage and potentially others.

OK.

> > (I'm using `utils_mount_volume'.  Maybe, if you want things to do
> 
> utils_mount_volume is missing things. See mount_volume and after it was
> called the volume size checks in ntfsresize. The error messages also
> answer FAQ's what got gone after I added the explanations/guesses to them.

I'll take a look at that.

Thanks,
Ian.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Anton A. <ai...@ca...>]

On Thu, 10 Apr 2003, Szakacsits Szabolcs wrote:
> Anton, how about if I start ntfsck as well based on the ntfsresize
> checks? It feels very stupid now, that I must ask people, please
> run 'ntfsresize -i device' to check your NTFS.

Sounds good.

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Anton A. <ai...@ca...>]

On Wed, 9 Apr 2003, Szakacsits Szabolcs wrote:
> On Wed, 9 Apr 2003, Ian Jackson wrote:
> > So I think that there is room for both facilities: the optimised image
> > copy, and the conversion to a stream representation.
> 
> I like simplicity. Your way seems to be the winner. If restoreimage can
> create a sparse file of the original image [metadata] that't fine with me
> (needed for debug, test, development). Now this is fast, reliable and
> works.

I agree that a simple image format is a good idea but I also think that
being able to create a simple sparse file for debugging purposes
especially a metadata only file is extremely valuable, too. If we can
combine the two utilities that would probably be the best thing. My
suggestion would be to create one utility rather than two different ones
and that utility for example (just picking at random, feel free to use
different) takes a first parameter "b" for create image/backup, "r" for
restore image/backup and then a second optional parameter say "s" for
working with a sparse file as the source ("r"/restore case) or destination
("b"/backup case). Much like the tar utility works in a way...

You could have long options, too like "--backup --sparse" or whatever if
you wanted in addition... 

Anyway just ideas...

> > I hope you'll accept my program which does write a stream format into
> > the linux-ntfs package.
> 
> Anton's decision but I support it. My wishes are only

Yes, of course. But ideally I would like to see the two functionalities in
one rather than having two separate imaging programs, one for sparse
files and one for custom image format...

>  - must have the consistency check. My experiences from ntfsresize
>    feedbacks is that there are many corrupted NTFS out there. chkdsk fixed
>    them nicely. In general no point and dangerous to backup a corrupted
>    NTFS (unless it's for recovery). This can be taken from my version if
>    you don't have [actually I'm planing to make it a library function].

Agreed.

>  - restoreimage should be able to create the image as a sparse file. I
>    don't think this would be a big issue.

I think this is the wrong level. I really don't want to have to create an
image to be able to create a sparse file from that, that would take twice
the space on disk and twice the time. Not worth it IMHO. Much better to
have the imaging program do it immediately on image creation. It would
also be really nice if on restore the imaging program can take the sparse
file and use that to restore the data to the partition. That allows for
example to run ntfsck (that will one day be written) and then to just
restore the metadata to the old state quickly by restoring the sparse
image with the metadata only option enabled.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Szakacsits S. <sz...@si...>]

On Wed, 9 Apr 2003, Ian Jackson wrote:

> > BTW, I strongly recommend, send your purposal for comments, review to the
> > linux-fsdevel mailing list
> > 	http://www.kernelnewbies.org/mailinglist.php3
>
> I've done that.

Thanks. I'll also read/think about it.

Only comment now is that the backup boot sector, the last 512 bytes of the
partition, isn't strictly part of the NTFS. Nowhere NTFS refers to it. So
imaging based on exclusively cluster bitmap isn't enough.

> >  - must have the consistency check. My experiences from ntfsresize
>
> I don't have a per-se objection to this, but I don't want to put
> anything in the ntfsimage.c file that's not related to the image
> stream generation.

It's a POW issue. I think it is a requirement before making any image.
Partimage doesn't have this, this is one of the reasons I'd never trust it.
The other one, it doesn't use the well-tested, widely used libntfs.

For me it came free: in the same session both consistency check and imaging
are done (except stdout case). So I guess you just use the cluster bitmap.
Without validating its correctness it's untrustable for reliable imaging,
works for you but not in widespread use. Also don't forget, NTFS is quite
complex, has several versions and was reverse engineered.

If you get reports that your tool destroyed/lost data, you might want to
figure out what went wrong. Some cases I had or were caught before
progressing,
 - inconsistent NTFS
 - bad RAM
 - kernel bugs
 - wrong IDE cable
 - partitioning tool bug
 - bugs in code _during_ development

> If there's a library function to do it, excellent.  Otherwise I'll punt
> on that for now.

OK, I'll make, actually it's also in progress as part of making the
inode/attrib/cluster walking code shared with many callback points for
ntfsresize/ntfsmeta/ntfsck/ntfsdefrag/my_ntfsimage and potentially others.

Anton, how about if I start ntfsck as well based on the ntfsresize
checks? It feels very stupid now, that I must ask people, please
run 'ntfsresize -i device' to check your NTFS.

> (I'm using `utils_mount_volume'.  Maybe, if you want things to do

utils_mount_volume is missing things. See mount_volume and after it was
called the volume size checks in ntfsresize. The error messages also
answer FAQ's what got gone after I added the explanations/guesses to them.

> consistency checks, it should be called from there ?)

Probably not. But I'd like to have locking there or actually in ntfs_mount.

> [sparse files]
> ...
> Yes, I think it has got worse (from your point of view).  Naturally I
> think this is an improvement :-).

I thought the same for a long time ... until I regularly started to work
with sparse files. Users have 30, 80, 200, etc GB filesystems with a couple
of MB metadata scattered on them. For debug, regression testing, etc they
are highly valuable.

I have a couple of GB. I need to cp, tar, bzip2, md5sum, ntfsresize, etc
many of them. Frequently time is spent just reading/writing empty buffers
You think it's an improvement, so how would you do these? I thought maybe I
try LD_PRELOAD a libsparse intercepting libc IO calls if feasible and
supporting the only fs providing the needed support, XFS.

	Szaka

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> I like simplicity. Your way seems to be the winner. If restoreimage can
> create a sparse file of the original image [metadata] that't fine with me
> (needed for debug, test, development). Now this is fast, reliable and
> works.

OK.

> BTW, I strongly recommend, send your purposal for comments, review to the
> linux-fsdevel mailing list
> 	http://www.kernelnewbies.org/mailinglist.php3

I've done that.

> Anton's decision but I support it. My wishes are only
> 
>  - must have the consistency check. My experiences from ntfsresize
>    feedbacks is that there are many corrupted NTFS out there. chkdsk fixed
>    them nicely. In general no point and dangerous to backup a corrupted
>    NTFS (unless it's for recovery). This can be taken from my version if
>    you don't have [actually I'm planing to make it a library function].

I don't have a per-se objection to this, but I don't want to put
anything in the ntfsimage.c file that's not related to the image
stream generation.  If there's a library function to do it,
excellent.  Otherwise I'll punt on that for now.

(I'm using `utils_mount_volume'.  Maybe, if you want things to do
consistency checks, it should be called from there ?)

>  - restoreimage should be able to create the image as a sparse file. I
>    don't think this would be a big issue.

Absolutely, that'll come out in the wash.  By default, restoreimage
will not write the absent clusters, instead seeking past them in the
output `file'.

>  - bugfree :)

:-).

[sparse files]
> Yes, I thought the situation became a bit better in the last 8 years or
> so ... I was wrong :(

Yes, I think it has got worse (from your point of view).  Naturally I
think this is an improvement :-).

Ian.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Szakacsits S. <sz...@si...>]

On Wed, 9 Apr 2003, Ian Jackson wrote:

> > I do it as:
> > 	ntfsimage -o - device | bzip2 ...
>
> That's much slower than simply using a format which encodes the
> sparseness.

Yes, I made some tests now also on an "average" box with "common" disk
partition sizes. It's _desperately_ slow. bzip2 compresses sparse files
extremely well but also extemely slow. Moreover it can't create sparse
file. tar can but can't be stdin streamed from ntfsimage ... So I'm
convinced.

> I'm not saying sparse files aren't sometimes a useful technique.  But
> in the past, file formats which only make sense as sparse files have
> turned out to be less useful because a smaller proportion of the
> standard utilities work well on them.

Yes, I thought the situation became a bit better in the last 8 years or
so ... I was wrong :(

> So I think that there is room for both facilities: the optimised image
> copy, and the conversion to a stream representation.

I like simplicity. Your way seems to be the winner. If restoreimage can
create a sparse file of the original image [metadata] that't fine with me
(needed for debug, test, development). Now this is fast, reliable and
works.

> > There is e2image in e2fsprogs, I don't know if it works with ext3 or not.
> > Probably.
>
> I hadn't heard of that.  As far as I can tell from its manpage it's
> not quite the same thing as I think ntfsimage is or should be - but it
> could benefit from being able to write a stream format.

Yes. Actually apparently only XFS has such dump feature [what doesn't it
have??? ok, undelete].

BTW, I strongly recommend, send your purposal for comments, review to the
linux-fsdevel mailing list
	http://www.kernelnewbies.org/mailinglist.php3

> There's really no reason why ntfsimage shouldn't be able to do both.

True.

> I hope you'll accept my program which does write a stream format into
> the linux-ntfs package.

Anton's decision but I support it. My wishes are only

 - must have the consistency check. My experiences from ntfsresize
   feedbacks is that there are many corrupted NTFS out there. chkdsk fixed
   them nicely. In general no point and dangerous to backup a corrupted
   NTFS (unless it's for recovery). This can be taken from my version if
   you don't have [actually I'm planing to make it a library function].

 - restoreimage should be able to create the image as a sparse file. I
   don't think this would be a big issue.

 - please also ask linux-fsdev.

 - bugfree :)

BTW, apparently I won't have the possibility to send my version today but
I'll do it ASAP.

Cheers,

	Szaka

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> I wrote it for disk-to-disk copy (debug) _and_ efficient streaming backup
> purpose _and_ considering my time constrains. It can stream the data
> however I'm sure it's slower than a special format but works with standard
> tools [and most importantly it took only hours to implement by some minimal
> changes and massive deletion to ntfsresize].

Well, quite.  But since I feel the need for something that can write a
stream sparse image format, I have the effort available :-).

> > If there is no separate image format then indeed you don't need a
> > restore utility.  But, if you want a stream image format suitable for
> > writing to tapes, file transfer, etc., then you have to have a decoder
> > as well as an encoder.
> 
> I do it as:
> 	ntfsimage -o - device | bzip2 ...

That's much slower than simply using a format which encodes the
sparseness.  I'm not even sure if my machine can run bzip2 at tape
streaming speed - certainly many machines won't be able to.  Also, and
I may be in a tiny minority here, but I don't like to put my backup
streams through a compression program because I think that's a bit
fragile.

> > I tend to think that sparse files are a bit of an anachronism, really.
> 
> I think sparse files are very useful in many cases but filesystem support
> is ackward for them. IMHO only XFS can provide info to userspace tools
> about the location of holes. This results, among others, extremely poor
> performances in many cases and people start to think sparse files are a bit
> of an anachronism ;)

Err, quite :-).

I'm not saying sparse files aren't sometimes a useful technique.  But
in the past, file formats which only make sense as sparse files have
turned out to be less useful because a smaller proportion of the
standard utilities work well on them.

So I think that there is room for both facilities: the optimised image
copy, and the conversion to a stream representation.

Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> On Tue, 8 Apr 2003, Ian Jackson wrote:
> > someone comes up with other uses.  ext3image anyone?)
> 
> There is e2image in e2fsprogs, I don't know if it works with ext3 or not.
> Probably.

I hadn't heard of that.  As far as I can tell from its manpage it's
not quite the same thing as I think ntfsimage is or should be - but it
could benefit from being able to write a stream format.

> >   * GNU tar's sparse file extensions - support 33-bit filesizes
> 
> 33? I've never noticed large file support wouldn't work ...

Maybe the web page I was reading was out of date.  It only had
12-character fields for the lengths and positions of holes.  Ghads,
yet more hideous complication in the tar format ?

> >   * partimage's image format - contains compression, CRCs, etc.,
> >      which I think this is the wrong layer for, and which add
> >      complexity.  (Also, I'm not 100% convinced of the quality of some
> >      of the partimage work.)
> 
> I only checked the ntfs part and I wouldn't trust it at all.

It's nice to have a second opinion who agrees.

> > So, please comment on my format, which is described below.  What form
> 
> First I should convince myself a new format make sense. I can see several
> drawbacks: more code, image can't be used with standard tools, at least two
> steps is needed to get a real image, etc but can't really see benefits.

There's really no reason why ntfsimage shouldn't be able to do both.

I hope you'll accept my program which does write a stream format into
the linux-ntfs package.  If not I'll be in a bit of a quandry: I can't
just distribute it separately because it wants to access various of
the utility code in ntfsprogs (and anyway, it conceptually ought to
live with the other tools).

Ian.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Szakacsits S. <sz...@si...>]

On Tue, 8 Apr 2003, Ian Jackson wrote:
> I don't think a sparse file is really the right answer for what I
> think ntfsimage is good for - although I can see it being useful when

sparse file + standard utils. tar, bzip2, etc.

> you want an optimised disk-to-disk copy.
>
> What I want is the ability to produce a stream representation of the
> original filesystem, eg for backup.

I wrote it for disk-to-disk copy (debug) _and_ efficient streaming backup
purpose _and_ considering my time constrains. It can stream the data
however I'm sure it's slower than a special format but works with standard
tools [and most importantly it took only hours to implement by some minimal
changes and massive deletion to ntfsresize].

> If there is no separate image format then indeed you don't need a
> restore utility.  But, if you want a stream image format suitable for
> writing to tapes, file transfer, etc., then you have to have a decoder
> as well as an encoder.

I do it as:
	ntfsimage -o - device | bzip2 ...

> I tend to think that sparse files are a bit of an anachronism, really.

I think sparse files are very useful in many cases but filesystem support
is ackward for them. IMHO only XFS can provide info to userspace tools
about the location of holes. This results, among others, extremely poor
performances in many cases and people start to think sparse files are a bit
of an anachronism ;)

> If you could send me the code you do have, it might help me at least

OK, sometime tomorrow.

	Szaka

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Andrew C. <cl...@gn...>]

On Tue, Apr 08, 2003 at 07:38:59PM +0100, Ian Jackson wrote:
>   * partimage's image format - contains compression, CRCs, etc.,
>      which I think this is the wrong layer for, and which add
>      complexity.  (Also, I'm not 100% convinced of the quality of some
>      of the partimage work.)

Ooops.  Sorry for not reading you properly.

Cheers,
Andrew

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Andrew C. <cl...@gn...>]

On Tue, Apr 08, 2003 at 07:38:59PM +0100, Ian Jackson wrote:
> I have a need for ntfsimage, which is listed as `not started' on the
> tools status page, so I decided to write it.

Doesn't partimage.sf.net have ntfs support?

Cheers,
Andrew

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Szakacsits S. <sz...@si...>]

On Tue, 8 Apr 2003, Ian Jackson wrote:

> someone comes up with other uses.  ext3image anyone?)

There is e2image in e2fsprogs, I don't know if it works with ext3 or not.
Probably.

>   * GNU tar's sparse file extensions - support 33-bit filesizes

33? I've never noticed large file support wouldn't work ...

>   * partimage's image format - contains compression, CRCs, etc.,
>      which I think this is the wrong layer for, and which add
>      complexity.  (Also, I'm not 100% convinced of the quality of some
>      of the partimage work.)

I only checked the ntfs part and I wouldn't trust it at all.

> So, please comment on my format, which is described below.  What form

First I should convince myself a new format make sense. I can see several
drawbacks: more code, image can't be used with standard tools, at least two
steps is needed to get a real image, etc but can't really see benefits.

	Szaka

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

Szakacsits Szabolcs writes ("Re: [Linux-NTFS-Dev] Implementing ntfsimage"):
> Mine saves to a sparse file either explictely given on the command line or
> outputs to stdout. It can also save only metadata. In both cases the image
> can be mounted by the NTFS driver or used by the ntfsprogs tools. With
> tar/bzip2 it can be compressed extremely well. The fs consistency also
> checked.

I don't think a sparse file is really the right answer for what I
think ntfsimage is good for - although I can see it being useful when
you want an optimised disk-to-disk copy.

What I want is the ability to produce a stream representation of the
original filesystem, eg for backup.

> > produces a file of some kind, but no restore utility.  I haven't
> 
> Is it really needed? Making the image:
> 	ntfsimage -o ntfs.img device
> restoring:
> 	ntfsimage -o device ntfs.img

If there is no separate image format then indeed you don't need a
restore utility.  But, if you want a stream image format suitable for
writing to tapes, file transfer, etc., then you have to have a decoder
as well as an encoder.

I tend to think that sparse files are a bit of an anachronism, really.
Using them makes assumptions about things that are really no business
of a userland program relying on the UN*X filesystem abstraction.
Perhaps others will disagree, and perhaps filesystem manipulation
tools have a case for messing around in these murky areas in between
layers.  But, even if a tool like your ntfsimage exists, I still think
something like my one needs to exist too (and quite likely they'll
want to share code, of course).

> I wrote already one for a while ago. I didn't sent it yet because it needs
> some cleanup for public use and nobody requested for it. On the contrary,
> people constantly ask for other things [being also worked on].

If you could send me the code you do have, it might help me at least
check that I'm doing roughly the right thing.  I've been trying to
write production-quality suitable-for-public-release code from the
start, but any additional code or information can only help.

Thanks,
Ian.

Re: [Linux-NTFS-Dev] Implementing ntfsimage

[Szakacsits S. <sz...@si...>]

On Tue, 8 Apr 2003, Ian Jackson wrote:

> I have a need for ntfsimage, which is listed as `not started' on the

I wrote already one for a while ago. I didn't sent it yet because it needs
some cleanup for public use and nobody requested for it. On the contrary,
people constantly ask for other things [being also worked on].

> * Firstly, status: I have a first cut of the imaging utility which

Mine saves to a sparse file either explictely given on the command line or
outputs to stdout. It can also save only metadata. In both cases the image
can be mounted by the NTFS driver or used by the ntfsprogs tools. With
tar/bzip2 it can be compressed extremely well. The fs consistency also
checked.

> produces a file of some kind, but no restore utility.  I haven't

Is it really needed? Making the image:

	ntfsimage -o ntfs.img device

restoring:

	ntfsimage -o device ntfs.img

     Szaka

[Linux-NTFS-Dev] Implementing ntfsimage

[Ian J. <ia...@da...>]

I have a need for ntfsimage, which is listed as `not started' on the
tools status page, so I decided to write it.  I'm posting here to let
you know to save anyone else the effort, to say where I'm at, and to
discuss a couple of design questions.

* Firstly, status: I have a first cut of the imaging utility which
produces a file of some kind, but no restore utility.  I haven't
stared hard at the image output, so I don't know whether the output of
the imager is right yet.

* I'm planning to produce two utilities: ntfsimage, which reads an
NTFS volume and spits out an image, and restoreimage, which restores
an image to a partition.  I wanted them to be separate programs
because the restoreimage functionality is not at all filesystem
dependent, so it's not clear that ntfstools is the right place for
it.  (Although it'll probably be sensible to ship it there until
someone comes up with other uses.  ext3image anyone?)

* I did some web browsing to try to find a suitable existing sparse
image format to use.  However, such a thing doesn't seem to exist.
This means I had to invent one - which is a bonus as it means I can
invent a nice simple format which is east to generate, but it also
means that I have to write the restore utility rather than just using
an existing tool.  Formats I rejected:
  * GNU tar's sparse file extensions - support 33-bit filesizes
    and holes only, which is no good for a filesystem image
  * partimage's image format - contains compression, CRCs, etc.,
     which I think this is the wrong layer for, and which add
     complexity.  (Also, I'm not 100% convinced of the quality of some
     of the partimage work.)
  * Apple have a disk image format UDIF which looks like it might have
    been a sensible standard to follow if Apple hadn't done the usual
    Apple thing and made it a secret format.
  * Ghost and its competitors don't seem to publish format specs
    either.
So, please comment on my format, which is described below.  What form
of documentation would be best for the image format specification ?
Should I write a section 5 manpage ?

* I'm slightly disturbed by the amount of option-parsing boilerplate
(and thus clone-and-hack) thats appears in the utilities in the
linux-ntfs 1.7.1 source tree.  For example, the stuff to implement the
force, help, verbose, quiet, and version options, and the code that's
needed when opterr=0.  Are there any plans to rationalise this, or
should I just follow the example of the other tools ?

* Please comment on the following option set:
    Usage: ntfsimage [options] device
        -c       --count        Print <image-total-bytes> <in-fs-clusters-used>*<cluster-size>.
        -f       --force        Use less caution
        -q       --quiet        Less output
        -v       --verbose      More output
        -V       --version      Print version number
        -h       --help         Print this help

* What copyright formalities are required, if any, for my code to make
it into the main linux-ntfs distribution ?

Thanks,
Ian.

Format description:
    at least 64 bytes of header
        offset  len
        0       16      ASCII magic number
                         "restoreimage" 0x0a 0x00 0x0d 0x1a
        16      16      random octets magic number
                         0xb5 0x0c 0xa9 0xb6 0x15 0x63 0xb5 0x18
                         0x9a 0x31 0x80 0x33 0xf1 0x14 0xf8 0x3c
        32      4       major version number, currently 0
                         readers must fail if major verion not known
        36      4       header size in bytes
                         must be multiple of cluster size (see below)
        40      4       minor version number, currently 0
                         readers must continue if major version not known
        44      4       reserved
        48      8       cluster size in bytes
        56      8       number of clusters
        64      ?       remainder of header is reserved
    zero or more cluster groups where a cluster group is:
        one cluster's worth of bitmap specifying whether
         each cluster is present (1=present, 0=absent), where:
         first cluster usage is ms bit of first byte
        the actual cluster data for the specified clusters

    everything is in network byte order (big endian)

    reserved fields are all-bits-zero on write, ignored on read (in v0.0)

    The last cluster group may be partial, if the number of clusters
    is not a multiple of the number of bits in a cluster.  In this
    case the leftover clusters past the end are _not_ marked as
    present, nor present

Re: [Linux-NTFS-Dev] NTFS Support for RedHat 9 (Shrike)

[Richard R. <nt...@fl...>]

Hi Szaka,

> > I've just released NTFS RPMs for RedHat 9 (Shrike).
> 
> BTW, what's the point distributing the old, broken and unmaintained NTFS
> driver instead of the new, correct and maintained one what Mandrake, Gentoo
> and others use?

Because I *cannot* compile the new driver into their kernel and make it
work.

To backport the new NTFS driver to 2.4 involves changing code outside
the ntfs code dir.   If I were to do this the module would not with
the redhat kernel.

So most users have the choice: the old driver or no support at all.

>    Red Hat said technical and legal issues currently prevent NTFS support.

I've talked to several RedHat technical folk.  They don't see any
problems with using NTFS.  Without breaking into a long rant, there's
not much else I can say.

Cheers,
  FlatCap (Rich)
  nt...@fl...

WWW: http://linux-ntfs.sourceforge.net
IRC: #ntfs on irc.freenode.net 

Re: [Linux-NTFS-Dev] NFS export

[Tom W. <tom...@nt...>]

> > I have to ask, as this is a showstopper for me at the moment, does the
> > new NTFS driver allow NFS exports yet?
>
> No. Does the old? That could be an argument for the old driver although I
> can't see it anywhere mentioned. I also didn't see much request for this
> [yours is the first] but seems reasonable and the TODO mentions it. Anton
> or Richard should know more about the issue.

Yes, it does.

> AFAIK you're right. Everybody has very limited free time and there are
> several frequently asked features to be worked on.

Sure, I understand.

> I don't know how difficult would be to implement NFS export, maybe I'll
> take a look if nobody comments.

I fancy taking a look myself, but I don't know how far I'll get...

Thanks,

Tom

[Linux-NTFS-Dev] NFS export

[Szakacsits S. <sz...@si...>]

On Fri, 4 Apr 2003, Tom Ward wrote:

> > BTW, what's the point distributing the old, broken and unmaintained NTFS
> > driver instead of the new, correct and maintained one what Mandrake, Gentoo
> > and others use?
>
> I have to ask, as this is a showstopper for me at the moment, does the new
> NTFS driver allow NFS exports yet?

No. Does the old? That could be an argument for the old driver although I
can't see it anywhere mentioned. I also didn't see much request for this
[yours is the first] but seems reasonable and the TODO mentions it. Anton
or Richard should know more about the issue.

> I remember reading that there was some technical problem, but can't
> remember the exact details, I don't suppose anyone's been working on
> this...

AFAIK you're right. Everybody has very limited free time and there are
several frequently asked features to be worked on.

I don't know how difficult would be to implement NFS export, maybe I'll
take a look if nobody comments.

	Szaka

Re: [Linux-NTFS-Dev] NTFS Support for RedHat 9 (Shrike)

[Szakacsits S. <sz...@si...>]

On 4 Apr 2003, Richard Russon wrote:

> I've just released NTFS RPMs for RedHat 9 (Shrike).

BTW, what's the point distributing the old, broken and unmaintained NTFS
driver instead of the new, correct and maintained one what Mandrake, Gentoo
and others use?

>   http://linux-ntfs.sourceforge.net/info/redhat.html

There might be also some addition to this statement on the above page:

  "Due to the uncertain legal status of using the NTFS driver, RedHat
   have chosen to leave the driver out of their kernels. "

From a recent ZDNet article at

	http://zdnet.com.com/2100-1104-994429.html

that was relayed also in CNET, BusinessWeek, etc:

   Red Hat said technical and legal issues currently prevent NTFS support.
   "We need to verify that Microsoft doesn't hold any patents in the area
   we might be violating. We're still doing investigation in that area,"
   said Matt Wilson, manager of Red Hat's base operating system, in an
   interview Tuesday. "Realistically, given the level of documentation
   Microsoft gives for NTFS, I don't know that I'd trust anyone's
   implementation except for Microsoft's," he added.

No comment. However since I'm also asked repeatedly about this issue, I
consider to quote and comment it on the ntfsresize FAQ page. I've asked
Matt Wilson if this would be ok but didn't get reply.

	Szaka

[Linux-NTFS-Dev] NTFS Support for RedHat 9 (Shrike)

[Richard R. <nt...@fl...>]

Hi All,

I've just released NTFS RPMs for RedHat 9 (Shrike).
I haven't had a chance to test them, so be careful.

  http://linux-ntfs.sourceforge.net/info/redhat.html

(detailed instructions included)

Cheers,
  FlatCap (Rich)
  nt...@fl...

WWW: http://linux-ntfs.sourceforge.net
IRC: #ntfs on irc.freenode.net 

Re: [Linux-NTFS-Dev] fdisk can't directly resize partitions

[Szakacsits S. <sz...@si...>]

On 25 Mar 2003, Jeremy Jackson wrote:

> I'd like to mention that EVMS combines the FS resize and partition
> resize, and has CLI, text, and GUI user interface.  It would be nice to
> have an NTFS module (and an FAT/FAT32 one) as this would make a real

Thanks, EVMS is very nice however I think the user base of EVMS and those
potentially 250+ million who might need NTFS resizing now (people want to
try out/migrate to Linux or setup a dual boot machine) are pretty distinct.

The EVMS would be an overkill for the later one, what they need is what
Mandrake did recently: select "Use unused space on Windwos to install
Linux" then click, use a slidebar to choose the new size, click, that's
all. Maybe even this is too annoying/confusing for them. Well, actually it
is.

> nice swiss army knife for installers to use, since it already supports
> ext2/3, reiser, swap, xfs.

I'm not aware any installer would use EVMS, maximum supports it. But FAT is
a "requirement" since the last 10+ years, just like NTFS became in the
recent years.

QTParted also integrated ntfsresize and I've heard rumors it might be
included in some distros running from CD or otherwise and/or Mandrake's
DiskDrake.

Of course I don't exclude your suggestion will make much more sense in the
future but at present I see it a very low priority and there are a lot much
more important ones.

Thanks,

	Szaka

Re: [Linux-NTFS-Dev] HostID field in PrivHead

[Richard R. <ld...@fl...>]

Hi Mark,

> I'm looking to be able to split mirrors and manage block level
> snapshots of dynamic disks.

Bloody hell, that's quite ambitious.

> I have a pretty good understanding of the LDM
> However, one field I'm sure I need to be able to modify
> is the HostID field in the PrivHead structure.

> What I can't work out is where this guid comes from.
> Can anybody tell me how Windows assigns that guid

I would guess it's just generated (there are standard algorithms for
creating guids).

> and where I might be able to find it elsewhere on a Windows system ?

Hmm, tricky.  Even though the disks are self-contained, I suppose it
must exist in Windows so it can tell which disks are its own, but I
haven't seen it.  I haven't really looked.  At worst case, you'll have
to import the "foreign" disks.

Is this program going to be GPLd?

Good luck,
  FlatCap (Rich)
  ld...@fl...

Re: [Linux-NTFS-Dev] fdisk can't directly resize partitions

[Richard R. <ld...@fl...>]

Hi Jeremy

> I'd like to mention EVMS
> It might accommodate LDM disk stuff easier than fdisk as well.

Yes, indeed.  But that's not likely to happen any time soon.  I wrote
the LDM driver for fun, but the feedback has been underwhelming.
If/when LDM use becomes more widespread, I'll start writing tools.

In the meantime, if anyone wants to create tools, I've documented
everything I've learnt from LDM (by reverse-engineering) and I'm happy
to teach.

  http://linux-ntfs.sf.net/ldm

Cheers,
  FlatCap (Rich)
  ld...@fl...