linux-ntfs-dev — Development discussions and bug reports.

Re: [Linux-NTFS-Dev] supported ntfs versions

[Szakacsits S. <sz...@si...>]

On Thu, 11 Jul 2002, Anton Altaparmakov wrote:

> > > BOOL ntfs_is_version_supported(ntfs_volume *vol)
> >
> >Name is ok (first I thought the same) but please no BOOL. It just
> >conflicts with everybody's own BOOL definition, int is better.
>
> We already have BOOL defined so as soon as you include an ntfs header you
> would have the conflict.

Yes but it will be easier to clean up later ;) Anyway vol must be
checked for NULL so BOOL isn't enough anymore.

	Szaka

[Linux-NTFS-Dev] Re: NTFS v1.1 -> v1.2 (was: Re: ntfsresize.c)

[Anton A. <ai...@ca...>]

At 15:42 11/07/02, Szakacsits Szabolcs wrote:
>On Thu, 11 Jul 2002, Anton Altaparmakov wrote:
>
> > I found the same in Windows NT 3.51, it formatted to older NTFS version and
> > then upgraded it to 1.2.
>
>Just curious, what version was the older?

I can't remember! Otherwise I would have written it... It was definitely 
1.something, but I can't remember if it was 1.0 or 1.1. It has been too 
long that I played with it. If I only could find the vmware image I had 
made with NT3.51 I could check but I have no idea where it is... Too many 
computers, too many harddrives... )-:

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] supported ntfs versions

[Anton A. <ai...@ca...>]

At 15:39 11/07/02, Szakacsits Szabolcs wrote:
>On Thu, 11 Jul 2002, Anton Altaparmakov wrote:
> > Yeah, Please adapt your function to accept all 2.x versions, too. (And
> > always treat them in your mind as if they were 3.x, but don't worry too
> > much as the probability of anyone ever running one of our utilities on a
> > system with NTFS 2.x is so tiny it is not worth thinking about!)
>
>Could you please elaborate what you mean? There is NTFS 2.x at all?

Yes. But only developers have it, and I doubt anyone in their sane mind 
would be using it... However everything will just work if you assume it is 
the same as ntfs 3.0. Ok? [You will encounter a few extra attributes which 
you can just ignore]

>Because if we don't know what it is, better to giveup (or user could
>use a "-f" force option) then hoping it will work.
>
> > >int ntfs_version_supported(u8 major, u8 minor)
> >
> > how about
> >
> > BOOL ntfs_is_version_supported(ntfs_volume *vol)
>
>Name is ok (first I thought the same) but please no BOOL. It just
>conflicts with everybody's own BOOL definition, int is better.

We already have BOOL defined so as soon as you include an ntfs header you 
would have the conflict. But if you insist leave it int for this function...

gcc-3.x (possibly not early gcc-3.x though) already have their own bool 
type in accordance with C99 I believe (or whatever standard it was/will be 
that defined/s it, I am not sure), so as soon as that becomes ubiquitously 
available we will switch to using that everywhere. I am also thinking of 
starting to rename everything now/in the near future and just do some magic 
to check if the compiler supports it, and if not typedef our own... But 
that is not too high priority. But thinking about it it may be a good idea 
to do it before releasing linux-ntfs-2.0.0...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Re: NTFS v1.1 -> v1.2 (was: Re: ntfsresize.c)

[Szakacsits S. <sz...@si...>]

On Thu, 11 Jul 2002, Anton Altaparmakov wrote:

> I found the same in Windows NT 3.51, it formatted to older NTFS version and
> then upgraded it to 1.2.

Just curious, what version was the older?

	Szaka

Re: [Linux-NTFS-Dev] supported ntfs versions

[Szakacsits S. <sz...@si...>]

On Thu, 11 Jul 2002, Anton Altaparmakov wrote:
> Yeah, Please adapt your function to accept all 2.x versions, too. (And
> always treat them in your mind as if they were 3.x, but don't worry too
> much as the probability of anyone ever running one of our utilities on a
> system with NTFS 2.x is so tiny it is not worth thinking about!)

Could you please elaborate what you mean? There is NTFS 2.x at all?
Because if we don't know what it is, better to giveup (or user could
use a "-f" force option) then hoping it will work.

> >int ntfs_version_supported(u8 major, u8 minor)
>
> how about
>
> BOOL ntfs_is_version_supported(ntfs_volume *vol)

Name is ok (first I thought the same) but please no BOOL. It just
conflicts with everybody's own BOOL definition, int is better.

	Szaka

[Linux-NTFS-Dev] Re: NTFS v1.1 -> v1.2 (was: Re: ntfsresize.c)

[Anton A. <ai...@ca...>]

At 15:17 11/07/02, Szakacsits Szabolcs wrote:
>On Tue, 9 Jul 2002, Szakacsits Szabolcs wrote:
> > On Tue, 9 Jul 2002, Anton Altaparmakov wrote:
> > > >BTW, to my big surprise the above mentioned NT4 SP3 build 1381
> > > >formatted NTFS volumes as NTFS v1.1. Both the DOS command and the Disk
> > > >Administrator. ntfsdoc mentions everywhere 1.2 and actually the
> > > >default/system drive is also v1.2 :-o
> > >
> > > It probably formats to 1.1 and then upgrades to 1.2 on mount.
> >
> > Hmmm, you mean I should reboot? AFAIR it remained 1.1 but I'll check
> > this explicitely.
>
>Yes, after reboot it had been upgraded to v1.2.

Excellent. (-: So it is just the formatter in SP3 which is old. The driver 
still uses 1.2.

I found the same in Windows NT 3.51, it formatted to older NTFS version and 
then upgraded it to 1.2.

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] supported ntfs versions

[Anton A. <ai...@ca...>]

At 14:18 11/07/02, Szakacsits Szabolcs wrote:
>I plan to commit the below [with comments] to volume.[ch] for general
>use and also convert ntfsfix.c to use this.

cool, see below for a few comments.

>However ntfsfix also accepts any version 2.X. Does that really needed?

Yeah, Please adapt your function to accept all 2.x versions, too. (And 
always treat them in your mind as if they were 3.x, but don't worry too 
much as the probability of anyone ever running one of our utilities on a 
system with NTFS 2.x is so tiny it is not worth thinking about!)

>I also plan to move log journal reset and

Either logfile.[hc] or volume.[hc], not sure what is appropriate... you 
decide...

Please, only one argument: ntfs_volume *vol, function does rest. Also, 
please don't use the code from ntfsfix and instead use ntfs_open_inode() + 
ntfs_open_attr() [you now know the size] + loop using ntfs_attr_pwrite() 
with a 4kiB or so buffer which is memset to -1 over the length of the 
attribute + close the attr and the inode. I have been meaning to do this 
for ages but never got round to it and ntfs_attr_pwrite() was not advanced 
enough back then, but it is now... Thanks! (-:

>marking volume dirty to the library.

volume.[hc]... For the volume flags, perhaps instead of having a function 
"ntfs_set_volume_dirty()" or whatever, it would be better to have a 
function "ntfs_set_volume_flags(ntfs_volume *vol, VOLUME_FLAGS flags)" and 
let the function do everything... You need to preserve old flags so when 
using the function, you would need to call it with (vol, vol->flags | 
VOLUME_IS_DIRTY).

Of course you could then have a static inline ntfs_set_volume_dirty(vol) 
function in volume.h just wrapping doing a simple "return 
ntfs_set_volume_flags(vol, vol->flags|VOLUME_IS_DIRTY);"...

Somewhere, we will need to set the VOLUME_MOUNTED_ON_NT4 flag if ntfs 
major > 1. Not sure when this should happen. Possibly at mount time if not 
mounted MS_RDONLY? It would be a convenient place as we already read the 
flags, so it is easy to or this bit and write the mft record straight 
back... Of course we wrap this with an if ntfs major > 1...

Further I think the journal emptying should actually happen at ntfs_mount() 
time, perhaps we could have a mount option to prevent this from happening? 
Although I don't see why would want to prevent it, so probably no need to 
bother. Again, we would only do this if not mountind MS_RDONLY.

How does that sound?

>Any objection?

No. (-:

>#define NTFS_V1_1(major, minor) ((major) == 1 && (minor) == 1)
>#define NTFS_V1_2(major, minor) ((major) == 1 && (minor) == 2)
>#define NTFS_V3_0(major, minor) ((major) == 3 && (minor) == 0)
>#define NTFS_V3_1(major, minor) ((major) == 3 && (minor) == 1)
>
>int ntfs_version_supported(u8 major, u8 minor)

how about

BOOL ntfs_is_version_supported(ntfs_volume *vol)

and return TRUE or FALSE... no need for errno... fits in with 
is_ntfs_bootsector() [or whatever it is called, can't remember exactly]

>{
>
>         if (NTFS_V1_1(major, minor) || NTFS_V1_2(major, minor))
>                 return 0;
>
>         if (NTFS_V3_0(major, minor) || NTFS_V3_1(major, minor))
>                 return 0;
>
>         errno = ENOTSUP;
>         return -1;
>}

Looks fine. I don't really mind if you go your way or with my 
suggestions... The only thing I would like to definitely be added is 
NTFS_V2_x() to be a valid version. Thanks.

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Re: NTFS v1.1 -> v1.2 (was: Re: ntfsresize.c)

[Szakacsits S. <sz...@si...>]

On Tue, 9 Jul 2002, Szakacsits Szabolcs wrote:
> On Tue, 9 Jul 2002, Anton Altaparmakov wrote:
> > >BTW, to my big surprise the above mentioned NT4 SP3 build 1381
> > >formatted NTFS volumes as NTFS v1.1. Both the DOS command and the Disk
> > >Administrator. ntfsdoc mentions everywhere 1.2 and actually the
> > >default/system drive is also v1.2 :-o
> >
> > It probably formats to 1.1 and then upgrades to 1.2 on mount.
>
> Hmmm, you mean I should reboot? AFAIR it remained 1.1 but I'll check
> this explicitely.

Yes, after reboot it had been upgraded to v1.2.

	Szaka

[Linux-NTFS-Dev] supported ntfs versions

[Szakacsits S. <sz...@si...>]

Hi,

I plan to commit the below [with comments] to volume.[ch] for general
use and also convert ntfsfix.c to use this. However ntfsfix also
accepts any version 2.X. Does that really needed? I also plan to move
log journal reset and marking volume dirty to the library. Any
objection? Thanks,

	Szaka

#define NTFS_V1_1(major, minor) ((major) == 1 && (minor) == 1)
#define NTFS_V1_2(major, minor) ((major) == 1 && (minor) == 2)
#define NTFS_V3_0(major, minor) ((major) == 3 && (minor) == 0)
#define NTFS_V3_1(major, minor) ((major) == 3 && (minor) == 1)

int ntfs_version_supported(u8 major, u8 minor)
{

	if (NTFS_V1_1(major, minor) || NTFS_V1_2(major, minor))
		return 0;

	if (NTFS_V3_0(major, minor) || NTFS_V3_1(major, minor))
		return 0;

	errno = ENOTSUP;
	return -1;
}


-- 

Re: [Linux-NTFS-Dev] Header file dependencies

[Anton A. <ai...@ca...>]

Hi Rich,

[snip cewl patch]

May I assume that you have tested compilation both for ./configure and 
./configure --enable-debug? If yes and there are no warnings then by all 
means, commit it. I only glanced over it but it looks good.

Cheers,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] Header file dependencies

[Richard R. <nt...@fl...>]

Hi Anton,

I've started looking at trying to break apart the evil
header dependencies.  Because everything's typedef'd it
makes it a rather nasty job.

This is just the first stage.  I've moved more of the basic
types into types.h and removed its dependency on layout.h
Also, I've moved the structs out of types.h.

If you're not completely horrified, I'll commit this and
keep going.  Lots more ideas :-)

 include/attrib.h         |   32 ++++++++++++++++++++
 include/bootsect.h       |    1 
 include/inode.h          |    4 ++
 include/layout.h         |   26 ----------------
 include/mft.h            |    1 
 include/mst.h            |    1 
 include/runlist.h        |   24 ++++++++++++++-
 include/types.h          |   74 +++++++++++++----------------------------------
 include/unistr.h         |    1 
 include/volume.h         |    4 ++
 ntfstools/ntfsundelete.c |    1 
 ntfstools/ntfsundelete.h |    2 -
 ntfstools/sd.c           |    1 
 13 files changed, 91 insertions(+), 81 deletions(-)

Cheers,
    FlatCap (Rich)
    nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net

Re: [Linux-NTFS-Dev] RFC: Changes to linux ntfs tools

[Richard R. <nt...@fl...>]

Hi Anton,

> >attrib.c is getting much too big.  it's more than triple the size of
> >anything else.  I want to move the the runlist code into runlist.c
> >(about a third of the code).
> 
> Great idea. I like the name. I take it you volunteer to do the split? (-; 
> btw. This is so obvious that it is silly to write it so I appologise in 
> advance, but please use a runlist.h header as well...

OK, the first stage is done.

> I would vote for ./configure --enable-debug -> print loads of stuff to 
> stderr. ./configure --disable-debug -> print to stderr only in really 
> serious cases like the segfault about to be caused.

OK

> >Testing.  This word is the bane of all programmers.  I hate it too, but
> >I understand the need for it.  We cannot anticipate every use of the
> >library, but we should have some form of unit testing.  The library is
> >still undergoing a lot of changes and it's hard to verify that nothing
> >has broken.  Any suggestions or guidelines would be welcome.
> 
> Sorry no suggestions or guidelines, but a regression test suite where we 
> just need to type "make test" would be very nice indeed...

I'll start thinking what we can automate.

> >Back to attrib.c.  It's a very complicated piece of code.
> Yes. It is code to edit a database... That is a complicated thing.
> 
> It depends what exactly you want to simplify. You had mentioned to me 
> before that you don't like the frequent offset additions like this:
> 
> FILE_NAME *fn;
> ATTR_RECORD *attr;
> 
> fn = (FILE_NAME*)((u8*)attr + le16_to_cpu(attr->value_offset));
> 
> I agree those are quite horrible and very long. Considering how they are 
> used we can make a generic macro, (please suggest a name!), lets call it 
> GF() for now just to show what it could look like.
> 
> fn = GF(attr, value_offset);
> #define GF(a, b) ...
> #define le_to_cpu(x) ...

That's the sort of thing I was toying with.  Terrifying, but it covers
adjusting the le_to_cpu sizes according to the struct member.  What
hadn't occurred to me, scribbling on bits of paper, was that because
it's all const, the whole thing would optimise away.  Nice.

> The prize question is what to call the GF() macro!

Naming's always the hardest bit :-)

GM  = get member 
GV  = get value
OFF = offset
PTR = pointer

Some combination of the above?

Cheers,
    FlatCap (Rich)
    nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net

Re: [Linux-NTFS-Dev] ntfs-tng question #1 :)

[Anton A. <ai...@ca...>]

At 12:14 11/07/02, Richard Russon wrote:
> > aops.c, line 250-257 (from cvs  i just checked out)

eeek! /me thinks the cvs repository needs to be destroyed!!!

>Argh!  The cvs repository is obsolete.  The latest code is kept
>in a BitKeeper repository.  We should have made this a bit clearer.
>
>   http://linux-ntfs.bkbits.net:8080/ntfs-tng-2.5

Yes, and there is the development repository as well:

   http://linux-ntfs.bkbits.net:8080/ntfs-devel

Note the two are usually very close together but the devel tree tends to 
receive things first. The ntfs-tng-2.5 tree only gets things pushed to it 
when I submit to Linus for inclusion. (He actually pulls from the 
ntfs-tng-2.5 tree so it has to always correspond to the patches I send to 
LKML.)

>Fortunately the piece of code you're referring to exists in both
>versions.  aops.c, line 388 (ntfs_readpage).
>
> > Not that this is a likely occurence or even legal, but to make sure I'm
> > sane.....
>
> > We're making the assumption here that a resident attribute cannot be of
> > length > PAGE_CACHE_SIZE, correct?  Because it seems to me that this code
> > does not take into account any offsets etc, and just returns the beginning
> > of the attribute no matter what... am I correct here?
>
>$Boot tells us how big the MFT records are, implying the value could
>vary, but we've only ever seen 1024 byte records.  For now the
>attributes can't be even close to a page size.

Apparently early versions of NTFS used 2048 byte MFT records. But even 
then, the smallest PAGE_CACHE_SIZE is 4096 bytes and the kernel throughout 
assumes this to be a fact, i.e. any architecture with PAGE_CACHE_SIZE < 
4096 would not actually work with the Linux kernel. So our code assumes the 
same.

Note that in super.c, at mount time, we actually do checks and abort if mft 
records are bigger (I think, and if we are not we definitely should be!)...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] ntfs-tng question #1 :)

[Richard R. <nt...@fl...>]

Hi Joe,

> aops.c, line 250-257 (from cvs  i just checked out)

Argh!  The cvs repository is obsolete.  The latest code is kept
in a BitKeeper repository.  We should have made this a bit clearer.

  http://linux-ntfs.bkbits.net:8080/ntfs-tng-2.5

Fortunately the piece of code you're referring to exists in both
versions.  aops.c, line 388 (ntfs_readpage).

> Not that this is a likely occurence or even legal, but to make sure I'm
> sane.....

> We're making the assumption here that a resident attribute cannot be of
> length > PAGE_CACHE_SIZE, correct?  Because it seems to me that this code
> does not take into account any offsets etc, and just returns the beginning
> of the attribute no matter what... am I correct here?

$Boot tells us how big the MFT records are, implying the value could
vary, but we've only ever seen 1024 byte records.  For now the
attributes can't be even close to a page size.

Cheers,
    FlatCap (Rich)
    nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net

[Linux-NTFS-Dev] ntfs-tng question #1 :)

[Joseph M. <jma...@st...>]

aops.c, line 250-257 (from cvs  i just checked out)

Not that this is a likely occurence or even legal, but to make sure I'm
sane.....

We're making the assumption here that a resident attribute cannot be of
length > PAGE_CACHE_SIZE, correct?  Because it seems to me that this code
does not take into account any offsets etc, and just returns the beginning
of the attribute no matter what... am I correct here?

-joe

Re: [Linux-NTFS-Dev] RFC: Changes to linux ntfs tools

[Anton A. <ai...@ca...>]

At 12:55 09/07/02, Richard Russon wrote:
>There hasn't been a release of the NTFS tools for some time and Anton's
>mentioned that he'd like to push for a 2.0 release some time soon.

Yep. Although I should add here that I am going to go via a beta phase, 
i.e. release 1.9.99beta or simillar first and if noone complains release 
2.0.0 a few weeks later. Also before releasing the beta, I want the code to 
be in freeze for a few weeks (at least the to be released code - other 
utilities can continue development; to be released code can continue 
develepment in a different CVS branch). During the freeze I envision 
testing of the utilities to happen as well as possibly writing test 
utilities for the library.

Please take my comments below as RFC, too. I am just saying what I think... 
As usual I do so in a very long winded way and this post is now getting 
close to winning the longest email ever prize. (-;

>I've got some ideas for changes, that I think we ought to do before we
>make a major release.  Some are trivial, some will require a bit of thought.

Cool. I was hoping to get comments from as many people as possible so we 
don't need another rewrite and a 3.0 release in a few months... (-;

>attrib.c is getting much too big.  it's more than triple the size of
>anything else.  I want to move the the runlist code into runlist.c
>(about a third of the code).

Great idea. I like the name. I take it you volunteer to do the split? (-; 
btw. This is so obvious that it is silly to write it so I appologise in 
advance, but please use a runlist.h header as well...

>Anton's done a great job of sorting out the APIs,

Thanks. (-:

>but there are a few places where the function naming isn't consistant.

Yes, I know. I have noticed that, too. It kind of just happened is the 
explanation. I agree we need to work on that.

>  e.g. ntfs_lookup_attr -> ntfs_attr_lookup, etc.  It looks a bit reverse 
> Polish, but the others are named that way.

Yes, I like that. First ntfs_, second the structure we operate on (here 
attr_), then the operation name.

I would like to see that applied to all of the API functions. (Please only 
ignore the remnants of the old library in attrib.c, i.e. 
get_attrbibute_value and set_attribute_value and there were some others I 
think, they are destined for /dev/null as soon as working replacements 
exist and are in use throughout, I think we may have the replacements 
already, just all users will need updating...)

>A few C things:  Do we need to declare basic types as const in function
>declarations?  e.g. function (const int i)  The variable is passed by
>value, so it doesn't matter to the outside.  Can any compilers do
>anything extra knowing the value is const?

gcc-2.x doesn't take advantage of this (so I am told). Don't know about 
gcc-3.x. I would like to have const everywhere possible in order to 
emphasize the "IN" parameter type but it is not strictly necessary. Do you 
have any particular reason not to want to put const? - I only don't put 
const when the compiler complains about it and/or when I want to recycle a 
variable...

The only thing gcc uses const for is to place static/global variables in 
read-only memory so they physically cannot be written to at all which is nice.

>Do we _need_ to use "register" anywhere?

gcc-2.x completely ignores "register". Other compilers don't ignore it. We 
don't strictly need to use it, but please keep the existing ones. There are 
only 4 (or 5?) present and they are located in very specially chosen places 
based on assembler level code optimizations.

>malloc/calloc/realloc/free.  Can I rename these to ntfs_*?  Then in
>debug we can easily intercept them and perform some memory tracking.

I would prefer not to. If we want to do something with memory we should do 
it properly with ElectricFence or simillar tools. They are way more 
powerful than we could ever make them without major efforts on our part 
which would be a waste of our time IMO.

>Pointers.  At the moment, we have control of all the source, but we have
>written a library.  We need to check all pointers given to us from the
>outside.

Correct. I believe all externally accessible functions are checked. 
Anywhere where they are not, I have missed something... I don't think there 
will be too many instances where we need to modify this. There may be some 
older functions but certainly all the new functions do all checking 
possible. I have been paying a lot of attention to make sure of that. Note 
in some places null checks do not exist but that is because the parameter 
is allowed to be null...

Note I don't want to checking (beyond catching our own bugs) to 
INTERNAL-flagged functions. The functions calling them should already have 
done all checking necessary.

>What form should the checking take?  if (!ptr) return error;
>or perhaps BUG_ON(!ptr);

IMO, a library should never, ever, do things like BUG(). A userspace 
library or utility that crashes like that is a Bad Thing(TM).

if (!ptr) {
         // do any necessary cleanup
         errno = EINVAL;
         return -1;
}

is the way to go for a library, i.e. always report errors via errno and 
returning -1. Obviously EINVAL is only for generic parameters being wrong. 
I always refernce man errno and check what the most appropriate error code 
for the current sitution is. e.g. file descriptor int f, if !f errno = EBADF...

Obviously there are some functions where we can't use -1 as an error return 
value. Special arrangements are required there. Examples include boolean 
functions which will return non-zero (true) on success.

>We need to document the comment blocks in the coding style

Would be a good idea.

>Error handling.  At the moment it's completely unstructured.

That is not quite true...

>On an error do we return 0, -1?

Depends on the function. Usually -1.

>Do we set errno?

For all functions with return -1, definitely. For others, it depends, but 
usually yes.

>When should the library display any output?

Good question. Ideally I would like to have the ability to compile or not 
compile debug output as it is now. In addition, I would like to be able to 
enable/disable error output via an option of some kind. (No I don't know 
how that is going to happen and if at all)

In general I think a library should not output messages at all. It should 
return an error and the program using the library should decide what it 
would like to do about this, perhaps print an error and perhaps not. But 
there maybe select cases where a message is on order. One example is where 
I am causing the library to segfault in the run list merging code if we run 
out of memory during a critical step at the end. There we want to tell the 
user we segfaulted on purpose or we will just get a bug report... 
Admittedly that induced segfault is awful. But it is only there until I (or 
someone else!) figure out how to be able to avoid the memory allocations 
too late in the process. I am very open to suggestions there...

>Minor errors, serious errors, never?  We need to
>adopt one approach and document it.

I would vote for ./configure --enable-debug -> print loads of stuff to 
stderr. ./configure --disable-debugt -> print to stderr only in really 
serious cases like the segfault about to be caused.

>Testing.  This word is the bane of all programmers.  I hate it too, but
>I understand the need for it.  We cannot anticipate every use of the
>library, but we should have some form of unit testing.  The library is
>still undergoing a lot of changes and it's hard to verify that nothing
>has broken.  Any suggestions or guidelines would be welcome.

Sorry no suggestions or guidelines, but a regression test suite where we 
just need to type "make test" would be very nice indeed...

>Back to attrib.c.  It's a very complicated piece of code.

Yes. It is code to edit a database... That is a complicated thing.

>On top of this it's obfuscated by the need for le16_to_cpu() function calls
>everywhere.

Well it is little endian... Can't do anything about it. We could shorten 
the names to something like LC() nad CL() but I don't want to put automatic 
type checking in there because I often use the macros to perform type 
conversion for me so I don't have to do additional casts. This would break 
when using LC()/CL()...

>The code needs simplifying, but how?  I've considered a few
>macro solutions, but they tended to be fairly specific.

It depends what exactly you want to simplify. You had mentioned to me 
before that you don't like the frequent offset additions like this:

FILE_NAME *fn;
ATTR_RECORD *attr;

fn = (FILE_NAME*)((u8*)attr + le16_to_cpu(attr->value_offset));

I agree those are quite horrible and very long. Considering how they are 
used we can make a generic macro, (please suggest a name!), lets call it 
GF() for now just to show what it could look like.

We would replace the above code with:

fn = GF(attr, value_offset);

And the macro would look like something like this (untested!):

#define GF(a, b)                                        \
({                                                      \
         (void *)((u8*)(a) + le_to_cpu((a)->(b)));       \
})

And again untested:

#define le_to_cpu(x)                    \
{(                                      \
         typeof(x) _x;                   \
         if (sizeof(x) == 1)             \
                 _x = (x);               \
         else if (sizeof(x) == 2)        \
                 _x = le16_to_cpu(x);    \
         else if (sizeof(x) == 4)        \
                 _x = le32_to_cpu(x);    \
         else if (sizeof(x) == 8)        \
                 _x = le64_to_cpu(x);    \
         _x;                             \
)}

In theory this should not only work, but it should also be completely 
optimised away by the compiler so that the remaining assembler code is 
identical to the original big, bad, and ugly line. (-;

The prize question is what to call the GF() macro! The function is clear, 
to get to a memory location pointed to by a little endian field in a 
structure. But I can't think of a name for it...

If anyone can come up with a sensible name we can test it and assuming it 
does really work both with gcc-2.96 and gcc-3.something I would be very 
happy to see the macro added to types.h or support.h or something and all 
that ugly code replaced...

Can you give examples of what else you would like to see simplified? - 
That's the only one I remember you complaining about. (-;

Cheers,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

[Linux-NTFS-Dev] RFC: Changes to linux ntfs tools

[Richard R. <nt...@fl...>]

Hi all,

There hasn't been a release of the NTFS tools for some time and Anton's
mentioned that he'd like to push for a 2.0 release some time soon.

I've got some ideas for changes, that I think we ought to do before we
make a major release.  Some are trivial, some will require a bit of thought.


I'm just looking for ideas, at the moment, not volunteers,
so please feel free to comment on anything.


attrib.c is getting much too big.  it's more than triple the size of
anything else.  I want to move the the runlist code into runlist.c
(about a third of the code).

Anton's done a great job of sorting out the APIs, but there are a few
places where the function naming isn't consistant. e.g. ntfs_lookup_attr
-> ntfs_attr_lookup, etc.  It looks a bit reverse Polish, but the others
are named that way.

A few C things:  Do we need to declare basic types as const in function
declarations?  e.g. function (const int i)  The variable is passed by
value, so it doesn't matter to the outside.  Can any compilers do
anything extra knowing the value is const?

Do we _need_ to use "register" anywhere?

malloc/calloc/realloc/free.  Can I rename these to ntfs_*?  Then in
debug we can easily intercept them and perform some memory tracking.

Pointers.  At the moment, we have control of all the source, but we have
written a library.  We need to check all pointers given to us from the
outside.  What form should the checking take?  if (!ptr) return error;
or perhaps BUG_ON(!ptr);

Comments.  Most functions now have a comment block preceding them.  This
also allows vim to perform folding.  I've tried using some autodoc
programs, but with limited success.  Does anyone know one of these
programs well enough to help us?

We need to document the comment blocks in the coding style

Error handling.  At the moment it's completely unstructured.  On an
error do we return 0, -1?  Do we set errno?  When should the library
display any output?  Minor errors, serious errors, never?  We need to
adopt one approach and document it.

Testing.  This word is the bane of all programmers.  I hate it too, but
I understand the need for it.  We cannot anticipate every use of the
library, but we should have some form of unit testing.  The library is
still undergoing a lot of changes and it's hard to verify that nothing
has broken.  Any suggestions or guidelines would be welcome.

Back to attrib.c.  It's a very complicated piece of code.  On top of
this it's obfuscated by the need for le16_to_cpu() function calls
everywhere.  The code needs simplifying, but how?  I've considered a few
macro solutions, but they tended to be fairly specific.


Cheers,
    FlatCap (Rich)
    nt...@fl...

WWW: http://linux-ntfs.sf.net
IRC: #ntfs on irc.openprojects.net

Re: [Linux-NTFS-Dev] Re: NTFS resize

[Andrew C. <cl...@gn...>]

On Sun, Jul 07, 2002 at 03:24:23PM +0100, Anton Altaparmakov wrote:
> At 14:25 07/07/02, Andrew Clausen wrote:
> >On Sat, Jul 06, 2002 at 06:07:56PM +0200, Szakacsits Szabolcs wrote:
> >> On Sat, 6 Jul 2002, Andrew Clausen wrote:
> >Also, I don't believe defragging requires any less knowledge of
> >the FS than reconstruction.  Aren't all references in NTFS done
> >via attributes?
> 
> Not knowledge but reconstructing an FS from a damaged state

Who mentioned anything about "damaged state"?
 
> >What about CHS/LBA addressing issues?  It's often convenient to have
> >a small boot partition at the start, etc.
> 
> No, that has not existed as a restriction in neither Windows nor Linux as 
> of a few years now. Both OS quite happily will boot from arbitrary 
> positions on disk / inside a partition. (Obviously arbitrary being a 
> limited number but the limit is well beyond what todays harddrives can 
> manage). (IIRC the partition table allows booting of partitions up to 2TiB 
> into the disk.)

Yeah, I guess you're right.

> I have looked at this before. I admit I didn't actually understand how the 
> hell it was supposed to work (the block remapper is totally beyond me, but 
> I haven't spent too much time trying to figure out how it works) so take 
> the following with a pinch of salt... - I don't think it can ever work for 
> ntfs at all because of compressed files, encrypted files, file data stored 
> within the inode structure, and most importantly because of no support for 
> FIBMAP...

Yeah... these things might be massage-able though, but it's not
straight-forward, certainly.

Cheers,
Andrew

Re: [Linux-NTFS-Dev] Re: NTFS resize

[Anton A. <ai...@ca...>]

>>IMHO ntfs_volume *vol would be better: version of NTFS and many others 
>>wouldn't be "lost" and some might be useful in the future.

I take back what I said before. I will use ntfs_volume *vol as you suggest 
as we need to know the clutser size and I don't like the idea of that being 
passed as yet another parameter...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: NTFS resize

[Anton A. <ai...@ca...>]

Hi Szaka,

At 16:07 07/07/02, Szakacsits Szabolcs wrote:
>On Sat, 6 Jul 2002, Anton Altaparmakov wrote:
> > get_size_for_mapping_pairs()    -> ntfs_get_size_for_mapping_pairs()
>
>ntfs_mapping_pairs_size()

This is nice but there is one problem: we don't know if it determines the 
size of existing mapping pairs or of the to be created mapping pairs.

ntfs_mapping_pairs_size() to me would suggest it is getting the size of the 
existing mapping pairs, which by the way is a function I want to implement 
at some point, I think it can be used nicely to optimize the allocation of 
run lists instead of having to reallocate memory all the time during 
decompression of the mapping pairs.

On second thought, maybe you are right. Then the function I describe above 
could be called ntfs_run_list_size() to be consistent with the naming scheme.

I still think the two names may be misleading but if you think the two 
names are ok I will switch. Or even feel free to rename it yourself if you 
like, so you can test if your write access to cvs works properly. (-;

> > Why do you need ntfs_rlwrite()?
>
>It exactly does what I needed. Not more, not less and it was there :)
>
>It's needed to write out the new updated bitmap pointed to by the
>truncated run list of the unamed data attribute of $Bitmap (unusual,
>improbable resident attribute case is not yet supported).
>
> > It is a very ugly, hacked together function
> > which I just came up with because it was the quickest way to do what I
> > wanted at the time and I was pushed for time when writing mkntfs... Can
> > none of the existing library functions (e.g. ntfs_attr_pwrite()) do?
>
>ntfs_attr_pwrite() does significantly more and actually using/shifting
>to its sanity checks would be useful however I suspect it won't work
>without some minor modification in its current form. Anyway I'll try
>and let you know.

Correct. It doesn't support resizing of attributes yet so it would be no 
use. What you would need is ntfs_attr_truncate() as well as 
ntfs_attr_pwrite(), but fully implemented so it can extend attributes as 
POSIX requires.

Ok, I will give you ntfs_rlwrite but I will call it ntfs_rl_pwrite() so it 
will basically have all the functionality you need and more. I will only 
remove the inited_size parameter and since you don't use it, that is ok. I 
will put it into disk_io.[ch] as it is too low level to go into attrib.[ch] 
IMHO. So I will use int fs rather than the ntfs volume to be consistent 
with the other disk_io functions.

> > (This is the one my eyes object to the most.) Also is it ok to
> > have ntfs_volume *vol instead of int fd? (Note sure about that,
> > may leave it to int fd...).
>
>IMHO ntfs_volume *vol would be better: version of NTFS and many others
>wouldn't be "lost" and some might be useful in the future.

That is true however the writing happens at a run list only level without 
any knowledge of which mft record, etc is being written so knowing which 
ntfs version it is, is not going to make any difference. This function is 
useful, I agree, but it is a very low level function. For example if the 
run list is partially unmapped or if it contain holes the function cannot 
do anything about it and must just return error. So the uses of this 
function are quite limited.

I think once we have ntfs_attr_pwrite() fully implemented and 
ntfs_attr_truncate() there will no longer be a need for ntfs_rl_pwrite() at 
all and if we find that we can always remove it from the library again...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: CopyFile

[Anton A. <ai...@ca...>]

At 14:43 07/07/02, Szakacsits Szabolcs wrote:

>On Sun, 7 Jul 2002, Anton Altaparmakov wrote:
>
> > of course). I am even planning a CopyFile system call which would need to
> > be done at VFS layer
>
>sendfile(2) worked for me in the past, imho there is even a copyfile()
>wrapper for it it newer glibc's.

I know about sendfile but I doubt it would be sufficient. Having said that 
I have never investigated it at all it being a relatively recent addition 
to the kernel so I may just not know what I am talking about...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: NTFS resize

[Szakacsits S. <sz...@si...>]

Hi Anton,

On Sat, 6 Jul 2002, Anton Altaparmakov wrote:

> Some are now done but renamed as shown below. I am open to suggestions on
> the function names... I still don't like some of the names too much but I
> think the new ones are more fitting to what the functions do. Note the
> functions are exported by attrib.h.
>
> get_min_bytes_width()           -> ntfs_get_nr_significant_bytes()
> get_size_for_mapping_pairs()    -> ntfs_get_size_for_mapping_pairs()

ntfs_get_mapping_pairs_size()
ntfs_mapping_pairs_size()

> write_min_bytes()               -> ntfs_write_significant_bytes()
> build_mapping_pairs()           -> ntfs_build_mapping_pairs()
>
> Some of the functions have been modified to detect errors and return -1 and
> errno set to the error code on error. See the bottom of attrib.c where I
> added the functions for details. As always, I have documented each function
> above its definition.

Cool!

> Let me know if the modified functions are not suitable for your needs. I
> think they will be just fine but just in case...

Works fine.

> Why do you need ntfs_rlwrite()?

It exactly does what I needed. Not more, not less and it was there :)

It's needed to write out the new updated bitmap pointed to by the
truncated run list of the unamed data attribute of $Bitmap (unusual,
improbable resident attribute case is not yet supported).

> It is a very ugly, hacked together function
> which I just came up with because it was the quickest way to do what I
> wanted at the time and I was pushed for time when writing mkntfs... Can
> none of the existing library functions (e.g. ntfs_attr_pwrite()) do?

ntfs_attr_pwrite() does significantly more and actually using/shifting
to its sanity checks would be useful however I suspect it won't work
without some minor modification in its current form. Anyway I'll try
and let you know.

> Having said that, I have actually been thinking about adding something
> similar to ntfs_rlwrite() but I am thinking of changing the interface so it
> isn't so ugly and also I want to make the interface more powerful so you
> don't have to start writing at the beginning of the run list for example.

Yes, it could be useful in the future.

> Can you tell me exactly what you need it for so I know what interface will
> be compatible? Or alternatively tell me what interface would be the minimum
> requirement for you... For example, do you need the s64 *inited_size
> parameter?

No. Usage is like
	size = ntfs_rlwrite(vol->fd, rl, lcn_bitmap.bm, bm_bsize, NULL);
	if (bm_bsize != size)
		... error handling ...

> (This is the one my eyes object to the most.) Also is it ok to
> have ntfs_volume *vol instead of int fd? (Note sure about that,
> may leave it to int fd...).

IMHO ntfs_volume *vol would be better: version of NTFS and many others
wouldn't be "lost" and some might be useful in the future.

	Szaka

[Linux-NTFS-Dev] Re: CopyFile

[Szakacsits S. <sz...@si...>]

On Sun, 7 Jul 2002, Anton Altaparmakov wrote:

> of course). I am even planning a CopyFile system call which would need to
> be done at VFS layer

sendfile(2) worked for me in the past, imho there is even a copyfile()
wrapper for it it newer glibc's.

	Szaka

Re: [Linux-NTFS-Dev] Re: NTFS resize

[Anton A. <ai...@ca...>]

At 14:25 07/07/02, Andrew Clausen wrote:
>On Sat, Jul 06, 2002 at 06:07:56PM +0200, Szakacsits Szabolcs wrote:
> > On Sat, 6 Jul 2002, Andrew Clausen wrote:
>Also, I don't believe defragging requires any less knowledge of
>the FS than reconstruction.  Aren't all references in NTFS done
>via attributes?

Not knowledge but reconstructing an FS from a damaged state requires a hell 
of a lot of logic which is not necessary for defragmenting. Admittedly, an 
intelligent defragmenter which doesn't just consolidate free space but for 
example orders files and inodes by directory, etc would also require a lot 
of logic. But the two logic sets are orthogonal.

>What about CHS/LBA addressing issues?  It's often convenient to have
>a small boot partition at the start, etc.

No, that has not existed as a restriction in neither Windows nor Linux as 
of a few years now. Both OS quite happily will boot from arbitrary 
positions on disk / inside a partition. (Obviously arbitrary being a 
limited number but the limit is well beyond what todays harddrives can 
manage). (IIRC the partition table allows booting of partitions up to 2TiB 
into the disk.)

> > If one (in some years probably most Win user) has such a box you think
> > he is care about "move the start" to be able to try out Linux?
>
>I think it's still a useful case to support.

It certainly is but it would be more useful to have a working program with 
less features than to not have a program at all. An existing resizer can 
always be extended to support more features at later date...

> > > Another thing that's worth looking at is:
> > >
> > >     http://tzukanov.narod.ru/convertfs/
> > >
> > > Is it possible to support NTFS via this route?  It would be very
> > > cool if it were...
> >
> > Clever indeed. I see many minor and two big problems:
> > - no NTFS write support in the kernel
>
>Obviously a big problem, right ;)

I have looked at this before. I admit I didn't actually understand how the 
hell it was supposed to work (the block remapper is totally beyond me, but 
I haven't spent too much time trying to figure out how it works) so take 
the following with a pinch of salt... - I don't think it can ever work for 
ntfs at all because of compressed files, encrypted files, file data stored 
within the inode structure, and most importantly because of no support for 
FIBMAP...

> > - lots of NTFS specific attribute/data (e.g. streams) would get lost
> >   during mv. I guess this is also true in the case of converting
> >   between different unix filesystems or even during resizing (see
> >   e.g. 'man 1 chattr' what I mean).
>
>Yeah, I'm wondering if this will be fixed.  Like some version of
>cp that copies attributes as well.  Anyway, it seems like a solvable
>problem.  (Although possibly painful, since many attributes in
>various fs's are "built in" and interpreted by the FS, etc. - but
>this shouldn't be an issue for resizing, only converting)

Yes I am definitely planning to have an ntcp utility which will copy all 
attributes when copying from one ntfs partition to another (or the same one 
of course). I am even planning a CopyFile system call which would need to 
be done at VFS layer but I suspect Al Viro would rip my head off for even 
suggesting something like that... )-:

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: [Linux-NTFS-Dev] Re: NTFS resize

[Andrew C. <cl...@gn...>]

On Sat, Jul 06, 2002 at 06:07:56PM +0200, Szakacsits Szabolcs wrote:
> On Sat, 6 Jul 2002, Andrew Clausen wrote:
> > > About cluster relocation/defrag in the future: I think partly "defrag"
> > > at the end of the volume is more then enough for two reasons:
> > > - less chance to screw things up (by a power failure of course ;)
> > > - could be significantly faster: people, who resize at e.g. install
> > >   or whatever time, are really not interested in full defrag however
> > >   fast operation.
> >
> > Actually, it's significantly slower, if you want to "move the start".
> 
> Agreed but two comments, keeping fs integrity is higher priority then
> performance.

I don't see how there is a difference wrt "fs integrity".  Actually,
my method (which I call "reconstructive" as opposed to "incremental")
is basically atomic, and doesn't require journaling, or sophisticated
tricks to prevent problems occuring on interruption.

Compare the parted's ext2 and fat resizers.  If you interrupt the
ext2 resizer (the traditional type of resizer you are proposing,
like defrag + superblock update), then e2fsck can recover.  If you
interrupt the FAT resizer, you get the old file system (except
in the very last phase which is very short).  No fsck required.

Also, I don't believe defragging requires any less knowledge of
the FS than reconstruction.  Aren't all references in NTFS done
via attributes?

> Second, "move the start" is not the typical customer
> scenario however quote from
> 	http://www.microsoft.com/hwdev/tech/storage/ntfs-preinstallP.asp
> 
> "NTFS is the recommended file system for computers running the
> Microsoft Windows XP and Windows .NET Server operating systems.
> Microsoft strongly encourages system manufacturers to manufacture
> single NTFS volumes on all systems where a 32-bit version of Windows
> ^^^^^^^^^^^^^^^^^^^
> XP is preinstalled. ... Feedback from customers and system manufacturers
> indicates that users want single volumes because they do not want to
> manage multiple volumes and do not understand multiple-volume usage
> scenarios."

What about CHS/LBA addressing issues?  It's often convenient to have
a small boot partition at the start, etc.

> If one (in some years probably most Win user) has such a box you think
> he is care about "move the start" to be able to try out Linux?

I think it's still a useful case to support.

> Well, NTFS isn't FAT (has its own oddities).

What are the oddities that make a difference?
(I have spent a lot of time reading NTFS docs, so you can throw
details at me, if you want)

> > Another thing that's worth looking at is:
> >
> > 	http://tzukanov.narod.ru/convertfs/
> >
> > Is it possible to support NTFS via this route?  It would be very
> > cool if it were...
> 
> Clever indeed. I see many minor and two big problems:
> - no NTFS write support in the kernel

Obviously a big problem, right ;)

> - lots of NTFS specific attribute/data (e.g. streams) would get lost
>   during mv. I guess this is also true in the case of converting
>   between different unix filesystems or even during resizing (see
>   e.g. 'man 1 chattr' what I mean).

Yeah, I'm wondering if this will be fixed.  Like some version of
cp that copies attributes as well.  Anyway, it seems like a solvable
problem.  (Although possibly painful, since many attributes in
various fs's are "built in" and interpreted by the FS, etc. - but
this shouldn't be an issue for resizing, only converting)

Cheers,
Andrew