corelinux-public

[Corelinux-public] Endianess

[Jim K. <jrk...@cs...>]

Is endian-ness an issue in Linux?  If a data file is created by a
Linux/Intel application, can it be read by the Linux/PowerPC version of
that application?

Should CoreLinux provide some facility for conversion of endian-ness in
cross-platform data?

Jim Koontz
jrk...@us...

Re: [Corelinux-public] Endianess

[Frank V. C. <fr...@co...>]

> Is endian-ness an issue in Linux?  If a data file is created by a
> Linux/Intel application, can it be read by the Linux/PowerPC version of
> that application?
>
> Should CoreLinux provide some facility for conversion of endian-ness in
> cross-platform data?
>
> Jim Koontz
> jrk...@us...

Data Files
--------------
There is an issue that has to be addressed by the applications persistence
implementation or any persistence that the library provides. From a numeric
stand point using network byte order consistently should avoid that, or of
course characters.

Source Files
------------------
There is an issue, I know bitfields and unions may be targets for analysis,
there may be more (I have been corrupted for too long by little-endianess)
but avoiding bit manipulation and unions is part of the CoreLinux standards.

Binary Libraries and Apps
------------------------------------
Big issue, I don't know of solutions.


I'm hoping there are some who are running on both type of Linux
implementations (big and little) that we can test/verify against early on in
the development stages.

Re: [Corelinux-public] Endianess

[Jim K. <jrk...@at...>]

On Thu, 9 Dec 1999, Frank V. Castellucci wrote:

> > Is endian-ness an issue in Linux?  If a data file is created by a
> > Linux/Intel application, can it be read by the Linux/PowerPC version of
> > that application?
> >
> > Should CoreLinux provide some facility for conversion of endian-ness in
> > cross-platform data?
> >
> > Jim Koontz
> > jrk...@us...
> 
> Data Files
> --------------
> There is an issue that has to be addressed by the applications persistence
> implementation or any persistence that the library provides. From a numeric
> stand point using network byte order consistently should avoid that, or of
> course characters.
> 
> Source Files
> ------------------
> There is an issue, I know bitfields and unions may be targets for analysis,
> there may be more (I have been corrupted for too long by little-endianess)
> but avoiding bit manipulation and unions is part of the CoreLinux standards.
> 
> Binary Libraries and Apps
> ------------------------------------
> Big issue, I don't know of solutions.
> 
> 
> I'm hoping there are some who are running on both type of Linux
> implementations (big and little) that we can test/verify against early on in
> the development stages.
> 
As far as the binary libraries and apps go, I think the only solution is
to build them on each platform. 

The way that I have seen endian-ness resolved in other operating systems
is to either save in one format, and provide code that does bit flipping
if the platform reading is opposite-endian, or to save in a canonical
format, that contains a flag field indicating the source platform.  If the
platform reading is opposite-endian from the platform that saved the data,
the bit flipping takes place.  In either case, macros that test for
endian-ness and perform the bit-flipping are provided as part of the
API.

Jim Koontz
jrk...@so...

Re: [Corelinux-public] Endianess

[Thomas M. <t.r...@wo...>]

I think an approach that encodes the ordering in the beginning of data files
and source files is an appropriate approach.  This would be akin to the
UNICODE non spacing non breaking code <0xfffe> used in the beginning of
unicode files to tell the reader the ordering.

----- Original Message -----
From: Jim Koontz <jrk...@at...>
To: <cor...@ma...>
Sent: Thursday, December 09, 1999 10:29 AM
Subject: Re: [Corelinux-public] Endianess


>
>
> On Thu, 9 Dec 1999, Frank V. Castellucci wrote:
>
> > > Is endian-ness an issue in Linux?  If a data file is created by a
> > > Linux/Intel application, can it be read by the Linux/PowerPC version
of
> > > that application?
> > >
> > > Should CoreLinux provide some facility for conversion of endian-ness
in
> > > cross-platform data?
> > >
> > > Jim Koontz
> > > jrk...@us...
> >
> > Data Files
> > --------------
> > There is an issue that has to be addressed by the applications
persistence
> > implementation or any persistence that the library provides. From a
numeric
> > stand point using network byte order consistently should avoid that, or
of
> > course characters.
> >
> > Source Files
> > ------------------
> > There is an issue, I know bitfields and unions may be targets for
analysis,
> > there may be more (I have been corrupted for too long by
little-endianess)
> > but avoiding bit manipulation and unions is part of the CoreLinux
standards.
> >
> > Binary Libraries and Apps
> > ------------------------------------
> > Big issue, I don't know of solutions.
> >
> >
> > I'm hoping there are some who are running on both type of Linux
> > implementations (big and little) that we can test/verify against early
on in
> > the development stages.
> >
> As far as the binary libraries and apps go, I think the only solution is
> to build them on each platform.
>
> The way that I have seen endian-ness resolved in other operating systems
> is to either save in one format, and provide code that does bit flipping
> if the platform reading is opposite-endian, or to save in a canonical
> format, that contains a flag field indicating the source platform.  If the
> platform reading is opposite-endian from the platform that saved the data,
> the bit flipping takes place.  In either case, macros that test for
> endian-ness and perform the bit-flipping are provided as part of the
> API.
>
> Jim Koontz
> jrk...@so...
>
>
> _______________________________________________
> Corelinux-public mailing list
> Cor...@li...
> http://lists.sourceforge.net/mailman/listinfo/corelinux-public
>

Re: [Corelinux-public] Endianess

[Jim K. <jrk...@at...>]

On Thu, 9 Dec 1999, Thomas Maguire wrote:

> I think an approach that encodes the ordering in the beginning of data files
> and source files is an appropriate approach.  This would be akin to the
> UNICODE non spacing non breaking code <0xfffe> used in the beginning of
> unicode files to tell the reader the ordering.
> 

I think that is a satisfactory solution, should this be incorporated into
the standards documents, or requirements?

Re: [Corelinux-public] Endianess

[Frank V. C. <fr...@co...>]

Jim Koontz wrote:
> 
> On Thu, 9 Dec 1999, Thomas Maguire wrote:
> 
> > I think an approach that encodes the ordering in the beginning of data files
> > and source files is an appropriate approach.  This would be akin to the
> > UNICODE non spacing non breaking code <0xfffe> used in the beginning of
> > unicode files to tell the reader the ordering.
> >
> 
> I think that is a satisfactory solution, should this be incorporated into
> the standards documents, or requirements?

There are a few requirements here:

1. Big/little endianess issues in the library development. I believe the
CoreLinux C++ standard addresses this.
2. CoreLinux++ should provide a persistence abstraction. A clear and
deisreable framework requirement.
3. The CoreLinux++ framework persistence implementation insures portable
content. For this I would ask the following:

Is our persistent data (the users application data I will assume) for
local storing? Exchange? Both?

The reason I ask this is because I assume that the framework will
consist of at least a suitable persistence abstraction and at minimum
one (1) implementation. 

If the implementation is a flat file one, then we can make IT'S
requirement and feature one that insures proper handling regardless of
the byte ordering in the content.

For a persistence implementation that uses XML (for example) this may be
a moot point.

For a persistence implementation that uses an underlying RDMS, I don't
know if it is moot or not.

While I agree with Thomas Maguire and you that it is a solution to a
data file exchange between big and little endian machines, I think it is
premature for us to make it a requirement of a implementation we don't
have a reqiurement for yet. 

When we get there, and if local file storage is the persistence
implementation, than we have this specification available which solves
the eventual issue.

Thoughts?

Re: [Corelinux-public] Endianess

[Rik H. <ri...@kd...>]

#if Frank V. Castellucci
> Is our persistent data (the users application data I will assume) for
> local storing? Exchange? Both?

I can offer examples from real life:

1. KDE uses a protocol called DCOP for communication. This was designed
as a replacement for using CORBA. Parameters are marshalled by streaming
into a 'QByteArray'. Here endianness is important.

2. Recently I created a class which gave the same functionality as GDBM.
Here again the data is streamed into a 'QByteArray'. This class is
used for efficiently indexing a mailbox.

-> I think persistent data must work on any byte-order. The advantage
of being able to marshal parameters for schemes such as KDE's DCOP, plus
the advantage of being able to read my mailbox indices on any machine
are great.

Cheers,
Rik

Re: [Corelinux-public] Endianess

[Frank V. C. <fr...@co...>]

----- Original Message ----- >
#if Frank V. Castellucci
> > Is our persistent data (the users application data I will assume) for
> > local storing? Exchange? Both?
>
> I can offer examples from real life:
>
> 1. KDE uses a protocol called DCOP for communication. This was designed
> as a replacement for using CORBA. Parameters are marshalled by streaming
> into a 'QByteArray'. Here endianness is important.
>
> 2. Recently I created a class which gave the same functionality as GDBM.
> Here again the data is streamed into a 'QByteArray'. This class is
> used for efficiently indexing a mailbox.
>
> -> I think persistent data must work on any byte-order. The advantage
> of being able to marshal parameters for schemes such as KDE's DCOP, plus
> the advantage of being able to read my mailbox indices on any machine
> are great.
>
> Cheers,
> Rik

Right, but doesn't the issue of what we implement as a default persistence
come into play?

I don't mind if we state here and now that CoreLinux++ framework will
provide both the abstractions and one (1) implementation for persistence as
a minimum. That said implementation will be flat file or stream based. That
said implementation will provide byte ordering protection.

Frank

Re: [Corelinux-public] Endianess

[Rik H. <ri...@kd...>]

#if Frank V. Castellucci
> Right, but doesn't the issue of what we implement as a default persistence
> come into play?

Yes, I'd say that there's nothing like streaming :)

Correct me if I'm wrong, but I tried to do something like this a while
back:

class Base {
  ...
  stream & operator << (stream & str, const Base &)
  {
    str << someData;
    return str;
  }
};

class Derived : public Base
{
  ...
  stream & operator << (stream & str, const Derived &)
  {
    Base::operator << (str, *this);
    str << someMoreData;
    return str;
  }
};

This didn't seem to work. It seems you have to do it this way:

class Base {
  ...
  void save(stream & str)
  {
    str << someData;
  }
};

class Derived : public Base
{
  ...
  void save(stream & str)
  {
    Base::save(str);
    str << someMoreData;
  }
};

I could have explained this in words but I couldn't work out which
ones ;)

What I'm trying to say is that you can't use operators << and >> in
a derived class and still stream the base, so you'd have to use
a method like save(stream &). That's the only niggle I have with
streaming. Apart from that it's very convenient.

By 'flatfile', are you suggesting that writing to a 'text' file is
an option ? If so, I don't know whether this is worth it, as if
you're using 16-bit Unicode then the file will just look like garbage
in most text editors, so there's no 'readability' advantage.

Did I miss the point ?

Cheers,
Rik