Bacula

On Monday 07 September 2009 16:31:52 lorenz schori wrote:
> Hi
>
> My approach of cross compiling the universal binary for mac os x 10.4
> onwards supporting Intel and PPC platforms is not working at the
> moment. Playing with the file daemon on 10.4/PPC I realized that there
> is a byteReverse Function in md5.c which gets compiled in on big
> endian machines but not on little endian boxes. I'll try to come up
> with a solution but until then universal-binary builds with the
> Makefile in platforms/osx will not be universal enough.

Yes, fixing md5.c to have a "null" runtime statement on bigendian machines 
should be rather trivial. One could simply setup a global variable when 
Bacula starts that is a bool and set according to the endian.  That variable 
could then be tested in the byteRevers function.  However, there may be other 
routines where similar things happen.  It isn't hard to fix them and make 
them runtime dependent, but it may be hard to find them all.

Kern

Hi Kern

On 07.09.2009, at 16:42, Kern Sibbald wrote:

> [...]
> could then be tested in the byteRevers function.  However, there may  
> be other
> routines where similar things happen.  It isn't hard to fix them and  
> make
> them runtime dependent, but it may be hard to find them all.


HAVE_BIGENDIAN is only used in md5.c:

$ find src -path src/win32 -prune -o -print0 | xargs -0 grep ENDIAN
src/lib/md5.c:#ifndef HAVE_BIGENDIAN

There is a bunch of stuff in win32 though but i'm not sure on whether  
we have to consider that.

Lorenz

On Monday 07 September 2009 16:51:51 lorenz schori wrote:
> Hi Kern
>
> On 07.09.2009, at 16:42, Kern Sibbald wrote:
> > [...]
> > could then be tested in the byteRevers function.  However, there may
> > be other
> > routines where similar things happen.  It isn't hard to fix them and
> > make
> > them runtime dependent, but it may be hard to find them all.
>
> HAVE_BIGENDIAN is only used in md5.c:
>
> $ find src -path src/win32 -prune -o -print0 | xargs -0 grep ENDIAN
> src/lib/md5.c:#ifndef HAVE_BIGENDIAN
>
> There is a bunch of stuff in win32 though but i'm not sure on whether
> we have to consider that.

No, I cannot see making universal binaries on Win32.  Nothing on Windows is 
compatible with anything else even within their own releases.

Kern

PS: the code in src/lib/serial.c already does dynamic endian testing.  It 
uses:

  if (htonl(1) == 1L) { 
      bigendian
  } else {
      littlendian
  }

If I remember right.

Hi Kern

On 07.09.2009, at 17:03, Kern Sibbald wrote:

> [..]
>
> PS: the code in src/lib/serial.c already does dynamic endian  
> testing.  It
> uses:
>
>  if (htonl(1) == 1L) {
>      bigendian
>  } else {
>      littlendian
>  }
>
> If I remember right.

Okay, i got bacula 3.0.2 universal binary on Mac OS X 10.4/PPC talking  
nicely to a director/storage daemon on FreeBSD-8/i386. Patches attached.

Lorenz

Hello Lorenz,

I don't understand very well why the current code doesn't work, what is the 
diffrence between the #ifdef and the if(htonl(1) == 1L) ?

They should produce the same thing, isn't it ?

Bye

Le Tuesday 08 September 2009 12:08:30 lorenz schori, vous avez écrit :
> Hi Kern
>
> On 07.09.2009, at 17:03, Kern Sibbald wrote:
> > [..]
> >
> > PS: the code in src/lib/serial.c already does dynamic endian
> > testing.  It
> > uses:
> >
> >  if (htonl(1) == 1L) {
> >      bigendian
> >  } else {
> >      littlendian
> >  }
> >
> > If I remember right.
>
> Okay, i got bacula 3.0.2 universal binary on Mac OS X 10.4/PPC talking
> nicely to a director/storage daemon on FreeBSD-8/i386. Patches attached.
>
> Lorenz

On Tuesday 08 September 2009 17:04:39 Eric Bollengier wrote:
> Hello Lorenz,
>
> I don't understand very well why the current code doesn't work, what is the
> diffrence between the #ifdef and the if(htonl(1) == 1L) ?
>
> They should produce the same thing, isn't it ?

The answer is yes, they should produce the same thing.  However, Lorenz wants 
to produce chip independent binaries for the Mac so the same binary can run 
on Intel and the older Macs (if I understand right), and to do so, byte order 
differences must be resolved at execution time rather than compile time.

For the moment, I am examining his patches, which I *think* are correct, but I 
am a bit unsure about bigendian/little endian in the md5.c code.  It appears 
that the byte order is reversed on bigendian machines which is the inverse of 
what we do in our serial code, so before applying it, I want to double/triple 
check ...

Kern

>
> Bye
>
> Le Tuesday 08 September 2009 12:08:30 lorenz schori, vous avez écrit :
> > Hi Kern
> >
> > On 07.09.2009, at 17:03, Kern Sibbald wrote:
> > > [..]
> > >
> > > PS: the code in src/lib/serial.c already does dynamic endian
> > > testing.  It
> > > uses:
> > >
> > >  if (htonl(1) == 1L) {
> > >      bigendian
> > >  } else {
> > >      littlendian
> > >  }
> > >
> > > If I remember right.
> >
> > Okay, i got bacula 3.0.2 universal binary on Mac OS X 10.4/PPC talking
> > nicely to a director/storage daemon on FreeBSD-8/i386. Patches attached.
> >
> > Lorenz

>>>>> On Tue, 8 Sep 2009 17:38:01 +0200, Kern Sibbald said:
> 
> On Tuesday 08 September 2009 17:04:39 Eric Bollengier wrote:
> > Hello Lorenz,
> >
> > I don't understand very well why the current code doesn't work, what is the
> > diffrence between the #ifdef and the if(htonl(1) == 1L) ?
> >
> > They should produce the same thing, isn't it ?
> 
> The answer is yes, they should produce the same thing.  However, Lorenz wants 
> to produce chip independent binaries for the Mac so the same binary can run 
> on Intel and the older Macs (if I understand right), and to do so, byte order 
> differences must be resolved at execution time rather than compile time.

Kind of -- it only runs configure once, so it can't be decided then.


> For the moment, I am examining his patches, which I *think* are correct, but I 
> am a bit unsure about bigendian/little endian in the md5.c code.  It appears 
> that the byte order is reversed on bigendian machines which is the inverse of 
> what we do in our serial code, so before applying it, I want to double/triple 
> check ...
> 
> Kern
> 
> >
> > Bye
> >
> > Le Tuesday 08 September 2009 12:08:30 lorenz schori, vous avez écrit :
> > > Hi Kern
> > >
> > > On 07.09.2009, at 17:03, Kern Sibbald wrote:
> > > > [..]
> > > >
> > > > PS: the code in src/lib/serial.c already does dynamic endian
> > > > testing.  It
> > > > uses:
> > > >
> > > >  if (htonl(1) == 1L) {

What a brilliant use of abstraction :-(

__Martin

Hi

On 08.09.2009, at 20:50, Martin Simmons wrote:

>>>>>> On Tue, 8 Sep 2009 17:38:01 +0200, Kern Sibbald said:
>>
>> On Tuesday 08 September 2009 17:04:39 Eric Bollengier wrote:
>>> Hello Lorenz,
>>>
>>> I don't understand very well why the current code doesn't work,  
>>> what is the
>>> diffrence between the #ifdef and the if(htonl(1) == 1L) ?
>>>
>>> They should produce the same thing, isn't it ?
>>
>> The answer is yes, they should produce the same thing.  However,  
>> Lorenz wants
>> to produce chip independent binaries for the Mac so the same binary  
>> can run
>> on Intel and the older Macs (if I understand right), and to do so,  
>> byte order
>> differences must be resolved at execution time rather than compile  
>> time.
>
> Kind of -- it only runs configure once, so it can't be decided then.

Yes, that's right. I thought about running the complete configure/make  
independently for each architecture and fusing together the binaries  
using the lipo command - or even abandon universal binaries and just  
put together an installer package for each architecture. If however  
the only problem with the current method is the one in md5.c, I  
happily stick with the current mechanism. The build process is faster  
and the Makefile less complex...

>>>>> [...]
>>>>> if (htonl(1) == 1L) {
>
> What a brilliant use of abstraction :-(

A more expressive endian check would be something like the following  
function (see also: http://c-faq.com/misc/endiantest.html)

#include <stdio.h>

inline int bigendian() {
   const int number_one = 1;
   const char first_byte_of_number_one = *(char*)&number_one;

   return first_byte_of_number_one==0;
}

int main() {
   printf("bigendian: %d\n",bigendian());
}

I'm not sure if inline and const are the proper way of doing this in c/ 
c++. I get the expected results on Mac OS X Intel/PPC, perhaps some of  
you could try this on less common OS and hardware?

Cheers
Lorenz

On Tuesday 08 September 2009 21:49:51 lorenz schori wrote:
> Hi
>
> On 08.09.2009, at 20:50, Martin Simmons wrote:
> >>>>>> On Tue, 8 Sep 2009 17:38:01 +0200, Kern Sibbald said:
> >>
> >> On Tuesday 08 September 2009 17:04:39 Eric Bollengier wrote:
> >>> Hello Lorenz,
> >>>
> >>> I don't understand very well why the current code doesn't work,
> >>> what is the
> >>> diffrence between the #ifdef and the if(htonl(1) == 1L) ?
> >>>
> >>> They should produce the same thing, isn't it ?
> >>
> >> The answer is yes, they should produce the same thing.  However,
> >> Lorenz wants
> >> to produce chip independent binaries for the Mac so the same binary
> >> can run
> >> on Intel and the older Macs (if I understand right), and to do so,
> >> byte order
> >> differences must be resolved at execution time rather than compile
> >> time.
> >
> > Kind of -- it only runs configure once, so it can't be decided then.
>
> Yes, that's right. I thought about running the complete configure/make
> independently for each architecture and fusing together the binaries
> using the lipo command - or even abandon universal binaries and just
> put together an installer package for each architecture. If however
> the only problem with the current method is the one in md5.c, I
> happily stick with the current mechanism. The build process is faster
> and the Makefile less complex...
>
> >>>>> [...]
> >>>>> if (htonl(1) == 1L) {
> >
> > What a brilliant use of abstraction :-(
>
> A more expressive endian check would be something like the following
> function (see also: http://c-faq.com/misc/endiantest.html)
>
> #include <stdio.h>
>
> inline int bigendian() {
>    const int number_one = 1;
>    const char first_byte_of_number_one = *(char*)&number_one;
>
>    return first_byte_of_number_one==0;
> }

I much prefer the code we have, but as I expect was Martin's point, I was 
already considering that we should probably move it into a an inline 
subroutine.  The place for such subroutines is in src/baconfig.h.  It would 
then make the code much easier to read.

inline bool bigendian() {
   return htonl(1) == 1L);
}

>
> int main() {
>    printf("bigendian: %d\n",bigendian());
> }
>
> I'm not sure if inline and const are the proper way of doing this in c/
> c++. I get the expected results on Mac OS X Intel/PPC, perhaps some of
> you could try this on less common OS and hardware?
>
> Cheers
> Lorenz

>>>>> On Wed, 9 Sep 2009 08:16:46 +0200, Kern Sibbald said:
> 
> On Tuesday 08 September 2009 21:49:51 lorenz schori wrote:
> > Hi
> >
> > On 08.09.2009, at 20:50, Martin Simmons wrote:
> > >>>>>> On Tue, 8 Sep 2009 17:38:01 +0200, Kern Sibbald said:
> > >>
> > >> On Tuesday 08 September 2009 17:04:39 Eric Bollengier wrote:
> > >>> Hello Lorenz,
> > >>>
> > >>> I don't understand very well why the current code doesn't work,
> > >>> what is the
> > >>> diffrence between the #ifdef and the if(htonl(1) == 1L) ?
> > >>>
> > >>> They should produce the same thing, isn't it ?
> > >>
> > >> The answer is yes, they should produce the same thing.  However,
> > >> Lorenz wants
> > >> to produce chip independent binaries for the Mac so the same binary
> > >> can run
> > >> on Intel and the older Macs (if I understand right), and to do so,
> > >> byte order
> > >> differences must be resolved at execution time rather than compile
> > >> time.
> > >
> > > Kind of -- it only runs configure once, so it can't be decided then.
> >
> > Yes, that's right. I thought about running the complete configure/make
> > independently for each architecture and fusing together the binaries
> > using the lipo command - or even abandon universal binaries and just
> > put together an installer package for each architecture. If however
> > the only problem with the current method is the one in md5.c, I
> > happily stick with the current mechanism. The build process is faster
> > and the Makefile less complex...
> >
> > >>>>> [...]
> > >>>>> if (htonl(1) == 1L) {
> > >
> > > What a brilliant use of abstraction :-(
> >
> > A more expressive endian check would be something like the following
> > function (see also: http://c-faq.com/misc/endiantest.html)
> >
> > #include <stdio.h>
> >
> > inline int bigendian() {
> >    const int number_one = 1;
> >    const char first_byte_of_number_one = *(char*)&number_one;
> >
> >    return first_byte_of_number_one==0;
> > }
> 
> I much prefer the code we have, but as I expect was Martin's point, I was 
> already considering that we should probably move it into a an inline 
> subroutine.  The place for such subroutines is in src/baconfig.h.  It would 
> then make the code much easier to read.
> 
> inline bool bigendian() {
>    return htonl(1) == 1L);
> }

Yes, that's what I meant really.  The algorithm is incidental, but it is
better to keep it in one place with a descriptive name.

__Martin