#15 Shapefile parsing is broken on big endian machines

[John Foerch]

Data in shapefiles is unpacked with the wrong byte
order on big endian machines. Note to self: be
mindful of the unpack bug in php4 (unsigned int32's
don't unpack!) when working on this.

[John Foerch]

Logged In: YES
user_id=983545

I had this informative conversation at irc://freenode/php
today. So this gives me an indication of what I need to do
for a workaround. Now I am thinking of the best way to go
about this, sacrificing as little speed as possible.

/////////////////////////

RetroJ hi, I am having some difficulty getting a script that
I wrote to work on big-endian byte order machines. I believe
that the problem is my use of unpack the unpack code "d",
which unpacks a double float in machine specific byte order.
I know that my data is stored in little endian byte order.
Am I on the right track with this diagnosis and can anybody
suggest how I can unpack this data correctly on a big endian
machine?

TML RetroJ: Don't use "d".

RetroJ what can I use instead?

TML RetroJ: You need to find a way to store something other
than the endian-dependent machine representation of a float.
Such as a string of numbers.

RetroJ TML: the data is not mine... I'm reading government
geographic data

RetroJ So I'm pretty much stuck with little endian packed
doubles. Any ready idea of how I can unpack them on a big
endian machine?

TML RetroJ: I can't think of any way to do it without going
down to C

RetroJ okay. I am able to detect the byte order of the
machine, so I think what I will try is to write a php
function to reverse the bytes of these doubles by repacking,
manipulating the string, and unpacking again

snake RetroJ: some problems with endianess ?

RetroJ yes snake. I have data packed as little endian
doubles that I need to unpack on a big endian machine

snake RetroJ: you got 2 solutions : using pack() or doing it
with bits operators

RetroJ can you give me a reference on bits operators?

TML snake: Be careful about advising people to use bitwise
operators in PHP. They have a tendency to lose resolution.

RetroJ TML: do you mean that the bitwise operators will lose
data?

snake TML: he is using integers

TML RetroJ: They can, yes.

TML snake: He's using double precision floats.

snake RetroJ: if you're using double pressision floats then
use pack() and unpack()

RetroJ I'll try that. Thank you both for your help

[John Foerch]

Logged In: YES
user_id=983545

Promising results this morning with a helper function I
wrote. The new function calls unpack, and then analyses the
format string and reverses the byte order of all the doubles
by re-pack-ing them, strrev-ing them, and unpacking them.
Other code makes a function pointer called
$unpack_workaround that points to 'unpack' on little endian
machines, but the special function on big endian machines.
Thus, almost no speed is sacrificed on little endian machines.

I am considering further study of this problem to write a
more generalized workaround. It would work for both floats
and doubles, and it would have new format codes for byte
order conversion.