#15 CalcSurfaceExtents: use inline asm to get x87 floating point, fixes lightmap glitches

[Eric Wasylishen]

I was testing a map for mfx and found the lightmaps were corrupted in places on OSX, but on windows they looked fine. Tracked it down to a floating-point rounding issue in CalcSurfaceExtents.

here's an example calculation of the variable 'val' I captured during debugging, with a mingw32 build (using x87 floating-point). This build looked correct:

(-2776.000000 * 0.752241) + (144.000000 * -0.752241) + (-360.000000 * 0.000000) + 2196.543457 = -0.000029

and the same calculation done with a OS X x86_64 build (uses sse2 floating point), this looked glitchy:

(-2776.000000 * 0.752241) + (144.000000 * -0.752241) + (-360.000000 * 0.000000) + 2196.543457 = 0.000000

These values are then floor()'ed, so for the x887 case you get -1 and the sse2 case you get 0, this leads to an off-by-one error in the lightmap bitmap size and the resulting visual glitch. Attached screenshots from mfxsp17, before and after the patch, on OSX. I expect the same thing happens with 64-bit windows/linux builds though haven't tested there.

Whatever calculation Quakespasm does here has to produce the same result as the light compiler , which did the same calculation. Since most map compilers and engines are win32, I figured the right fix was using x87 explicitly with inline asm, as ugly as that is.

[Eric Wasylishen]

Forgot to mention, I tested the inline asm on a recent version mingw32 on Windows 8, and Xcode 5.1.1's version of clang on OS X 10.9.

This will break the visual studio project I guess, assuming i386 or x86_64 are defined in VS

[Ozkan Sezer]

The problem may be real but I don't like the non-portable solution, and it breaks msvc. No other way around this at all?

[Eric Wasylishen]

ok, seems like I was overly conservative, and simply adding casts to double so that the intermediate adds/multiplies are kept at 64-bits is good enough to fix the glitches with SSE2, at least on clang / OS X, and with mfx's maps.

Need to test this on Visual Studio/x86_64 too. I added the savegame from mfxsp17 I used to make those screenshots.

[Ozkan Sezer]

Tested the patch with an x86 build (with -msse and -mfpmath=sse added to CFLAGS), and it seems to work. Will test on a real x86_64 tomorrow. Waiting for your further results with msvc.

PS: there is something with the save you put here? the models are 'sunken' to floor when I load it. I run like "qs -quoth -game mfxsp17 +load mfxsp17", the map I dl'ed from https://www.quaddicted.com/filebase/mfxsp17.zip

[Eric Wasylishen]

I just tested the new patch on mingw32 (with build_cross_win32.sh, uses x87 floating point) and in VS2013 with both x86 and x64, and it seems to work on all of those configurations.

Yes, the entities in the save are messed up; I think I saved it without using quoth.

[Ozkan Sezer]

Applied (r908.) Thanks.