[Algorithms] Filtering

[Andreas B. <and...@gm...>]

Hi,

I have a texture in which I use the R, G and B channel to store a
value in the [0, 1] range with very high precision. The value is
extracted like this in the (Cg) shader:

float
extractValue(float2 pos) {
    float4 temp = tex2D(buffer, pos);
    return (temp.x * 16711680.0 + temp.y * 65280.0 + temp.z * 255.0) *
(1.0 / 16777215.0);
}

I now want to sample this value with bilinear filtering but when I do
this I don't get a correct result. If I do the filtering manually like
this:

float
sampleValue(float2 pos) {
	float2 ipos = floor(pos);
	float2 fracs = pos - ipos;
	float d0 = extractValue(ipos);
	float d1 = extractValue(ipos + float2(1, 0));
	float d2 = extractValue(ipos + float2(0, 1));
	float d3 = extractValue(ipos + float2(1, 1));
	return lerp(lerp(d0, d1, fracs.x), lerp(d2, d3, fracs.x), fracs.y);
}

everything works as expected. The values in the buffer can be seen as
a linear combination of three constants:

value = (C0 * r + C1 * g + C2 * b)

If we use the built in texture filtering we should get the following
if we sample somewhere between two texels: {r0, g0, b0} and {r1, g1,
b1}. For simplicity we just look at filtering along one axis:

filtered value = lerp(r0, r1, t) * C0 + lerp(g0, g1, t) * C1 +
lerp(b0, b1, t) * C2;

Doing the filtering manually:

filtered value = lerp(r0 * C0 + b0 * C1 + g0 * C2, r1 * C0 + g1 * C1 +
b1 * C2, t)  =
                   = (r0 * C0 + b0 * C1 + g0 * C2) * (1 - t) + (r1 *
C0 + g1 * C1 + b1 * C2) * t =
                   = (r0 * C0) * (1 - t) + (r1 * C0) * t + ... =
                   = lerp(r0, r1, t) * C0 + ...

So in the world of non floating point numbers these two should be
equivalent right?

My theory is that the error is caused by an unfortunate order of
floating point operations. I've tried variations like:

(temp.x * (16711680.0 / 16777215.0) + temp.y * (65280.0/16777215.0) +
temp.z * (255.0/16777215.0))

and

(((temp.x * 256.0 + temp.y) * 256.0 + temp.z) * 255.0) * (1.0 / 16777215.0)

but all exhibit the same problem. What do you think; is it possible to
solve this problem?

Regards Andreas