Issue with transparency and lighting

[James W. <ja...@fr...>]

I thought I was correctly handling transparency with multiple lighting 
passes in QORenderer::Renderer::RenderTransparent, but it turns out that 
in some cases it can be visibly incorrect.  I'll try to explain a 
simplified scenario.

Suppose we have an nearly-opaque object with color A, and in front of 
that there is a transparent object with color B.  There are 2 lighting 
passes, contributing brightnesses L and M respectively.  Then the 
resulting color value ought to be B*(L+M) + (1-alpha)*A*(L+M).  But if 
we have to handle the lights in different passes, the first pass 
produces B*L + (1-alpha)*A*L.  If we use glBlendFunc( GL_ONE, GL_ONE ) 
in the second pass, as RenderTransparent currently does, then we just 
add B*M + A*M, resulting in a total of B*(L+M) + (1-alpha)*A*L + A*M, 
which has too much contribution from A.

I don't see that any choice of blend function will produce the right 
result in all cases.

In many cases where I use a texture with an alpha channel, the alpha 
channel is pretty much either 1 or 0.  In such a case, I think there 
would be pretty good results if we turned on the alpha test, and let 
opaque pixels write to the depth buffer, and didn't use blending.

Thoughts?
-- 
   James W. Walker, Innoventive Software LLC
   <http://www.frameforge3d.com/>