There are a couple of issues with how a Layered Texture blends info together, esp. with regard to bump mapping. There is some unexpected behavior regarding blend modes. Each example screengrab includes the texture settings that produced the render.
The pictures, one at a time. Please go by the file names:
GetTextureSpec opts out early when the texture layer is fuly opaque.(1) should be easily fixable by having LayeredTexture line 462 add a clause to not opt out early if layer mode is Overlay (BA)
(3) & (4) are not as easy. My prefered solution would be to re-write the entire method so that it builds the spec up from the lowest layer, otherwise we have to reference the blending mode of the layer above the one we are working on. I get the feeling that the method was written as it is as an optimization, but how much time is actually spent in this code? IE, is it worth the obfuscation?
Diff:
Been working on this a bit and would like clarification on how the over overlay mode is supposed to work. My understanding is that an overlay texture at 100% opacity will set all of the texture spec to match itself, except where there is transparency. Then, the color of the underlying layer shows through. Does this only affect the diffuse channel, or, for example, does the specular part also pass through a bit? Does this change if the overlay fraction of less than 100%?
The intended behavior is that, where the top layer is transparent, all aspects of the lower layer show through. A 100% transparent texture layered over another texture in overlay mode should look exactly the same as if that upper layer weren't there at all.