Rainbow Six transparent textures with DxWnd

[gho]

Thanks.
I'm trying to focus on this particular situation: there's a mysterious surface 0x485ddb8 in the log, that seems coming out from nowhere (there's no former log of its creation). This is where is first referenced:

Blt(4): dest=0x485deb8 src=0x485ddb8 dwFlags=0x1000000(DDBLT_WAIT) destrect=(NULL) srcrect=(NULL)
GetColorKey(4): lpdds=0x485ddb8 flags=0x2(DDCKEY_DESTBLT)
GetColorKey: ERROR lpdds=0x485ddb8 flags=0x2 res=0x887600d7(DDERR_NOCOLORKEY)
GetColorKey(4): lpdds=0x485ddb8 flags=0x8(DDCKEY_SRCBLT)
GetColorKey: colors=(L:0xfeb4,H:0xfeb4)
GetAttachedSurface(4): lpdds=0x485ddb8 caps=0x401000(DDSCAPS_TEXTURE+MIPMAP) caps2=0 caps3=0 caps4=0
GetAttachedSurface(4): ERROR res=0x887600ff(DDERR_NOTFOUND) at 3131
Release(S4): lpdds=0x485ddb8
Release(S): lpdds=0x485ddb8 ref=1

This patterns then is repeated several times and probably is related with texture loading.
The same thing happens on my computer (obviuously the surface id is different):

Blt(4): dest=0x13183bd8 src=0x13183218 dwFlags=0x1000000(DDBLT_WAIT) destrect=(NULL) srcrect=(NULL)
GetColorKey(4): lpdds=0x13183218 flags=0x2(DDCKEY_DESTBLT)
GetColorKey: ERROR lpdds=0x13183218 flags=0x2 res=0x887600d7(DDERR_NOCOLORKEY)
GetColorKey(4): lpdds=0x13183218 flags=0x8(DDCKEY_SRCBLT)
GetColorKey: ERROR lpdds=0x13183218 flags=0x8 res=0x887600d7(DDERR_NOCOLORKEY)
GetAttachedSurface(4): lpdds=0x13183218 caps=0x401000(DDSCAPS_TEXTURE+MIPMAP) caps2=0 caps3=0 caps4=0
GetAttachedSurface(4): ERROR res=0x887600ff(DDERR_NOTFOUND) at 3131
Release(S4): lpdds=0x13183218
Release(S): lpdds=0x13183218 ref=1

Curiously, on my computer the color key is absent at the beginning, but later on the assumes a value

GetColorKey(4): lpdds=0x13183218 flags=0x2(DDCKEY_DESTBLT)
GetColorKey: ERROR lpdds=0x13183218 flags=0x2 res=0x887600d7(DDERR_NOCOLORKEY)
GetColorKey(4): lpdds=0x13183218 flags=0x8(DDCKEY_SRCBLT)
GetColorKey: colors=(L:0xf776,H:0xf776)

The color value for my computer (0xf776) is quite different from Nath's one (0xfeb4).
In the log there are many SetColorKey operations as well, but none associate to this strange surface, so I assume its key color will remain the same.
Dumping the color values we have this:

0xf776 = (binary)1111-0111-0111-0110
0xfeb4 = (binary)1111-1110-1011-0100

So what? It's hard to say, there's no clear relation between the two colors and their values could be coming from some video card defaults or who knows what? The only thing I can guess is that where a texture has a color identical with one of these, its pixels will become transparent and this may explain the differences. In any case, I don't think I can say I got everything, the whole matter is still dark to me.
Any ideas?

[BEEN_Nath_58]

the log I provided was from v1. 04 which had the single pixel transparency. Should I log the 1.03 version

[gho]

Uhm, maybe I got an idea ....
Since
1) the problem seems to come from uncontrolled color key values, and
2) the game seems able to work also when the color key is not supported
the idea is simply to pretend that we have no color key always (retcode = DDERR_NOCOLORKEY).
The attached dxwnd.dll does this unconditionally (if it works, later I will add a dedicated flag) and seems to work correctly on my computer. Does this happen on Nath's one? If so, that could be a good enough fix!

[dippy dipper]

Regarding the different color gradients in Rainbow6:
There is definitely a noticeable difference between running the game in Dxwnd or not. However it might not be noticeable on a poor quality / badly calibrated monitor. Or with poor eye sight.

However this is fairly simple to test. Running the game outside Dxwnd take a screenshot using the "PrtScr" button and a second screenshot using the in-game "F2" screenshot button. At least on my system the screenshot taken with the games internal method shows more color banding similar to how the game looks with Dxwnd. I think the difference could be that the GPU is doing some post processing to the image which gets bypassed when the game runs through Dxwnd.

The gif animation shows the exact same spot looking at the white canvas in the first mission alternating between the two different color gradients.

[BEEN_Nath_58]

In the last page, I posted comparisons and there is some bending visible in the wall colours.

[gho]

@Saptadeep Nath: please, can you test my last patch dxwnd.2.05.72.nocolorkey.rar posted here above? There is a possibility that it may fix the problem, but I need to know for sure before I move the fix to a stable release.

[BEEN_Nath_58]

Nothing changes when "Suppress DX common errors" in unchecked. The texture is still transparent. Also the textures remain transparent with this option checked.

[BEEN_Nath_58]

Also here's the log, tested on v1.03 with this patch you gave

[BEEN_Nath_58]

Without the colour key fix and with only the "Suppress DX common error" parameter enable, there is still transparency in ground. As soon as I disable it, the glasses become transparent and the ground transparency is gone. It's working vice-versa for me, one is fixed and the other becomes unfixed:

[BEEN_Nath_58]

Ran the game with Software acceleration with and the parameter disabled, the game every texture loads fine (grass, glass). As soon as I enable the parameter, every texture has holes:

[gho]

Well, yes, differences between hw and sw modes are many. In these two screenshots you can see for instance that hw mode is better for texture filtering (color gradients are smooth and not pixellated as in sw mode) but sw mode has a much better transparency handling (look at the fences). Also, hw mode is way faster than sw mode.
My idea is that Nath's hardware has imperfect handling of transparencies, it seems that full black should be considered as transparent, but this hw considers transparent also quasi-black pixels (or maybe makes full-black the darker pixels?). Another possible attack could be to alter the texture pixels and recolor all the pixels that risk to become transparent. Or maybe we could trim the video capabilities and make the game avoid using key colors. Too bad that I can't replicate all this on my computer, things would be much easier.

[BEEN_Nath_58]

the color key patch didn't fix anything, nonetheless do you think it's happening for the hardware or the driver. I am using a 1050Ti with the latest driver 466.xx. Things tend to change mostly because of driver, in case of Amd hardware and if you have a similar hw and old driver may you could try updating drv

[gho]

Sorry, I'm using a portable pc with integrated video card and I can't change hw, nor willing to change my working sw.

[BEEN_Nath_58]

i think i will look up in a vm if dxwnd works and see if thr texture is fixed.

apart from that how do you see the color bending, both dippy and me noticed this happening, is it some kind of post processing from either of the software?

[gho]

Surely there could be some video post processing. One possible example is the gamma ramp control that can make your pixel lighter or darker and is allowed only for the whole screen, not for the inside of a window. And I'm pretty sure there could be a lot of other "proprietary" features of the kind. But once captured, a pixel color is just a binary value no matter how the video card processes it, so it may happen that two screenshots from different video output could be not that much different. Anyway, all this is usually out of a sw tool reach, so DxWnd can't be of much help.

Back to the transparent pixels, I saw that the sw mode eliminates the problem but the rendering is much slower. Instead, if you leave the hw rendering but set the "ddraw:ForcesHEL" tweak the output seems identical to sw mode but the rendering seems faster. You could try this ...

Another possible experiment (if you didn't do that already) could be to set the "DirectX(2) / Set minimal caps".

[dippy dipper]

That "ddraw:ForcesHEL" trick is neat! It results in a fluent gameplay with the software renderer.

[BEEN_Nath_58]

I don't know what I am doing wrong, SW mode with this option changes nothing for me, there is little increase in performance yet it cannot reach what HW offers. :/

[gho]

you must use it with hw mode!

[BEEN_Nath_58]

No change for me in terms of graphics and performance with HEL whether I run it with HW mode or SW mode(it looks a lot like SW as you said). Game runs at 20fps, although thiss maybe the maximum possible by this since i don't know.

[BEEN_Nath_58]

A summary of what I got till now:

Retail (1.03)

Without "Suppress DX common errors" option-

Glass has holes
A few grass pixels have holes

With "Suppress DX common errors" option-

Glass has no holes
Grass has holes all over the place

GOG (1.04)

Without "Suppress DX common errors" option-

Glass has holes
A few grass pixels have holes

With "Suppress DX common errors" option-

1 glass pixel has hole
No hole in any part of grass

Edit: HEL does the same thing to both the games, the game runs at around 20fps(more than what is achieved with plain SW) but completely looks like SW even when HW is enabled.

[BEEN_Nath_58]

@dippy-dipper could you please report if you have a few transparent ground textures with "Suppress DX common errors" enabled, i.e. when you encounter transparency in glass. You should see it around here

[dippy dipper]

Yes, with “v2_05_71_build.rar” and enabling the "Suppress DX common errors" flag I can see those transparent pixels on the ground textures. However not with the “dxwnd.2.05.72.nocolorkey.rar” test build.

I will need to arrange some tests runs on other machines with newer hardware. It may take a while though.

In the mean time a compromise could be to not mess with the code and maybe just add a note in the .dxw file for Rainbow6 to try the "Suppress DX common errors" flag if there are issues with transparent pixels on the textures.

P.S.
Have you tried Rainbow6 with DDrawCompat?
https://github.com/narzoul/DDrawCompat/releases

[dippy dipper]

Tested on an even older PC just for fun and it showed no transparent pixel issues with or without the "Suppress DX common errors" flag using "v2_05_71_build.rar". It had an integrated ATI graphics card. So the issue seems complicated and highly dependent on GPU and/or it's drivers.

P.S.
I am still running on Windows 10 version 20H2. I have not yet upgraded to 21H1. Hopefully that makes no difference.

[BEEN_Nath_58]

I had tried DDrawCompat earlier but it didn't do me any good then so I decided to keep it away. This did wonders now and now I can't find any texture holes in game.

With the base DxWnd + provided .dxw profile + "Suppress DX common errors" enabled + tweak "hook:ReplaceDirectXDLLs" I don't find any texture issues in game except the transparent grill.

[BEEN_Nath_58]

Even the GOG version's single pixel transparency doesn't occur anymore.

However SW mode should be played without DDrawCompat and by forcing Hardware Emulation Layer through the tweaks menu as gho suggested. With HEL enabled and DDrawCompat hook, the graphics will be perfect as earlier but framerate will be as low as with HEL disabled. To prevent the mouse trails in SW mode, GDI rendering has to be used over primary surface.

These provide best results for me.