Rainbow Six transparent textures with DxWnd
Window hooker to run fullscreen programs in window and much more...
Status: Beta
Brought to you by:
ghotik
Menu
▾
▴
I have tried out the game on SW mode on a different machine and the same settings had to be used as I mentioned earlier to achieve the best results.
However the game didn't run on DxWnd HW emulation for some reason and would only work in SW until DDrawCompat was hooked. With DDrawCompat hooked and "Suppress DX common errors" game didn't encounter any issues, with the latter setting disabled the wall grills would become black(as gho had encountered in the no Color key patch).
I hope SW will work similar on your machines too, it's the HW mode causing problems, as the PC where I tested now was very weak and used Windows 7 and didn't have the best and latest DX capabilities.
The picture below shows some pixel hunting. I wanted to verify that the transparent pixels all came from full black texture pixels. So I took a screenshot of the scene with the blue transparent pixels on the window, then I dumped the game textures to find the texture bitmap of the window and finally I used the DxWnd palette viewer tool to check the color of the transparent pixels.
The experiment confirmed the theory: though the window appear fully black, most of the pixels are simply very dark but not black (color 0x000000) and the few fully black pixels appeared transparent in the rendered scene.
This could suggest a possible workaround by recoloring the fully black pixels to a very dark color (es. 0x010101) that wouldn't be interpreted as transparent in any case.
Last edit: gho 2021-06-13
For example the pixels at co-ordinates: 64-65,56 for the window texture come out as RGB 0,0,0. Flood filling the matching colors with magenta produces a pattern that can be seen in-game.
Note that not all textures use the same color for transparent pixels.
Look at the staircase column and the comparison to the texture. Also some unintentional holes in the texture? Or maybe the house just has a termite problem. Here the transparent color is RGB 206,203,206. (or CECBCE in hex)
I feel like the issue might be with how the game engine handles the colorkey and alpha for the textures. It may be complicated and not too accurate depending on the GPU and drivers. For example the iron gate texture looks fine in the texture dump but renders poorly in-game.
I also found this old nvidia document detailing some of the perils of using color keyed textures in Direct3D.
http://developer.download.nvidia.com/assets/gamedev/docs/ColorKey.pdf
On an Intel GPU machine running the game with DDrawCompat, + DxWnd makes the transparency disappear. Just as my main PC.
In case coloring the texture becomes a problem for majority who have this problem, DDrawCompat alternative may be used.
Also I noted that some GPUs cannot render it in HW, and instead have broken visuals including misplaced textures, high amount of transparency, which may be too cumbersome to fix with coloring.
After many attempts, I'm still without ideas. I tried different approaches to eliminate transparent pixels, but the problem is this: how am I supposed to tell if a black pixel is supposed to be transparent or not? The attached DxWnd release adds a fancy and useless tweak "ddraw:DisableColorKey" that bypasses the color key operations. This way, no pixel becomes transparent, fixing all holes in the windows and in the grass, but sadly deleting the necessary transparencies in game menus and transparent textures like the grids.
I made this .dxw profile for the SW version which should give good mediocre performance, no menu cursor trails and no transparency. Under the 3 PCs I tested this didn't impose any problem.
Even if you disable "Force Software" from in-game it will render in Software. Have a try :)
Where am I supposed to click?
Edit: I crossed this but no change in transparency + a new problem in tree
Last edit: BEEN_Nath_58 2021-06-14
My super low-end Windows 7 machine when hooked to use DDrawCompat for HW rendering has this problem (this is the default DxWnd release):
However all of my machines which support DxWnd HW rendering and DDrawCompat render this game properly with no transparency at all except this grill.
Last edit: BEEN_Nath_58 2021-06-14
Running the game on Voodoo2 over my main PC shows it now has more problems, the tress have less transparency in old machines:
I doubt it's possible to fix the texture transparency problem in this game with this combination of hardware (the video card) and drivers. The game uses key colors correctly, but it seems the hw rendering doesn't handle the situation correctly. In my last release dxwnd.2.05.72.nockey3.rar the "ddraw:DisableColorKey" tweak is just a useless experiment: it should eliminate the usage of key colors, but in my testbed the result is awful: the texture holes (that I don't see ...) should disappear, but also all other texture transparencies will disappear, so it's better to forget about it.
The only way to fix the problem could be the texture hack feature, by finding, manually editing and replacing all textures that have holes by recoloring in dark gray the black pixels. We could maybe start with the window holes that are a limited number of pixels easily detectable.
Well that'll will take a long time. I am not surprised to find that PCs over 11 years ago had this problem already: https://www.youtube.com/watch?v=fWJtNp6JteU
Some people 10 yrs ago also didn't have this problem: https://www.youtube.com/watch?v=AyLUK55waq4&t=118s
However I am more surprised many PCs currently don't have this problem, as indicated in this speedrun published 1 month ago: https://www.youtube.com/watch?v=QzbTlObXCjU
Some people at VOGONS gave me the best solution, to use dgVoodoo2. However dgVoodoo2 doesn't run being hooked through DxWnd in this game. dgVoodoo2 can run the game separately without DxWnd. Can you look into it, because CD audio is another feature I need from Rainbow Six that was discussed earlier in the group?
FIXED, here you go. I just had to hook WineD3D through DxWnd. Surprisingly I tried this days ago and didn't work, possible suspect is savegame got corrupted(and the game didn't run at all). But anyways its working completely fine now.
Another good news, game works on "leaked" Windows 11 too, a relief if you thought some amount of backward compatibility will be removed in the OS. PS: VMware can't run on HW acceleration so its SW acceleration footage.
Last edit: BEEN_Nath_58 2021-06-16
Very good.
But be ready for some possible complications. WineD3D replacement rely on OpenGL, and OpenGL has its own drivers and releases, the most recent ones also supporting pixel shaders. This, again, means that depending on your video card and the opengl drivers support you can get out of WineD3D either an excellent hw support or a poor sw implementation, pretty out of your control. In any case, it seems to me that opengl support is more stable than DirectX support (incredible, isn't it?) so in the end the WineD3D con easily be the best option.
afaik ogl implementation on nvidia is better than old DX implementation at least for the cards released in the cards released in the past 10yrs. until someone is playing it on a 2005 gpu there shouldn't be much issues. However I am unaware of the current AMD side of things. I will try to look into it if possible.
The "suppress dx common errors" is no longer reqd so you could update the rainbow six dxw profile saying it's for the HW version and requires WineD3D. As for SW rendering, this file works and doesn't require wined3d or anything:
Slight change to my previous statement, DxWnd can't actually hook WineD3D so the when the game ran for me it picked up the WineD3D files for which it worked and dgVoodoo2 hook gives an error. The solution is to not use DirectX API hook in DxWnd by selecting it to None, which results in both dgVoodoo2 and WineD3D working since the game uses them directly. Along with it I had to change a few mouse setting or else it will result in an infinite spin loop. Also a problem existed by which the desktop cursor would reappear which should be fixed by the profile provided:
Last edit: BEEN_Nath_58 2021-09-01
If you want to experiment with ddraw replacements (WineD3D or dgVoodoo2) you should be aware that there's a little trick: all dlls of the chosen replacement must be "seen" by the program, and this brings some problems with some dependency, like wined3d.dll in the case of WineD3D. So, this file must be copied either in DxWnd folder or in the game folder.
I apologize for not explaining the matter before or in the help documentation (I will recover). In attach a picture of the game taken with WineD3D replacement: in my case I had no transparent pixels, but the screenshot shows a perfect fence (like in SW mode) but the game runs in opengl hw accelerated mode and the bricks texture shows bilinear filtering, so definitely this is my best result so far.
P.s. if you have any doubt about what dll to drop and where, you can experiment also by forgetting the dxwnd tweak and dropping the whole ddraw replacement in the game folder.
Last edit: gho 2021-06-18
It was the problem that DxWnd didn't notice wined3d.dll as you mentioned. I put a note in the HW .dxw profile to put the .dll files in the game directories. The DxWnd tweaks "hook:ReplaceDirectXDLLs" is no longer required such.
I made sure id everything right now. You can put these 2 files in your next DxWnd release. Hope it helps.
I updated the 2 .dxw profiles I posted. I verified the colour bending and it was caused by dissimilarities in the ddraw.dll. I added some notes to let people know not to use WineD3D or dgVoodoo2 for SW rendering. Also dgVoodoo2 doesn't have the best support with AMD GPUs(it gives a black and white colour with no logo in AMD GPUs) so I had to change some universal settings.
Last edit: BEEN_Nath_58 2021-06-18
Pinging this thread again, I made some further changes and now it works on more PCs. And I also replaced renderer from GDI to primary buffer as asked. It now even works on 1st gen Intel HD graphics on both HW mode and SW mode (also supports wined3d which works extremely well, and dgvoodoo2). I believe this is the way the vanilla game should work.
Curiously, you mention updates to some attach but I see nothing attached in your posts ...
It is worth also to be aware that SF portal may have strange behaviors when re-posting files with the same name: it may assume that a file with identical name is already available and not update it. I personally always try to use this little trick (for instance with different releases of dxwnd.dll): I envelope the file within a compressed archive with an unique filename, so that there will be no doubt about the content of your download.
So, I would suggest to compress your final versions of .dxw files in some archive with original name (like rainbow_1.rar) and post it again here. Then, if it will ever happen that you will make updates, you will post the new files within a rainbow_2.rar and so forth.
Last edit: gho 2021-09-09
As asked:
thanks, perfect.
p.s. by the way, your comments on a single text line showed me a little DxWnd.exe defect: since the line wrap was not enabled most of the text was invisible and needed to be scrolled. I changed the EditBox behavior, now it will be no longer necessary to split the text into more lines, the line wrap will be automatic. Thanks for making me notice that.