Deadlock 2 under Windows 10

[PC gamer x32]

Having tried the suggestions from the "mouse problems" thread on Deadlock 2, I think Windows 10 might present a new challenge. In full screen mode (the default), many of the game screens go blank at the slightest move of the mouse. Having tried 50 different settings in DXWnd, I finally found one with an interesting impact:

Running DXWnd in Windows 10 (64-bit). On "DirectX" tab, in "Emulation" section, click "locked surface" (instead of the default "primary surface"). Then run Deadlock 2, hit spacebar 5 times (skips intro/videos) to arrive at main game screen. Now when the mouse is moved, it leaves a trail of black blocks instead of blanking out the whole screen. (Warning: I have zero DirectX experience).

You can get a similar problem adding "-w" to the command ("deadlock.exe -w") and running in windowed mode. Where before I was seeing two problems on Windows 10 (mouse smear vs complete screen blanking), there's now a flag that makes it only one problem.

Anyone discovered tricks that work for Windows 10 in running Deadlock 2?
(I've read the Windows 7/8 thread about "mouse problems" without luck)

[gho]

I'm getting a lot of problems in Win10. Just to shorten the hunt, does this happen on the terrible (from DxWnd point of view) patched game or on the original game as well?
Do you have examples of other games that have a similar problem?
Setting "Locked surface" could be a partial solution, because this mode doesn't emulate color depth, so any game with less than 32BPP color depth is out!

[PC gamer x32]

A user on gog.com working on a new version of ddraw.dll has gotten everything working for me. He made source code changes to ddraw.dll, which I tested, and no longer see glitches. For a slight interest in the discussion:
discussion of black screen issue

For those interested in either ddraw.dll or the source code fixing the problem:
dll and source code of solution

And for those who want to diff against the prior version of ddraw code:
earlier version of ddraw code

I've attached the ddraw.dll for posterity, in case the original disappears.

[gho]

Great job! Let's cross fingers and hope that finally I can understand this puzzle of the Deadlock2 patch. Not being able to understand nor fixing the problem was one of my greatest disappointment.
Thank you, gamer.

[gho]

@Narzoul: you did a wonderful job. To me, it seems that the overall idea is similar to the one implemented by the DxWnd "Emulate device context" flag, but with something extra that eliminates flickering and black trails.
It seems possible that the trick is in the new files CompatGdiDc.cpp and is related to a caching of the DC surface to manage clipped regions, maybe owned by other threads, to allow recovering black-filled regions.
I'd like to replicate your schema in DxWnd, also because it's not possible to run DxWnd and your ddraw wrapper contemporarily (I tried, but the result was not good!).
Since the inner structure of DxWnd and your ddraw wrapper are quite different, I can't simply ask permission to copy your code in my project. May I ask you, instead, to explain the basic idea behind the fix so that I could try to reimplement it within DxWnd?
Of course, you'll deserve my public thanks.

[Anonymous]

oh, i already know this project since last month, and I had intended to introduce it for you because it looks indeed great.
Look at this page, gho:
https://www.gog.com/forum/general/ddrawcompat_yet_another_directdraw_wrapper/page1

narzoul said that his wrapper fixed broken Battle.net interface on StarCraft: Brood War, and do you think that his wrapper may fix the menu glitches in Diablo 1?
http://www.accursedfarms.com/forums/viewtopic.php?f=17&p=239459

This will be a big benefit if we implemented the functions of ddrawcompat for Dxwnd. Simply create an account from GOG and send him a message.

[gho]

I just did it .... there I'm ghotik2002.

[Narzoul]

Hi, I'm the developer of DDrawCompat. (Uhh, this is going to be a long post...)

First of all, my initial approach to this problem is partially documented at the end of the readme on GitHub. It may still work for Deadlock 2, but it was suffering from heavy GDI resource leaks in StarCraft so I abandoned that approach. It seems like some of the ReleaseDC calls were not being caught by hotpatching - perhaps being called only internally by some kernel mode driver like win32k.sys. Unfortunately the leak was heavy enough that it even broke my desktop rendering after a longer test session (even after exiting the game) and the only way to recover was to log off from my Windows account. (Strange, I thought the number of GDI handles is nowadays only limited by system memory, at least according to some docs I remember reading... maybe something else was broken and the problem wasn't caused by the leak?). I couldn't think of a good enough solution for this (I know next to nothing about kernel hooking) so I changed approach for the upcoming v0.2.0 version.

Now instead of hooking the various GetDC functions and ReleaseDC, I'm experimenting with hooking only the GDI rendering methods (e.g. BitBlt, DrawText, etc). This requires only a temporary replacement of the HDC parameters with redirected DCs which can be immediately freed (or returned to the cache, as is the case) after the operation completes, avoiding any leaks.

I've pushed part 2 of the GDI overhaul to GitHub now so you can check a more complete version of the changes (I'm not adding a binary release for now), though for Deadlock 2 the changes in part 1 would probably be sufficient, as the game only seems to rely on BitBlt. Part 2 hooks additional methods and I'm now getting usable results with it in StarCraft, though some minor glitches still remain.

The basic idea is to redirect all GDI operations to the DirectDraw primary surface, like it used to be in the days before desktop composition. I assume that you also create the game's primary surface as an off-screen surface and periodically copy it to the real primary surface to make it easier to support color conversion, flips, etc. But most of this may work with the real primary surface too.

Basically you can use IDirectDrawSurface's GetDC to get a compatible DC for your primary surface. Then you need to take care of adjusting the client origin of the new DC to that of the original. I use GetDCOrgEx and SetWindowOrgEx for this. If you plan to support operations other than BitBlt, you would also need to copy over some other DC attributes and selected objects, see copyDcAttributes in CompatGdiDc.cpp.
Finally you need to replicate the original clipping region. Unfortunately you cannot set the "system region" that you can obtain with GetRandomRgn, but you can intersect it with the user clipping region (ExtSelectClipRgn) which should provide the same end result. Also, GetRandomRgn with desktop composition enabled no longer accounts for overlapping top level windows, so you have to remove these manually from the clipping region to avoid drawing on them. See the EnumThreadWindows call for this. It is assumed that it enumerates windows in front-to-back Z-order (so you can stop once you reach the target window), although this doesn't seem to be mentioned in the offical documentation.

Now, there are some technical difficulties with the above. For example, you can only create one DC from a DirectDraw surface. To work around this, I use a trick (more like a hack): I create "client-memory surface" clones (see DX7 SDK) that all map to the primary surface's video memory, then each of those can provide a separate DC. Now, this no longer works on Windows 10 without locking the primary surface for the duration of the GDI rendering (it seems that after unlocking, anything written to that memory will not be reflected in video memory, like it used to be the case on Windows 8 and earlier), so watch out for that. Also consider that the locked surface memory address might change from lock to lock for video memory surfaces, though in practice this only seems to happen after the first DirectDraw Blt (which is why I have fixSurfacePtrs in CompatDirectDrawSurface.cpp).

Keeping the DCs cached is not necessary. I originally did that so that I can create a certain number of cached DCs up-front so that I can avoid locking the DirectDraw critical section every time for GDI rendering, which can lead to deadlocks. But now that on Win 10 locking the primary surface is anyway necessary, it seems there is no easy way to avoid accessing DirectDraw anyway. (You could of course force an emulated primary surface into system memory, in which case the locking is probably not needed - I will need to try this in the future as I'm anyway having performance issues with software palette conversion in some games...)
I kept the cache around because there is some cost to creating a fresh client-memory surface and a DC for that every time (about 200-300 microseconds according to some measurements I took). Of course, you should make sure you clean up the DCs before returning them to a cache - at least the SelectObject parts (or maybe it's sufficient to clean that up only when releasing it from the cache).
Another thing to watch out for is palette changes, in which case the simplest route is again to clear the cache and create new client-memory surfaces with the new palette.

The main drawback of this approach is that not all rendering can be caught. Again, probably some of it is done internally in win32k.sys and I know of no obvious ways to hook that. I already only use hotpatching as that seems to catch the most calls and has the least interference from Microsoft's own compatibility hooks. For the unhookable internally rendered parts (caret, WM_ERASEBKGND) I use some manual workarounds, you can find these in my source code (CompatGdiCaret and CompatGdiWinProc). There are still some minor missing stuff, like the title bar of the save dialog that appears in StarCraft's Battle.net menus when pressing print screen. I haven't added all the necessary workarounds for those yet.

The above may also cause other problems for you, because DDrawCompat uses the "DXPrimaryEmulation -DisableMaxWindowedMode" compatiblity shim, so that all those uncaught internally rendered things don't show up in full screen games at all. In windowed mode things might be different and DDrawCompat currently doesn't support windowed GDI interworking for that reason. It's even possible that the interference would actually have some positive effects on the end result...

There are of course alternative solutions to try that I haven't had time to research sufficiently yet. For example, in theory, you could redirect all DirectDraw rendering to the GDI back buffers instead (using GetDC with all visible top level windows to obtain those back buffers), then this might work better in windowed mode. But it may have performance problems, especially when you need to provide Lock or other read access (e.g. Blt from primary surface). I also don't know if those back buffers can be made to reside in system memory. Some MS articles make it sound like that starting with Win 7 they only exist in video memory, and prior to that there was a system memory copy as well which was synchronized with the video memory copy. Perhaps that behavior could be restored by the NoGdiHWAcceleration shim, or by some other odd features like WS_EX_COMPOSITED style rendering.

As for how GDI interworking works by default on Windows, I'm not exactly sure. I could swear that earlier I was able to capture some API calls (with the API Monitor from rohitab) that showed apphelp.dll or (some similar compatibility dll) creating a WS_EX_COMPOSITED window overlayed on top of the main application window (it was even named something like "Compatibility Layer"), but for the life of me I can no longer reproduce it. Anyway, my theory is that such an overlay may be used by Windows to render GDI content on top of the DirectDraw primary surface, and it's possible that parts of it get erased (i.e. set to transparent) when DirectDraw updates that portion of the primary surface. However this would also mean that neither surface can see what is on the other surface, which is why transparent rendering doesn't work correctly (e.g. the text on button in StarCraft).

All of the above assumes you want to handle some of the more complex scenarios as well (StarCraft...). For Deadlock 2, you may really only need to redirect BitBlt.
I think I also used to have an early test version (without DXPrimaryEmulation) which simply copied the main window's DC to the DirectDraw primary surface before each Lock operation and that also seemed to fix the black screen issue. But that may have had performance problems as well, I can't remember.

Feel free to "borrow" any source code from me. I also need not be mentioned in any credits. If you need further clarification, just ask.

[gho]

To begin with, thank you a lot for the explanations.
I'll need some time to "digest" this all, but for sure I may easily come with further questions, hoping not to annoy you.
I'll spend just a few words to explain how DWnd handles the situation:
Currently, DxWnd uses several ways to manage GDI and ddraw interactions, but none of theses seems perfect in all cases. Let me begin telling that usuallly I prefer release game support in surface emulated mode, that means that the program works on fake primary / backbuffer surfaces, built in memory and with original size and color depth, then these are to be rendered to screen in different possible ways. So, there are basically three types of GDI handling:
1) let it go directly on the window surface, maybe properly scaled, and hoping it won't interfere with ddraw....
2) make ddraw and GDI use a common DC using the fake primary surface DC, though this may easily produce deadlocks
3) build a memory DC for GDI with same characteristics as ddraw fake primary, let GDI work on this and then merge the results. This DC can be cached ("Reuse emulated DC" option) or created and destroyed each time: depending on the game, sharing can be faster but give locks.

For Deadlock II, the strange thing is that version 1.0 works pretty well in the first crude way, but the 1.2 patch has problems in all possible modes.

About StarCraft, I'd like to fix that as well, but battlenet seems to check for some CRC of the storm.dll content, so that patching it, even at runtime, is detected and banned.

I'll keep in touch
bye
GHO

[Narzoul]

Ok, I forgot to mention this important bit, which may also be responsible for a lot of the deadlocks you are experiencing.

As you may already know, DirectDraw internally uses a global CRITICAL_SECTION object for thread-safety, and it's used by most (all?) of its methods.
IDirectDrawSurface::GetDC is special in that it enters this global critical section and will not leave it until the corresponding ReleaseDC is called. Which means you won't be able to use DirectDraw from other threads as long as you have unreleased DirectDraw DCs, and that can be a significant source of deadlocks (had this issue myself).

Fortunately, there is a simple workaround. You have some basic access to this critical section through these two exported functions in ddraw.dll:
AcquireDDThreadLock
ReleaseDDThreadLock

So you can call ReleaseDDThreadLock immediately after GetDC and then later AcquireDDThreadLock immediately before ReleaseDC to avoid a major source of deadlocks, if you intend to keep the DC around for a while (like keeping it cached).

About StarCraft, I'm still not sure what detection everyone is referring to but if it's that message that comes up about being unable to detect the Battle.net version (or something like that), then it does not get triggered with DDrawCompat. In my experience it may not be entirely dependent on the hooking technique you use, because even with DDrawCompat's hooking (hotpatching) I could sometimes get this message when I botched something up seriously in the rendering. So maybe it checks some of the actually rendered content? In any case I didn't touch storm.dll at all, why would you want to?
I haven't actually tried starting a game from Battle.net, but logging in, chatting and things like that work (in the unreleased version and also in the older "leaky" releases), doesn't that mean those checks didn't kick me out, or does it happen at a later point?

[gho]

About Starcraft, I do believe Battle.net checks the memory segment of the loaded storm.dll to verify it is inaltered. You don't experience this problem because the hacking is in an external wrapper, so it is the ddraw segment to be modified. Another way to act unnoticed is to use what I called (just to give it a name) "hot patching", that is inserting a detour assembly code at the beginning of the called routine. But DxWnd mainly uses a patching of the IAT (Import Address Table?) in the calling modules, then the STORM memory segment do get modified. Just as an experiment, I tried to "hot patch" all the calls, and Battle.net didn't complain, but DxWnd didn't work correctly this way.....

[gho]

@Narzoul: Hi!
I've made some first very rude attempt to use your suggestions, and the result has been surprising! It's a copernican revolution: I no longer get a service DC by GetDC or BeginPaint to be handed to all other calls, but I leave GetDC and BeginPaint untouched and redirect the DC inside the blitting API. Reusing the same calls that I referenced to switch the DC the result was at least as good as before, with no more deadlocks, and I didn't even begin to fix these routines with the clipping region, coordinate origin and other compensations that you suggested. I'm excited at the idea of building a single emulation mode that copes with all possible cases!
More than this, the solution seems ideal to concentrate in a single set of routines (AcquireServiceDC and ReleaseServiceDC) all flavours of handling the matter.
I have a few issues for you, just in case you are interested or you may give suggestions

1) I'd like to use this solution to handle threee possible cases, that are
a) there is no ddraw session and you could redirect to a virtual DC created for this purpose, with same size and color depth as the program required
b) derive a virtual DC from the current ddraw virtual primary buffer,
c) derive a virtual DC from the surface of the D3D device.
The first two ones were somehow already implemented, though with the problems of my previous architecture, but I don't know if it is possible to catch the equivalent od a virtual primary surface in D3D8 or D3D9, since that seems not existing. Ah, in theory I could have the same problem also in case someone mixed GDI and OpenGL, though I do hope noone did it. A good example (in reality, the only one I encountered so far) of D3D + GDI is the 1998 version (a nice 3D remake, and also released free!) of Battlezone.

2) You suggested to detect and compensate the DC origin by using GetDCOrgEx and SetWindowOrgEx. I used SetWindowOrgEx to compensate the case where the program is not blitting directly on the primary surface, but it's doing this on a overlapped child window, setting the origins as the displacement betwaan the (0,0) coordinates of the two windows. Likely, a more general solution should detect the surface origins and compensate for the composition of different origin and different window position. A game that shows this problem is "Star Trek Armada" in its initial menues. I don't know if you considered this situation or if you could be interested in what I'm saying, but just in case ...

3) I put beforehands that I haven't had time to analyze all your code yet, so I may say something already considered, but I wonder if between all DC characteristics one should consider also a difference in measure units: there's a window of the detailed map in "Imperialism 1" (the one that is activated by the magnifying glass) that I never properly managed and always causes a game crash. I suspect that not managing the different measurement units of the background could be the cause.

[Narzoul]

1) c) Actually there seems to be a GetDC for D3D9 as well (https://msdn.microsoft.com/en-us/library/windows/desktop/bb205894(v=vs.85).aspx) but not for D3D8. But as long as you can lock the surfaces, I guess you could always fall back to DirectDraw for providing DCs even for D3D8/9 surfaces. You just have to pass the surface width, height, pixel format, pitch and surface memory pointer to IDirectDraw7::CreateSurface (actually I'm not sure what's the lowest interface version that supports this), as it is explained in the "Creating Client Memory Surfaces" section of the DX7 SDK docs. Here's an online version for WinCE which is probably pretty much the same: https://msdn.microsoft.com/en-us/library/aa451271.aspx

Something similar may actually be available for OpenGL too, as there seems to be a function in WDDM that does exactly this (D3DKMTCreateDCFromMemory: https://msdn.microsoft.com/en-us/library/windows/hardware/ff546826(v=vs.85).aspx). In fact, the DirectDraw implementation may actually be using this when suitable drivers are available, as you can find this function in ddraw.dll's import table.

2) I'm not sure what you mean here. According to the documentation of GetDCOrgEx (https://msdn.microsoft.com/en-us/library/windows/desktop/dd144874(v=vs.85).aspx) it should always give the screen coordinates corresponding to the the logical (0,0) coordinates. Although on the same page there is a comment that this doesn't actually work this way for WS_EX_COMPOSITED style windows. Perhaps Armada is using this? In any case the only time I got seemingly inaccurate results with this method so far is for popup menus, and I haven't yet investigated how to fix that (or even if this was the issue). Maybe ClientToScreen could be used to correct any differences, if that is actually more reliable...

3) You could take a look at the various coordinate space and transformation functions (https://msdn.microsoft.com/en-us/library/dd183476(v=vs.85).aspx) and try to copy over all of the transformations and settings from there as well. I haven't bothered with this myself because I didn't think any games would actually use these, but who knows...

[gho]

I made some experiments recently (one of the reasons why I didn't write as much as usual) and discovered a few interesting, though annoying, things about Deadlock2 v1.2 mouse handling.
One must say that the initial version 1.0 has none of htese problems, and it's really a pity that a good gameplay patch made also the game maybe impossible to handle in window mode.
The initial problem, that became then more complicated than ever, var black trail following the cursor, as if the surface where the cursor was moved didn't get updated with the background before blitting.
Tracing (with APITrace) all system calls, it seems that the cursor surface is generated starting from a GDI CreateCompatibleDC call starting from the desktop window dc. Usually, that call generates a 1x1 pixel hdc that must be characterized with attributes by selecting a proper size, colo depth, image and so on. But in this case nothing of this is performed, apparently because of a peculiarity of the call on desktop dc ( https://msdn.microsoft.com/it-it/library/windows/desktop/dd183489%28v=vs.85%29.aspx ): "If this handle is NULL, the function creates a memory DC compatible with the application's current screen.". So, it seems that this dc is given birth with the property of mapping the desktop graphic no matter when or where it is created! On Win7, using the attached configuration, the game runs almost perfectly (only problem, I couldn't delete the system cursor even configuration DxWnd to delete it!). One may try it and see. The configuration simply tells no ddraw emulation (otherwise the desktop dc would be covered by the ddraw layer) and no scaling. This may imply a desktop color depth change, but that is not too bad.
But if you try to replace the 0,0 size (that means no scaling, size equal to resolution) with anything different, e.g. 800x600, or you simply stretch the window grabbing the borders, the blitted cursor seems to inherit the background from some bad location and the screen becomes a mess.
And what even worse, when I tried the same experiment at home on my Win10, the back trails/screens reappeared, as if the DCM added a dc layer or changed somehow the rules.
Another significand difference is that Deadlock 2 v1.2 can work perfectly in fullscreen mode on Win7, but it doesn't in Win10, where Narzoul dll is required to make it working correctly.
I still don't know what to think about it, but I wonder if Narzoul dll approach would work in a scaled situation.

[gho]

Good news, at last! The radical change sggested by Narzoul is starting to produce good side effects, though I'm not above 50% of implementation (likely, much less....).
Applying the release under development for ms Golf '98 I got the unexpected surprise that it handles perfectly the Deadlock 2 v.1.2 mouse, even in scaled windows (the one in the screenshot is 800x600) and surface emulation, compatible with Win10 and evey other OS since XP.
Deadlock 2 fans will have to wait a little more because I saw that there are at least clipping problems, likely because of the part of code I didn't migrate yet, but this is a GREAT result: thank you, Narzoul!

[gho]

I'm in the middle of a huge revolution in gdi32 and user32 handling (it's probably more than a hundred calls to restructure!), but I could leak this beta that is good both for Deadlock 2 Shrine Wars version 1.0 (as it was before) and for the patched version v1.2, that was having so many cursor and flickering problems. Work is far from finished, as shown by the need of two completely different configuration files, but I hope to get a configuration working with everything to use as a default, sooner or later.
Be careful that compatibility with other games is not garanteed, so you'd better install in a separate folder on top of another copy of v2.03.52 latest release.