Re: [DirectPython] Sending keys

[Heikki S. <ho...@gm...>]

> -----Original Message-----
> From: Skepsis [mailto:sk...@bl...] 
> Sent: 30. marraskuuta 2007 22:28
> To: ho...@gm...
> Subject: Re: [DirectPython] Sending keys
> 
> Sorry forthe direct mail ... I can not seem to reply tothe 
> discussion board?
> 
> Thanks for the mail. I will check them out but the problem is 
> that the normal methods of sending input like win32 send 
> message / keyboard_event mouse_event even
> the neat Sendkeys module,  only work   for 'normal' windows.  
> I need to
> control game program windows and these are not affected by 
> the normal methods.
> 
> I have yet to try post message and the sendInput you pointed 
> me to once I understand it some :) and you are right I dont 
> know C or C++.
> 
> Thanks again
> Adrian
> 

I added the mailing list to receivers, just in case someone else is
interested about this. 

As you have noticed, keyboard_event() and others only work with native
Windows controls. Most games don't use them because they usually need some
special styles or effects. Games running in fullscreen can't use them even
if they wanted (with some minor exceptions).

So the usual solution is to implement "windowless" controls. They create
custom windows which are specific to the application. Windows (the operating
system, that is) does not know anything about them. The d3dgui module which
comes with DirectPython is one example. Some normal applications also use
windowless controls (like Internet Explorer).

So how do they expose themself to the user? Normally they don't have to do
much. When a user moves the mouse (or presses a key etc.), the main window
receives the event and the application then figures out what widget should
receive it. Everything works fine as long as the user is conserned. Problems
arise when you need to use these things without normal (mouse and keyboard)
methods.

For example testing, usability support and in you case voice control are
harder with windowless applications. There are several ways in which you can
dig information about these controls that are normally invisible to the
operating system. These are some common ways in which windowless
applications expose their "internals":

-Implement Windows Active accessibility API. Implement IAccessible
COM-object that provides information about the custom controls. This is a
normal way to support screen readers and other accessibility applications.

-Implement custom COM/OLE support. Microsoft Office, Internet explorer and
some other applications can be controlled programmatically and information
can be asked about their contents.

-Implement some other accessibility protocol.

-No support, use OCR and other methods to dig info about the application.

-No support, you must reverse engineer the data structures used by the
program. This is difficult and dangerous, but sometimes there are no
alternatives if you have to do it. 

If you only need to support windowless applications created using Python and
they use open-source windowless GUI-libraries, there is another way. It
might sound weird or difficult, but it should usually work quite well. Not
for the faint of heart, though! It also requires some C-experience, altough
it is be possible to write dll's with other langueages.

Suppose you want to get information about the process A, which uses
d3dgui-library (or it can use some pygame GUI, it does not matter much) for
it's controls. No normal method works, so you have to get creative. Your
voice control application is process B.

Create a very basic .dll which is then injected into procees A. The dll uses
Python C-API to find the module d3dgui and overrides (for example)
Manager.render() class method or adds a property which just monitors the
custom controls and notifies process B about the existing controls (like
position and text). Different GUI-libraries require different approaches,
but the principle is same. You can also use a debugger to figure out how the
application works internally. 

You could also try to modify the application before it starts (especially if
it is done by py2exe or some similar method), but it can cause some
problems. However, that could be easier than doing it on the fly. Without
knowing more accurately what your application does, I can't give much more
hints. It's up to you to decide what you need are how much work are you
willing to do for it.

--
Heikki Salo