[Emacs-vr-mode-devel] reimplementing vr-mode (long)

[Jan R. <ja...@ry...>]

Hi,

Having recently decided that I really need dictation in my XEmacs, I
have taken a close look at VR-mode and started hacking on it. I have
spent about a week on it, and it's time to write about my experiences
and bounce some ideas around.

First of all, many thanks to Barry Jaspan. Without his code, I would not
have been able to get started with this work.

I started by just trying to run VR-mode under XEmacs. That would not
work for a number of reasons. The code has really been written with GNU
Emacs in mind -- it uses overlays, which XEmacs does not have (instead
providing extents). It also seems to be quite old and works only with
older versions of Emacs.

The next logical step was to try to modify the code and adapt it to
XEmacs. But the more I looked at the code, the more I realized that is
very complex and that I simply cannot understand it. So, I decided to
start from scratch on the XEmacs side and promptly proceeded to rip out
all the code which I could not understand. Needless to say, that has
left me with very little code :-)

In the process of rebuilding the code I have learned quite a bit about
how it works, also diving into VR.exe along the way. I have discovered
many things which were rather puzzling to me -- for example, in spite of
the efforts to keep all changes synchronized after every buffer
modification, a complete resynchronization is done at the beginning of
each recognition.

The first changes to VR.exe were rather simple -- since my goal was to
dictate into Emacs/XEmacs running anywhere (not necessarily on the same
Windows desktop), I had to change the way I dictation was activated. In
principle, I wanted to dictate "into" the VR.exe window. The advantage
of these changes is that it let me get rid of the "hook.DLL" dependency.

As I was getting a better understanding of the code, I was also trying
to simplify it. The first thing to go away on the VR.exe side were
multiple Clients. The original code tried to maintain a separate Client
object for each Emacs frame, with each client object maintaining a list
of buffers. This doesn't really reflect very well how Emacs manages
frames and buffers, and introduces a lot of complexity. So, I have
reworked the code to use only a single client.

After that, I got the synchronization in a single buffer to work. It
actually wasn't as difficult as I had feared. I am now at a point where
I can dictate into an XEmacs buffer, use the select and say
functionality, and use "scratch that". It works for simple text buffers,
also with auto-fill enabled. Of course, you can also do edits on the
XEmacs side and mix them with dictation freely. There are still some
bugs remaining, but the basic functionality is there.

Then, I proceeded to implement dictation in multiple buffers. This
started getting tricky: I had to be very careful which buffers are
active in which corresponding custom dictation objects are
activated. And then, I finally hit a real problem. What I wanted to
achieve was flawless integration. I wanted to be able to dictate into
Emacs at any time, not just into "voice-activated" buffers. Basically,
whenever I can see the point on the screen, I should be able to dictate.

The problem that appears is that if you simply start dictating into a
new buffer, VR.exe still thinks you're dictating into the old buffer and
it requests the contents of the old buffer for resynchronization. There
is no easy way around that, because the new custom dictation object for
the new buffer doesn't exist yet, and even if we create it at this
moment, it is already too late -- dictation has started and DNS things
we are dictating into the old custom dictation object.

Of course, there are some ways of artificially working around that --
but I started thinking: why do we bother with storing the complete
contents of several buffers on the DNS side. It takes a lot of effort,
and a lot of communication, and is rather wasteful of resources. I also
have serious doubts whether DNS actually uses this contents for anything
-- I suspect only the immediate context is used by Dragon for things
like capitalization, spacing and punctuation, and the rest is never
looked at. So, I think the next step for me will be to rework VR.exe to
just use a single custom dictation object, whose contents would be
synchronized to whatever is currently needed. I also think that I will
not bother with synchronizing entire buffers. Instead, I will only
synchronized a small area -- closely related to what is actually visible
on the users screen. Comments and suggestions are appreciated.

Another thing which got me thinking was that there are two ways of
interfacing to Dragon NaturallySpeaking: you can use the Dragon native
API or SAPI. From what I can see, the Dragon native API is slightly
easier to use, but also assumes a lot about what you're trying to do
(for example, it automatically manages spacing for you, and there is no
way to turn it off), while SAPI is more complex, but also more
flexible. An example of this flexibility is the "Phrase Hypothesis"
method, which is supposed to provide several hypotheses for a given
utterance. This method might be very useful if we know a bit about the
context, which we often do in Emacs. However, I have no idea how well
Dragon supports SAPI and whether all methods are actually implemented. I
also don't know what is the strategic direction for Dragon
NaturallySpeaking -- which API will continue to be supported in the
future.

And finally (this e-mail is already long enough) -- does anybody know if
there is a way to implement things like the QuickCorrect menu, adding
words to the dictionary, or word pronunciation training without the
Windows desktop? It seems to me that the Dragon API and assumes that the
user is going to sit in front of a Windows desktop. You can request a
dialog box to be shown, but you can't simply access the functionality
yourself. But perhaps I simply haven't found a way to do it yet.

Your thoughts and suggestions would be very much appreciated.

--J.