amforth-devel

[Amforth] Filename completion for the amforth shell

[Matthias T. <mt...@we...>]

Hi Keith,

I've played with filename completion  for your remarkable
shell. First the good news: it works (for me). The bad news is that
the code is by far not as readable as yours. But I tried my best.

What does it do

Upon startup (and whenver a general update seems appropriate) it
reads the whole directory tree under the current directory and
all directories the environment variable AMFORTH_LIB points to.

While doing so it tries to recognize double occurencies of the
same file (e.g. if ./lib/test.frt and $AMFORTH_LIB/test.frt are
the same file it counts only as one -- difficult to check). For
every file the directories are stored in canonical form in a map
(self._filedirs). Whenever a filename needs to be processed
the program checks if it is unique over all directories (the
map contains 1 entry for that file).

If that check succeeds, the file is processed the usual way.
If not, an error message is printed.

In interactive mode, the #include command works the same.
With filename completion (without telling the number of
directories however).

The drawback of the current code is that you cannot give
directory names any more. Only the filenames itself are
allowed. This is fine for me, but may bother the users.

mt@ayla:~/amforth/trunk/tools$ ./amforth-shell.py -p /dev/ttyUSB0
--no-error-on-output
|I=Entering amforth interactive interpreter
|I=using device.py for atmega328p
(ATmega328P)> #include test<TAB>
test2.frt           tester-amforth.frt  tester.frt
(ATmega328P)> #include test2.frt
|D=#include test2.frt
|I=using test2.frt from /home/mt/amforth/lib
|I=mcudef
|I=using device.py for atmega328p
|F=/home/mt/Projekte/avr/forth/amforth/lib/test2.frt
|C|    1|\ this is a test
|S|    2|100 200 + .
|O|    2|300
|S|    3|1000 ms
(ATmega328P)>

The current code is in the repository (rev 1251)

Matthias

Re: [Amforth] Filename completion for the amforth shell

[Keith A. <ca...@pi...>]

{-- Thu, 12 Jul 2012 20:50:28 +0200: Matthias <mt...@we...> wrote: --}

  Matthias> Hi Keith, I've played with filename completion for your
  Matthias> remarkable shell. First the good news: it works (for
  Matthias> me). The bad news is that the code is by far not as readable
  Matthias> as yours. But I tried my best.

Nice.  Maintaining completion while supporting search directories
invalidates my original solution for completion and implementing it is
definitely a jump in complexity.

<description of _filedirs map based implementation omitted>

  Matthias> The drawback of the current code is that you cannot give
  Matthias> directory names any more. Only the filenames itself are
  Matthias> allowed. This is fine for me, but may bother the users.

I think another limitation is that the use of the #cd directive to
change directories won't work in files any longer.  Preserving this is
slightly tricky because of the way the code tracks nested directory
changes in files included with the #include directory, but to do so I
would replace the call to:

 os.path.normpath(os.path.join(os.path.join(wd, filename))

near the top of the upload_files() method with a call to a function
like:

def find_upload_file(self, filename, working_directory):
    if os.path.isabs(filename):
        return filename   # Don't search if given absolute path
    for p in self._search_path:
        fn = os.path.normpath(os.path.join(p, filename))
        if not os.path.isabs(fn):
           fn = os.path.normpath(os.path.join(working_directory, fn))
        if os.path.exists(fn):
             return fn
    return filename        # The attempt to open later will fail...

I think this should handle everything required to get the search
behavior working if at startup you split the AMFORTH_LIB environment
variable contents into the "self._search_path" instance variable.  Note
I haven't actually tried to implement just typed it in.  It would be an
alternative to the way you have it implemented right now.

The completion support is trickier.  Roughly, I would replace the code
in the _rl_completer function for #include and #edit directives with
something like:

if os.path.isabs(text):
   self._rl_matches = glob.glob(text + "*")
else:
   self._rl_matches = itertools.chain(map(lambda p: \
        glob.glob(os.path.join(p, text) + "*"), self._search_path))

This will essentially return the union of the glob matches for the
supplied completion text in all possible search directories if the
completion text is not already an absolute filename.  It might get slow
if you have a lot of large directories on your search path, but I
suspect it will be fine.  I also expect I didn't get it really right but
I think it should be close enough that you could get it working if you
started with this.

It's really nice to see these additional features being added!

       --- Keith

Re: [Amforth] Filename completion for the amforth shell

[Matthias T. <mt...@we...>]

Hi Keith,

> Nice.  Maintaining completion while supporting search directories 
> invalidates my original solution for completion and implementing it 
> is definitely a jump in complexity.

But its very convenient. Simply saying "#include marker.<TAB>" and
to get it without having trouble to find the right directory is really
cool.

> I think another limitation is that the use of the #cd directive to 
> change directories won't work in files any longer.

I dont think that this is a showstopper for the fully recursive
file lookup. You dont need to do a change dir if all files are
directly available. The list of the base directories could
be expanded at runtime, maybe.

> os.path.normpath(os.path.join(os.path.join(wd, filename))

python's path magic is only for the brave.

> This will essentially return the union of the glob matches for the 
> supplied completion text in all possible search directories if the 
> completion text is not already an absolute filename.  It might get 
> slow if you have a lot of large directories on your search path, but 
> I suspect it will be fine.  I also expect I didn't get it really 
> right but I think it should be close enough that you could get it 
> working if you started with this.

I'll give it a try. Thanks.

Matthias

Re: [Amforth] Filename completion for the amforth shell

[Keith A. <ca...@pi...>]

{-- Sat, 14 Jul 2012 20:33:29 +0200: Matthias <mt...@we...> wrote: --}

  >> Nice.  Maintaining completion while supporting search directories
  >> invalidates my original solution for completion and implementing it
  >> is definitely a jump in complexity.

  Matthias> But its very convenient. Simply saying "#include
  Matthias> marker.<TAB>" and to get it without having trouble to find
  Matthias> the right directory is really cool.

Certainly!  It's definitely worth the jump in complexity.  The code just
has to be a bit messier to handle it.

  >> I think another limitation is that the use of the #cd directive to
  >> change directories won't work in files any longer.

  Matthias> I dont think that this is a showstopper for the fully
  Matthias> recursive file lookup. You dont need to do a change dir if
  Matthias> all files are directly available. The list of the base
  Matthias> directories could be expanded at runtime, maybe.

I agree that it isn't a show-stopper but it is still useful if you have
some directories with code that you don't normally need and so keep off
the search path but still want to access conveniently.

  >> os.path.normpath(os.path.join(os.path.join(wd, filename))

  Matthias> python's path magic is only for the brave.

Hmmm.... I suspect that was a typing mistake, especially since there
aren't enough closing brackets.  Probably there was only supposed to be
one of those os.path.join calls...  ;-)

  >> This will essentially return the union of the glob matches for the
  >> supplied completion text in all possible search directories if the
  >> completion text is not already an absolute filename.  It might get
  >> slow if you have a lot of large directories on your search path,
  >> but I suspect it will be fine.  I also expect I didn't get it
  >> really right but I think it should be close enough that you could
  >> get it working if you started with this.

  Matthias> I'll give it a try. Thanks.

I'd love to try it out myself but I haven't had time to get my
microcontrollers unpacked and setup again after we moved recently.  Let
me know how it goes.

       --- Keith