#20 Support for custom completion

[Blackwell]

It would be nice to have a drop-dead-simple mechanism to add custom completion functionality to PyCmd.

I am thinking of completing something like my excessively long Git branch names for example, so that when I type...

git checkout [TAB]

...I will be presented the list of possible completions (==branch names).

[Horea Haitonic]

Totally agreed, this is a long-standing "future plan" in the wish list (see README.txt).

The simplest (and most powerful) thing would be to allow the user to create individual Python functions for some commands: complete_dir, complete_git etc.

PyCmd would split the user input into separate commands (think "cd myrepo && git checkout -b mybr"), detect the actual command (say, "git"), and then call the appropriate complete_git() with the rest of the command as arguments (['checkout', '-b', 'mybr']).

The custom complete_git() has to know how to parse the arguments and return a list of suitable completions for the last one (say, ['mybranch', 'mybrokenbranch']).

This list returns to PyCmd and gets handled just as the results of complete_file() and complete_env_vars() are handled now.

Does this sound reasonable?

[Nobody/Anonymous]

Hello Horea.

Yes, that sounds reasonable.

As a first step I created the simplest (and not quite correct, or rather incomplete) variants (not matching your specification/description yet) of a Git completer and code to invoke it that I could come up with.

Your input on it would be very welcome.

[Horea Haitonic]

Thanks for the patch, I managed to get it to work after some tweaks; here are my remarks:

1) relying on sys.argv[0] to determine the location of the PyCmd installation is risky, as it is not guaranteed to contain a full path -- I can easily get it to crash if I start PyCmd.exe using something like .\PyCmd\PyCmd.exe (this is why I had to tweak the installation directory by hand)

2) For my pycmd repo, the completion is not very reliable; here is some example output:

D:\Users\h\pycmd> git branch <Tab>
alt-slash
bdu-prop
master
proposed
run-improvements
^^^ These are my current branches, all ok up to now

D:\Users\h\pycmd> git branch m<Tab>
master
run-improvements
D:\Users\h\pycmd> git branch
^^^ Note how the completion list is wrong, and the "m" that I typed is gone

3) The substring-based completion that you suggest is a good idea in principle. This is yet another completion style that we might or might not want to support for the general completion mechanism. Emacs uses this approach to completing, even for file names; sometimes it's nice, sometimes it's annoying. Different ways of having this in PyCmd come to mind:
a. Add this as a global option that modifies the general behavior of completions across PyCmd (generic -- files, or custom)
b. Use a different key (Shift-Tab?) for this substring-based completion (global, for all completions in PyCmd)
c. Allow each custom completion function to define its own approach, seeing as different modes make sense for different completion use cases

My favorite right now is option c (the one that you already implemented), but I am interested in your opinion about this. Would you find the substring-based completion useful in general, or just for some particular commands?

4) I really think that, in the end, the custom completions mechanism should be more general:
* use something like complete_git() instead of complete_git_branch()
* split the line before, and call the appropriated completion function only
* use the existing line-splitting code for the above, as it has a lot of work put into it (this is a harder task than one might guess)

Apart from the above, you seem to be on to something. Let's see if we can get this right...

[Blackwell]

Hello Horea.

1) relying on sys.argv[0] to determine the location of the PyCmd
installation is risky, as it is not guaranteed to contain a full path -- I
can easily get it to crash if I start PyCmd.exe using something like
.\PyCmd\PyCmd.exe (this is why I had to tweak the installation directory by
hand)

Ah yes, that was a quick hack to make it work out of the box, not properly thought through, not meant for production.

On this topic I was wondering whether it would make sense to allow to have "completers" in the PyCmd installation folder as well as in the PyCmd data folder or so (global vs. local).

2) For my pycmd repo, the completion is not very reliable; here is some
example output:

[...]

^^^ Note how the completion list is wrong, and the "m" that I typed is
gone

Ah, thanks for reporting this, will check it.

3) The substring-based completion that you suggest is a good idea in
principle. This is yet another completion style that we might or might not
want to support for the general completion mechanism. Emacs uses this
approach to completing, even for file names; sometimes it's nice, sometimes
it's annoying.

Yes, I guess it is up to user preference often. (Substring-based completion only occurred to me while playing around, and you have set the example with PyCmd anyway. But it is a good example of me having to play around with things before I have a good idea of what I like or dislike.)

Different ways of having this in PyCmd come to mind:
a. Add this as a global option that modifies the general behavior of
completions across PyCmd (generic -- files, or custom)
b. Use a different key (Shift-Tab?) for this substring-based completion
(global, for all completions in PyCmd)
c. Allow each custom completion function to define its own approach,
seeing as different modes make sense for different completion use cases

Good list, good to get an overview.

My favorite right now is option c (the one that you already implemented),
but I am interested in your opinion about this. Would you find the
substring-based completion useful in general, or just for some particular
commands?

I can't say at the moment, will have to think about it some more, sorry. There may be people with heaps of branches, for those it would not be good to have it complete per substring I guess.

I guess it should be configurable per completer to be able to make everyone happy, and probably in PyCmd, rather than leaving the decision up to the completer. (Thus voting for PyCmd having the final say.)

4) I really think that, in the end, the custom completions mechanism
should be more general:
use something like complete_git() instead of complete_git_branch()
split the line before, and call the appropriated completion function
only

Hm yes, that seems simple to implement.

• use the existing line-splitting code for the above, as it has a lot
  of work put into it (this is a harder task than one might guess)

Definitely.

Clemens

[Horea Haitonic]

Hi,

In case you are still working on this, I added

pycmd_install_dir = os.path.dirname(os.path.abspath(sys.argv[0]))

as a global variable in PyCmd.py. It is currently used for locating init/config files, but you could use it for locating the custom completion functions. It works for both the compiled PyCmd.exe and running the script directly via 'python PyCmd.py'.

[Blackwell]

Hi,

it is still on my list. I have made some first changes to alter the design according to your raised points, but it is not finished yet.

Thanks for pointing out the change/addition. (I have a habit to blindly pull from time to time anyway, always expected a nice surprise (changes, new features) anyway.)

[Blackwell]

Hello Horea,

you wrote that I could use the variable...

pycmd_install_dir = os.path.dirname(os.path.abspath(sys.argv[0]))

...but there is a problem with that:

When completion.py imports PyCmd I get this:

Traceback (most recent call last):
File "C:\Users\clackwell\src\pycmd\PyCmd.py", line 8, in <module>
from completion import complete_file, complete_wildcard, complete_env_var, find_common_prefix, has_wildcards, wildcard_to_regex
File "C:\Users\clackwell\src\pycmd\completion.py", line 11, in <module>
import PyCmd
File "C:\Users\clackwell\src\pycmd\PyCmd.py", line 8, in <module>
from completion import complete_file, complete_wildcard, complete_env_var, find_common_prefix, has_wildcards, wildcard_to_regex
ImportError: cannot import name complete_file

I have duplicated that line of code instead now, so we should fix it once we know how.

I will upload a revised patch (that still combines custom completion support and git branch name completion; eventually these should be split into two separate commits).

Clemens

[Blackwell]

2nd generation of custom completion support patch

[Horea Haitonic]

Hi,

Thanks, this works better than the previous one.

The pycmd_install_dir will be moved into common so that it can be imported from anywhere within PyCmd -- you can use the duplicate for now, although it's sometimes unreliable, I'll make the move after your changes are merged.

Please give the remaining suggestions some thought: generic complete_git() for completing commands, branches etc + only call the completer that makes sense for the given command.

Thank you so much for the work, Clemens, and sorry for the delays in my answers... Keep' em coming :)

[Blackwell]

Hello,

don't be sorry for the delays. I suffer from time shortage, too.

Regarding the "remaining" suggestions - are you sure that you suggested those things before? (Yes, I am trying to defend myself from feature creep-in!)

The Git completer is currently definitely only tailored towards my own, most pressing little needs. It seems completing the Git commands themselves is possibly easy to add.

However, I am wondering whether it is perhaps better to provide smaller, more specific completers instead of one which tries to complete in all possible (use) cases. This in turn makes it hard for me to decide on cementing a "one completer per command" scheme into PyCmd, because there could be unforeseen needs where someone wants to step in and complete only in one specific case, and adding that to a large block of existing code is probably harder than we would want it to be.

What do you think?

What I still want to do at any rate is some kind of unit testing and splitting up into separate commits.

clemens

[Horea Haitonic]

The "remaining" suggestions are from the 3rd comment on this Issue, list item 4.

I just realized that we each have his way of viewing a completion mechanism:

A) mine is - I think - the common one (used in bash, fish):
1. tokenize the command
2. find out the command to complete and its arguments. E.g. for cd ~\proj && git checkout mas<Tab> the command is 'git' and the arguments are 'checkout' and 'mas'
3. Search for and call complete_git(), with the arguments
4. Use (expand, list) the list of possible completions as returned by complete_git().

B) yours is - I think - a fresher approach that might actually be a good idea:
1. tokenize the command
2. <you should="" add="" this="" in=""> if this is a compound command (cd ~\proj && git checkout mas<Tab>), split the command to get the one that's actually to complete (git checkout mas<Tab>)
3. call all the completers you can find with the (tokenized) form of this last command as argument
4. select only the the first non-empty, or gather all results in a list
5. Use the list (expand, list)

Here are my pros and cons:

Option A:
+ fast (we only call a single completer)
+ keeps completion code organized, i.e. all the code for completing git commands is in one place (complete_git(), maybe other code in complete_git.py)
+ tried-and-true: this is the approach used by other shell since many years; one might easily translate e.g. from fish's completion function into python-based completers for PyCmd
- might lead to duplicated code, e.g. complete_unzip() and complete_7zip() will all have some code to look for .zip files; this might be alleviated by extracting some common functionality, but it's some extra complexity

Option B:
+ powerful: one might have a single completer that, for several known (or even just inferred) commands adds some completions that make sense
+ easier to write in some respects: complete when you know how (e.g. the args match some pattern), otherwise return an empty list
+ flexibile: for some input lines, one could get completions from more than one completers (is this ever useful?)
- related code gets split into multiple completers; e.g. complete_git_branch, complete_git_command, complete_git_options etc

What are your ideas on these choices? Can you think of some concrete use-cases which show how one is better than the other?

[Blackwell]

Hello Horea.

I changed things to invoke "complete_<command_name>()" (if defined by the respective completer) in the last patch.

I definitely need to split the input line to find the "real" command. I looked but could not find code for that right away. You implied that there is code for this already, which file and function should I look at?

Just to clarify, I named the file "got_completer.py" like that to express the intended functionality that it provides. It could just as well have been "foo.py" and should still work for completing "git".

Differences in views aside, I think with the current state of the patch the technical difference is this:

• Load "the one" command specific completer and invoke its complete_() function.

• Load every completer and invoke their complete_() function - if defined.

The latter leaves the decision whether to improve my tiny Git completer to provide full fledged Git completion to me, and avoids forcing me to integrate with an alternative Git completer that you may decide to ship with PyCmd. (If PyCmd would only support one completer per command I would have to patch your completer. And my custom completion needs might be so odd that you could refuse to add them to your completer, so I would have to keep patching your completer (or the code that invokes the completers).)

Because currently only the complete_() of the completers is being invoked when it is defined, I think the performance hit for the invocations themselves are not so expensive as to worry about them.

On the other hand, if there are several completers for the same command, and assuming they do affect the performance (perhaps because each of them invokes "git" 5 times), then I think it would be okay to assume that the user will contact the authors of these completers to ask for a solution (put more functionality into one completer and remove the completers that are obsolete then).

Regarding a use case:

I cannot think of concrete use cases where one of the approaches is better than the other (except perhaps the above example where additional completion functionality above what existing completers provide are desired), but I think allowing more than one completer for a command gives more freedom while not sacrificing much (aside of one or more additional function calls, a few checks, a few more files to load).

PS: I am sure the above is pretty redundant, hard to follow, inconsistent, etc. I apologize in advance, it is too early in the morning, sorry.

[Blackwell]

Hello Horea,

I would like to move forward with the invocation of custom completers. I think there are two open points for that task:

1. Splitting user input to get the right command to complete even with input like "cd .. & git". You said there is code for this already?

2. Deciding whether to...
   2.1 Look for and load "completers\git.py" and invoke its "complete_git()" function.
   2.1 Load all completers and invoke each's "complete_git()" function, if defined.

(Choosing 2.1 would imply to collect all completions from all completers I guess? Otherwise it would be limiting.)

Kind regards

Clemens Anhuth

[Horea Haitonic]

Hi,

You're right, I've been slowing you down; thanks for pushing this forward. My suggestions:

1. Maybe I spoke too highly of this; there's just a basic check in completion.py: 83 -- there's not very much to it, though. You should probably extract it into a function that we can refine later if needed.

2. Out of the two 2.1 versions, I'd choose the first one :) Let's go to with the classic approach, and see if we can benefit from completion code from other shells. I don't want to drop your alternative idea of having more general, cross-command completers; we can maybe add it later and try to support both approaches?

Thanks and good luck!

[Blackwell]

Hello Horea,

thank you for the pointer to the code, I'll move it out into a function.

1. Out of the two 2.1 versions, I'd choose the first one :)

Oh, I screwed up the numbering then. :) Let's assume 2.1 and 2.2.

Let's go to
with the classic approach, and see if we can benefit from completion code
from other shells.

I do not understand this point. Does this mean other shells have completion code written in Python? Then you probably mean code re-use?

I don't want to drop your alternative idea of having
more general, cross-command completers; we can maybe add it later and try
to support both approaches?

Hm, I have a gut feeling that I miserably failed at selling 2.2, because it does support 2.1 as well. (On the other hand though it is trivial to change the code from doing 2.1 to doing 2.2. So I'll implement 2.1. Time will tell if anybody ever asks for 2.2.)

(By the way, I never meant one completer file to contain completers for multiple commands. That is a possibility that comes along with approach 2.2, but I would not agree with doing that actually.)

[Horea Haitonic]

I do not understand this point. Does this mean other shells have
completion code written in Python? Then you probably mean code re-use?

Other shells have custom completion code written in their own language (fish script, bash script, etc). They could be a good starting point for writing completions for PyCmd -- of course there's translation involved, but it's easier to translate something than write it from scratch. Also, some sort of automatic translation (at least partial) might be possible.

Hm, I have a gut feeling that I miserably failed at selling 2.2, because
it does support 2.1 as well. (On the other hand though it is trivial to
change the code from doing 2.1 to doing 2.2. So I'll implement 2.1. Time
will tell if anybody ever asks for 2.2.)
The problem was probably at my end -- my mindset was stuck on the approach that I saw other shells use, and it took me quite a while to realize that you have a different idea. This being said, I clearly understand that 2.2 supports 2.1; the important difference here is how we organize the code, and how we tell users to organize their own custom code. Let's try to support the familiar approach first, and then see where we end up going in the future.

(By the way, I never meant one completer file to contain completers for
multiple commands. That is a possibility that comes along with approach
2.2, but I would not agree with doing that actually.)
Not several completers for multiple commands, but one completer that can be useful in the context of multiple commands. E.g. a completer that completes zip files might work for both unzip and 7 zip -- it would be its responsibility to figure out if the current user input is something that it knows how to handle. This is, in my opinion, the main selling point for 2.2

[Horea Haitonic]

Hi,

What's your status here?

I don't necessarily think we should hurry up with this issue, since it's a heavy feature that needs a lot of work. But I'd like to proceed with the renaming of the project (from "PyCmd" to "Zinc") and this will create a lot of conflicts with any unapplied work -- and I don't want to make you go through this.

How do you think we should go about this?

[Blackwell]

Hello Horea,

I did not get to this again yet, but please go right ahead with any changes. The current patch is pretty isolated and easy add back.