pydev-code

[Pydev-code] Pydev completion participant

[Marcelo T. <mai...@gm...>]

Hello.
My name is Marcelo Taube.
In my organization we use python a lot. I am in charge of the python
environment and do some tool development or installing for my users.
We have decided we will install Pydev and probably acquire a Pydev
extensions license.  The environment will be heavily used inside our
company.
In spite of pydev being quite good we have decided that we need some extra
features, I think that some of the features require that I develop a plugin
extension to pydev and some other might just be solved by better
configuration. I know that some of the extensions I will write will be
insteresting to pydev and maybe you will accept a patch in the future, some
others will be too specific to my company.


This mail is to ask for some initial advise on one of the issues I am
interested.

One of the most important requirements for my users is completion according
to our needs. The problems comes from two fronts
 1) In our python installation we have modules which were wrote by us in the
C language, we would like our users to get proper completion proposals.
Logically this wont work in the initial installation because pydev has not
python code to parse.
2) Sometimes we would like our users to have completion on function return
values and even concatenations like in " foo(3,4).bar([3,4],5).doit(). ".

If I am right the first problem might be solvable by just configuring the
builtin completion in pydev (right?). The second problem is not problem with
most of python code since i saw that pydev has good code analysis. However
both together are a problem since I don't think pydev is able to check the
return type of function written in C.

If there is a current solution to this i want to know. But in any case i
would like to be able to add my own completions in order to be able to fix
problems if they appear.

I think there might be two approaches, one is being able to connect to the
logical engine of pydev and suppling extra information like the return type
of some functions. The other approach would be to supply textual completions
by a separate engine.

I have not discarded the first approach but it seems to be easier to
implement the second one since i noticed that you have an extension point in
pydev to add completions participants, so i started investigating that
option.

I have studyed a little the code of pydev as a way of learning. since i have
not found material about the desing of your completion and how to extend it.
I readed the classes PyCodeCompletion, PythonCompletionProcessor,
CompletionState, CompletionRequest and most important
IPyDevCompletionParticipant.
Please correct me if i am not learning in the proper way.

I also wrote a kind of example completion participant to add some dumb
completion.
My example would always return a token for the completion 'abcdef', not
depending in the activation token. It seems that pydev correctly filters out
the completion if the activation token is not a prefix of the proposal.
Anyway, the example worked, but now I found an extra problem. PyDev would
only allow me to add completion proposals to global variables but would not
allow me to give proposals which include dots. If i return the proposal
'abcdef.ghi' and the user writes 'abcedf.' + Ctrl +Space my proposal does
not appear.
After some debugging i noticed my code is never called and after reading
what is happening in PyCodeCompletion i realized that there two distinct
cases, one for global variables and one for tokens. It seems that in the
second case contributors are never called. Is that a bug?

Thank you for your time and your answers, i hope you can help me and also
that i will be able to contribute back for this help.

An thanks to Fabio for all the completion code he wrote :)

Have a nice day,
Marcelo Taube

Re: [Pydev-code] Pydev completion participant

[Fabio Z. <fa...@es...>]

> Hello.
> My name is Marcelo Taube.

Hello Marcelo,

> One of the most important requirements for my users is completion according
> to our needs. The problems comes from two fronts
>  1) In our python installation we have modules which were wrote by us in the
> C language, we would like our users to get proper completion proposals.
> Logically this wont work in the initial installation because pydev has not
> python code to parse.
> 2) Sometimes we would like our users to have completion on function return
> values and even concatenations like in " foo(3,4).bar([3,4],5).doit(). ".
>
> If I am right the first problem might be solvable by just configuring the
> builtin completion in pydev (right?). The second problem is not problem with
> most of python code since i saw that pydev has good code analysis. However
> both together are a problem since I don't think pydev is able to check the
> return type of function written in C.

You're right:
- You can configure the forced builtins to do what you want by
spawning a shell and dynamically analyzing your dlls (
http://fabioz.com/pydev/manual_101_interpreter.html )
- pydev has no way of getting return types of c-functions (that's not
available in the shell code-completion introspection -- right now,
sometimes pydev is able to get the arguments by analyzing the
docstrings from those dlls, but that's also not always available)

> If there is a current solution to this i want to know. But in any case i
> would like to be able to add my own completions in order to be able to fix
> problems if they appear.
>
> I think there might be two approaches, one is being able to connect to the
> logical engine of pydev and suppling extra information like the return type
> of some functions. The other approach would be to supply textual completions
> by a separate engine.
>
> I have not discarded the first approach but it seems to be easier to
> implement the second one since i noticed that you have an extension point in
> pydev to add completions participants, so i started investigating that
> option.

There are plans to add a way to the engine so that you can provide a
file with the completions -- right now you could probably generate
python stubs for the c code you have and add those as a .zip to the
pythonpath -- but creating a plugin to add more completions would
surely work too.

> I have studyed a little the code of pydev as a way of learning. since i have
> not found material about the desing of your completion and how to extend it.
> I readed the classes PyCodeCompletion, PythonCompletionProcessor,
> CompletionState, CompletionRequest and most important
> IPyDevCompletionParticipant.
> Please correct me if i am not learning in the proper way.

Reading the code/asking in the list is really the best way of doing it.

> I also wrote a kind of example completion participant to add some dumb
> completion.
> My example would always return a token for the completion 'abcdef', not
> depending in the activation token. It seems that pydev correctly filters out
> the completion if the activation token is not a prefix of the proposal.
> Anyway, the example worked, but now I found an extra problem. PyDev would
> only allow me to add completion proposals to global variables but would not
> allow me to give proposals which include dots. If i return the proposal
> 'abcdef.ghi' and the user writes 'abcedf.' + Ctrl +Space my proposal does
> not appear.
> After some debugging i noticed my code is never called and after reading
> what is happening in PyCodeCompletion i realized that there two distinct
> cases, one for global variables and one for tokens. It seems that in the
> second case contributors are never called. Is that a bug?

It's more like a feature-request (as you can see in
IPyDevCompletionParticipant, it specifically names the methods for
globals and arguments) -- also, when getting things from a token, you
usually need to know more things, such as what that token actually
represents, so, the code-completion for clients there would need good
use-cases to know what would be needed for a client to properly do a
code-completion there.

> Thank you for your time and your answers, i hope you can help me and also
> that i will be able to contribute back for this help.

;-)

Cheers,

Fabio

Re: [Pydev-code] Pydev completion participant

[Marcelo T. <mai...@gm...>]

Fabio,
Thank you for your fast answer.
As far as i understand i cannot write a plug in for more completions since
the completion participant could only add global completions, which is not
what i need.

I know in the general case is difficult to know what a participant should
get because he has to know the type of the activation token, like in:
<code>
   a = SomeClass()
   a.<--- CTRL + SPACE
</code>

However in my case it might be enogh to just get the string used for
completion and then return a string which includes dots, like getting "a."
and returning "a.one_attribute". This is because i have close contact with
my users and I might explain them that they should do "SomeClass(). <---
CTRL + SPACE" and only then separate to a different variable (It sounds
wierd but people is willing to accept this if there is no better option)

In any case i understand that is not logical that you change your current
working plans to extend the IPyDevCompletionParticipant interface.
But i thought that if we could agree on some interface (and use case) I
could add the feature myself and send a patch to you.

Anyway, i will also try inspecting the solution of creating stubs and I tell
you after that.

I am curious about the power of pydev, is it always able to index each
python function and its return value? is it able to understand functions
which have more than one return type?

Also, are you planning to add support for parameter type deducing?

Thank you
Marcelo

On Mon, Jun 22, 2009 at 1:26 AM, Fabio Zadrozny <fa...@es...> wrote:

> > Hello.
> > My name is Marcelo Taube.
>
> Hello Marcelo,
>
> > One of the most important requirements for my users is completion
> according
> > to our needs. The problems comes from two fronts
> >  1) In our python installation we have modules which were wrote by us in
> the
> > C language, we would like our users to get proper completion proposals.
> > Logically this wont work in the initial installation because pydev has
> not
> > python code to parse.
> > 2) Sometimes we would like our users to have completion on function
> return
> > values and even concatenations like in " foo(3,4).bar([3,4],5).doit(). ".
> >
> > If I am right the first problem might be solvable by just configuring the
> > builtin completion in pydev (right?). The second problem is not problem
> with
> > most of python code since i saw that pydev has good code analysis.
> However
> > both together are a problem since I don't think pydev is able to check
> the
> > return type of function written in C.
>
> You're right:
> - You can configure the forced builtins to do what you want by
> spawning a shell and dynamically analyzing your dlls (
> http://fabioz.com/pydev/manual_101_interpreter.html )
> - pydev has no way of getting return types of c-functions (that's not
> available in the shell code-completion introspection -- right now,
> sometimes pydev is able to get the arguments by analyzing the
> docstrings from those dlls, but that's also not always available)
>
> > If there is a current solution to this i want to know. But in any case i
> > would like to be able to add my own completions in order to be able to
> fix
> > problems if they appear.
> >
> > I think there might be two approaches, one is being able to connect to
> the
> > logical engine of pydev and suppling extra information like the return
> type
> > of some functions. The other approach would be to supply textual
> completions
> > by a separate engine.
> >
> > I have not discarded the first approach but it seems to be easier to
> > implement the second one since i noticed that you have an extension point
> in
> > pydev to add completions participants, so i started investigating that
> > option.
>
> There are plans to add a way to the engine so that you can provide a
> file with the completions -- right now you could probably generate
> python stubs for the c code you have and add those as a .zip to the
> pythonpath -- but creating a plugin to add more completions would
> surely work too.
>
> > I have studyed a little the code of pydev as a way of learning. since i
> have
> > not found material about the desing of your completion and how to extend
> it.
> > I readed the classes PyCodeCompletion, PythonCompletionProcessor,
> > CompletionState, CompletionRequest and most important
> > IPyDevCompletionParticipant.
> > Please correct me if i am not learning in the proper way.
>
> Reading the code/asking in the list is really the best way of doing it.
>
> > I also wrote a kind of example completion participant to add some dumb
> > completion.
> > My example would always return a token for the completion 'abcdef', not
> > depending in the activation token. It seems that pydev correctly filters
> out
> > the completion if the activation token is not a prefix of the proposal.
> > Anyway, the example worked, but now I found an extra problem. PyDev would
> > only allow me to add completion proposals to global variables but would
> not
> > allow me to give proposals which include dots. If i return the proposal
> > 'abcdef.ghi' and the user writes 'abcedf.' + Ctrl +Space my proposal does
> > not appear.
> > After some debugging i noticed my code is never called and after reading
> > what is happening in PyCodeCompletion i realized that there two distinct
> > cases, one for global variables and one for tokens. It seems that in the
> > second case contributors are never called. Is that a bug?
>
> It's more like a feature-request (as you can see in
> IPyDevCompletionParticipant, it specifically names the methods for
> globals and arguments) -- also, when getting things from a token, you
> usually need to know more things, such as what that token actually
> represents, so, the code-completion for clients there would need good
> use-cases to know what would be needed for a client to properly do a
> code-completion there.
>
> > Thank you for your time and your answers, i hope you can help me and also
> > that i will be able to contribute back for this help.
>
> ;-)
>
> Cheers,
>
> Fabio
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
>
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code
>

Re: [Pydev-code] Pydev completion participant

[Fabio Z. <fa...@gm...>]

> As far as i understand i cannot write a plug in for more completions since
> the completion participant could only add global completions, which is not
> what i need.
>
> I know in the general case is difficult to know what a participant should
> get because he has to know the type of the activation token, like in:
> <code>
>    a = SomeClass()
>    a.<--- CTRL + SPACE
> </code>
>
> However in my case it might be enogh to just get the string used for
> completion and then return a string which includes dots, like getting "a."
> and returning "a.one_attribute". This is because i have close contact with
> my users and I might explain them that they should do "SomeClass(). <---
> CTRL + SPACE" and only then separate to a different variable (It sounds
> wierd but people is willing to accept this if there is no better option)
>
> In any case i understand that is not logical that you change your current
> working plans to extend the IPyDevCompletionParticipant interface.
> But i thought that if we could agree on some interface (and use case) I
> could add the feature myself and send a patch to you.

Actually, what I would think is that clients of
IPyDevCompletionParticipant are called but already knowing about the
information that could be resolved already.

E.g.:

in the token:
mod1.mod2.ClassA.Foo.

it could know that mod1.mod2.ClassA is already a given module/class
and then just goes on asking clients to complete with Foo (which it
doesn't know about).

> Anyway, i will also try inspecting the solution of creating stubs and I tell
> you after that.
>
> I am curious about the power of pydev, is it always able to index each
> python function and its return value? is it able to understand functions
> which have more than one return type?

When it finds

a = MyFunc()

it'll go on to analyze the return statements available for that
function and for each return it'll try to get the completions for that
token (so, in that case, yes, it's able to properly understand
functions which have more than one return type).

> Also, are you planning to add support for parameter type deducing?

Not currently... I already thought about that, but I have some
concerns on whether it's feasible to do it with acceptable levels of
performance.


Cheers,

Fabio




>
> Thank you
> Marcelo
>
> On Mon, Jun 22, 2009 at 1:26 AM, Fabio Zadrozny <fa...@es...> wrote:
>>
>> > Hello.
>> > My name is Marcelo Taube.
>>
>> Hello Marcelo,
>>
>> > One of the most important requirements for my users is completion
>> > according
>> > to our needs. The problems comes from two fronts
>> >  1) In our python installation we have modules which were wrote by us in
>> > the
>> > C language, we would like our users to get proper completion proposals.
>> > Logically this wont work in the initial installation because pydev has
>> > not
>> > python code to parse.
>> > 2) Sometimes we would like our users to have completion on function
>> > return
>> > values and even concatenations like in " foo(3,4).bar([3,4],5).doit().
>> > ".
>> >
>> > If I am right the first problem might be solvable by just configuring
>> > the
>> > builtin completion in pydev (right?). The second problem is not problem
>> > with
>> > most of python code since i saw that pydev has good code analysis.
>> > However
>> > both together are a problem since I don't think pydev is able to check
>> > the
>> > return type of function written in C.
>>
>> You're right:
>> - You can configure the forced builtins to do what you want by
>> spawning a shell and dynamically analyzing your dlls (
>> http://fabioz.com/pydev/manual_101_interpreter.html )
>> - pydev has no way of getting return types of c-functions (that's not
>> available in the shell code-completion introspection -- right now,
>> sometimes pydev is able to get the arguments by analyzing the
>> docstrings from those dlls, but that's also not always available)
>>
>> > If there is a current solution to this i want to know. But in any case i
>> > would like to be able to add my own completions in order to be able to
>> > fix
>> > problems if they appear.
>> >
>> > I think there might be two approaches, one is being able to connect to
>> > the
>> > logical engine of pydev and suppling extra information like the return
>> > type
>> > of some functions. The other approach would be to supply textual
>> > completions
>> > by a separate engine.
>> >
>> > I have not discarded the first approach but it seems to be easier to
>> > implement the second one since i noticed that you have an extension
>> > point in
>> > pydev to add completions participants, so i started investigating that
>> > option.
>>
>> There are plans to add a way to the engine so that you can provide a
>> file with the completions -- right now you could probably generate
>> python stubs for the c code you have and add those as a .zip to the
>> pythonpath -- but creating a plugin to add more completions would
>> surely work too.
>>
>> > I have studyed a little the code of pydev as a way of learning. since i
>> > have
>> > not found material about the desing of your completion and how to extend
>> > it.
>> > I readed the classes PyCodeCompletion, PythonCompletionProcessor,
>> > CompletionState, CompletionRequest and most important
>> > IPyDevCompletionParticipant.
>> > Please correct me if i am not learning in the proper way.
>>
>> Reading the code/asking in the list is really the best way of doing it.
>>
>> > I also wrote a kind of example completion participant to add some dumb
>> > completion.
>> > My example would always return a token for the completion 'abcdef', not
>> > depending in the activation token. It seems that pydev correctly filters
>> > out
>> > the completion if the activation token is not a prefix of the proposal.
>> > Anyway, the example worked, but now I found an extra problem. PyDev
>> > would
>> > only allow me to add completion proposals to global variables but would
>> > not
>> > allow me to give proposals which include dots. If i return the proposal
>> > 'abcdef.ghi' and the user writes 'abcedf.' + Ctrl +Space my proposal
>> > does
>> > not appear.
>> > After some debugging i noticed my code is never called and after reading
>> > what is happening in PyCodeCompletion i realized that there two distinct
>> > cases, one for global variables and one for tokens. It seems that in the
>> > second case contributors are never called. Is that a bug?
>>
>> It's more like a feature-request (as you can see in
>> IPyDevCompletionParticipant, it specifically names the methods for
>> globals and arguments) -- also, when getting things from a token, you
>> usually need to know more things, such as what that token actually
>> represents, so, the code-completion for clients there would need good
>> use-cases to know what would be needed for a client to properly do a
>> code-completion there.
>>
>> > Thank you for your time and your answers, i hope you can help me and
>> > also
>> > that i will be able to contribute back for this help.
>>
>> ;-)
>>
>> Cheers,
>>
>> Fabio
>>
>>
>> ------------------------------------------------------------------------------
>> Are you an open source citizen? Join us for the Open Source Bridge
>> conference!
>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>> $250.
>> Need another reason to go? 24-hour hacker lounge. Register today!
>>
>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>> _______________________________________________
>> pydev-code mailing list
>> pyd...@li...
>> https://lists.sourceforge.net/lists/listinfo/pydev-code
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code
>
>