pydev-code

[Pydev-code] Add 'global functions' to PyDev analyzer

[Andreas P. <an...@fr...>]

Hi,

I'm currently working on improving our usage of PyDev, in particular 
starting to let PyDev not only syntax highlight but also analyze the 
code our users write. Unfortunately this has one drawback, we use a 
custom interpreter and add several global functions to it (this is done 
by having a module for these functions and then importing everything 
from this module by default into the global namespace).

I know about the setting for the PyDev analyzer to not show errors on 
certain undefined functions/variables, but the number of functions makes 
this single lineedit/preference pretty cumbersome to extend.

I'm wondering wether there's a different way (via code maybe?) to teach 
PyDev about a list of functions it should consider to be 'predefined', 
similar to the builtin modules?

I'm currently using PyDev 2.6, but could upgrade to 2.7.5 if that makes 
it easier to achieve what I want.

Andreas

-- 
Andreas Pakulat sq...@fr...
froglogic GmbH - Automated UI and Web Testing

Re: [Pydev-code] Add 'global functions' to PyDev analyzer

[Fabio Z. <fa...@es...>]

Hi Andreas,

Actually, if you really have a custom interpreter and this is added in its
startup, it's strange that it doesn't work already (PyDev always spawns a
shell for the completion and upon startup it'll load the __builtin__ -- or
builtins -- module and do a dir() in it).

So, if it's custom interpreter, it's mostly a matter of filling the
__builtins__ at the startup... If that's not the case, you could change
PyDev to add some custom code at the module scope at
/org.python.pydev/pysrc/pycompletionserver.py (but you'll have to maintain
your own fork if you choose this path).

Cheers,

Fabio




On Fri, Jul 19, 2013 at 12:15 PM, Andreas Pakulat <an...@fr...>wrote:

> Hi,
>
> I'm currently working on improving our usage of PyDev, in particular
> starting to let PyDev not only syntax highlight but also analyze the
> code our users write. Unfortunately this has one drawback, we use a
> custom interpreter and add several global functions to it (this is done
> by having a module for these functions and then importing everything
> from this module by default into the global namespace).
>
> I know about the setting for the PyDev analyzer to not show errors on
> certain undefined functions/variables, but the number of functions makes
> this single lineedit/preference pretty cumbersome to extend.
>
> I'm wondering wether there's a different way (via code maybe?) to teach
> PyDev about a list of functions it should consider to be 'predefined',
> similar to the builtin modules?
>
> I'm currently using PyDev 2.6, but could upgrade to 2.7.5 if that makes
> it easier to achieve what I want.
>
> Andreas
>
> --
> Andreas Pakulat sq...@fr...
> froglogic GmbH - Automated UI and Web Testing
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code
>

Re: [Pydev-code] Add 'global functions' to PyDev analyzer

[Andreas P. <an...@fr...>]

Hi Fabio,

thanks for the hint about the pycompletionserver.py. Currently we 
already have a change or two to PyDev and roll our own but we would 
rather use the official version.

Unfortunately our interpreter is embedded into a commandline tool that 
does not support being used like a normal python interpreter. So thats 
why  choosing it as an interpreter is not suitable at the moment. What 
it does is basically load the main script file (i.e. parse it but does 
not run it immediately), then add a few modules to the python 
interpreter via C API, import those modules into the global namespace of 
the main script file and also import several functions into the global 
namespace. This was done as a convenience for the users but now clearly 
bites back at us when trying to teach PyDev about these specialties.

Its of course also possible to import those registered modules in the 
script file or in one that is being imported into the main script.

What I ended up doing now which seems to work with 2 caveats (see 
further down) is that I'm using the extension points 
pydev_interpreter_new_custom_entries, pydev_modules_observer and the 
completion extension point. The first one registers the names of the 
modules we add via the C API as 'builtins', the second one adds tokens 
to the __builtin__ module for those modules and the functions we import 
into the global namespace. It also populates our modules with their 
functions when the PyDev code needs them (a script importing our squish 
module for example). All this seems to avoid errors being flagged for 
our Squish API, but does not yet provide the modules and functions for 
completion. Hence using the completion extension point to support that.

The two minor issues with this are:

1. For the builtins registration using the 
pydev_interpreter_new_custom_entries extension point the users need to 
remove and re-add the interpreter so that PyDev fetches the builtins 
again.

Maybe there's a way to do the above programatically upon startup of a 
plugin?

2. When using 'import squish' in a script now a warning is flagged 
about this shadowing a builtin module which is correct from what PyDev 
knows but not really happening in the actual interpreter that executes 
the tests.

The second point is not much of a problem as its just a warning and 
hence does not jump out at the user as much as an error (in particular 
an error in script code that Squish just generated).

Andreas


On 2013-07-22 22:39, Fabio Zadrozny wrote:
> Hi Andreas,
>
> Actually, if you really have a custom interpreter and this is added
> in its startup, it's strange that it doesn't work already (PyDev
> always spawns a shell for the completion and upon startup it'll load
> the __builtin__ -- or builtins -- module and do a dir() in it).
>
> So, if it's custom interpreter, it's mostly a matter of filling the
> __builtins__ at the startup... If that's not the case, you could
> change PyDev to add some custom code at the module scope at
> /org.python.pydev/pysrc/pycompletionserver.py (but you'll have to
> maintain your own fork if you choose this path).
>
> Cheers,
>
> Fabio
>
> On Fri, Jul 19, 2013 at 12:15 PM, Andreas Pakulat 
> <an...@fr...> wrote:
>
>> Hi,
>>
>> I'm currently working on improving our usage of PyDev, in particular
>> starting to let PyDev not only syntax highlight but also analyze the
>> code our users write. Unfortunately this has one drawback, we use a
>> custom interpreter and add several global functions to it (this is 
>> done
>> by having a module for these functions and then importing everything
>> from this module by default into the global namespace).
>>
>> I know about the setting for the PyDev analyzer to not show errors 
>> on
>> certain undefined functions/variables, but the number of functions 
>> makes
>> this single lineedit/preference pretty cumbersome to extend.
>>
>> I'm wondering wether there's a different way (via code maybe?) to 
>> teach
>> PyDev about a list of functions it should consider to be 
>> 'predefined',
>> similar to the builtin modules?
>>
>> I'm currently using PyDev 2.6, but could upgrade to 2.7.5 if that 
>> makes
>> it easier to achieve what I want.
>>
>> Andreas
>>
>> --
>> Andreas Pakulat sq...@fr...
>> froglogic GmbH - Automated UI and Web Testing
>>
>> 
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from 
>> AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>> 
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk 
>> [1]
>> _______________________________________________
>> pydev-code mailing list
>> pyd...@li...
>> https://lists.sourceforge.net/lists/listinfo/pydev-code [2]
>
>
>
> Links:
> ------
> [1]
> 
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&amp;iu=/4140/ostg.clktrk
> [2] https://lists.sourceforge.net/lists/listinfo/pydev-code
>
> 
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from 
> AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> 
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code

-- 
Andreas Pakulat sq...@fr...
froglogic GmbH - Automated UI and Web Testing

Re: [Pydev-code] Add 'global functions' to PyDev analyzer

[Fabio Z. <fa...@es...>]

On Tue, Jul 23, 2013 at 4:46 AM, Andreas Pakulat <an...@fr...>wrote:

> Hi Fabio,
>
> thanks for the hint about the pycompletionserver.py. Currently we
> already have a change or two to PyDev and roll our own but we would
> rather use the official version.
>
> Unfortunately our interpreter is embedded into a commandline tool that
> does not support being used like a normal python interpreter. So thats
> why  choosing it as an interpreter is not suitable at the moment. What
> it does is basically load the main script file (i.e. parse it but does
> not run it immediately), then add a few modules to the python
> interpreter via C API, import those modules into the global namespace of
> the main script file and also import several functions into the global
> namespace. This was done as a convenience for the users but now clearly
> bites back at us when trying to teach PyDev about these specialties.
>

Ok, so, it's not really a custom interpreter, just a different main... Not
sure if that's the best approach (I'd probably go toward a more common
approach of having a different main entry which then would run the script
after configuring the environment, such as most web frameworks do or maybe
a custom sitecustomize).

Still, if you bundle it as an executable, it shouldn't be all that hard to
make it look like a real python interpreter (i.e.: one that could really be
used as an interpreter -- that way you wouldn't need to change PyDev at all
and everything should just work). This is probably the preferred approach.

Now, if that's difficult for you, the
pycompletionserver.py/importsTipper.pyadjustment probably requires
less fiddling to integrate.

Regarding a better approach within PyDev itself, please create a feature
request for that in https://sw-brainwy.rhcloud.com/tracker/PyDev/ (this is
also a request web2py users have -- for which there's a discussed
workaround detailed at the old tracker:
http://sourceforge.net/p/pydev/feature-requests/534/).


>
> Its of course also possible to import those registered modules in the
> script file or in one that is being imported into the main script.
>
> What I ended up doing now which seems to work with 2 caveats (see
> further down) is that I'm using the extension points
> pydev_interpreter_new_custom_entries, pydev_modules_observer and the
> completion extension point. The first one registers the names of the
> modules we add via the C API as 'builtins', the second one adds tokens
> to the __builtin__ module for those modules and the functions we import
> into the global namespace. It also populates our modules with their
> functions when the PyDev code needs them (a script importing our squish
> module for example). All this seems to avoid errors being flagged for
> our Squish API, but does not yet provide the modules and functions for
> completion. Hence using the completion extension point to support that.
>
> The two minor issues with this are:
>
> 1. For the builtins registration using the
> pydev_interpreter_new_custom_entries extension point the users need to
> remove and re-add the interpreter so that PyDev fetches the builtins
> again.
>
> Maybe there's a way to do the above programatically upon startup of a
> plugin?
>
> 2. When using 'import squish' in a script now a warning is flagged
> about this shadowing a builtin module which is correct from what PyDev
> knows but not really happening in the actual interpreter that executes
> the tests.
>
> The second point is not much of a problem as its just a warning and
> hence does not jump out at the user as much as an error (in particular
> an error in script code that Squish just generated).
>
> Andreas
>
>
> On 2013-07-22 22:39, Fabio Zadrozny wrote:
> > Hi Andreas,
> >
> > Actually, if you really have a custom interpreter and this is added
> > in its startup, it's strange that it doesn't work already (PyDev
> > always spawns a shell for the completion and upon startup it'll load
> > the __builtin__ -- or builtins -- module and do a dir() in it).
> >
> > So, if it's custom interpreter, it's mostly a matter of filling the
> > __builtins__ at the startup... If that's not the case, you could
> > change PyDev to add some custom code at the module scope at
> > /org.python.pydev/pysrc/pycompletionserver.py (but you'll have to
> > maintain your own fork if you choose this path).
> >
> > Cheers,
> >
> > Fabio
> >
> > On Fri, Jul 19, 2013 at 12:15 PM, Andreas Pakulat
> > <an...@fr...> wrote:
> >
> >> Hi,
> >>
> >> I'm currently working on improving our usage of PyDev, in particular
> >> starting to let PyDev not only syntax highlight but also analyze the
> >> code our users write. Unfortunately this has one drawback, we use a
> >> custom interpreter and add several global functions to it (this is
> >> done
> >> by having a module for these functions and then importing everything
> >> from this module by default into the global namespace).
> >>
> >> I know about the setting for the PyDev analyzer to not show errors
> >> on
> >> certain undefined functions/variables, but the number of functions
> >> makes
> >> this single lineedit/preference pretty cumbersome to extend.
> >>
> >> I'm wondering wether there's a different way (via code maybe?) to
> >> teach
> >> PyDev about a list of functions it should consider to be
> >> 'predefined',
> >> similar to the builtin modules?
> >>
> >> I'm currently using PyDev 2.6, but could upgrade to 2.7.5 if that
> >> makes
> >> it easier to achieve what I want.
> >>
> >> Andreas
> >>
> >> --
> >> Andreas Pakulat sq...@fr...
> >> froglogic GmbH - Automated UI and Web Testing
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> See everything from the browser to the database with AppDynamics
> >> Get end-to-end visibility with application monitoring from
> >> AppDynamics
> >> Isolate bottlenecks and diagnose root cause in seconds.
> >> Start your free trial of AppDynamics Pro today!
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> >> [1]
> >> _______________________________________________
> >> pydev-code mailing list
> >> pyd...@li...
> >> https://lists.sourceforge.net/lists/listinfo/pydev-code [2]
> >
> >
> >
> > Links:
> > ------
> > [1]
> >
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&amp;iu=/4140/ostg.clktrk<http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk>
> > [2] https://lists.sourceforge.net/lists/listinfo/pydev-code
> >
> >
> >
> ------------------------------------------------------------------------------
> > See everything from the browser to the database with AppDynamics
> > Get end-to-end visibility with application monitoring from
> > AppDynamics
> > Isolate bottlenecks and diagnose root cause in seconds.
> > Start your free trial of AppDynamics Pro today!
> >
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> >
> > _______________________________________________
> > pydev-code mailing list
> > pyd...@li...
> > https://lists.sourceforge.net/lists/listinfo/pydev-code
>
> --
> Andreas Pakulat sq...@fr...
> froglogic GmbH - Automated UI and Web Testing
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code
>

Re: [Pydev-code] Add 'global functions' to PyDev analyzer

[Andreas P. <an...@fr...>]

Hi Fabio,

On 2013-07-23 13:39, Fabio Zadrozny wrote:
> On Tue, Jul 23, 2013 at 4:46 AM, Andreas Pakulat 
> <an...@fr...> wrote:
>> thanks for the hint about the pycompletionserver.py. Currently we
>> already have a change or two to PyDev and roll our own but we would
>> rather use the official version.
>>
>> Unfortunately our interpreter is embedded into a commandline tool 
>> that
>> does not support being used like a normal python interpreter. So 
>> thats
>> why  choosing it as an interpreter is not suitable at the moment. 
>> What
>> it does is basically load the main script file (i.e. parse it but 
>> does
>> not run it immediately), then add a few modules to the python
>> interpreter via C API, import those modules into the global 
>> namespace of
>> the main script file and also import several functions into the 
>> global
>> namespace. This was done as a convenience for the users but now 
>> clearly
>> bites back at us when trying to teach PyDev about these specialties.
>
> Ok, so, it's not really a custom interpreter, just a different
> main... Not sure if that's the best approach (I'd probably go toward 
> a
> more common approach of having a different main entry which then 
> would
> run the script after configuring the environment, such as most web
> frameworks do or maybe a custom sitecustomize).
>
> Still, if you bundle it as an executable, it shouldn't be all that
> hard to make it look like a real python interpreter (i.e.: one that
> could really be used as an interpreter -- that way you wouldn't need
> to change PyDev at all and everything should just work). This is
> probably the preferred approach.

There are a few more technical details involved that cannot be easily 
changed which make it a bit harder to turn the tool into something that 
can just run python code as-is.

Howerver I now realized that even that will not be sufficient to fully 
support our usecase. Squish tests are naturally running on a particular 
application that is to be tested. In various of the Squish editions we 
expose the complete internals of the Application (or rather the UI 
toolkit used) to the script side. This means a user testing an Eclipse 
application has full access to the Java and Eclipse API and can write 
code like this in his scripts:

mystr = java_lang_String("Hello World")
findObject("Shell").setText(mystr)

And the binding of these classes, modules and functions is only done 
once an application is actually started. In addition the binding is 
somewhat dynamic, so if the application loads additional code lazily 
then Squish will only know about that once the code is actually loaded.

This is as far as I can see not really supportable, so I now went the 
route of simply moving errors being reported by PyDev to warnings so our 
users scripts don't have 90% of their lines flagged in red :)

Thanks for the tips though, we be able to use them to make at least 
some of the problems go away in the future provided the user has the 
application running.

Andreas

-- 
Andreas Pakulat sq...@fr...
froglogic GmbH - Automated UI and Web Testing

Re: [Pydev-code] Add 'global functions' to PyDev analyzer

[Jonah G. <jo...@ki...>]

Hi,

I need to solve a similar problem. I have some scripts that are user
editable, but are run by an execfile with some other magic wrapped
around it. In that magic some new globals are added, which seem in a
similar way to how web2py is described in
http://sourceforge.net/p/pydev/feature-requests/534/

What I would like to do is add a new extension point to extend the
behaviour of CompiledModule:setTokens() and have it essentially add
new values to "array" within this if statement?

            //as we will use it for code completion on sources that
map to modules, the __file__ should also
            //be added...
            if (array.size() > 0 && (name.equals("__builtin__") ||
name.equals("builtins"))) {
                array.add(new CompiledToken("__file__", "", "", name,
IToken.TYPE_BUILTIN));
                array.add(new CompiledToken("__name__", "", "", name,
IToken.TYPE_BUILTIN));
                array.add(new CompiledToken("__builtins__", "", "",
name, IToken.TYPE_BUILTIN));
                array.add(new CompiledToken("__dict__", "", "", name,
IToken.TYPE_BUILTIN));
// extension point data insertion would be here[1]
            }

Does that sound like a correct solution? Does that seem generally
useful? If so I can put an initial pull request together for review.

[1] Should the extension point users return AbstractToken, or is there
a subset of that API that should be exposed (e.g. just name). My
preference would be AbstractToken, to allow full range of options for
extension point user.

Jonah


On 23 July 2013 12:39, Fabio Zadrozny <fa...@es...> wrote:
> On Tue, Jul 23, 2013 at 4:46 AM, Andreas Pakulat <an...@fr...>
> wrote:
>>
>> Hi Fabio,
>>
>> thanks for the hint about the pycompletionserver.py. Currently we
>> already have a change or two to PyDev and roll our own but we would
>> rather use the official version.
>>
>> Unfortunately our interpreter is embedded into a commandline tool that
>> does not support being used like a normal python interpreter. So thats
>> why  choosing it as an interpreter is not suitable at the moment. What
>> it does is basically load the main script file (i.e. parse it but does
>> not run it immediately), then add a few modules to the python
>> interpreter via C API, import those modules into the global namespace of
>> the main script file and also import several functions into the global
>> namespace. This was done as a convenience for the users but now clearly
>> bites back at us when trying to teach PyDev about these specialties.
>
>
> Ok, so, it's not really a custom interpreter, just a different main... Not
> sure if that's the best approach (I'd probably go toward a more common
> approach of having a different main entry which then would run the script
> after configuring the environment, such as most web frameworks do or maybe a
> custom sitecustomize).
>
> Still, if you bundle it as an executable, it shouldn't be all that hard to
> make it look like a real python interpreter (i.e.: one that could really be
> used as an interpreter -- that way you wouldn't need to change PyDev at all
> and everything should just work). This is probably the preferred approach.
>
> Now, if that's difficult for you, the pycompletionserver.py/importsTipper.py
> adjustment probably requires less fiddling to integrate.
>
> Regarding a better approach within PyDev itself, please create a feature
> request for that in https://sw-brainwy.rhcloud.com/tracker/PyDev/ (this is
> also a request web2py users have -- for which there's a discussed workaround
> detailed at the old tracker:
> http://sourceforge.net/p/pydev/feature-requests/534/).
>
>>
>>
>> Its of course also possible to import those registered modules in the
>> script file or in one that is being imported into the main script.
>>
>> What I ended up doing now which seems to work with 2 caveats (see
>> further down) is that I'm using the extension points
>> pydev_interpreter_new_custom_entries, pydev_modules_observer and the
>> completion extension point. The first one registers the names of the
>> modules we add via the C API as 'builtins', the second one adds tokens
>> to the __builtin__ module for those modules and the functions we import
>> into the global namespace. It also populates our modules with their
>> functions when the PyDev code needs them (a script importing our squish
>> module for example). All this seems to avoid errors being flagged for
>> our Squish API, but does not yet provide the modules and functions for
>> completion. Hence using the completion extension point to support that.
>>
>> The two minor issues with this are:
>>
>> 1. For the builtins registration using the
>> pydev_interpreter_new_custom_entries extension point the users need to
>> remove and re-add the interpreter so that PyDev fetches the builtins
>> again.
>>
>> Maybe there's a way to do the above programatically upon startup of a
>> plugin?
>>
>> 2. When using 'import squish' in a script now a warning is flagged
>> about this shadowing a builtin module which is correct from what PyDev
>> knows but not really happening in the actual interpreter that executes
>> the tests.
>>
>> The second point is not much of a problem as its just a warning and
>> hence does not jump out at the user as much as an error (in particular
>> an error in script code that Squish just generated).
>>
>> Andreas
>>
>>
>> On 2013-07-22 22:39, Fabio Zadrozny wrote:
>> > Hi Andreas,
>> >
>> > Actually, if you really have a custom interpreter and this is added
>> > in its startup, it's strange that it doesn't work already (PyDev
>> > always spawns a shell for the completion and upon startup it'll load
>> > the __builtin__ -- or builtins -- module and do a dir() in it).
>> >
>> > So, if it's custom interpreter, it's mostly a matter of filling the
>> > __builtins__ at the startup... If that's not the case, you could
>> > change PyDev to add some custom code at the module scope at
>> > /org.python.pydev/pysrc/pycompletionserver.py (but you'll have to
>> > maintain your own fork if you choose this path).
>> >
>> > Cheers,
>> >
>> > Fabio
>> >
>> > On Fri, Jul 19, 2013 at 12:15 PM, Andreas Pakulat
>> > <an...@fr...> wrote:
>> >
>> >> Hi,
>> >>
>> >> I'm currently working on improving our usage of PyDev, in particular
>> >> starting to let PyDev not only syntax highlight but also analyze the
>> >> code our users write. Unfortunately this has one drawback, we use a
>> >> custom interpreter and add several global functions to it (this is
>> >> done
>> >> by having a module for these functions and then importing everything
>> >> from this module by default into the global namespace).
>> >>
>> >> I know about the setting for the PyDev analyzer to not show errors
>> >> on
>> >> certain undefined functions/variables, but the number of functions
>> >> makes
>> >> this single lineedit/preference pretty cumbersome to extend.
>> >>
>> >> I'm wondering wether there's a different way (via code maybe?) to
>> >> teach
>> >> PyDev about a list of functions it should consider to be
>> >> 'predefined',
>> >> similar to the builtin modules?
>> >>
>> >> I'm currently using PyDev 2.6, but could upgrade to 2.7.5 if that
>> >> makes
>> >> it easier to achieve what I want.
>> >>
>> >> Andreas
>> >>
>> >> --
>> >> Andreas Pakulat sq...@fr...
>> >> froglogic GmbH - Automated UI and Web Testing
>> >>
>> >>
>> >>
>> >> ------------------------------------------------------------------------------
>> >> See everything from the browser to the database with AppDynamics
>> >> Get end-to-end visibility with application monitoring from
>> >> AppDynamics
>> >> Isolate bottlenecks and diagnose root cause in seconds.
>> >> Start your free trial of AppDynamics Pro today!
>> >>
>> >>
>> >> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> >> [1]
>> >> _______________________________________________
>> >> pydev-code mailing list
>> >> pyd...@li...
>> >> https://lists.sourceforge.net/lists/listinfo/pydev-code [2]
>> >
>> >
>> >
>> > Links:
>> > ------
>> > [1]
>> >
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&amp;iu=/4140/ostg.clktrk
>> > [2] https://lists.sourceforge.net/lists/listinfo/pydev-code
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > See everything from the browser to the database with AppDynamics
>> > Get end-to-end visibility with application monitoring from
>> > AppDynamics
>> > Isolate bottlenecks and diagnose root cause in seconds.
>> > Start your free trial of AppDynamics Pro today!
>> >
>> >
>> > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> >
>> > _______________________________________________
>> > pydev-code mailing list
>> > pyd...@li...
>> > https://lists.sourceforge.net/lists/listinfo/pydev-code
>>
>> --
>> Andreas Pakulat sq...@fr...
>> froglogic GmbH - Automated UI and Web Testing
>>
>>
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> _______________________________________________
>> pydev-code mailing list
>> pyd...@li...
>> https://lists.sourceforge.net/lists/listinfo/pydev-code
>
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code
>

Re: [Pydev-code] Add 'global functions' to PyDev analyzer

[Fabio Z. <fa...@es...>]

On Fri, Aug 9, 2013 at 8:19 PM, Jonah Graham <jo...@ki...> wrote:

> Hi,
>
> I need to solve a similar problem. I have some scripts that are user
> editable, but are run by an execfile with some other magic wrapped
> around it. In that magic some new globals are added, which seem in a
> similar way to how web2py is described in
> http://sourceforge.net/p/pydev/feature-requests/534/
>
> What I would like to do is add a new extension point to extend the
> behaviour of CompiledModule:setTokens() and have it essentially add
> new values to "array" within this if statement?
>
>             //as we will use it for code completion on sources that
> map to modules, the __file__ should also
>             //be added...
>             if (array.size() > 0 && (name.equals("__builtin__") ||
> name.equals("builtins"))) {
>                 array.add(new CompiledToken("__file__", "", "", name,
> IToken.TYPE_BUILTIN));
>                 array.add(new CompiledToken("__name__", "", "", name,
> IToken.TYPE_BUILTIN));
>                 array.add(new CompiledToken("__builtins__", "", "",
> name, IToken.TYPE_BUILTIN));
>                 array.add(new CompiledToken("__dict__", "", "", name,
> IToken.TYPE_BUILTIN));
> // extension point data insertion would be here[1]
>             }
>
> Does that sound like a correct solution? Does that seem generally
> useful? If so I can put an initial pull request together for review.
>
> [1] Should the extension point users return AbstractToken, or is there
> a subset of that API that should be exposed (e.g. just name). My
> preference would be AbstractToken, to allow full range of options for
> extension point user.
>

I think it sounds like a proper solution... when getting that, it may be
nice providing the interpreter itself, the shell and the result of
manager.getCompletePythonPath(
 nature.getProjectInterpreter(), nature.getRelatedInterpreterManager())
which are in that same context, as PyDev itself could check if web2py is
installed when asked for the builtins and if it is it could provide the
web2py tokens to support it properly.

The extension point users should probably return a List<IToken>.

Cheers,

Fabio


>
> Jonah
>
>
> On 23 July 2013 12:39, Fabio Zadrozny <fa...@es...> wrote:
> > On Tue, Jul 23, 2013 at 4:46 AM, Andreas Pakulat <an...@fr...>
> > wrote:
> >>
> >> Hi Fabio,
> >>
> >> thanks for the hint about the pycompletionserver.py. Currently we
> >> already have a change or two to PyDev and roll our own but we would
> >> rather use the official version.
> >>
> >> Unfortunately our interpreter is embedded into a commandline tool that
> >> does not support being used like a normal python interpreter. So thats
> >> why  choosing it as an interpreter is not suitable at the moment. What
> >> it does is basically load the main script file (i.e. parse it but does
> >> not run it immediately), then add a few modules to the python
> >> interpreter via C API, import those modules into the global namespace of
> >> the main script file and also import several functions into the global
> >> namespace. This was done as a convenience for the users but now clearly
> >> bites back at us when trying to teach PyDev about these specialties.
> >
> >
> > Ok, so, it's not really a custom interpreter, just a different main...
> Not
> > sure if that's the best approach (I'd probably go toward a more common
> > approach of having a different main entry which then would run the script
> > after configuring the environment, such as most web frameworks do or
> maybe a
> > custom sitecustomize).
> >
> > Still, if you bundle it as an executable, it shouldn't be all that hard
> to
> > make it look like a real python interpreter (i.e.: one that could really
> be
> > used as an interpreter -- that way you wouldn't need to change PyDev at
> all
> > and everything should just work). This is probably the preferred
> approach.
> >
> > Now, if that's difficult for you, the
> pycompletionserver.py/importsTipper.py
> > adjustment probably requires less fiddling to integrate.
> >
> > Regarding a better approach within PyDev itself, please create a feature
> > request for that in https://sw-brainwy.rhcloud.com/tracker/PyDev/ (this
> is
> > also a request web2py users have -- for which there's a discussed
> workaround
> > detailed at the old tracker:
> > http://sourceforge.net/p/pydev/feature-requests/534/).
> >
> >>
> >>
> >> Its of course also possible to import those registered modules in the
> >> script file or in one that is being imported into the main script.
> >>
> >> What I ended up doing now which seems to work with 2 caveats (see
> >> further down) is that I'm using the extension points
> >> pydev_interpreter_new_custom_entries, pydev_modules_observer and the
> >> completion extension point. The first one registers the names of the
> >> modules we add via the C API as 'builtins', the second one adds tokens
> >> to the __builtin__ module for those modules and the functions we import
> >> into the global namespace. It also populates our modules with their
> >> functions when the PyDev code needs them (a script importing our squish
> >> module for example). All this seems to avoid errors being flagged for
> >> our Squish API, but does not yet provide the modules and functions for
> >> completion. Hence using the completion extension point to support that.
> >>
> >> The two minor issues with this are:
> >>
> >> 1. For the builtins registration using the
> >> pydev_interpreter_new_custom_entries extension point the users need to
> >> remove and re-add the interpreter so that PyDev fetches the builtins
> >> again.
> >>
> >> Maybe there's a way to do the above programatically upon startup of a
> >> plugin?
> >>
> >> 2. When using 'import squish' in a script now a warning is flagged
> >> about this shadowing a builtin module which is correct from what PyDev
> >> knows but not really happening in the actual interpreter that executes
> >> the tests.
> >>
> >> The second point is not much of a problem as its just a warning and
> >> hence does not jump out at the user as much as an error (in particular
> >> an error in script code that Squish just generated).
> >>
> >> Andreas
> >>
> >>
> >> On 2013-07-22 22:39, Fabio Zadrozny wrote:
> >> > Hi Andreas,
> >> >
> >> > Actually, if you really have a custom interpreter and this is added
> >> > in its startup, it's strange that it doesn't work already (PyDev
> >> > always spawns a shell for the completion and upon startup it'll load
> >> > the __builtin__ -- or builtins -- module and do a dir() in it).
> >> >
> >> > So, if it's custom interpreter, it's mostly a matter of filling the
> >> > __builtins__ at the startup... If that's not the case, you could
> >> > change PyDev to add some custom code at the module scope at
> >> > /org.python.pydev/pysrc/pycompletionserver.py (but you'll have to
> >> > maintain your own fork if you choose this path).
> >> >
> >> > Cheers,
> >> >
> >> > Fabio
> >> >
> >> > On Fri, Jul 19, 2013 at 12:15 PM, Andreas Pakulat
> >> > <an...@fr...> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I'm currently working on improving our usage of PyDev, in particular
> >> >> starting to let PyDev not only syntax highlight but also analyze the
> >> >> code our users write. Unfortunately this has one drawback, we use a
> >> >> custom interpreter and add several global functions to it (this is
> >> >> done
> >> >> by having a module for these functions and then importing everything
> >> >> from this module by default into the global namespace).
> >> >>
> >> >> I know about the setting for the PyDev analyzer to not show errors
> >> >> on
> >> >> certain undefined functions/variables, but the number of functions
> >> >> makes
> >> >> this single lineedit/preference pretty cumbersome to extend.
> >> >>
> >> >> I'm wondering wether there's a different way (via code maybe?) to
> >> >> teach
> >> >> PyDev about a list of functions it should consider to be
> >> >> 'predefined',
> >> >> similar to the builtin modules?
> >> >>
> >> >> I'm currently using PyDev 2.6, but could upgrade to 2.7.5 if that
> >> >> makes
> >> >> it easier to achieve what I want.
> >> >>
> >> >> Andreas
> >> >>
> >> >> --
> >> >> Andreas Pakulat sq...@fr...
> >> >> froglogic GmbH - Automated UI and Web Testing
> >> >>
> >> >>
> >> >>
> >> >>
> ------------------------------------------------------------------------------
> >> >> See everything from the browser to the database with AppDynamics
> >> >> Get end-to-end visibility with application monitoring from
> >> >> AppDynamics
> >> >> Isolate bottlenecks and diagnose root cause in seconds.
> >> >> Start your free trial of AppDynamics Pro today!
> >> >>
> >> >>
> >> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> >> >> [1]
> >> >> _______________________________________________
> >> >> pydev-code mailing list
> >> >> pyd...@li...
> >> >> https://lists.sourceforge.net/lists/listinfo/pydev-code [2]
> >> >
> >> >
> >> >
> >> > Links:
> >> > ------
> >> > [1]
> >> >
> >> >
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&amp;iu=/4140/ostg.clktrk
> >> > [2] https://lists.sourceforge.net/lists/listinfo/pydev-code
> >> >
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > See everything from the browser to the database with AppDynamics
> >> > Get end-to-end visibility with application monitoring from
> >> > AppDynamics
> >> > Isolate bottlenecks and diagnose root cause in seconds.
> >> > Start your free trial of AppDynamics Pro today!
> >> >
> >> >
> >> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> >> >
> >> > _______________________________________________
> >> > pydev-code mailing list
> >> > pyd...@li...
> >> > https://lists.sourceforge.net/lists/listinfo/pydev-code
> >>
> >> --
> >> Andreas Pakulat sq...@fr...
> >> froglogic GmbH - Automated UI and Web Testing
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> See everything from the browser to the database with AppDynamics
> >> Get end-to-end visibility with application monitoring from AppDynamics
> >> Isolate bottlenecks and diagnose root cause in seconds.
> >> Start your free trial of AppDynamics Pro today!
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> pydev-code mailing list
> >> pyd...@li...
> >> https://lists.sourceforge.net/lists/listinfo/pydev-code
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > See everything from the browser to the database with AppDynamics
> > Get end-to-end visibility with application monitoring from AppDynamics
> > Isolate bottlenecks and diagnose root cause in seconds.
> > Start your free trial of AppDynamics Pro today!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> > _______________________________________________
> > pydev-code mailing list
> > pyd...@li...
> > https://lists.sourceforge.net/lists/listinfo/pydev-code
> >
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> pydev-code mailing list
> pyd...@li...
> https://lists.sourceforge.net/lists/listinfo/pydev-code
>