PYTHONPATH and code completion

2005-02-13
2013-03-15
  • Hi,

    Last week I filed bug# 1118129 about from ... import ... and we ended with a discussion about code completion and the PYTHONPATH...

    Now I don't have my PYTHONPATH set explicitly in the environment, but Python itself nevertheless seems to work OK.
    'Restore PYTHONPATH' works (gives me a path that looks OK), the sys.path in a console looks OK.

    Flavio asked me to check if built-ins work: When I'm on an empty line in a Python file, and hit ctrl-space, I get a long list of built-ins (including str, etc).

    So I believe that there's no real problem with my pythonpath setting.

    But when I import 'datetime' (to stay with that example), and type 'datetime.', I get no code-completions for the datetime module. (Same for module 'os' for instance).

    What are the further suggestions?
    I don't see anything in the error log. (Well, I do, but not from today, and I don't expect past errors to influence functioning now.)

    cheers,

    --Tim

     
    • Fabio Zadrozny
      Fabio Zadrozny
      2005-02-14

      Well, datetime is a .pyd file in windows, located (in my python distribution) at python/dlls, so, make sure that this folder is in your pythonpath (in the enviroment pythonpath and in the project pythonpath - check the FAQ if you have any doubts regarding that).

      cheers,

      Fabio

       
    • Hi Fabio,

      I double-checked a number of things, but only partial progress...

      I checked the PYTHONPATH in Eclipse, and it got lost. Restoring it gives me the correct-looking path, including the python2.4/dlls directory.

      I now get code-comletions for the os module, but somehow still not for the datetime module (no matter how I write the import statement).

      I believe that everything should be set up correctly, both for the project and for the environment.

      Now I tried to switch the PYTHONPATH environment to python2.3 and to that end I changed the project properties to select the python2.3 runtime, and tried to restore the pythonpath, but it's still pointing at 2.4 for everything. To get a PYTHONPATH for a different Python version, I have to change the order in the debug - preferences. I'd say that this is a bug, b/c a generic setting overrides what I set for the project?

      Anyways, as soon as I set the PYTHONPATH for Python2.3 the code-completion now works for the datetime module! For both forms of the import statement:

      import datetime

      and:

      from datetime import datetime

      Both now work!

      BUT when I switch back to Python2.4 again as the default Python runtime, and restore PYTHONPATH for Python2.4 again, then I loose my code-completion on the datetime module once again.

      So now that I've reached this conclusion, I guess I should file a bug-report for this issue?

      regards,

      --Tim

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-14

        Yep, seems a bug to me. I haven't switched to 2.4 yet, so, right now I don't know why this happens to 2.4. After I switch (still no date though), I will take a look at it.

        cheers,

        Fabio

         
    • Hi,

      Unfortunately I don't see an error appearing in the PYDEV error log.

      I don't know if perhaps the Python stdout might give us some clues about what's going wrong but I don't know if / how to check that?

      cheers,

      --Tim

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-14

        Well, one thought: After switching versions, you have to 'kill' the python shells that the pydev plugin lauches (that is, restart eclipse and erase all your .pyc files in eclipse). This may have something to do about it...

        Fabio

         
    • Hi Fabio,

      Well, that's something which I'll try... I did get code-completion for some items but I'll check this later today.
      thanks!

      --Tim

       
    • Ok Fabio,

      I tried removing all *.pyc files and no luck. Still no code-completion on the 'datetime' module. I didn't check out which other DLL based modules exist or if they work. I did check the module 'os' and it's sub-module 'os.path', and that works.

      cheers,

      --Tim

       
    • Hi Fabio,

      If I find some time to do some debugging, do you then mind me reworking the code to use the logging API? And if I do so, would you accept patches of me to introduce logging into the code?

      Or you prefer to stay away from the logging API, and should I turn on logging with print statements again?

      cheers,

      --Tim

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-15

        Hi Tim,

        you're saying the JAVA logging API right? I think it would be good to introduce it, yes... how would you intend to configure it?

        cheers,

        Fabio

         
    • Hi Fabio,

      I meant the Python logging API -

      http://www.python.org/doc/2.3.5/lib/module-logging.html

      I thought to configure it from the code for a start; configuration using config files can be added later.

      --Tim

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-15

        For a start, configuring through the code seems ok, but I think it really should be configured through the pydev preferences (or at least point to the config file and output file from the preferences).

        Fabio

         
    • Hi Fabio,

      So far with the logging I added, I get the impression that the pycompletionserver just hangs after listening on the socket...

      The pycompletionserver starts twice, and doesn't give me any messages that it's about to stop. It also doesn't give me any messages that it received any messages.

      Unfortunately it doesn't work now with python2.3 whereas it did work before; I removed all *.pyc but still no luck. (Error from Eclipse: cannot connect to external python process, when I try to update the PYTHONPATH.) So I don't have sufficient comparison material yet to present any conclusions.

      I'm too busy with other things to get into it now but will post back when I have any findings.

      cheers,

      --Tim

       
    • With the latest PyGtk version, there are no longer crashes when getting completions for the Gtk import object....

      That means that the pythoncompletionserver is being called and sending back results; however I don't see evidence of that in my logs!
      Given the logging that I'm doing, I *should* see something in the log when a connection has been established, and when a message has been received. But the last thing I see in the log, is what I'm logging just before doing the 'socket.bind'. (That's already in the thread that's started.)
      That leaves me actually kind of clueless as to what's going wrong, and what code actually gets executed.

      Fabio, can you give me some hints perhaps? What kind of thing could happen that logging stops working??

      (datetime, which doesn't work for me in python2.4, is a .pyd file. Gtk is also a .pyd file. So something in python2.4 makes the datetime.pyd break, while it doesn't break the _gtk.pyd ... I don't understand it at all!)

      --Tim

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-16

        Just a question: are the gtk completions being gotten without problems? (I didn't understand if it just did not crash or if it did function correctly).

         
    • Hi Fabio!

      Sorry for being so unclear. I wanted to post on my partial results but didn't have time to try make myself clearer.

      To make it clear: I got the gtk completions without problems now, thanks to the new pygtk version.

      I also worked out by now why my modificiations to pycompletionserver.py broke python2.3; and I'm reworking them to be py2.3 compatible.

      But I haven't yet worked out why:
      A) I'm getting duplicate log messages (python is started twice but is that normal?)
      B) I'm not getting any logmessages anymore after the bind!

      I'm trying to get at least something out of the logs but was hoping you -- or someone else -- could perhaps provide me with clues why I see nothing in the logs after binding to the socket. (Even when I do get completions, while performing an action that previously resulted in a crashing python.exe. Meaning: the pycompletionserver *must* be doing *something*)

      cheers,

      --Tim

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-16

        Tim,

        Yes. it is started twice: one shell is created for code completion and another for refactoring and other things.

        As for the log messages not being shown, I can give a look in your changes to the code if you send it to me (without looking at it, it is hard to guess anything - can you send me a patch of it?).

        cheers,

        Fabio

         
    • Hi Fabio!

      I don't have now yet a patch for you with my changes, but I do have now a logfile for you with an error.

      (I modified the code to write to stderr as well as file, and re-activated your code to redirect stderr to a file. And oh joy, the logging-to-file breaks but logging-to-stderr-redirected-to-file works!)

      As you can see in the log, the logging to file breaks (file closed) and completion for 'datetime' fails... b/c there's no such module 'datetime'.

      Perhaps this helps you somewhat in solving my original problem: that I don't get completions for the datetime module in combination with py2.4!

      --Tim

      Traceback (most recent call last):
        File "d:\Python24\lib\logging\__init__.py", line 712, in emit
          self.stream.write(fs % msg)
      ValueError: I/O operation on closed file
      2005-02-16 19:01:35,184 [4688] pydev.pycompletionserver ERROR Got exception during message processing!
      Traceback (most recent call last):
        File "d:\java\eclipse\plugins\org.python.pydev_0.9.0\PySrc\pycompletionserver.py", line 197, in run
          comps = importsTipper.GenerateTip(data)
        File "d:\java\eclipse\plugins\org.python.pydev_0.9.0\PySrc\importsTipper.py", line 70, in GenerateTip
          return GenerateImportsTip(splitted)
        File "d:\java\eclipse\plugins\org.python.pydev_0.9.0\PySrc\importsTipper.py", line 106, in GenerateImportsTip
          return GenerateImportsTipForModule(newTokenList, mod)
        File "d:\java\eclipse\plugins\org.python.pydev_0.9.0\PySrc\importsTipper.py", line 122, in GenerateImportsTipForModule
          mod = getattr(mod, tokenList.pop(0))
      AttributeError: 'module' object has no attribute 'datetime'

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-16

        can you log the values to the parameters passed to the function: GenerateImportsTip in importsTipper.py?

        --tokenList=?
        --pth=?
        --completeModule=?

        and for GenerateTip

        --data=?

        Fabio

         
    • Hi Fabio,

      Here's the logging you asked for.

      2005-02-16 20:45:16,885 [4240] pydev.pycompletionserver DEBUG received message:
      @@IMPORTS:__builtin__.datetime
      END@@
      2005-02-16 20:45:16,885 [4240] pydev.importsTypper DEBUG params: data="__builtin__.datetime
      "
      2005-02-16 20:45:16,885 [4240] pydev.importsTypper DEBUG params: tokenList="['__builtin__', 'datetime']", pth="['D:\\java\\e3wkspc2\\PyTestProj\\pysrc', 'd:\\java\\eclipse\\plugins\\org.python.pydev_0.9.0\\PySrc', 'd:\\java\\eclipse\\plugins\\org.python.pydev_0.9.0\\PySrc\\ThirdParty\\brm', 'C:\\WINDOWS\\System32\\python24.zip', 'd:\\java\\eclipse\\plugins\\org.python.pydev_0.9.0\\PySrc', 'D:\\Python24\\DLLs', 'D:\\Python24\\lib', 'D:\\Python24\\lib\\plat-win', 'D:\\Python24\\lib\\lib-tk', 'D:\\Python24', 'D:\\Python24\\lib\\site-packages', 'D:\\Python24\\lib\\site-packages\\gtk-2.0', 'D:\\Python24\\lib\\site-packages\\win32', 'D:\\Python24\\lib\\site-packages\\win32\\lib', 'D:\\Python24\\lib\\site-packages\\Pythonwin']", completeModule=""

       
      • Fabio Zadrozny
        Fabio Zadrozny
        2005-02-17

        Ok, I guess I see the problem, but just check one last thing: what is the project pythonpath you're using (in the project properties).

         
    • Hi,

      After a number of experiments, some things to think about... (I'll sort out my logfiles later to see what can be useful from them)

      1) With Python2.4, logging to file stops (file closed) after opening a socket. With Python 2.3, logging to file continues to work. I'll have to stir this up on the Python mailing list.

      2)  I've been getting inconsistent results with 'from datetime import datetime' with Python2.3. Sometimes it works, sometimes it only works after first importing the datetime module. Can't quite figure it out but something seems to go wrong sometimes.
      'from os import path' works all the time in all Python versions (and also doesn't give pylint errors). But here, 'path' is not a class but a module inside the module... Which of course is a difference.

      Have you meanwhile got any ideas why there's breakage with Python2.4?

      I suspect it's got something to do with:
      A) The fact that I'm trying to import 1 class from a module
      B) This class has the same name as that module.

      --Tim

       
    • Hi Fabio,

      Here's the pythonpath of that project:

      D:\Python24\lib\site-packages\win32
      D:\Python24\lib\site-packages\win32\lib
      D:\Python24\DLLs
      D:\Python24\lib\site-packages\gtk-2.0
      D:\Python24
      D:\Python24\lib\lib-tk
      d:\java\eclipse\plugins\org.python.pydev_0.9.0\PySrc
      D:\Python24\lib\site-packages\Pythonwin
      D:\Python24\lib
      D:\Python24\lib\site-packages

       
      • Ken Pier
        Ken Pier
        2005-02-17

        I've been monitoring this forum and just have to ask a basic question:

        Where should the pythonpath appear in the project properties?  My python project properties has only a tiny text box labelled "Include path" .  Is this where the pythonpath should go?

        pydev0.8.0  Eclipse3.0.1 on Windows XP

         
        • Hi Ken,

          For your version of pydev, things still work a little differently (I believe).

          Upgrade to pydev 0.9, and look at the Pydev properties in your Project's property page.

          cheers,

          --Tim