pydev-code

[Pydev-code] Another take at symlinks

[chris v. <chr...@gm...>]

Hello,

I'm currently trying to make a patch which adresses the long-standing
symlink problem on Unix platforms, as I am using Ubuntu Feisty and they
have almost every library symlinked...

I have debugged this a little (using Eclipse 3.2 and the 1.3.5 zipped
src) and I've found out that this is neither a Java problem nor an
Eclipse problem; both of them can in fact handle symlinks just as one
would expect them to (I just mention this because I've seen statements
on this list claiming otherwise). It is actually Pydev's usage of
canonical vs. absolute paths which is causing the problem. pygtk.py is a
good example to show what is happening:

The file pygtk.py is physically located at
/usr/share/python-support/python-gobject/, while the pythonpath points
to /var/lib/python-support/python2.5/. But since
/var/lib/python-support/python2.5/pygtk.py is a symlink to the physical
location, python will accept an "import pygtk" without complaining.

Testing with the aforementioned file, the following behaviour can be seen:

    File f = new File("/var/lib/python-support/python2.5/pygtk.py");
    
    f.isFile() returns true
    f.getCanonicalPath() returns
"/usr/share/python-support/python-gobject/pygtk.py"

This is in line with the Java Documentation which states that
getCanonicalPath() resolves symlinks. ModulesManager.changePythonPath()
contains the following code (f is pointing to the symlinked location
"/var/lib/.." retrieved by scanning the pythonpath):

    String fileAbsolutePath = REF.getFileAbsolutePath(f);
        String m = pythonPathHelper.resolveModule(fileAbsolutePath);

REF.getFileAbsolutePath() canonicalizes the path passed to it, so
resolveModule is called with "/usr/share/python-support/...". But
resolveModule() internally matches the passed file against pythonpath,
and since this is different from the real path (because of the resolved
symlink), it is dropped.

My question is now, why this matching is done in the first place? To
determine the package name, one could just walk up the directories and
look for __init.py__, that would be not only faster but also remove the
symlink problem. Are there any special reasons why the canonical path of
a module must be compared against pythonpath?

regards,
 chris

Re: [Pydev-code] Another take at symlinks

[Fabio Z. <fa...@gm...>]

On 6/25/07, chris vigelius <chr...@gm...> wrote:
>
> Hello,


Hi Chris,

I'm currently trying to make a patch which adresses the long-standing
> symlink problem on Unix platforms, as I am using Ubuntu Feisty and they
> have almost every library symlinked...
>
>  ...

>
> REF.getFileAbsolutePath() canonicalizes the path passed to it, so
> resolveModule is called with "/usr/share/python-support/...". But
> resolveModule() internally matches the passed file against pythonpath,
> and since this is different from the real path (because of the resolved
> symlink), it is dropped.
>
> My question is now, why this matching is done in the first place? To
> determine the package name, one could just walk up the directories and
> look for __init.py__, that would be not only faster but also remove the
> symlink problem. Are there any special reasons why the canonical path of
> a module must be compared against pythonpath?


The current implementation is the way it is because it can resolve module
names even if the file does not exist. E.g.: When a file is removed, the
code-completion builder will have to remove the AST cache for that file --
but when that happens, the file does not actually exist anymore... you can
see that the signature it has is:

public String resolveModule(String fullPath, final boolean
requireFileToExist)

So, if the file does not exist, it might be difficult to resolve it...

Still, I thought it didn't actually resolve symlinks, but you're right about
getCanonicalPath, so, I think this can be worked on -- initially making it
resolve the modules in the changePythonPath, when we know that the files
exist -- fixing it in the builder (for projects) may be a different story
(at: ProjectModulesManagerBuild.rebuildModule)... anyway, 1st things 1st...

The solutions available (I think) would be:
- Creating a new resolveModule that would work on existing files
- Resolving the module when we're looking for the files in the pythonpath
(currently in listFilesForCompletion, it uses
pythonPathHelper.getModulesBelow to get a list of files and then resolves
it, but maybe this could be changed to resolve the File and the actual
module name at the same time).

I think that the second approach would probably be better... would you be
willing to take a look at it?

Cheers,

Fabio

Re: [Pydev-code] Another take at symlinks

[chris v. <chr...@gm...>]

Hi Fabio,

Fabio Zadrozny schrieb:
> The solutions available (I think) would be:
> - Creating a new resolveModule that would work on existing files
> - Resolving the module when we're looking for the files in the
> pythonpath (currently in listFilesForCompletion, it uses
> pythonPathHelper.getModulesBelow to get a list of files and then
> resolves it, but maybe this could be changed to resolve the File and
> the actual module name at the same time).
>
> I think that the second approach would probably be better... would you
> be willing to take a look at it?

I'll try over the weekend.

regards,
 chris

Re: [Pydev-code] Another take at symlinks

[chris v. <chr...@gm...>]

Hi Fabio,

> I'll try over the weekend.
>   
I've created a new implementation for listFilesForCompletion (had to
change a little bit of changePythonPath, too), and it seems to work here
locally, all modules are found regardless whether linked or not.

Now I'm testing a little bit, and found out the following does NOT work:

  from xml.dom import minidom
  doc = minidom.parse("test.xml")
  e = doc. <-- no completion on "doc"

On the other hand, if I write

 doc2 = minidom.Document()

code completion on doc2 works as expected.

Could this be related to my patch, or is it a general problem?

regards,
 chris

Re: [Pydev-code] Another take at symlinks

[Fabio Z. <fa...@gm...>]

Hi Chris,

I've created a new implementation for listFilesForCompletion (had to
> change a little bit of changePythonPath, too), and it seems to work here
> locally, all modules are found regardless whether linked or not.
>
> Now I'm testing a little bit, and found out the following does NOT work:
>
>   from xml.dom import minidom
>   doc = minidom.parse("test.xml")
>   e = doc. <-- no completion on "doc"
>
> On the other hand, if I write
>
> doc2 = minidom.Document()
>
> code completion on doc2 works as expected.
>
> Could this be related to my patch, or is it a general problem?


It's a 'general problem'. Basically, pydev will try to analyze the parse
function for its return value and is unable to discover the likely value in
the return, whereas in the other option, it can readily find the assignment
to Document.

Cheers,

Fabio