Correction:  I didn't check to see if completion worked in my Python project when I used ede-cpp-root-project.  I'll try that today.

Also, I'd be very interested in helping enhance the Python module import features in semantic.  There are certainly others who are better equipped, but I have the time and perhaps they don't. I just have no feel for what the best and most "CEDET-like" strategy would be.

Best,
David


On Thu, Dec 19, 2013 at 2:54 PM, David Ventimiglia <dventimi@gmail.com> wrote:
Hi Eric,

As always, thanks to you and the other David for the lucid explanations.  You might consider removing the generic "project root" variable in semantic since it may promote confusion, but of course that's your call.  You might also consider extracting the non-C++-specific parts of ede-cpp-root-project into a more generic superclass that handles defining a project and its include and system paths, but little else.  Actually, you probably already have considered that.  I'd be happy to help with that if need be.  In any case, I did try to use ede-cpp-root-project for a Python project and it worked pretty much as you anticipated.  Semantic could then find the files for imported modules within my project and I could navigate to those files (I check to see if completion worked).  Having said that, like in your experience it failed to find anything related to Python's sys module, but that's unsurprising.  Python has builtin modules that are defined in C (at least, in CPython they are).  In the case of the "sys" module it's defined in sysmodule.h (do a locate on that name).  The situation is a little like it is with Java, insofar as there are certain packages that are built-in and which you'll never be able to find a file that you can parse in that language.  I.e., in Java, "java.lang" is in a compiled JAR file that you can't parse using the semantic Java parser, and in Python, "sys" is in the python binary and in sysmodule.h, neither of which can be parsed using the semantic Python parser.  In the case of Java, I believe you solved that problem using "javap", either the actual binary program that ships with the JDK or a facsimile thereof written in Elisp (I don't remember).  The same thing might have to happen in Python:  get symbols using the python executable itself and some Python "lifting code."  In fact, that may be the appropriate strategy for all Python imports, for several reasons.
  • Unlike with Java, Python has no essential compile step.  This means that types defined in Python source code within your project are immediately available to python (provided PYTHONPATH is set appropriately) and so could be lifted out using a little Python introspection code.  Contrast that with Java, where types defined in Java source code within your project are not available to java or javap until a compile step has occurred.
  • Python module loading can be more complicated than it is in Java.  One module can itself import other modules and then export the symbols so defined as if they were its own.  For semantic's Python parser to know this, it seems like it would have to recursively parse a module's imports and then treat the submodule's symbols as if they belong to the importing module.  Perhaps this is already how semantic's Python parser works out of the box.  That would be great, but I would be unsurprised if it doesn't quite work that way.

As for the other option you considered, which is to add an ede-generic project type so long as one can identify some common file that exists at the root of all python projects, this touches on another aspect of EDE that I don't quite understand.  When I invoke "M-x ede-new" that ends up creating a "Project.ede" file in the root of my project.  Doesn't that file already define the root of a project?  Why does one need some other file (e.g., README, Makefile, etc.) to define root?  The answer may be in the docs, which I'll re-read again.   Thanks!

Best,

David






On Wed, Dec 18, 2013 at 8:06 PM, Eric M. Ludlam <eric@siege-engine.com> wrote:
On 12/18/2013 05:18 PM, David Ventimiglia wrote:
    It's really messy. The nicest solution would be to have something like
    ede-cpp-root-project for python. But at least I could live with some
    ad-hoc solution inside semantic, as long as Eric doesn't veto it.


If I had a vote (which I don't), it would be for the "right" solution
(be that EDE) in lieu of something expedient but ad-hoc.  I'm in no
hurry.  But, I'm afraid I don't (yet) understand the EDE stuff very
well.  Is there no generic EDE project type not tied to a particular
language that says, "This file (say, Project.ede) defines the root of a
project.  In it, you can specify one or more source directories.
Semantic will look for include files relative to those source
directories, in a way appropriate for the given language."?

Hi David,

As the other David mentioned, I'm not too fond of the generic "project root" variable in semantic.  It is one of those things that seemed like a great idea that could do "anything", but then all the things it couldn't do kept showing up, which is why I started focusing on things like EDE which had the API fortitude to handle all the subtle differences in language and source organization.

I also took a run at a generic project which I had called ede-simple. It was eventually removed from CEDET as other simple projects like ede-cpp-root could do a better job with less code by being specific.

The C/C++ support in EDE ended up using a pretty generic API which I hoped could handle multiple languages.  The "include" handling in semantic for C/C++ was similarly generic.  This did not work very well for Java which needed it's own classpath support in EDE and Semantic to work well which is the other language I know pretty well.

Anyway, the point is that a side effect is that some folks have had success using the cpp-root project with other languages for the purpose of marking the root of a project.  If python includes are similarly modeled to c++, (ie - it matches a file on disk you are likely to have access too) you might even be able to use the same C++ slots for system includes etc and point them at python directories.  I was going through a python tutorial a while back, so I tried this:

(ede-cpp-root-project "pythontutorial" :file "/home/eric/python/tutorial/test.py"
     :include-path '( "" )
     :system-include-path (semantic-python-get-system-include-path)
     )

It could find 'cookielib', but it didn't find 'sys'.  I looked for sys.py but I didn't find the file myself with a system-wide 'locate' call.

Of course, setting the system-include-path via EDE is completely redundant since python sets up the generic one already, but I did see that it added these paths a second time.

Of course, ede-cpp-root may try to do c++ things to your python, and won't do any python specific stuff.  That really will need a custom EDE project to avoid, but perhaps this is a good experiment.  If it handles the basics, then you could run with that and collect all the missing features and apply that to a new python project type later.

Lastly, it might be possible to add an ede-generic project type if there is some common file that exists at the root of all python projects, the way "Makefile" shows up for some C projects.  That allows interactive configuration of the local include paths and is more dynamic.

I hope this helps.
Eric