CEDET

Dan Debertin writes:
> After a bit of exploratory programming, I've decided to work on
> getting CEDET to recognize Scala programs. As I mentioned before, I
> have converted the entire Scala grammar to BNF and gotten Wisent to
> parse the thing and generate tags for variables, classes, traits,
> objects and methods/functions. They even show up in the methods window
> of ECB (which is very gratifying! ;-).

That's great!

> Are there any other tags I should generate that would be useful?  I
> see that other grammars (notably Python) generate CODE-TAGs all over
> the place; how (if at all) are those used by CEDET?

I don't think the Python parser really uses them. It's just that the
Python grammar actually does parse code (instead of just structures and
types), so at least it does not hurt to generate code tags. They could
be useful for stuff like refactoring.

> The next step is to define overrides for the functions that will allow
> completion, analysis of the context, etc. I know about
> semantic-analyze-current-context and semantic-get-local-variables; are
> there any others I should look at for full CEDET support?

There's no general answer for that. There are many, *many* functions
which you can override; doing a grep for 'define-mode-local-override' on
files like bovine/c.el will show you some of them. Most of these
functions have a default implementation which might or might not work,
depending on the language and its major-mode. The default implementation
is pretty general, but usually has C++ in mind, which also happens to be
a good upper bound on the complexity of programming languages in
general.

The best approach is to take something that's currently not working and
then try to see which functionality has to be overridden. If you need
help with that, just ask here; you sometimes have to dig real deep to
see what's missing. It might also be that the major-mode is missing some
feature. For instance, Semantic often depends on stuff like
`forward-sexp' to work correctly. Also, a correct syntax-table is very
important.

Good luck,
David

On Mon, Nov 12, 2012 at 1:14 PM, David Engster <deng@...> wrote:

>> Are there any other tags I should generate that would be useful?  I
>> see that other grammars (notably Python) generate CODE-TAGs all over
>> the place; how (if at all) are those used by CEDET?
>
> I don't think the Python parser really uses them. It's just that the
> Python grammar actually does parse code (instead of just structures and
> types), so at least it does not hurt to generate code tags. They could
> be useful for stuff like refactoring.

My reason for asking is that parsing the entire language adds quite a
bit of complexity to the grammar (upwards of 200 productions), not to
mention a number of pesky S/R and R/R conflicts I have to chase down,
for not much real gain AFAICT. If (like I did in CEDET's Ruby grammar)
I could reduce it to just an infrastructure grammar it would simplify
things quite a bit, but I will keep the full grammar if it would
enable useful functionality.

As I recently realized when I sat down to implement it, "smart
completion" will probably never work with Scala. Since scala uses type
inference, many variable and function types don't have to be declared
at all in the code, so it's impossible (well, really hard) to know a
variable's type until compile-time. I can see now why ENSIME invokes
the compiler directly and uses reflection to introspect types instead
of trying to infer them from the code. Given that, I am not sure what
benefit my work has over the ctags-based implementation.

I am, per your advice, looking at the various overrides and
determining what would be useful to implement, but with somewhat
reduced enthusiasm.

-Dan D
--
debertin@...

Dan Debertin writes:
> My reason for asking is that parsing the entire language adds quite a
> bit of complexity to the grammar (upwards of 200 productions), not to
> mention a number of pesky S/R and R/R conflicts I have to chase down,
> for not much real gain AFAICT. If (like I did in CEDET's Ruby grammar)
> I could reduce it to just an infrastructure grammar it would simplify
> things quite a bit, but I will keep the full grammar if it would
> enable useful functionality.

A 'tagging grammar' is usually sufficient for most tasks. I could
imagine full grammars to be useful for things like refactoring or even
static code analysis, but really, no one has done that yet using
Semantic, so that's just me daydreaming.

> As I recently realized when I sat down to implement it, "smart
> completion" will probably never work with Scala. Since scala uses type
> inference, many variable and function types don't have to be declared
> at all in the code, so it's impossible (well, really hard) to know a
> variable's type until compile-time. 

Yes, that's a hard problem. C++11 now has the same thing with the new
'auto' keyword. There's no easy way out; you have to look at the
right-hand side and infer the type. I think this will be possible for
C++, but I'm mostly afraid that it will make parsing much slower.

I've already looked at libclang, but this one has the problem of not
being fault-tolerant, which is problematic when parsing unfinished code.

> I am, per your advice, looking at the various overrides and
> determining what would be useful to implement, but with somewhat
> reduced enthusiasm.

Well, you got a chicken/egg problem here. You won't be able to properly
infer types as long as you don't have a working parser. Unfortunately, I
know nothing about Scala, so I can't really help you here or even tell
you how hard this might be. But you might be right that this is just not
feasible. Another approach might be to ask a compiler, like ENSIME does.

-David

On 11/07/2012 12:59 PM, Dan Debertin wrote:
> Are there any other tags I should generate that would be useful? I see
> that other grammars (notably Python) generate CODE-TAGs all over the
> place; how (if at all) are those used by CEDET?

Hi Dan,

David did a good job answering your other questions, but I can answer 
this one.  Sorry for the late reply.

CODE tags are used by the parser to mark blocks of code that can't be 
tied to a function  or other construct directly.  The CODE tag's primary 
use is with the incremental parser defining some bounds within which to 
restrict the partial reparse.  If the code was instead marked as 
unmatched, then your edits might affect other live tags nearby, and more 
code would need to be partially reparsed.  I can't recall all the rules 
there, but tagging code in the Emacs Lisp parser this way was helpful. 
It probably helps with the local-context parsers too.

What that means is, if you already have all your functions defined, and 
the function includes the code block, then no need to keep going.  If 
Scala really has code just sitting about unowned by some other well 
defined construct, then a code tag would be useful, but there is no real 
need to parse the code.  Just identify the space it takes up, and 
possibly identify a name.

Hope that helps.
Eric