CEDET

Hi Richard (& Eric),

 > The following summarizes my understanding of the benefits of
 > semantic. Please correct me if I'm mistaken. If I'm on the
 > righttrack, I would use something like this in an introductory
 > chapter of the semantic manual to give the reader an overview of
 > semantic.
[...]

Having a diagram that positions concepts and tools is a grand idea!
IMO, you got the essence of Semantic's concepts ;-)

I just redraw your diagram in a different manner to illustrate the
target architecture of Semantic (Eric, correct me if I am wrong ;-)

Here it is:

+--------------------------------------------------------------------+
|                        Languages/Grammars                          |
|=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8|
| C/C++, Java, Python, ELisp, Scheme, Texinfo, BNF, WY, ...          |
+--------------------------------------------------------------------+
                                 |
                                 v
                    +-------------------------+
                    |  Common Grammar Spec.   |
                    +-------------------------+
                                 |
           +---------------------+---------------------+
           |                     |                     |
           v                     v                     v
    +--------------+      +--------------+      +--------------+
    | LL Grammar   |      | LALR Grammar |      | ... Grammar  |
    | Framework    |      | Framework    |      | Framework    |
    +--------------+      +--------------+      +--------------+
           |                     |                     |
           v                     v                     v
    +--------------+      +--------------+      +--------------+=20
    | LL Parser    |      | LALR Parser  |      | ... Parser   |
    +--------------+      +--------------+      +--------------+
           |                     |                     |
           +---------------------+---------------------+
                                 |
                                 v
                    +-------------------------+
                    |        Parser API       |
                    |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8|
                    | Common Parser Framework |
                    +-------------------------+
                                 |
                                 v
                    +-------------------------+
                    | Common Parse Tree Spec. |
                    |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8|
                    |      Parse Tree API     |
                    +-------------------------+
                                 |
                                 v
+--------------------------------------------------------------------+
|                           Applications                             |
|=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8|
| SemanticDB, Senator, Speedbar, Document, Analyse, COGRE, ECB,      |
| JDEE, ...                                                          |
+--------------------------------------------------------------------+

Thanks!
David

That is a great summary.  I have a couple additions/questions below.
You can ignore them if they are "too much detail" for the audience
you are targeting.

>>> "Richard Y. Kim" <ryk@...> seems to think that:
>Eric and David,
>
>The following summarizes my understanding of the benefits of
>semantic. Please correct me if I'm mistaken. If I'm on the right
>track, I would use something like this in an introductory chapter
>of the semantic manual to give the reader an overview of semantic.
>
>-------------------------------------------------------------------
>
>The following diagram illustrates the benefits of using semantic:
>
>* Semantic parses a language (for which a parser is available) and
>  produces a common parse tree regardless of the original source
>  language.

I tend to think of the "parse tree" as "tags", though "parse tree"
sounds cooler and more technical.

>* Semantic provides a common API based on which applications can be
>  written in a language independent way.  This means that the
>  application will work for ALL languages for which a semantic parser
>  can be witten.
>
>* Semantic provids good set of tools and examples for the appplication
>  writers.
>
>* Semantic also provies extensive tools for the parser developers as well.
>  For many languages, those that can be described using a context-free
>  grammer, all that one needs to do is write a grammar file along with
>  appropriate semantic rules.

Didn't you take python's YACC file and port it?  This would be useful
to mention here.

With wisent, you can aso:

* Create language translations (see calc-wy.wy)

And possibly too much detail:

* semantic provides an API for parsing `local context' (see
  semantic-ctxt.el) for intellisense and completion.

Thanks!
Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org

>>> David Ponce <david@...> seems to think that:
>Hi Richard (& Eric),
>
> > The following summarizes my understanding of the benefits of
> > semantic. Please correct me if I'm mistaken. If I'm on the
> > righttrack, I would use something like this in an introductory
> > chapter of the semantic manual to give the reader an overview of
> > semantic.
>[...]
>
>Having a diagram that positions concepts and tools is a grand idea!
>IMO, you got the essence of Semantic's concepts ;-)
>
>I just redraw your diagram in a different manner to illustrate the
>target architecture of Semantic (Eric, correct me if I am wrong ;-)

Good idea David.  I'd augment your diagram like this though:

Eric

>Here it is:
>
+--------------------------------------------------------------------+
|                        Languages/Grammars                          |
|=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8|
| C/C++, Java, Python, ELisp, Scheme, Texinfo, BNF, WY, ...          |
+--------------------------------------------------------------------+
                                 |                                  |
                                 v				     V
                    +-------------------------+       +-----------------+
                    |  Common Grammar Spec.   |       | External Parser |
                    +-------------------------+       +-----------------+
                                 |				     | 	 =20
           +---------------------+---------------------+	     |
           |                     |                     |	     |
           v                     v                     v	     |
    +--------------+      +--------------+      +--------------+    |=20
    | LL Grammar   |      | LALR Grammar |      | ... Grammar  |    |
    | Framework    |      | Framework    |      | Framework    |    |
    +--------------+      +--------------+      +--------------+    |
           |                     |                     |	     |
           v                     v                     v	     |
    +--------------+      +--------------+      +--------------+    |
    | LL Parser    |      | LALR Parser  |      | ... Parser   |    |
    +--------------+      +--------------+      +--------------+    |
           |                     |                     |	     |
           +---------------------+---------------------+	     |
                                 |			       	     |
                                 v				     |
                    +-------------------------+		     |
                    |        Parser API       |		     |
                    |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8|                     |
                    | Common Parser Framework |       		     |
                    +-------------------------+		     |
                                 |----------------------------------+
                                 v				     =20
                    +-------------------------+		     =20
                    | Common Parse Tree Spec. |		     =20
                    |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8|		     =20
                    |      Parse Tree API     |=09
                    +-------------------------+=09
                                 |		=09
                                 v		=09
+--------------------------------------------------------------------+
|                           Applications                             |
|=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8|
| SemanticDB, Senator, Speedbar, Document, Analyse, COGRE, ECB,      |
| JDEE, ...                                                          |
+--------------------------------------------------------------------+
 [ ... ]

Eric,

 > Good idea David.  I'd augment your diagram like this though:

Good point!

Are you thinking to Elisp, for which the parser is the built-in one
(`read')?  If so I think it uses the parser framework too (parse-tree
management, hooks, partial re-parse, etc.).  So I will change the
diagram like this:

         +-----------------------------------------------------------+
         |                        Languages/Grammars                 |
         |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8|
         | C/C++, Java, Python, ELisp, Scheme, Texinfo, BNF, WY, ... |
         +-----------------------------------------------------------+
                         |                                  |
                         v                                  |
            +-------------------------+                     |
            |  Common Grammar Spec.   |                     |
            +-------------------------+                     |
                         |                                  |
        +----------------+----------------+                 |
        |                |                |                 |
        v                v                v                 |
+--------------+ +--------------+ +--------------+          |
| LL Grammar   | | LALR Grammar | | ... Grammar  |          |
| Framework    | | Framework    | | Framework    |          |
+--------------+ +--------------+ +--------------+          |
        |                |                |                 |
        v                v                v                 v
+--------------+ +--------------+ +--------------+ +-----------------+
| LL Parser    | | LALR Parser  | | ... Parser   | | External Parser |
+--------------+ +--------------+ +--------------+ +-----------------+
        |                |                |                 |
        +----------------+----------------+                 |
                         |                                  |
                         v                                  |
             +-------------------------+                    |
             |        Parser API       |                    |
             |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8|                    |
             | Common Parser Framework |<-------------------+
             +-------------------------+
                          |
                          v
             +-------------------------+
             | Common Parse Tree Spec. |
             |=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8|
             |      Parse Tree API     |
             +-------------------------+
                          |
                          v
+---------------------------------------------------+
|                    Applications                   |
|=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=A8=
=A8=A8|
| SemanticDB, Senator, Speedbar, Document, Analyse, |
| COGRE, ECB, JDEE, ...                             |
+---------------------------------------------------+

David

>>>>> "EL" == Eric M Ludlam <eric@...> writes:
    EL> 
    EL> That is a great summary.  I have a couple
    EL> additions/questions below. You can ignore them if
    EL> they are "too much detail" for the audience you are
    EL> targeting.

I don't think we should ignore the things you point out.
They all should be mentioned somewhere in the manual.
The question is where in the manual.

To borrow David's characterization, what I tried to do was
to capture the "essence" of semantic. I think the
introductory section should concisely summarize the
"product" so that the reader can see the broad picture
of the product without too much details to
detract and confuse the reader.

    EL> I tend to think of the "parse tree" as "tags",
    EL> though "parse tree" sounds cooler and more
    EL> technical.

Not only cooler and more technical, it also conveys the intended
message to programmers very concisely, since the meaning of "parse
tree" is widely understood.

Besides, semantic/wisent can perform full parses. Python is an
example.  Although the proper semantic rules are yet to be
written, as far as I know, the parser performs full parse,
i.e., no semantic-list token is ever generated.  We may
decide to change this later for performance reasons, but
using the term "parse tree" is not a stretch at all.

    EL> Didn't you take python's YACC file and port it?
    EL> This would be useful to mention here.

I will try to write up my experience with porting Python
grammar to semantic/wisent grammer.  Perhaps that can serve
as a case study which can be part of the semantic manual or
an addendum.

    EL> With wisent, you can aso:
    EL> 
    EL> * Create language translations (see calc-wy.wy)

Again this and any other features of semantic/wisent should
be documented somewhere in the manual.

    EL> And possibly too much detail:
    EL> 
    EL> * semantic provides an API for parsing `local
    EL>   context' (see semantic-ctxt.el) for intellisense
    EL>   and completion.

I don't think I have learned semantic-ctxt.el yet.
One of these days I'll get to this.

>>>>> "DP" == David Ponce <david@...> writes:

    DP> Hi Richard (& Eric),
    DP> 
    DP> Having a diagram that positions concepts and tools is a grand idea!
    DP> IMO, you got the essence of Semantic's concepts ;-)

Thanks.  Since we can include image files to texinfo, we may
create better looking diagrams someday, but for now ascii
diagrams get the job done in many cases.

    DP> I just redraw your diagram in a different manner to illustrate the
    DP> target architecture of Semantic (Eric, correct me if I am wrong ;-)
    DP> Here it is:

This is a nice diagram.
One question though.
I thought BNF and WY grammars had differnt syntax.
So what do you mean by "Common Grammar Spec."?

    DP> +--------------------------------------------------------------------+
    DP> |                        Languages/Grammars                          |
    DP> |¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨|
    DP> | C/C++, Java, Python, ELisp, Scheme, Texinfo, BNF, WY, ...          |
    DP> +--------------------------------------------------------------------+
    DP>                                  |
    DP>                                  v
    DP>                     +-------------------------+
    DP>                     |  Common Grammar Spec.   |
    DP>                     +-------------------------+
    DP>                                  |
    DP> [...]

Hi Richard,  Eric,

>This is a nice diagram.
>
Thanks!

>One question though.
>I thought BNF and WY grammars had differnt syntax.
>So what do you mean by "Common Grammar Spec."?
>
This is the goal ;-)  I planned to work on merging BNF & WY into a 
common spec.!
 I  will probably  use WY spec. as the base for the common spec., 
because WY is already well
described by a LALR grammar , very close of the Bison's one :-)  Also WY 
don't suffer of some
 BNF limitations on syntax (for example in BNF you can't use the lisp #' 
notation).

I wonder if we should find another name for the common grammar spec.?
WY is not general enough.  BNF is good but maybe it should be kept as it 
is now for compatibility?
I don't know if other BNF grammars exist not in semantic distro.  If I 
correctly remember an XPath
grammar exists which use an old BNF spec. for wisent.

Any thought?  My favorite is to keep the nice "Bovine Normal Form" name, 
even if the new spec.
 is not compatible.  IMO only a few grammars exist around, outside of 
semantic.

David

>>> "Richard Y. Kim" <ryk@...> seems to think that:
>>>>>> "EL" == Eric M Ludlam <eric@...> writes:
  [ ... ]
>    EL> Didn't you take python's YACC file and port it?
>    EL> This would be useful to mention here.
>
>I will try to write up my experience with porting Python
>grammar to semantic/wisent grammer.  Perhaps that can serve
>as a case study which can be part of the semantic manual or
>an addendum.

Something useful for someone to know in a summary may be that it can
be done.  If someone just reads the web-site they may decide to
implement the grammar if they know they don't have to write the
grammar from scratch.

Thus, in the introductory text, it could just say "such as a grammar
for yacc or bison."  or "you can port a yacc/bison grammar to
wisent." or some-such.

  [ ... ]
>    EL> And possibly too much detail:
>    EL> 
>    EL> * semantic provides an API for parsing `local
>    EL>   context' (see semantic-ctxt.el) for intellisense
>    EL>   and completion.
>
>I don't think I have learned semantic-ctxt.el yet.
>One of these days I'll get to this.
  [ ... ]

It is both easier and harder than a grammar.  It is easier because it
is all Emacs Lisp (something you have shown much skill in), and
harder because it tends to be C-like in what/how it asks for things
and some languages don't like being wedged into that.

Eric

-- 
          Eric Ludlam:                 zappo@..., eric@...
   Home: http://www.ultranet.com/~zappo            Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net               GNU: http://www.gnu.org