opencxx-users

[Opencxx-users] [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

Hi,

In his e-mail Stefan Reuther offered several patches to OpenC++ parser.
Those patches are welcome, however I think we again hit recurring problem:

Conceptually OpenC++ consists of three sybsystems:

    Frontend --- reads preprocessed sources, makes parse tree
        out of them; possibly OpenC++ static analyzer also belongs
        in this subsystem

    Translator --- traverses syntax tree and transforms it
        according to rules described by metaclasses

    Driver --- runs preprocessor, Frontend, Translator,
        native compiler and linker.

As I understand Michael and Stefan work with Frontend and are not interested
in the rest of the system. I believe, that this is the most frequently
reused part of OpenC++ and I know of at least three more projects that
reuse just this part.

Thus I suggest that we take it out and make it a separate library.
Otherwise I can see serious problems with testing.

As you now, out present methodology is regression testing, e.g. we run
'make test' each time we modify anything.

This is very important, as this allows somebody who is not a guru to come
and commit some fixes without fear that he/she breaks some distant parts
of the package. Our testsuite is pathetically small at the moment, but
it already helps and I believe that it can grow.

But now comes the problem: when we just extend Frontend itself, without
proper support in Translator (e.g. your patches will make 'if ({...}) '
parse, but Translator will choke on it), we do not have way to add tests
that make sure the new Frontend functionality works.

With separate Frontend library, we can test the library on its own, and
this is what we need at the moment, because we somehow have to add tests
for the new features implemented by patches from Fiasco project.

BTW: Over a year ago there was a resolution on this forum to factor out the
Driver subsystem and make it a shell script (for easier maintenance
and to make underlying compiler/preproc/linker settable with Autoconf).

Let me know what you think.

Best regards
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

Yes this is a good idea. 
there is an open task for Stefan and I to modularize openc++.

To be honest, this has not been done. If someone want to take that
over, that would be fine.

The only issue that I see is the replacing of constants like "4" with
things like OPEN_CXX_PARAMETER_NAME so that the code is more reable.

mike

--- Grzegorz Jakacki <ja...@he...> wrote:
> 
> Hi,
> 
> In his e-mail Stefan Reuther offered several patches to OpenC++
> parser.
> Those patches are welcome, however I think we again hit recurring
> problem:
> 
> Conceptually OpenC++ consists of three sybsystems:
> 
>     Frontend --- reads preprocessed sources, makes parse tree
>         out of them; possibly OpenC++ static analyzer also belongs
>         in this subsystem
> 
>     Translator --- traverses syntax tree and transforms it
>         according to rules described by metaclasses
> 
>     Driver --- runs preprocessor, Frontend, Translator,
>         native compiler and linker.
> 
> As I understand Michael and Stefan work with Frontend and are not
> interested
> in the rest of the system. I believe, that this is the most
> frequently
> reused part of OpenC++ and I know of at least three more projects
> that
> reuse just this part.
> 
> Thus I suggest that we take it out and make it a separate library.
> Otherwise I can see serious problems with testing.
> 
> As you now, out present methodology is regression testing, e.g. we
> run
> 'make test' each time we modify anything.
> 
> This is very important, as this allows somebody who is not a guru to
> come
> and commit some fixes without fear that he/she breaks some distant
> parts
> of the package. Our testsuite is pathetically small at the moment,
> but
> it already helps and I believe that it can grow.
> 
> But now comes the problem: when we just extend Frontend itself,
> without
> proper support in Translator (e.g. your patches will make 'if ({...})
> '
> parse, but Translator will choke on it), we do not have way to add
> tests
> that make sure the new Frontend functionality works.
> 
> With separate Frontend library, we can test the library on its own,
> and
> this is what we need at the moment, because we somehow have to add
> tests
> for the new features implemented by patches from Fiasco project.
> 
> BTW: Over a year ago there was a resolution on this forum to factor
> out the
> Driver subsystem and make it a shell script (for easier maintenance
> and to make underlying compiler/preproc/linker settable with
> Autoconf).
> 
> Let me know what you think.
> 
> Best regards
> Grzegorz
> 
> ##################################################################
> # Grzegorz Jakacki                       Huada Electronic Design #
> # Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
> # tel. +86-10-64365577 x2074               Beijing 100015, China #
> # Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
> ##################################################################
> 


=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
http://webhosting.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Thu, 13 Mar 2003, James Michael DuPont wrote:

> Yes this is a good idea.
> there is an open task for Stefan

Stefan Seefeld, that is

> and I to modularize openc++.
>
> To be honest, this has not been done. If someone want to take that
> over, that would be fine.
>
> The only issue that I see is the replacing of constants like "4" with
> things like OPEN_CXX_PARAMETER_NAME so that the code is more reable.

Could you explain what do you mean? I think I have much cleaner solution,
but I am not sure if this is the solution to the problem you mention :-)

Best regards
Grzegorz

>
> mike
>
> --- Grzegorz Jakacki <ja...@he...> wrote:
> >
> > Hi,
> >
> > In his e-mail Stefan Reuther offered several patches to OpenC++
> > parser.
> > Those patches are welcome, however I think we again hit recurring
> > problem:
> >
> > Conceptually OpenC++ consists of three sybsystems:
> >
> >     Frontend --- reads preprocessed sources, makes parse tree
> >         out of them; possibly OpenC++ static analyzer also belongs
> >         in this subsystem
> >
> >     Translator --- traverses syntax tree and transforms it
> >         according to rules described by metaclasses
> >
> >     Driver --- runs preprocessor, Frontend, Translator,
> >         native compiler and linker.
> >
> > As I understand Michael and Stefan work with Frontend and are not
> > interested
> > in the rest of the system. I believe, that this is the most
> > frequently
> > reused part of OpenC++ and I know of at least three more projects
> > that
> > reuse just this part.
> >
> > Thus I suggest that we take it out and make it a separate library.
> > Otherwise I can see serious problems with testing.
> >
> > As you now, out present methodology is regression testing, e.g. we
> > run
> > 'make test' each time we modify anything.
> >
> > This is very important, as this allows somebody who is not a guru to
> > come
> > and commit some fixes without fear that he/she breaks some distant
> > parts
> > of the package. Our testsuite is pathetically small at the moment,
> > but
> > it already helps and I believe that it can grow.
> >
> > But now comes the problem: when we just extend Frontend itself,
> > without
> > proper support in Translator (e.g. your patches will make 'if ({...})
> > '
> > parse, but Translator will choke on it), we do not have way to add
> > tests
> > that make sure the new Frontend functionality works.
> >
> > With separate Frontend library, we can test the library on its own,
> > and
> > this is what we need at the moment, because we somehow have to add
> > tests
> > for the new features implemented by patches from Fiasco project.
> >
> > BTW: Over a year ago there was a resolution on this forum to factor
> > out the
> > Driver subsystem and make it a shell script (for easier maintenance
> > and to make underlying compiler/preproc/linker settable with
> > Autoconf).
> >
> > Let me know what you think.
> >
> > Best regards
> > Grzegorz
> >
> > ##################################################################
> > # Grzegorz Jakacki                       Huada Electronic Design #
> > # Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
> > # tel. +86-10-64365577 x2074               Beijing 100015, China #
> > # Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
> > ##################################################################
> >
>
>
> =====
> James Michael DuPont
> http://introspector.sourceforge.net/
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Web Hosting - establish your business online
> http://webhosting.yahoo.com
>
>

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Michael H. <ho...@sa...>]

Grzegorz Jakacki <ja...@he...> writes:

> Conceptually OpenC++ consists of three sybsystems:
>
>     Frontend --- reads preprocessed sources, makes parse tree
>         out of them; possibly OpenC++ static analyzer also belongs
>         in this subsystem
>
>     Translator --- traverses syntax tree and transforms it
>         according to rules described by metaclasses
>
>     Driver --- runs preprocessor, Frontend, Translator,
>         native compiler and linker.
>
> As I understand Michael and Stefan work with Frontend and are not interested
> in the rest of the system. I believe, that this is the most frequently
> reused part of OpenC++ and I know of at least three more projects that
> reuse just this part.

(Just as an aside: Would you mind posting a list of the projects you
know that use OpenC++ to this list or the project's homepage?  I would
like to have a look at them, and I really learned a lot from looking
at the Synopsis source code, for example.)

> Thus I suggest that we take it out and make it a separate library.

There is a fourth part that lives somewhere in the middle between
Frontend and Translator (maybe that's what you mean by ``static
analyzer''): OpenC++'s symbol-table implementation, encapsulated in
classes Environment, Encoding and friends.  We had trouble
understanding and reusing particularly this part, and in the end
decided not to use it, so it should be separable as well.

> Otherwise I can see serious problems with testing.
[...]
> With separate Frontend library, we can test the library on its own, and
> this is what we need at the moment, because we somehow have to add tests
> for the new features implemented by patches from Fiasco project.

I think what you are proposing can be achieved by some
build-infrastructure (and maybe some terminology and documentation)
changes, plus very little.  You actually do not need complete
separation of the four parts comprising OpenC++ in order to facilitate
testing -- separation would just be convenient for reuse.

In my opinion, what's needed is just:

1. An abstract base class for Walker, which just provides the
   interface for visiting a Ptree hierarchy and is completely
   independent from the Translator and symbol-table parts.  (As an
   added bonus, its visit functions should support return types other
   than Ptree*, so it probably should be a template.)

2. A simple unit-testing framework, to which tests for new
   functionality (on a per-module level) can be added easily

For our VFiasco project <http://www.vfiasco.org>, we use a wrapper
around Walker which provides (1), and (2) consists of a set of
unit-test programs that output test data and a simple Makefile (for
GNU Make) that tests for regressions.

All in all, I think that the changes I proposed should be gradually
added to the source base, but nothing more.  Frequent changes to the
build system and library interface actually make reuse harder, not
easier, and besides the points I mentioned, there isn't really that
much of an itch to scratch.

> BTW: Over a year ago there was a resolution on this forum to factor out the
> Driver subsystem and make it a shell script (for easier maintenance
> and to make underlying compiler/preproc/linker settable with Autoconf).

I don't really care about the driver, but I don't understand how
completely replacing a perfectly-well-working part makes anything more
maintainable.  It certainly is no problem to make the underlying tools
configurable for the existing C++ driver, is it?

Kind regards,
Michael
-- 
ho...@sa..., ho...@in...
http://home.pages.de/~hohmuth/

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Thu, 13 Mar 2003, Michael Hohmuth wrote:

> Grzegorz Jakacki <ja...@he...> writes:
>
> > Conceptually OpenC++ consists of three sybsystems:
> >
> >     Frontend --- reads preprocessed sources, makes parse tree
> >         out of them; possibly OpenC++ static analyzer also belongs
> >         in this subsystem
> >
> >     Translator --- traverses syntax tree and transforms it
> >         according to rules described by metaclasses
> >
> >     Driver --- runs preprocessor, Frontend, Translator,
> >         native compiler and linker.
> >
> > As I understand Michael and Stefan work with Frontend and are not interested
> > in the rest of the system. I believe, that this is the most frequently
> > reused part of OpenC++ and I know of at least three more projects that
> > reuse just this part.
>
> (Just as an aside: Would you mind posting a list of the projects you
> know that use OpenC++ to this list or the project's homepage?  I would
> like to have a look at them, and I really learned a lot from looking
> at the Synopsis source code, for example.)

* Introspector
* Synopsis
* the commercial project I used to work on (non-open)
* I was told that 'Sds' projects also uses OpenC++, but I never contacted
  them
* Juan Carlos Arevalo-Baeza (should be on this list) once used
  OpenC++ in his project
* Walter Cazzola (should be on this list) once used
  OpenC++ in his project

> > Thus I suggest that we take it out and make it a separate library.
>
> There is a fourth part that lives somewhere in the middle between
> Frontend and Translator (maybe that's what you mean by ``static
> analyzer''): OpenC++'s symbol-table implementation, encapsulated in
> classes Environment, Encoding and friends.  We had trouble
> understanding and reusing particularly this part, and in the end
> decided not to use it, so it should be separable as well.

Right.

> > Otherwise I can see serious problems with testing.
> [...]
> > With separate Frontend library, we can test the library on its own, and
> > this is what we need at the moment, because we somehow have to add tests
> > for the new features implemented by patches from Fiasco project.
>
> I think what you are proposing can be achieved by some
> build-infrastructure (and maybe some terminology and documentation)
> changes, plus very little.  You actually do not need complete
> separation of the four parts comprising OpenC++ in order to facilitate
> testing -- separation would just be convenient for reuse.
>
> In my opinion, what's needed is just:
>
> 1. An abstract base class for Walker, which just provides the
>    interface for visiting a Ptree hierarchy and is completely
>    independent from the Translator and symbol-table parts.  (As an
>    added bonus, its visit functions should support return types other
>    than Ptree*, so it probably should be a template.)

But perhaps not called 'Walker', unfortunately, due to code base.

> 2. A simple unit-testing framework, to which tests for new
>    functionality (on a per-module level) can be added easily
>
> For our VFiasco project <http://www.vfiasco.org>, we use a wrapper
> around Walker which provides (1), and (2) consists of a set of
> unit-test programs that output test data and a simple Makefile (for
> GNU Make) that tests for regressions.

Could you part of your infrastructure to OpenC++? If so, can you put it
somewhere on the web so that we can review it and see what can OpenC++ get
from you?

> All in all, I think that the changes I proposed should be gradually
> added to the source base, but nothing more.  Frequent changes to the
> build system and library interface actually make reuse harder, not
> easier, and besides the points I mentioned, there isn't really that
> much of an itch to scratch.

I think that separation of Parser as a module should be done sooner or
later. In many applications you do not need the rest, why should it be
bundled? Moreover, I do not think that this separation *requires* much
of interface change (BTW: OpenC++ interface is a subjest for another
story).

> > BTW: Over a year ago there was a resolution on this forum to factor out the
> > Driver subsystem and make it a shell script (for easier maintenance
> > and to make underlying compiler/preproc/linker settable with Autoconf).
>
> I don't really care about the driver, but I don't understand how
> completely replacing a perfectly-well-working part makes anything more
> maintainable.

It is not working perfectly, see e.g. my problems with running regression
on gcc-3.2, originating from the fact that underlying compiler is
hardcoded and selected with system-specific defines. We hit this problem
a year ago, when Pavel Krusina was working on Irix port.

Another issue are shared libraries, which are currently not trurly
supported. libtool has all we need to make shared or dlopenable library,
while at the same time driver.cc/driver2.cc duplicates this functionality,
and is far less perfect than libtool (the fact is it was created much
earlier than libtool).

> It certainly is no problem to make the underlying tools
> configurable for the existing C++ driver, is it?

It is possible, but it is more work than it is worth (especially for
shared libs).

Best regards
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Walter C. <ca...@di...>]

On Fri, 14 Mar 2003, Grzegorz Jakacki wrote:

> > (Just as an aside: Would you mind posting a list of the projects you
> > know that use OpenC++ to this list or the project's homepage?  I woul=
d
> > like to have a look at them, and I really learned a lot from looking
> > at the Synopsis source code, for example.)

[CUT]

> * Walter Cazzola (should be on this list) once used
>   OpenC++ in his project

Yep, but I have suspended my experiments (almost a year ago) because the
OpenC++'s (reflective) template support was not as I need and I had not
the time to work on its extension. Now (i.e., when the template supports
has been improved) the project is still suspended because new projects
are started and I don't have the time to complete my old work.

Walter

--=20
Walter Cazzola, PhD - Assistant Professor, DICo, University of Milano
E-mail ca...@di... Ph.: +39 010 353 6637  Fax: +39 010 353 6699
=B7 =B7 =B7 --------------------------- =B7 =B7 =B7 ---------------------=
------ =B7 =B7 =B7
               ... recursive: adjective, see recursive ...
=B7 =B7 =B7 --------------------------- =B7 =B7 =B7 ---------------------=
------ =B7 =B7 =B7

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

--- Grzegorz Jakacki <ja...@he...> wrote:
> On Thu, 13 Mar 2003, Michael Hohmuth wrote:
> > (Just as an aside: Would you mind posting a list of the projects
> you
> > know that use OpenC++ to this list or the project's homepage?  I
> would
> > like to have a look at them, and I really learned a lot from
> looking
> > at the Synopsis source code, for example.)
> 
> * Introspector

I have not gotten up to using OpenC++ just yet. However I do see
OpenC++ as an inspiriation. Hopefully I can help, and am ashamed that I
have only been making empty promises. 

Currently I am working on www.dotgnu.org and www.Swig.org, when that
work is completed, I will be able to propose some interfaces into
openc++ that will preserve the openc++, but also tie it into the global
metadata exchange that I envision.

mike

=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
http://webhosting.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Sun, 16 Mar 2003, James Michael DuPont wrote:

>
> --- Grzegorz Jakacki <ja...@he...> wrote:
> > On Thu, 13 Mar 2003, Michael Hohmuth wrote:
> > > (Just as an aside: Would you mind posting a list of the projects
> > you
> > > know that use OpenC++ to this list or the project's homepage?  I
> > would
> > > like to have a look at them, and I really learned a lot from
> > looking
> > > at the Synopsis source code, for example.)
> >
> > * Introspector
>
> I have not gotten up to using OpenC++ just yet. However I do see
> OpenC++ as an inspiriation.

Sorry, I was under impression that you are using OpenC++ as code base.

Best regards
Grzegorz

> Hopefully I can help, and am ashamed that I
> have only been making empty promises.
>
> Currently I am working on www.dotgnu.org and www.Swig.org, when that
> work is completed, I will be able to propose some interfaces into
> openc++ that will preserve the openc++, but also tie it into the global
> metadata exchange that I envision.
>
> mike
>
> =====
> James Michael DuPont
> http://introspector.sourceforge.net/
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Web Hosting - establish your business online
> http://webhosting.yahoo.com
>
>

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Stefan R. <sr...@ma...>]

Hello,

On Thu, Mar 13, 2003 at 01:52:52PM +0800, Grzegorz Jakacki wrote:
> This is very important, as this allows somebody who is not a guru to come
> and commit some fixes without fear that he/she breaks some distant parts
> of the package. Our testsuite is pathetically small at the moment, but
> it already helps and I believe that it can grow.

I think, there are two problems, at least with the parser. The
interface consists of two parts, namely the Ptree descendants
and their contents.

1) Every Ptree descendant has a method "Translate" which calls
   a method of Walker. Adding a new Ptree class means adding a
   method to Walker. User code must override that method, or
   they'll (accidentially, maybe) run into the default behaviour
   which just recurses into the subtrees. This problem can be
   solved with an abstract base class, so people who want to be
   paranoid can derive from that. Their code will automatically
   fail to compile when we add a new node type, which is a Good
   Thing(tm), IMHO.

   So far, everything nice.

2) The bigger problem is with the content of the nodes. For
   example, the new wide-char constants turn into Leafs. Old
   programs might assume that all Leafs are either numeric, char
   or string literals, and break now. Or, my "if(decl)" patch[1]
   would add the new property that the third child of "if" can
   be a declaration instead of an expression. I don't see a good
   solution for that problem. Okay, one could go away from the
   Lispy data structure towards a more "explicit" syntax, i.e.
     class PtreeIfStatement {
       Ptree *if_keyword, *left_paren, *right_paren;
       PtreeExpressionOrDeclaration *condition;
       PtreeStatement *controlled_statement;
     };
   but that would need an enormous amount of work and break
   everything. Plus, OpenC++ started as a meta-object protocol
   and not as a parser library, so I'm perfectly happy when it
   fulfills its primary purpose best :-)

[1] Since the CVS code currently can't be built, I'll wait with
checking it in.

> But now comes the problem: when we just extend Frontend itself, without
> proper support in Translator (e.g. your patches will make 'if ({...}) '
> parse, but Translator will choke on it), we do not have way to add tests
> that make sure the new Frontend functionality works.

My 1) is a partial solution. 2) can be solved by very defensive
programming. My code checks every Ptree node it encounters
whether it fits into its imagination. For example, my "if"
handling code starts with (the equivalent of)
   assert(p->First()->Eq("if"));
   assert(p->Second()->Eq('('));
   assert(p->Nth(3)->Eq(')'));
so if someone now adds an "unless" statement which also
generates a PtreeIfStatement ("it's almost the same"), my code
will see that it can't handle it. I think we can't get much
better.

> BTW: Over a year ago there was a resolution on this forum to factor out the
> Driver subsystem and make it a shell script (for easier maintenance
> and to make underlying compiler/preproc/linker settable with Autoconf).

Personally, I don't care whether the driver is in C, Perl, Shell
or Intercal.

What about generating a ".h" file and including that in the
driver? In another project of mine, I have this in "configure.in":
----8<----
define(CREATE1, [AC_OUTPUT_COMMANDS([echo "creating $1..."
cat >conftest.tmp <<_EOF
$2
_EOF
if cmp -s $1 conftest.tmp; then
  echo $1 is unchanged
  rm -f conftest.tmp
else
  rm -f $1
  mv conftest.tmp $1
fi])])dnl

CREATE1(arch/platform.h, [#ifndef PCC_V2_ARCH_PLATFORM_H
#define PCC_V2_ARCH_PLATFORM_H
#define PCC2_SRCDIR "`cd "$srcdir" && pwd`"
#define PCC2_DATADIR "$pdatadir"
#include "arch/unix/platform.h"
#endif
])
----8<----
This creates a file ("arch/platform.h") which contains the
specified code, with shell variables expanded (Disclaimer: I'm
not an Autoconf expert, maybe what I did can be done much
easier).

In addition, setting these variables with options to the driver
is a nice thing to have. Shouldn't be too hard. I'm currently
fiddling around with symlinks in $HOME/bin. Not too great :-P
On the other hand, there's always the option to just translate
the source (-E) and compile manually.


  Stefan

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Fri, 14 Mar 2003, Stefan Reuther wrote:

> Hello,
>
> On Thu, Mar 13, 2003 at 01:52:52PM +0800, Grzegorz Jakacki wrote:
> > This is very important, as this allows somebody who is not a guru to come
> > and commit some fixes without fear that he/she breaks some distant parts
> > of the package. Our testsuite is pathetically small at the moment, but
> > it already helps and I believe that it can grow.
>
> I think, there are two problems, at least with the parser. The
> interface consists of two parts, namely the Ptree descendants
> and their contents.
>
> 1) Every Ptree descendant has a method "Translate" which calls
>    a method of Walker. Adding a new Ptree class means adding a
>    method to Walker.

This is the first sign of too little isolation between visitor (Walker)
and visited class hierarchy (Ptree and derived classes) --- you cannot
extend Ptree hierarchy without modifying Walker code.

>    User code must override that method, or
>    they'll (accidentially, maybe) run into the default behaviour
>    which just recurses into the subtrees.

I am not sure if I understand you here. When the user adds the method of
Walke, he/she is free to define the default behavious as he/she wants,
right? Or maybe you mean, that *some* default behavious has to be
defined, since the member function cannot be just left abstract, because
otherwise user would make Walker class abstract?

>    This problem can be
>    solved with an abstract base class, so people who want to be
>    paranoid can derive from that.

Right, but then they have to define traversals in all 'Translate()'
overloads by themselves.

>    Their code will automatically
>    fail to compile when we add a new node type, which is a Good
>    Thing(tm), IMHO.

It depends. When I am adding new node to store '__attribute__' GNU
extension, I do not want old code to fail. I would prefere a design,
in which I can do it so that new code can see '__attribute__' node,
while old code does not see it.

>
>    So far, everything nice.
>
> 2) The bigger problem is with the content of the nodes. For
>    example, the new wide-char constants turn into Leafs. Old
>    programs might assume that all Leafs are either numeric, char
>    or string literals, and break now. Or, my "if(decl)" patch[1]
>    would add the new property that the third child of "if" can
>    be a declaration instead of an expression. I don't see a good
>    solution for that problem. Okay, one could go away from the
>    Lispy data structure towards a more "explicit" syntax, i.e.
>      class PtreeIfStatement {
>        Ptree *if_keyword, *left_paren, *right_paren;
>        PtreeExpressionOrDeclaration *condition;
>        PtreeStatement *controlled_statement;
>      };

Precisely. In fact OpenC++ does not use the "Lispy data structure"
(I really like this term!). The abstract datatypes that you envision
are already present --- look at the names of Metaclass methods ---
they are candidates for the new syntax hierarchy. Such hierarchy
would be much safer to play with: instead of
"ifthenelse->Car()->Car()->Cdr()", which potentially can coredump
at each arrow, one would conveniently write "ifthenelse->GetThen()".
Also if sometime in the future the representation of "PtreeIfStatement"
node changes, then client using "ifthenelse->GetThen()" would feel good,
while the client using "ifthenelse->Car()->Car()->Cdr()" would have
to fix his/her Car()s and Cdr()s.

>    but that would need an enormous amount of work and break
>    everything.

If you just throw away Ptree's and introduce new hierarchy, then yes.

But there is much better way: one can wrap raw tree implementation (like
Ptree) into better-structured one using wrapper types (which are
implementation of Proxy pattern). The technique I am trying to come up with
for it is mainly based on so-called traits (classes similar to e.g.
numeric_limits<> from Standard Library) and type-safe visitation (similar to
technique used in boost::visitor).

Introduction of wrappers does not break any existing code, but makes the new
code much better isolated from the actual tree representation, so changes
in the tree representation (even adding new nodes) does not break the new
code.

>    Plus, OpenC++ started as a meta-object protocol
>    and not as a parser library, so I'm perfectly happy when it
>    fulfills its primary purpose best :-)

As C++ changes the meta-object protocol has to change also. Apart of that,
OpenC++ is the most reasonable non-GPL'd C++ frontend approximation
I know of.

> [1] Since the CVS code currently can't be built, I'll wait with
> checking it in.

Ok.

> > But now comes the problem: when we just extend Frontend itself, without
> > proper support in Translator (e.g. your patches will make 'if ({...}) '
> > parse, but Translator will choke on it), we do not have way to add tests
> > that make sure the new Frontend functionality works.
>
> My 1) is a partial solution.

But this requires adding *some* support of new nodes to the translator
anyway, right?

> 2) can be solved by very defensive
> programming.

Which is a maintenance nightmare and I would like to avoid it.

> My code checks every Ptree node it encounters
> whether it fits into its imagination. For example, my "if"
> handling code starts with (the equivalent of)
>    assert(p->First()->Eq("if"));
>    assert(p->Second()->Eq('('));
>    assert(p->Nth(3)->Eq(')'));
> so if someone now adds an "unless" statement which also
> generates a PtreeIfStatement ("it's almost the same"), my code
> will see that it can't handle it. I think we can't get much
> better.

Not with the design in place.

> > BTW: Over a year ago there was a resolution on this forum to factor out the
> > Driver subsystem and make it a shell script (for easier maintenance
> > and to make underlying compiler/preproc/linker settable with Autoconf).
>
> Personally, I don't care whether the driver is in C, Perl, Shell
> or Intercal.

I do, because I am responding to bug reports :-)
(Seriously: shell is the only solution that does not have to duplicate
libtool functionality, e.g. to know how to install shared library on the
given platform).

> What about generating a ".h" file and including that in the
> driver? In another project of mine, I have this in "configure.in":
> ----8<----
> define(CREATE1, [AC_OUTPUT_COMMANDS([echo "creating $1..."
> cat >conftest.tmp <<_EOF
> $2
> _EOF
> if cmp -s $1 conftest.tmp; then
>   echo $1 is unchanged
>   rm -f conftest.tmp
> else
>   rm -f $1
>   mv conftest.tmp $1
> fi])])dnl
>
> CREATE1(arch/platform.h, [#ifndef PCC_V2_ARCH_PLATFORM_H
> #define PCC_V2_ARCH_PLATFORM_H
> #define PCC2_SRCDIR "`cd "$srcdir" && pwd`"
> #define PCC2_DATADIR "$pdatadir"
> #include "arch/unix/platform.h"
> #endif
> ])
> ----8<----
> This creates a file ("arch/platform.h") which contains the
> specified code, with shell variables expanded (Disclaimer: I'm
> not an Autoconf expert, maybe what I did can be done much
> easier).

AC_DEFINE is the solution you are looking for.

What you suggest is reasonable for e.g. setting compiler or setting the
proper shared lib extension (.so, .dynlib etc.), but apart of those there is
*functionality* which is necessary in driver. How will you pass the
knowledge about the correct way of installing shared library? On some
platforms you just do 'cp'. On some you also have to set up dynamic linker
somehow. On yet other platforms you have to run special program on the new
shared library to activate it. On some you have to make appropriate
symlinks. I do not want to maintain code that takes care of all this,
especially that this code has already been written and is called libtool.

> In addition, setting these variables with options to the driver
> is a nice thing to have. Shouldn't be too hard. I'm currently
> fiddling around with symlinks in $HOME/bin.

I do not quite understand. What do you use symlinks for?

> Not too great :-P
> On the other hand, there's always the option to just translate
> the source (-E) and compile manually.

And that was basically what I would like occ executable to do. No more, no
less.

Best regards
Grzegorz

PS: I promised to prepare a write-up on the solution I have in mind, but
although I was trying hard thorughout the weekend, I still do not have it in
satisfactory form. Hopefully I will be able to mail it tomorrow. G.


##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Stefan R. <sr...@ma...>]

Hello,

On Mon, Mar 17, 2003 at 04:43:53PM +0800, Grzegorz Jakacki wrote:
> On Fri, 14 Mar 2003, Stefan Reuther wrote:
> > I think, there are two problems, at least with the parser. The
> > interface consists of two parts, namely the Ptree descendants
> > and their contents.
> >
> > 1) Every Ptree descendant has a method "Translate" which calls
> >    a method of Walker. Adding a new Ptree class means adding a
> >    method to Walker.
> 
> This is the first sign of too little isolation between visitor (Walker)
> and visited class hierarchy (Ptree and derived classes) --- you cannot
> extend Ptree hierarchy without modifying Walker code.

I don't think it can be done much different. Ptree and Walker
have to "know each other". What needs to be improved is that it
can be detected at compile-time that a Walker descendant does no
longer match our tree.

> >    User code must override that method, or
> >    they'll (accidentially, maybe) run into the default behaviour
> >    which just recurses into the subtrees.
> 
> I am not sure if I understand you here. When the user adds the method of
> Walke, he/she is free to define the default behavious as he/she wants,
> right? Or maybe you mean, that *some* default behavious has to be
> defined, since the member function cannot be just left abstract, because
> otherwise user would make Walker class abstract?

Right now, the default behaviour for every Walker::TranslateXYZ
method is to recurse into the subtrees, and return the changed
tree. I'm trying to say that this behaviour is not optimal,
because it might break user code's expectations. Say, I write a
Walker where each TranslateXYZ method returns a particular node,
like this:
   Ptree* MyWalker::TranslateIfStatement(Ptree* p) {
       return new MyFunkyPtreeDescendant(p, 0);
   }
To be sure, I take walker.h and override all methods of Walker.
So I can now rightfully assume that this
  dynamic_cast<MyFunkyPtreeDescendant*>(walker.Translate(p)) != 0
holds.

If a new TranslateXYZ method is added to Walker, my code still
compiles but does no longer fulfill the above assertion. The
wired-in default behaviour doesn't match my use-case. I can't
say what to do with nodes I don't know about.

Solutions I propose:

- add a AbstractWalker class, where all methods are pure
  virtual. People concerned with the above problem (like me) can
  use that as a base class. Code using that class breaks when we
  add new productions, and forces people to update their code
  (or stay with the old version). Walker would be derived from
  that class.
- add a DefaultWalker class, where all TranslateXYZ methods
  call another function, like this:
     Ptree* DefaultWalker::TranslateIfStatement(Ptree* p) {
         return DefaultHandler(p);
     }
  This would let people choose their default behaviour. Having a
  Ptree::Clone would come in handy for this one.

Another nice thing to have would be a Walker returning other
things than Ptrees. I'm still unsure how that could work. We
have such a thing in the VFiasco project, but (Michael, please
look away) it's not perfect. We also have that which I called
"DefaultWalker" here.

> >    This problem can be
> >    solved with an abstract base class, so people who want to be
> >    paranoid can derive from that.
> 
> Right, but then they have to define traversals in all 'Translate()'
> overloads by themselves.

A good thing, in my opinion. When I want to generate code, I
want to be sure not to miss any node. For example, the recent
addition of the "typeid" operator caused our code to silently
translate
   typeinfo ti = typeid(p);
as
   typeinfo ti = p;

> >    Their code will automatically
> >    fail to compile when we add a new node type, which is a Good
> >    Thing(tm), IMHO.
> 
> It depends. When I am adding new node to store '__attribute__' GNU
> extension, I do not want old code to fail. I would prefere a design,
> in which I can do it so that new code can see '__attribute__' node,
> while old code does not see it.

That's what I meant with "node contents" below.

> > 2) The bigger problem is with the content of the nodes. For
> >    example, the new wide-char constants turn into Leafs. Old
> >    programs might assume that all Leafs are either numeric, char
> >    or string literals, and break now. Or, my "if(decl)" patch[1]
> >    would add the new property that the third child of "if" can
> >    be a declaration instead of an expression. I don't see a good
> >    solution for that problem. Okay, one could go away from the
> >    Lispy data structure towards a more "explicit" syntax, i.e.
> >      class PtreeIfStatement {
> >        Ptree *if_keyword, *left_paren, *right_paren;
> >        PtreeExpressionOrDeclaration *condition;
> >        PtreeStatement *controlled_statement;
> >      };
> 
> Precisely. In fact OpenC++ does not use the "Lispy data structure"
> (I really like this term!). The abstract datatypes that you envision
> are already present --- look at the names of Metaclass methods ---
> they are candidates for the new syntax hierarchy. Such hierarchy
> would be much safer to play with: instead of
> "ifthenelse->Car()->Car()->Cdr()", which potentially can coredump
> at each arrow, one would conveniently write "ifthenelse->GetThen()".

Sir, where do I sign? :-)

Still, in the *current* code, TranslateIfStatement takes a
Ptree* and not a PtreeIfStatement*, although the latter would be
a safe assumption for most cases.

> Also if sometime in the future the representation of "PtreeIfStatement"
> node changes, then client using "ifthenelse->GetThen()" would feel good,
> while the client using "ifthenelse->Car()->Car()->Cdr()" would have
> to fix his/her Car()s and Cdr()s.

Far future, because nowadays everyone uses Car() and Cdr().

One problem we'd still have to solve is that people can still
construct invalid trees. For example, if I have a Ptree
representing "if (cond) statement;", someone could Snoc a "&" at
the end, although that'd yield invalid syntax, and nothing could
prevent it (although adding an "else" and another statement
still must be valid). That's what I meant with "Lispy" data
structure: in Lisp, you have structured lists, too, but no-one
can prevent you from mangling them somehow.

> >    but that would need an enormous amount of work and break
> >    everything.
> 
> If you just throw away Ptree's and introduce new hierarchy, then yes.
> 
> But there is much better way: one can wrap raw tree implementation (like
> Ptree) into better-structured one using wrapper types (which are
> implementation of Proxy pattern). The technique I am trying to come up with
> for it is mainly based on so-called traits (classes similar to e.g.
> numeric_limits<> from Standard Library) and type-safe visitation (similar to
> technique used in boost::visitor).
> 
> Introduction of wrappers does not break any existing code, but makes the new
> code much better isolated from the actual tree representation, so changes
> in the tree representation (even adding new nodes) does not break the new
> code.

I think adding those accessor functions ("GetCondition",
"GetThen", "GetIf") would be a good start, although it doesn't
work nicely for everything: for example, PtreeDeleteExpr can
have the following forms:
         [:: delete expr]
         [:: delete \[ \] expr]
         [delete expr]
         [delete \[ \] expr]
I'm unsure whether we should "regularize" that, for example, as
         [::  delete nil     expr]
         [::  delete [\[ \]] expr]
         [nil delete nil     expr]
         [nil delete [\[ \]] expr]
Still, this would break a lot of stuff. There are quite a number
of places like this in the parse tree. I could (try to) make up
a complete list. In my original posting, I mentioned only
"un-regular" places in Declarators.

Another nice thing to have would be a "better" class hierarchy.
Improvements I see so far are:
- all PtreeFooExpr should share a common base class,
  PtreeExpression
- all PtreeFooStatement should share a common base class,
  PtreeStatement. Hmmm, this seems to collide with PtreeBrace
  and PtreeDeclaration, so maybe leave it out.
- PtreeDeclaration and PtreeExpression should have a common base
  class (needed for my if(condition) thing)
Anything else? This should not break anything.

> > [1] Since the CVS code currently can't be built, I'll wait with
> > checking it in.
> 
> Ok.

It's in now.

Change in the parse tree: the third child of if/while/switch may
now be a PtreeDeclaration instead of an expression :-)

By the way, I get a strange message I can't interpret:
# Mailing ope...@li......
# Generating notification message...
# Generating notification message... done.
# Mailing ope...@li......
# Generating notification message...
# Traceback (innermost last):
#   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 322, in ?
#     main()
#   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 315, in main
#     blast_mail(subject, people, specs[1:], contextlines, fromhost)
#   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 240, in blast_mail
#     print calculate_diff(file, contextlines)
#   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 139, in calculate_diff
#     file, oldrev, newrev = string.split(filespec, ',')
# ValueError: unpack list of wrong size

Doesn't look like any interpreter I know :-)

> > > But now comes the problem: when we just extend Frontend itself, without
> > > proper support in Translator (e.g. your patches will make 'if ({...}) '
> > > parse, but Translator will choke on it), we do not have way to add tests
> > > that make sure the new Frontend functionality works.
> >
> > My 1) is a partial solution.
> 
> But this requires adding *some* support of new nodes to the translator
> anyway, right?

Yes, but I can't seem to find a way around that, because there
is no universal fallback. Some want just to recurse into the new
nodes, some do not want to miss them.

> > > BTW: Over a year ago there was a resolution on this forum to factor out the
> > > Driver subsystem and make it a shell script (for easier maintenance
> > > and to make underlying compiler/preproc/linker settable with Autoconf).
> >
> > Personally, I don't care whether the driver is in C, Perl, Shell
> > or Intercal.
> 
> I do, because I am responding to bug reports :-)
> (Seriously: shell is the only solution that does not have to duplicate
> libtool functionality, e.g. to know how to install shared library on the
> given platform).

Okay, I overlooked that shared library thing.

> > What about generating a ".h" file and including that in the
> > driver? In another project of mine, I have this in "configure.in":
[...]
> > This creates a file ("arch/platform.h") which contains the
> > specified code, with shell variables expanded (Disclaimer: I'm
> > not an Autoconf expert, maybe what I did can be done much
> > easier).
> 
> AC_DEFINE is the solution you are looking for.

Whoops? I wonder how I could have missed that :-P

> What you suggest is reasonable for e.g. setting compiler or setting the
> proper shared lib extension (.so, .dynlib etc.), but apart of those there is
> *functionality* which is necessary in driver. How will you pass the
> knowledge about the correct way of installing shared library? On some
> platforms you just do 'cp'. On some you also have to set up dynamic linker
> somehow. [snip]

I was mainly thinking about configuring compiler names etc. into
occ. I have "g++-2.95" and "g++-3.2", and to select one I
install a symlink in $HOME/bin. I'd like to be able to choose
that when I compile or - even better - run occ. For shared
libraries, you're absolutely right.


  Stefan

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Tue, 18 Mar 2003, Stefan Reuther wrote:

> Hello,
>
> On Mon, Mar 17, 2003 at 04:43:53PM +0800, Grzegorz Jakacki wrote:
> > On Fri, 14 Mar 2003, Stefan Reuther wrote:
> > > I think, there are two problems, at least with the parser. The
> > > interface consists of two parts, namely the Ptree descendants
> > > and their contents.
> > >
> > > 1) Every Ptree descendant has a method "Translate" which calls
> > >    a method of Walker. Adding a new Ptree class means adding a
> > >    method to Walker.
> >
> > This is the first sign of too little isolation between visitor (Walker)
> > and visited class hierarchy (Ptree and derived classes) --- you cannot
> > extend Ptree hierarchy without modifying Walker code.
>
> I don't think it can be done much different. Ptree and Walker
> have to "know each other".

I believe that it is possible to introduce a layer in between,
which would encapsualte the actual implementation of Ptree.

> What needs to be improved is that it
> can be detected at compile-time that a Walker descendant does no
> longer match our tree.

Yes, but "our tree" does not necessarily have to mean phisical
implementation.

> > >    User code must override that method, or
> > >    they'll (accidentially, maybe) run into the default behaviour
> > >    which just recurses into the subtrees.
> >
> > I am not sure if I understand you here. When the user adds the method of
> > Walke, he/she is free to define the default behavious as he/she wants,
> > right? Or maybe you mean, that *some* default behavious has to be
> > defined, since the member function cannot be just left abstract, because
> > otherwise user would make Walker class abstract?
>
> Right now, the default behaviour for every Walker::TranslateXYZ
> method is to recurse into the subtrees, and return the changed
> tree. I'm trying to say that this behaviour is not optimal,
> because it might break user code's expectations. Say, I write a
> Walker where each TranslateXYZ method returns a particular node,
> like this:
>    Ptree* MyWalker::TranslateIfStatement(Ptree* p) {
>        return new MyFunkyPtreeDescendant(p, 0);
>    }
> To be sure, I take walker.h and override all methods of Walker.
> So I can now rightfully assume that this
>   dynamic_cast<MyFunkyPtreeDescendant*>(walker.Translate(p)) != 0
> holds.
>
> If a new TranslateXYZ method is added to Walker, my code still
> compiles but does no longer fulfill the above assertion. The
> wired-in default behaviour doesn't match my use-case. I can't
> say what to do with nodes I don't know about.
>
> Solutions I propose:
>
> - add a AbstractWalker class, where all methods are pure
>   virtual. People concerned with the above problem (like me) can
>   use that as a base class. Code using that class breaks when we
>   add new productions, and forces people to update their code
>   (or stay with the old version). Walker would be derived from
>   that class.
> - add a DefaultWalker class, where all TranslateXYZ methods
>   call another function, like this:
>      Ptree* DefaultWalker::TranslateIfStatement(Ptree* p) {
>          return DefaultHandler(p);
>      }
>   This would let people choose their default behaviour.

Ok, I do not see any issues.

> Having a
>   Ptree::Clone would come in handy for this one.

This should be easy to add too.

>
> Another nice thing to have would be a Walker returning other
> things than Ptrees. I'm still unsure how that could work.

In fact Walker is implementation of pattern called Visitor.
I am not sure if I have already mentioner "Design Patterns"
by Gamma, Johnson, Ralph & Vlissides. If you have it around that may
be a good moment to have a look at chapters "Visitor" and "Composite"
(chapters are quite compact and not too long). Personally I found it very
enlightening.

Walker (or more generally visitor) can return arbitrary thing. I assume
that what you have in mind is that it returns nodes of another tree
(or rather handles/pointers to such nodes), thus implementing some
homomoprhic tree transformation. It that it?

> We
> have such a thing in the VFiasco project, but (Michael, please
> look away) it's not perfect.

Could I have a look?

> We also have that which I called
> "DefaultWalker" here.
>
> > >    This problem can be
> > >    solved with an abstract base class, so people who want to be
> > >    paranoid can derive from that.
> >
> > Right, but then they have to define traversals in all 'Translate()'
> > overloads by themselves.
>
> A good thing, in my opinion. When I want to generate code, I
> want to be sure not to miss any node.

But I am still thinking about ways to "derive" tree traversals from
simpler ones.

> For example, the recent
> addition of the "typeid" operator caused our code to silently
> translate
>    typeinfo ti = typeid(p);
> as
>    typeinfo ti = p;

Why was that (I could go figure out myself, but probably you still
remember)?

> > >    Their code will automatically
> > >    fail to compile when we add a new node type, which is a Good
> > >    Thing(tm), IMHO.
> >
> > It depends. When I am adding new node to store '__attribute__' GNU
> > extension, I do not want old code to fail. I would prefere a design,
> > in which I can do it so that new code can see '__attribute__' node,
> > while old code does not see it.
>
> That's what I meant with "node contents" below.
>
> > > 2) The bigger problem is with the content of the nodes. For
> > >    example, the new wide-char constants turn into Leafs. Old
> > >    programs might assume that all Leafs are either numeric, char
> > >    or string literals, and break now. Or, my "if(decl)" patch[1]
> > >    would add the new property that the third child of "if" can
> > >    be a declaration instead of an expression. I don't see a good
> > >    solution for that problem. Okay, one could go away from the
> > >    Lispy data structure towards a more "explicit" syntax, i.e.
> > >      class PtreeIfStatement {
> > >        Ptree *if_keyword, *left_paren, *right_paren;
> > >        PtreeExpressionOrDeclaration *condition;
> > >        PtreeStatement *controlled_statement;
> > >      };
> >
> > Precisely. In fact OpenC++ does not use the "Lispy data structure"
> > (I really like this term!). The abstract datatypes that you envision
> > are already present --- look at the names of Metaclass methods ---
> > they are candidates for the new syntax hierarchy. Such hierarchy
> > would be much safer to play with: instead of
> > "ifthenelse->Car()->Car()->Cdr()", which potentially can coredump
> > at each arrow, one would conveniently write "ifthenelse->GetThen()".
>
> Sir, where do I sign? :-)

You mean NDA ? No, I do not have the code :-)

Those abstract data types are still, well, abstract. However
they exist in the desing, they correspond to 'TranslateXYZ()' methods.
However, we can move towards them, at the end is my write-up on how I
think it can be done.

> Still, in the *current* code, TranslateIfStatement takes a
> Ptree* and not a PtreeIfStatement*, although the latter would be
> a safe assumption for most cases.

There is a little bit of confusion, since lispy data structure is mixed up
with 'PtreeIfStatement'-like classes, which on the other hand do not
provide methods like "GetThen()". Why cannot we change
TranslateIfStatement so that it takes "Ptree*" argument?

> > Also if sometime in the future the representation of "PtreeIfStatement"
> > node changes, then client using "ifthenelse->GetThen()" would feel good,
> > while the client using "ifthenelse->Car()->Car()->Cdr()" would have
> > to fix his/her Car()s and Cdr()s.
>
> Far future, because nowadays everyone uses Car() and Cdr().

How can anyone use any other access method, while this is the only one
available?

I think that we touched a general problem of interface changes in the
presence of legacy code.

In my opinion the evolution is inevitable, revolution is not.

If for several versions you provide clients with both old and new
interface, then at some moment you may start deprecating the old one
(i.e. hesitate no more to change implementation of the syntax tree
if the change would not be visible through the new interface).

The time, when two interfaces are available simultaneously enables clients
to *gradually* move from the old to the new interface, most likely running
the code with both new and old interface binding for quite some time.

> One problem we'd still have to solve is that people can still
> construct invalid trees. For example, if I have a Ptree
> representing "if (cond) statement;", someone could Snoc a "&" at
> the end, although that'd yield invalid syntax, and nothing could
> prevent it (although adding an "else" and another statement
> still must be valid). That's what I meant with "Lispy" data
> structure: in Lisp, you have structured lists, too, but no-one
> can prevent you from mangling them somehow.

This cannot be avoided if there is lispy interface to create the tree
nodes available to the client. I think we should just assert or throw once
such a situation (invalid tree) is identified.

> > >    but that would need an enormous amount of work and break
> > >    everything.
> >
> > If you just throw away Ptree's and introduce new hierarchy, then yes.
> >
> > But there is much better way: one can wrap raw tree implementation (like
> > Ptree) into better-structured one using wrapper types (which are
> > implementation of Proxy pattern). The technique I am trying to come up with
> > for it is mainly based on so-called traits (classes similar to e.g.
> > numeric_limits<> from Standard Library) and type-safe visitation (similar to
> > technique used in boost::visitor).
> >
> > Introduction of wrappers does not break any existing code, but makes the new
> > code much better isolated from the actual tree representation, so changes
> > in the tree representation (even adding new nodes) does not break the new
> > code.
>
> I think adding those accessor functions ("GetCondition",
> "GetThen", "GetIf") would be a good start, although it doesn't
> work nicely for everything: for example, PtreeDeleteExpr can
> have the following forms:
>          [:: delete expr]
>          [:: delete \[ \] expr]
>          [delete expr]
>          [delete \[ \] expr]
> I'm unsure whether we should "regularize" that, for example, as
>          [::  delete nil     expr]
>          [::  delete [\[ \]] expr]
>          [nil delete nil     expr]
>          [nil delete [\[ \]] expr]
> Still, this would break a lot of stuff.

I would say that unfortunately for the time being (= unless Ptree* is
deprecated) the burden of figuring out how to handle the tree lies on the
accessor function. This is what client code does now anyways.

Accessor functions can be refactored in the future, if the tree
implementation is changed/normalized.

(BTW: I do not think that accessors should be free functions; see the
write-up).

> There are quite a number
> of places like this in the parse tree. I could (try to) make up
> a complete list.

If you want to invest your time in it, than it would syrely help in
developing the accessors.

> In my original posting, I mentioned only
> "un-regular" places in Declarators.
>
> Another nice thing to have would be a "better" class hierarchy.
> Improvements I see so far are:
> - all PtreeFooExpr should share a common base class,
>   PtreeExpression

What would this class contain? (Not to discourage you, I am just looking
for arguments).

> - all PtreeFooStatement should share a common base class,
>   PtreeStatement. Hmmm, this seems to collide with PtreeBrace
>   and PtreeDeclaration, so maybe leave it out.

Could you elaborate on where is the problem exactly?

> - PtreeDeclaration and PtreeExpression should have a common base
>   class (needed for my if(condition) thing)

Where would this help? In function signatures, I guess, but at the moment
I cannot see where exactly.

> Anything else?

The problem I can see is that fixes to syntax tree are always inevitable.
There is no universally good class hierarchy for C++ syntax, and there
will never be. The class hierarchy depends on the application. I am trying
to address this issue by introducing a "view" --- something in between the
parse tree and Walker.

> This should not break anything.
>
> > > [1] Since the CVS code currently can't be built, I'll wait with
> > > checking it in.
> >
> > Ok.
>
> It's in now.

Great. However could I ask you to add the testcases to verify this
functionality? There are instructions somewhere under ./testsuite, perhaps
those tests should go into ./testsuite/comp.

> Change in the parse tree: the third child of if/while/switch may
> now be a PtreeDeclaration instead of an expression :-)
>
> By the way, I get a strange message I can't interpret:
> # Mailing ope...@li......
> # Generating notification message...
> # Generating notification message... done.
> # Mailing ope...@li......
> # Generating notification message...
> # Traceback (innermost last):
> #   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 322, in ?
> #     main()
> #   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 315, in main
> #     blast_mail(subject, people, specs[1:], contextlines, fromhost)
> #   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 240, in blast_mail
> #     print calculate_diff(file, contextlines)
> #   File "/cvsroot/sitedocs/CVSROOT/cvstools/syncmail", line 139, in calculate_diff
> #     file, oldrev, newrev = string.split(filespec, ',')
> # ValueError: unpack list of wrong size
>
> Doesn't look like any interpreter I know :-)

Python?

This is Michael's fault, I believe he is looking into it, but most likely
you are better informed in this regard :-)

> > > > But now comes the problem: when we just extend Frontend itself, without
> > > > proper support in Translator (e.g. your patches will make 'if ({...}) '
> > > > parse, but Translator will choke on it), we do not have way to add tests
> > > > that make sure the new Frontend functionality works.
> > >
> > > My 1) is a partial solution.
> >
> > But this requires adding *some* support of new nodes to the translator
> > anyway, right?
>
> Yes, but I can't seem to find a way around that, because there
> is no universal fallback. Some want just to recurse into the new
> nodes, some do not want to miss them.

I am thinking about adding the code into the "view" layer (between tree
and walker), what would express the new node in terms of old nodes
(or at least abort).

>
> > > > BTW: Over a year ago there was a resolution on this forum to factor out the
> > > > Driver subsystem and make it a shell script (for easier maintenance
> > > > and to make underlying compiler/preproc/linker settable with Autoconf).
> > >
> > > Personally, I don't care whether the driver is in C, Perl, Shell
> > > or Intercal.
> >
> > I do, because I am responding to bug reports :-)
> > (Seriously: shell is the only solution that does not have to duplicate
> > libtool functionality, e.g. to know how to install shared library on the
> > given platform).
>
> Okay, I overlooked that shared library thing.
>
> > > What about generating a ".h" file and including that in the
> > > driver? In another project of mine, I have this in "configure.in":
> [...]
> > > This creates a file ("arch/platform.h") which contains the
> > > specified code, with shell variables expanded (Disclaimer: I'm
> > > not an Autoconf expert, maybe what I did can be done much
> > > easier).
> >
> > AC_DEFINE is the solution you are looking for.
>
> Whoops? I wonder how I could have missed that :-P
>
> > What you suggest is reasonable for e.g. setting compiler or setting the
> > proper shared lib extension (.so, .dynlib etc.), but apart of those there is
> > *functionality* which is necessary in driver. How will you pass the
> > knowledge about the correct way of installing shared library? On some
> > platforms you just do 'cp'. On some you also have to set up dynamic linker
> > somehow. [snip]
>
> I was mainly thinking about configuring compiler names etc. into
> occ. I have "g++-2.95" and "g++-3.2", and to select one I
> install a symlink in $HOME/bin. I'd like to be able to choose
> that when I compile or - even better - run occ.

I think this kind of stuff should go into client's Makefile. You know the
best what compiler and how you want to call. Possibly you also want to do
some tricks with auto-generated dependencies etc. I think it would be
cleaner, when occ is just 'occ -E'. We could provide "out-of-the-box
Makefile" or script that would do the job for lazy users, who just want to
compile straight with the default compiler.

Best regards
Grzegorz


----------------------------
Hi,

As promised, this is a write-up of thoughts I had wrt. improvind the
design of OpenC++ in evolutionary way.

As I already wrote, there is too much coupling between Walker and Ptree
hierarchy (this is usual problem with Visitor pattern).

I was thinking about providing more isolation by introducing a level of
indirection. In particular I would like Ptree* to be replaced by some
proxy type (a.k.a. "smart pointer", "handler" or "wrapper").

The key point is that if 'p' is a pointer, the expression like 'p.sth' does
not compile. Thus we can migrate from Ptree* to some proxy type, that
behaves like Ptree*. For example:

template <class T>
class PtreePtr
{
public:
    T& operator*()  { return *ptr_; }
    T* operator->() { return ptr_; }
private:
    T* ptr_;
};

We can migrate to this solution in small steps, with very small changes in
API (instead of Ptree* you get something that behaves almost like Ptree*).

The next step would be to get rid of 'new Ptree...' operators in the code.
It should be gradually replaced by something like 'create<Ptree...>()',
which is supposed to give back "something that behaves like Ptree",
wihtout exposing details on what it really is to the client (not even
its type, which should be available through, say, create<Ptree...>::type
or separate designated trait class).

With the above code "injected" into OpenC++ we can start
specializing PtreePtr for particular Ptree kinds,
adding methods for accessing respective subtrees, e.g.:

template <>
class PtreePtr<PtreeIfStmt>
{
    ...
    PtreePtr<Ptree> GetThen()
    {
        return PtreePtr<Ptree>(ptr_->Cdr()->Car()->Cdr());
    }
};

Now if you write code using PtreePtr and use those new accessors (like
'GetThen()'), your code is isolated from the changes in actual Ptree
implementation --- the changes in implementation can be compensated by
changes in PtreePtr specializations, so that the client does not notice
them.

As for visitors (aka. walkers): currently the "view" of the tree that
visitor "sees" is fixed by the tree implementation. However, not every
visitor wants to see the same --- e.g. sometimes you want to see comments
or whitespace as nodes, sometimes not. It is especially true when you have
deveopled several visitors and you add some new functionality to the
parser (e.g. "explicit" keyword), so you need to add syntax node, but you
would like to keep your old visitors running (and e.g. ignore the new
node). Another example are parenthesis --- if you have visitor that
simplifies arithmetic expressions, you most likely are not interested in
it seeing the nodes representing parenthesis.

This can be taken care of by the use of tag classes. One can define a set
of tag classes (which represent a "view" for visitor) and trait class that
maps node implementation type to a tag (in a given view), e.g.:

    class BinaryNode {};
    ...
    class BasicViev;

    template <class View, class Node> class tree_traits;

    template<> struct tree_traits<BasicView,PtreePtr<PtreeAssign> >
    {
        typedef BinaryNode type;
    }
    ...

    tree_traits<BasicView,Node>::tag    <--- what 'Node' means in 'BasicView'
                ^^^^^^^^^ ^^^^
             view          implementation type

To make the mechanism complete, one needs also a "dispatcher" or
"destructor" --- function or functor, that for given view, visitor and node
determines the node "nature" --- e.g. given Ptree* determines if it is
"if statment", "assignment" etc., casts the node representation to
appropriate type (or wraps it in appropriate proxy) and calls the visitor
on it.

Visitor makes use of the traits class to perform compile-time dispatching,
instead of overload resolution suggested in "Design Patterns". This is
basically analogon of a dreaded 'switch' statement, but it is not dreaded,
because

  (1) unlike ordinary switch it fails to compile, when
      you add new tags, that are not covered

  (2) you never want to add new tags, because instead of
      it you leave the view as it was, and add new view,
      that exposes the new node tag.

Now you can have multiple visitors, each of which sees the tree differently.
In particular, when you add a new node, you can make all old visitors work
as before, and add a new view exposing the new node for the new visitors,
which are going to exploit it. (Looks like I am repeating myself, but this
is cruicial).

That's more or less it. Sorry for being chaotic: "This letter is longer than
usual because I lack time to make it short."

I would very much appreciate any feedback.

Best regards
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

--- Grzegorz Jakacki <ja...@he...> wrote:
> On Tue, 18 Mar 2003, Stefan Reuther wrote:
> 
> > Hello,
> >
> > On Mon, Mar 17, 2003 at 04:43:53PM +0800, Grzegorz Jakacki wrote:
> > > On Fri, 14 Mar 2003, Stefan Reuther wrote:
> > > > I think, there are two problems, at least with the parser. The
> > > > interface consists of two parts, namely the Ptree descendants
> > > > and their contents.
> > > >
> > > > 1) Every Ptree descendant has a method "Translate" which calls
> > > >    a method of Walker. Adding a new Ptree class means adding a
> > > >    method to Walker.
> > >
> > > This is the first sign of too little isolation between visitor
> (Walker)
> > > and visited class hierarchy (Ptree and derived classes) --- you
> cannot
> > > extend Ptree hierarchy without modifying Walker code.
> >
> > I don't think it can be done much different. Ptree and Walker
> > have to "know each other".
> 
> I believe that it is possible to introduce a layer in between,
> which would encapsualte the actual implementation of Ptree.

Exactly my thoughts. We need a cleaner tree.
Have you looked at treecc from Rhys W?
http://www.southern-storm.com.au/treecc.html

=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
Yahoo! Platinum - Watch CBS' NCAA March Madness, live on your desktop!
http://platinum.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Thu, 20 Mar 2003, James Michael DuPont wrote:

>
> --- Grzegorz Jakacki <ja...@he...> wrote:
> > On Tue, 18 Mar 2003, Stefan Reuther wrote:
> >
> > > Hello,
> > >
> > > On Mon, Mar 17, 2003 at 04:43:53PM +0800, Grzegorz Jakacki wrote:
> > > > On Fri, 14 Mar 2003, Stefan Reuther wrote:
> > > > > I think, there are two problems, at least with the parser. The
> > > > > interface consists of two parts, namely the Ptree descendants
> > > > > and their contents.
> > > > >
> > > > > 1) Every Ptree descendant has a method "Translate" which calls
> > > > >    a method of Walker. Adding a new Ptree class means adding a
> > > > >    method to Walker.
> > > >
> > > > This is the first sign of too little isolation between visitor
> > (Walker)
> > > > and visited class hierarchy (Ptree and derived classes) --- you
> > cannot
> > > > extend Ptree hierarchy without modifying Walker code.
> > >
> > > I don't think it can be done much different. Ptree and Walker
> > > have to "know each other".
> >
> > I believe that it is possible to introduce a layer in between,
> > which would encapsualte the actual implementation of Ptree.
>
> Exactly my thoughts. We need a cleaner tree.
> Have you looked at treecc from Rhys W?
> http://www.southern-storm.com.au/treecc.html

I had a look. However, I believe that there are no "the best" tree.
Everybody needs slightly different tree:

  * OpenC++ MOP needs "lispy" Ptree structure to be able to introduce
    text chunks into parse tree, which not necessarily are legal C++
    (i.e. to be able to insert anything using
    YourMetaclass::TranslateXYZ())

  * Application which extracts class hierarchy, does not need a tree
    which exposes function bodies (they possibly have to be parsed,
    but application is not interested in walking/visiting them at all,
    and they are only a burden in application's walker)

  * Syntax-highlighting editor needs very detailed hierarchy and even
    more --- it wants to be able to store annotations with each
    (or some) nodes.

What I have in mind for all those examples above is one C++ parser and
three instances of intermediate layer --- each of which adapts
(on-the-fly) the tree "view" to the application needs. I also believe that
such layers can be somehow "derived" from each other, so if two
application's requirements are only slightly different, they can reuse the
layer components (which could at some moment constitute a library).

In general I see two solution when implementing such intermediate layer,
each with its own trade-ofs:

(1) using "visitor adapters"

    template <class Adapter>
    class VisitorAdapterHidingNodeX : public Adapter
    {
        virtual void VisitNodeX(NodeX*) {
             ... creates the translation of NodeX
             ... and visits it
        }
    };

(2) using traits class for identifying node types visible through
    particular "view" and using generic visitation to implement
    visitor (ala boost::variant)

I am still working on both, I will be glad to post my thoughts when I
arrive at some substantial conclusions.

Best regards
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

--- Grzegorz Jakacki <ja...@he...> wrote:
> On Thu, 20 Mar 2003, James Michael DuPont wrote:
> 
> >
> > --- Grzegorz Jakacki <ja...@he...> wrote:
> > > On Tue, 18 Mar 2003, Stefan Reuther wrote:
> > >
> > > > Hello,
> > > >
> > > > On Mon, Mar 17, 2003 at 04:43:53PM +0800, Grzegorz Jakacki
> wrote:
> > > > > On Fri, 14 Mar 2003, Stefan Reuther wrote:
> > > > > > I think, there are two problems, at least with the parser.
> The
> > > > > > interface consists of two parts, namely the Ptree
> descendants
> > > > > > and their contents.
> > > > > >
> > > > > > 1) Every Ptree descendant has a method "Translate" which
> calls
> > > > > >    a method of Walker. Adding a new Ptree class means
> adding a
> > > > > >    method to Walker.
> > > > >
> > > > > This is the first sign of too little isolation between
> visitor
> > > (Walker)
> > > > > and visited class hierarchy (Ptree and derived classes) ---
> you
> > > cannot
> > > > > extend Ptree hierarchy without modifying Walker code.
> > > >
> > > > I don't think it can be done much different. Ptree and Walker
> > > > have to "know each other".
> > >
> > > I believe that it is possible to introduce a layer in between,
> > > which would encapsualte the actual implementation of Ptree.
> >
> > Exactly my thoughts. We need a cleaner tree.
> > Have you looked at treecc from Rhys W?
> > http://www.southern-storm.com.au/treecc.html
> 
> I had a look. However, I believe that there are no "the best" tree.
> Everybody needs slightly different tree:
> 
>   * OpenC++ MOP needs "lispy" Ptree structure to be able to introduce
>     text chunks into parse tree, which not necessarily are legal C++
>     (i.e. to be able to insert anything using
>     YourMetaclass::TranslateXYZ())
> 
>   * Application which extracts class hierarchy, does not need a tree
>     which exposes function bodies (they possibly have to be parsed,
>     but application is not interested in walking/visiting them at
> all,
>     and they are only a burden in application's walker)
> 
>   * Syntax-highlighting editor needs very detailed hierarchy and even
>     more --- it wants to be able to store annotations with each
>     (or some) nodes.
> 
> What I have in mind for all those examples above is one C++ parser
> and
> three instances of intermediate layer --- each of which adapts
> (on-the-fly) the tree "view" to the application needs. I also believe
> that
> such layers can be somehow "derived" from each other, so if two
> application's requirements are only slightly different, they can
> reuse the
> layer components (which could at some moment constitute a library).
> 
> In general I see two solution when implementing such intermediate
> layer,
> each with its own trade-ofs:
> 
> (1) using "visitor adapters"
> 
>     template <class Adapter>
>     class VisitorAdapterHidingNodeX : public Adapter
>     {
>         virtual void VisitNodeX(NodeX*) {
>              ... creates the translation of NodeX
>              ... and visits it
>         }
>     };
> 
> (2) using traits class for identifying node types visible through
>     particular "view" and using generic visitation to implement
>     visitor (ala boost::variant)
> 
> I am still working on both, I will be glad to post my thoughts when I
> arrive at some substantial conclusions.


Ok, 
As far as I can tell. We have a few data sources , 

1. the preprocessor
2. the lexer
3. the yacc
4. the trees (ast in gcc)
5. the rtl (in gcc)
6. the debug information(in gcc)

All of these are just producing messages.
we need to have a callback filtering system so that the user can
register and subscribe to get the information needed. That will
predicate a identifier system that allows addressing of any part of the
system with an identifier.

This query system could be just a set of callbacks, each callback would
be a differnet type of function.

of course that could be made into a set of classes with methods you can
override. Maybe even a generic function that would allow you to get a
fully instanciated object of a given type.

The treecc is good for defining all the types of nodes. the question of
limiting the view and scope of data is a separate one.

mike

=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
http://tax.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Mon, 14 Apr 2003, James Michael DuPont wrote:

[...]
> 1. the preprocessor
> 2. the lexer
> 3. the yacc
> 4. the trees (ast in gcc)
> 5. the rtl (in gcc)
> 6. the debug information(in gcc)
>
> All of these are just producing messages.
> we need to have a callback filtering system so that the user can
> register and subscribe to get the information needed. That will
> predicate a identifier system that allows addressing of any part of the
> system with an identifier.
>
> This query system could be just a set of callbacks, each callback would
> be a differnet type of function.
>
> of course that could be made into a set of classes with methods you can
> override. Maybe even a generic function that would allow you to get a
> fully instanciated object of a given type.

In the framework I have in mind there is no callback registration.
The user of the framework supplies an object whose type is a class with
overloaded operator(). Overloads are distinguished based on the
kind of nodes in the tree. Kinds of nodes can be just nodes types,
but not necessarily --- one kind can be implemented by many types
and vice versa. The framework takes tree node implementation, decides
what kind is it and calls the appropriate overload on user-supplied
object.

> The treecc is good for defining all the types of nodes. the question of
> limiting the view and scope of data is a separate one.

I do not think that there is the universal set of types of nodes, perhaps
cons/nil is the only common denominator. It is very much domain dependent
and the framewok should not fix it. Take a look at SLT, which defines just
requirements on types (concepts) of containers. STL algorithms can be used
with any types, not only the ones defined in STL (standard containers'
iterators). For an example: imagine somebody prepared universal set of nodes
for C++ before Chiba invented OpenC++. Now comes Chiba and wants to have
some nodes which represent arbitrary chunks of text inserted into C++ parse
tree. It is very unlikely, that the person who focused on inventing
"universal" C++ parse tree predicted Chiba's requirement. We are in similar
situation.

Thanks for your input.
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

--- Grzegorz Jakacki <ja...@he...> wrote:
> On Mon, 14 Apr 2003, James Michael DuPont wrote:
> 
> [...]
> > 1. the preprocessor
> > 2. the lexer
> > 3. the yacc
> > 4. the trees (ast in gcc)
> > 5. the rtl (in gcc)
> > 6. the debug information(in gcc)
> >
> > All of these are just producing messages.
> > we need to have a callback filtering system so that the user can
> > register and subscribe to get the information needed. That will
> > predicate a identifier system that allows addressing of any part of
> the
> > system with an identifier.
> >
> > This query system could be just a set of callbacks, each callback
> would
> > be a differnet type of function.
> >
> > of course that could be made into a set of classes with methods you
> can
> > override. Maybe even a generic function that would allow you to get
> a
> > fully instanciated object of a given type.
> 
> In the framework I have in mind there is no callback registration.

> The user of the framework supplies an object whose type is a class
> with
> overloaded operator(). Overloads are distinguished based on the
> kind of nodes in the tree. Kinds of nodes can be just nodes types,
> but not necessarily --- one kind can be implemented by many types
> and vice versa. The framework takes tree node implementation, decides
> what kind is it and calls the appropriate overload on user-supplied
> object.

So that is a form of a callback. By passing an object that supports a
given interface where virtual methods are called, you have a call back
scenario.
> 
> > The treecc is good for defining all the types of nodes. the
> question of
> > limiting the view and scope of data is a separate one.
> 
> I do not think that there is the universal set of types of nodes,
> perhaps
> cons/nil is the only common denominator. It is very much domain
> dependent
> and the framewok should not fix it. Take a look at SLT, which defines
> just
> requirements on types (concepts) of containers. STL algorithms can be
> used
> with any types, not only the ones defined in STL (standard
> containers'
> iterators). For an example: imagine somebody prepared universal set
> of nodes
> for C++ before Chiba invented OpenC++. Now comes Chiba and wants to
> have
> some nodes which represent arbitrary chunks of text inserted into C++
> parse
> tree. 

Then he should create new nodes derived from the right part. (or supply
template classes as you point out)

>It is very unlikely, that the person who focused on inventing
> "universal" C++ parse tree predicted Chiba's requirement. We are in
> similar
> situation.

I am not saying we should handle all the nodes, but the ones we know
about.

mike

=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Stefan R. <sr...@ma...>]

Hi *.*,

On Wed, Mar 19, 2003 at 01:38:26PM +0800, Grzegorz Jakacki wrote:
> On Tue, 18 Mar 2003, Stefan Reuther wrote:
> > I don't think it can be done much different. Ptree and Walker
> > have to "know each other".
> 
> I believe that it is possible to introduce a layer in between,
> which would encapsualte the actual implementation of Ptree.

Another possibility would be to put that knowledge into the
Walker interface. Here, we have such a thing for
PtreeDeclaration. Depending upon whether a PtreeDeclaration is a
class declaration ("class foo;"), a name declaration ("int x;")
or a function implementation ("void x() { }"), it nicely
dissects the tree node and calls different functions. "Class"
and "ClassWalker" already seem to do some other parts of this.

Backwards-compatibility is not too hard with that (it can be
implemented atop Walker). Forward-compatibility works, too: when
the tree is changed, we add a new function which dissects the
new node, and translates it into a call to old functions.

I'm unsure which ("mine" or "yours") I like better.

> > Another nice thing to have would be a Walker returning other
> > things than Ptrees. I'm still unsure how that could work.
> 
> In fact Walker is implementation of pattern called Visitor.
> I am not sure if I have already mentioner "Design Patterns"
> by Gamma, Johnson, Ralph & Vlissides.

Damn, I wanted to read that for ages, but didn't get around
fetching it from the library :-) But it is more the
implementation than the understanding which worries me.

> Walker (or more generally visitor) can return arbitrary thing. I assume
> that what you have in mind is that it returns nodes of another tree
> (or rather handles/pointers to such nodes), thus implementing some
> homomoprhic tree transformation. It that it?

I even want to return complex things. For example, when
"compiling" an expression, I want to return
  class Expr_result {
      Ptree*            tree;     // result tree
      Type              type;     // result type
      Function_symbol*  symbol;   // function symbol, if result is an ambiguous function name
      Kind              kind;     // whether it's l- or r-value,
                                  // function, bound member function, ...
      Ptree*            obj_tree; // associated object for a member function call
      // methods omitted
  };

Michael Hohmuth solved that by making a template class
Ptree_visitor<T> which inherits from Walker. Instead of using
the Ptree* return value somehow, it saves the return value:
    Ptree* TranslateClassSpec(Ptree* p) {
       _saved = visit_classspec(dynamic_cast<PtreeClassSpec*>(p));
       return 0;
    }
and returns it later, approximately like this:
    T visit(Ptree* p) {
       Walker::Translate(p);
       return _saved;
    }

In full epic detail: <http://os.inf.tu-dresden.de/~sr21/c++/> That's
all my code. It's still incomplete. Michael's visitor is
<http://os.inf.tu-dresden.de/~sr21/cgi-bin/zipserv/c++/c++annotator-0.1.zip/c++annotator-0.1/comp/ptree_visitor.cpp>

> > For example, the recent addition of the "typeid" operator
> > caused our code to silently translate
> >    typeinfo ti = typeid(p);
> > as
> >    typeinfo ti = p;
> 
> Why was that (I could go figure out myself, but probably you still
> remember)?

Outline: a Walker to translate expressions, with the semantics:
to translate an expression, call "Translate()" and it will
deposit code to do that expression somewhere.

So "- x" is translated as "translate x, then negate". Because the
default action for "typeid" is to recurse (instead of generating
code), only the expression "p" is translated, and we have no way
of knowing that.

> > > > 2) The bigger problem is with the content of the nodes.
> > > > [...]
> > > "ifthenelse->Car()->Car()->Cdr()", which potentially can coredump
> > > at each arrow, one would conveniently write "ifthenelse->GetThen()".
> >
> > Sir, where do I sign? :-)
> 
> You mean NDA ?

Sign your petition for a better interface :-) Got tired of just
saying "Yes".

> > Still, in the *current* code, TranslateIfStatement takes a
> > Ptree* and not a PtreeIfStatement*, although the latter would be
> > a safe assumption for most cases.
> 
> There is a little bit of confusion, since lispy data structure is mixed up
> with 'PtreeIfStatement'-like classes, which on the other hand do not
> provide methods like "GetThen()". Why cannot we change
> TranslateIfStatement so that it takes "Ptree*" argument?

Currently, TranslateIfStatement takes a Ptree* argument. As far
as I can tell, it is only called with PtreeIfStatement* nodes,
so I would opt for changing that. Type safety, nothing more.

It can hurt on some pathological cases, but I don't think that's
really a problem: I had one place or two where I wanted to put a
simple Ptree* into TranslateName (which should, following my
logics, take a PtreeName*), but that's easily solvable by adding
a private third method.

> > One problem we'd still have to solve is that people can still
> > construct invalid trees. For example, if I have a Ptree
> > representing "if (cond) statement;", someone could Snoc a "&" at
> > the end, although that'd yield invalid syntax, and nothing could
> > prevent it (although adding an "else" and another statement
> > still must be valid). That's what I meant with "Lispy" data
> > structure: in Lisp, you have structured lists, too, but no-one
> > can prevent you from mangling them somehow.
> 
> This cannot be avoided if there is lispy interface to create the tree
> nodes available to the client. I think we should just assert or throw once
> such a situation (invalid tree) is identified.

Alternatively, allow to construct valid trees only. Things like
  class PtreeIfStatement : public PtreeStatement {
      PtreeIfStatement(Ptree* keyword, Ptree* lparen,
                       PtreeExprOrDecl* decl, Ptree* rparen,
                       PtreeStatement* statement);
      void AddElseClause(Ptree* keyword, PtreeStatement* statement);
  }
and deprecating (or "protected"ing) "Snoc", "Nconc" and friends.

I think having the "lispy" Ptree representation is nice for
traversal, output, GetLineNumber, etc, but allowing everyone to 
manipulate them might not be good.

> > I think adding those accessor functions ("GetCondition",
> > "GetThen", "GetIf") would be a good start, although it doesn't
> > work nicely for everything: for example, PtreeDeleteExpr can
> > have the following forms:
[...]
> Accessor functions can be refactored in the future, if the tree
> implementation is changed/normalized.
> 
> (BTW: I do not think that accessors should be free functions; see the
> write-up).

I didn't mean that, sorry. Accessors would be members of
PtreeDeleteExpr.

> > Another nice thing to have would be a "better" class hierarchy.
> > Improvements I see so far are:
> > - all PtreeFooExpr should share a common base class,
> >   PtreeExpression
> 
> What would this class contain? (Not to discourage you, I am just looking
> for arguments).

It would just be a base class for type safety. When I write
something which deals with expressions, I want to specify that
in the interface:
   void DoSomethingWith(PtreeExpression* p) { ... }
If I take a raw "Ptree", people can give me a PtreeIfStatement
or something, and I'll rightfully choke on it.

It's not a pressing problem, though. AbstractWalker would help
more for the instant.

> Where would this help? In function signatures, I guess, but at the moment
> I cannot see where exactly.

I guess we mean the same thing.

> > - all PtreeFooStatement should share a common base class,
> >   PtreeStatement. Hmmm, this seems to collide with PtreeBrace
> >   and PtreeDeclaration, so maybe leave it out.
> 
> Could you elaborate on where is the problem exactly?

Right now, PtreeBlock derives from PtreeBrace, which is IMHO a
sensible relation.

Since PtreeBlock can be a statement, it would have to derive
from PtreeStatement somehow. So we'd have to either cut the
relation between PtreeBlock and PtreeBrace, or make PtreeBrace
derive from PtreeStatement which is not good.

> > > > [1] Since the CVS code currently can't be built, I'll wait with
> > > > checking it in.
> > >
> > > Ok.
> >
> > It's in now.
> 
> Great. However could I ask you to add the testcases to verify this
> functionality? There are instructions somewhere under ./testsuite, perhaps
> those tests should go into ./testsuite/comp.

I'm working on it. I'll probably have to learn a bit more about
the meta-object protocol.

Essentially I want to call a Walker on some code. Any hints how
to do that most easily? Put it into a class, and override
TranslateFunctionImplementation, or is there a simpler way?

> > > But this requires adding *some* support of new nodes to the translator
> > > anyway, right?
> >
> > Yes, but I can't seem to find a way around that, because there
> > is no universal fallback. Some want just to recurse into the new
> > nodes, some do not want to miss them.
> 
> I am thinking about adding the code into the "view" layer (between tree
> and walker), what would express the new node in terms of old nodes
> (or at least abort).

Agreed.

> As promised, this is a write-up of thoughts I had wrt. improvind the
> design of OpenC++ in evolutionary way.
> 
> As I already wrote, there is too much coupling between Walker and Ptree
> hierarchy (this is usual problem with Visitor pattern).
> 
> I was thinking about providing more isolation by introducing a level of
> indirection. In particular I would like Ptree* to be replaced by some
> proxy type (a.k.a. "smart pointer", "handler" or "wrapper").

Short question: why? I would just put the accessor methods:
> template <>
> class PtreePtr<PtreeIfStmt>
> {
>     ...
>     PtreePtr<Ptree> GetThen()
>     {
>         return PtreePtr<Ptree>(ptr_->Cdr()->Car()->Cdr());
>     }
> };
into the respective Ptree nodes. It's all the same to me whether
I write p->GetThen() or p.GetThen().

Maybe there are subtle implications for forward- and backward
compatibility, but I'm currently unable to see them.

Anyway, if someone wants to add a new form of If, he can still
make PtreeIfStatement abstract (maybe with virtual GetThen())
and derive a PtreeNewIfStatement and PtreeOldIfStatement. Or am
I missing something here?

> The next step would be to get rid of 'new Ptree...' operators in the code.
> It should be gradually replaced by something like 'create<Ptree...>()',
> which is supposed to give back "something that behaves like Ptree",

Okay. Maybe constructors already suffice? But right, if the
returned type changes (PtreeIfStatement ->
PtreeNewFunkyIfStatement), constructors are problematic.

> As for visitors (aka. walkers): currently the "view" of the tree that
> visitor "sees" is fixed by the tree implementation. However, not every
> visitor wants to see the same --- e.g. sometimes you want to see comments
> or whitespace as nodes, sometimes not. It is especially true when you have
> deveopled several visitors and you add some new functionality to the
> parser (e.g. "explicit" keyword), so you need to add syntax node, but you
> would like to keep your old visitors running (and e.g. ignore the new
> node). Another example are parenthesis --- if you have visitor that
> simplifies arithmetic expressions, you most likely are not interested in
> it seeing the nodes representing parenthesis.
> 
> This can be taken care of by the use of tag classes. One can define a set
> of tag classes (which represent a "view" for visitor) and trait class that
> maps node implementation type to a tag (in a given view), e.g.:
> 
>     class BinaryNode {};
>     ...
>     class BasicViev;
> 
>     template <class View, class Node> class tree_traits;
> 
>     template<> struct tree_traits<BasicView,PtreePtr<PtreeAssign> >
>     {
>         typedef BinaryNode type;
>     }
>     ...
> 
>     tree_traits<BasicView,Node>::tag    <--- what 'Node' means in 'BasicView'
>                 ^^^^^^^^^ ^^^^
>              view          implementation type
> 
> To make the mechanism complete, one needs also a "dispatcher" or
> "destructor" --- function or functor, that for given view, visitor and node
> determines the node "nature" --- e.g. given Ptree* determines if it is
> "if statment", "assignment" etc., casts the node representation to
> appropriate type (or wraps it in appropriate proxy) and calls the visitor
> on it.

That's what Walker does, right? I think we can just take Walker
and derive a number of views from it. If someone wants to skip
over whitespace, their view would override TranslateWhitespace
to do nothing.

Maybe I'm too inexperienced to see the great advantages. I
usually aim to minimize work, and "your" approach sounds like a
lot of work :)


  Stefan

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

--- Stefan Reuther <sr...@ma...> wrote:
> Hi *.*,
> 
> On Wed, Mar 19, 2003 at 01:38:26PM +0800, Grzegorz Jakacki wrote:
> > On Tue, 18 Mar 2003, Stefan Reuther wrote:
> > > I don't think it can be done much different. Ptree and Walker
> > > have to "know each other".
> > 
> > I believe that it is possible to introduce a layer in between,
> > which would encapsualte the actual implementation of Ptree.
> 
> Another possibility would be to put that knowledge into the
> Walker interface. Here, we have such a thing for
> PtreeDeclaration. Depending upon whether a PtreeDeclaration is a
> class declaration ("class foo;"), a name declaration ("int x;")
> or a function implementation ("void x() { }"), it nicely
> dissects the tree node and calls different functions. "Class"
> and "ClassWalker" already seem to do some other parts of this.


The walker has only one problem.
It does not only walk, but also has to interpret things.
It has to know how to extract the data out of the parser

How do you propose to hide the details of the parser from the walker?

mike

=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
http://tax.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

On Mon, 14 Apr 2003, James Michael DuPont wrote:

>
> --- Stefan Reuther <sr...@ma...> wrote:
> > Hi *.*,
> >
> > On Wed, Mar 19, 2003 at 01:38:26PM +0800, Grzegorz Jakacki wrote:
> > > On Tue, 18 Mar 2003, Stefan Reuther wrote:
> > > > I don't think it can be done much different. Ptree and Walker
> > > > have to "know each other".
> > >
> > > I believe that it is possible to introduce a layer in between,
> > > which would encapsualte the actual implementation of Ptree.
> >
> > Another possibility would be to put that knowledge into the
> > Walker interface. Here, we have such a thing for
> > PtreeDeclaration. Depending upon whether a PtreeDeclaration is a
> > class declaration ("class foo;"), a name declaration ("int x;")
> > or a function implementation ("void x() { }"), it nicely
> > dissects the tree node and calls different functions. "Class"
> > and "ClassWalker" already seem to do some other parts of this.
>
>
> The walker has only one problem.
> It does not only walk, but also has to interpret things.
> It has to know how to extract the data out of the parser
>
> How do you propose to hide the details of the parser from the walker?

I think it is possible by usage of user defined handles instead of
pointers. Pointers reveal actual implementation types used in syntax tree.
User-defined handles do not have to.

Best regards
Grzegorz

>
> mike
>
> =====
> James Michael DuPont
> http://introspector.sourceforge.net/
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Tax Center - File online, calculators, forms, and more
> http://tax.yahoo.com
>
>

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Stefan R. <sr...@ma...>]

On Mon, Apr 14, 2003 at 04:32:57AM -0700, James Michael DuPont wrote:
> --- Stefan Reuther <sr...@ma...> wrote:
> > Another possibility would be to put that knowledge into the
> > Walker interface. Here, we have such a thing for
> > PtreeDeclaration. Depending upon whether a PtreeDeclaration is a
> > class declaration ("class foo;"), a name declaration ("int x;")
> > or a function implementation ("void x() { }"), it nicely
> > dissects the tree node and calls different functions. "Class"
> > and "ClassWalker" already seem to do some other parts of this.
> 
> The walker has only one problem.
> It does not only walk, but also has to interpret things.
> It has to know how to extract the data out of the parser
> 
> How do you propose to hide the details of the parser from the walker?

I would not hide the details.

Someone has to know about the Ptree structure, right? My point
was making Walker the one who knows it.

Like this:
   // users can override this method
   virtual Ptree* TranslateNamespaceCooked(Ptree* keyword,
                                           Ptree* name,
                                           Ptree* content);
   // this method is called by PtreeNamespace::Translate
   virtual Ptree* TranslateNamespaceRaw(PtreeNamespace* p) {
       return TranslateNamespaceCooked(p->Car()    /* "namespace" */,
                                       p->Second() /* the namespace name */,
                                       p->Third()  /* the contents */);
   }

I'm not exactly sure, but maybe that's not too flexible (one
can't inspect a given Ptree except by making a custom Walker and
calling it).

The other solution,
  class PtreeNamespace : public Ptree {
    public:
      Ptree* GetKeyword() { return Car(); }
      Ptree* GetName() { return Second(); }
      Ptree* GetContent() { return Third(); }
  };
puts that knowledge into the Ptree itself, which I like, but I
feel there are performance problems for things like "GetName" in
a PtreeDeclarator (it has to skip over a number of "*", "&",
"class::*" and cv-qualifiers to find the name or sub-declarator).
But maybe that's not at all a problem (caching?).

Using handle classes (Grzegorz' way) would definitely work nice,
but it's a lot of work to implement it.


  Stefan

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[James M. D. <mdu...@ya...>]

--- Stefan Reuther <sr...@ma...> wrote:
> On Mon, Apr 14, 2003 at 04:32:57AM -0700, James Michael DuPont wrote:
> > --- Stefan Reuther <sr...@ma...> wrote:
> > > Another possibility would be to put that knowledge into the
> > > Walker interface. Here, we have such a thing for
> > > PtreeDeclaration. Depending upon whether a PtreeDeclaration is a
> > > class declaration ("class foo;"), a name declaration ("int x;")
> > > or a function implementation ("void x() { }"), it nicely
> > > dissects the tree node and calls different functions. "Class"
> > > and "ClassWalker" already seem to do some other parts of this.
> > 
> > The walker has only one problem.
> > It does not only walk, but also has to interpret things.
> > It has to know how to extract the data out of the parser
> > 
> > How do you propose to hide the details of the parser from the
> walker?
> 
> I would not hide the details.
> 
> Someone has to know about the Ptree structure, right? My point
> was making Walker the one who knows it.
> 
> Like this:
>    // users can override this method
>    virtual Ptree* TranslateNamespaceCooked(Ptree* keyword,
>                                            Ptree* name,
>                                            Ptree* content);
>    // this method is called by PtreeNamespace::Translate
>    virtual Ptree* TranslateNamespaceRaw(PtreeNamespace* p) {
>        return TranslateNamespaceCooked(p->Car()    /* "namespace" */,
>                                        p->Second() /* the namespace
> name */,
>                                        p->Third()  /* the contents
> */);
>    }
> 
> I'm not exactly sure, but maybe that's not too flexible (one
> can't inspect a given Ptree except by making a custom Walker and
> calling it).

Ok, that is one way to do this. 

> 
> The other solution,
>   class PtreeNamespace : public Ptree {
>     public:
>       Ptree* GetKeyword() { return Car(); }
>       Ptree* GetName() { return Second(); }
>       Ptree* GetContent() { return Third(); }
>   };
> puts that knowledge into the Ptree itself, which I like, but I
> feel there are performance problems for things like "GetName" in
> a PtreeDeclarator (it has to skip over a number of "*", "&",
> "class::*" and cv-qualifiers to find the name or sub-declarator).
> But maybe that's not at all a problem (caching?).

I like that better. I dont want to have to make two passes to have a
clean model.

> 
> Using handle classes (Grzegorz' way) would definitely work nice,
> but it's a lot of work to implement it.

I dont know yet what the handle classes would look like. The idea
sounds good. In the end we need to have to hide this whole first and
second interface. 

I like the idea of a XML/DOM like structure that also allows for
lookups on the names. Each type of node that we know about gets a
proper c++ class that is attached to the generic node :


so it looks like this :

ptree -->pImp--> TreeClassDecl

the TreeClassDecl is the class that implements the classdecl
information. The ptree is the generic tree. The pImp points to the
TreeClassDecl because you may have to delete and recreate the
implementation class.

what do you think?
mike

=====
James Michael DuPont
http://introspector.sourceforge.net/

__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Michael H. <ho...@sa...>]

Hi Stefan!

> Damn, I wanted to read that for ages, but didn't get around
> fetching it from the library :-) 

Du kannst das Buch von mir ausborgen.

Michael
-- 
ho...@sa..., ho...@in...
http://home.pages.de/~hohmuth/

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Michael H. <ho...@sa...>]

I wrote:

> [...]

Sorry for sending a private message to the mailing list.  Of course,
it's not my fault ;-) it's almost always a bad idea to have
mailing-list software set a Reply-To header in emails it passes on.

Kind regards,
Michael
-- 
ho...@sa..., ho...@in...
http://www.sax.de/~hohmuth/

[Opencxx-users] Re: [opencxx] Let's break OpenC++ up

[Grzegorz J. <ja...@he...>]

Hi,

On Wed, 16 Apr 2003, Michael Hohmuth wrote:

> Hi Stefan!
>
> > Damn, I wanted to read that for ages, but didn't get around
> > fetching it from the library :-)
>
> Du kannst das Buch von mir ausborgen.

Now when both of you have read it I will finaly post my long-baked reply
to Stefan's e-mail :-)

Seriously --- sorry for the delay. I have just returned from ETAPS
conference in Warsaw (with Easter at home as a bonus...) back to the SARS
inferno and many small things keep me a little bit busy.

Something along the lines we have been talking about was in fact discussed
at Compiler Construction Conference (part of ETAPS) as a reusable compiler
framework in Java. I tried to develop details of both your solution and
also template-based solution that I have in mind and discuss it with
several persons. I will try to collect all those thought and post them,
hopefully this week.

Best regards
Grzegorz

>
> Michael
> --
> ho...@sa..., ho...@in...
> http://home.pages.de/~hohmuth/
>
>

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2002 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################