opencxx-users

[Opencxx-users] OpenC++ Core Lib -- prototype

[Grzegorz J. <ja...@ac...>]

Hi,

The new module in CVS called 'occ-core' contains my first attempt at
launching the "OpenC++ Core" initiative. Throughout the last week
Stefan Seefeld and myself were discussing the possibilities of having
a common repo for the core part of OpenC++ reused by Synopsis, but some
unsolved issues still remain, thus I decided to demo my idea in an actual
code.

As outlined in "OpenC++ Core Lib MANIFESTO", OpenC++ has problems with
advancing the quality of code analysis (templates and overloading),
because along with advancement in this areas, other parts of the system 
need to be accordingly maintained, which distracts the resources. Thus 
the idea to take out the core functionality pertaining to code analysis
and make it into compact, self-contained and self-testable library.
The division would separated code into "OpenC++ Core" (lexer, parser,
code object model ("Class", "Member"), type elaboration) and
"Greater OpenC++" (translation, metaclasses).

'occ-core' is a first attempt to group the "OpenC++ Core" and make it
self-contained. To checkout (non-developers):

   export CVSROOT=:pserver:ano...@cv...:/cvsroot/opencxx
   cvs login
   cvs -z3 co occ-core

To build and run exemplary tests:

   make test

Makefile will require some tweaking if you are not using g++. However
build process is very straighforward, so that should not be an issue.

All developers interested in advancing templates and overloading
functionality are invited to have a look and comment. Please remember, 
that this is by no means a ready product, but rather a workplace for the 
actual work.

Best regards
Grzegorz

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:
> Hi,
> 
> The new module in CVS called 'occ-core' contains my first attempt at
> launching the "OpenC++ Core" initiative. Throughout the last week
> Stefan Seefeld and myself were discussing the possibilities of having
> a common repo for the core part of OpenC++ reused by Synopsis, but some
> unsolved issues still remain, thus I decided to demo my idea in an actual
> code.
> 
> As outlined in "OpenC++ Core Lib MANIFESTO", OpenC++ has problems with
> advancing the quality of code analysis (templates and overloading),
> because along with advancement in this areas, other parts of the system 
> need to be accordingly maintained, which distracts the resources. Thus 
> the idea to take out the core functionality pertaining to code analysis
> and make it into compact, self-contained and self-testable library.
> The division would separated code into "OpenC++ Core" (lexer, parser,
> code object model ("Class", "Member"), type elaboration) and
> "Greater OpenC++" (translation, metaclasses).
> 
> 'occ-core' is a first attempt to group the "OpenC++ Core" and make it
> self-contained. To checkout (non-developers):
> 
>   export CVSROOT=:pserver:ano...@cv...:/cvsroot/opencxx
>   cvs login
>   cvs -z3 co occ-core

'no such module' is all I get. That's probably because the anonymous
access is seeing changes with some hours of delay. Using my developer
login works.

Did you have a look at the redesign I did inside Synopsis ? How does
your occ-core compare to that ?

I'd generally much prefer to start with an assessment (i.e. documentation
in terms of actual text, unit tests, and examples) of the existing design.
Some months ago I proposed a rigorous separation between 'parser' (including
ptree API) and higher level classes (Environment, TypeInfo, Class, etc.).

Now that my understanding of occ internals is a little better, I would
revise my suggestions I then made. Still, I believe that I'm not the only
one whose understanding is limitted, and so I believe in order to be able
to make 'educated suggestions' we should start by discussing the existing
design.
In particular, how does parsing work in detail ? What is the Walker's role ?
How are Encoding and TypeInfo related ?
Without better understanding these and other questions I'm unable to suggest
where to draw the line between low level and high level.

Regards,
		Stefan

PS: I consider the redesign I'm doing in Synopsis as an iterative process.
     I work my way up starting from the lowest layers, and while I get a better
     understanding on the individual classes I'm able to improve their design.

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Grzegorz J. <ja...@ac...>]

Hi!

Stefan Seefeld wrote:
> Did you have a look at the redesign I did inside Synopsis ? How does
> your occ-core compare to that ?

Yes I have. I have not got down to visitors, so this is one difference. 
I got rid of metaclass- and translation-related stuff, as I think this 
is the domain of a client (Greater OpenC++) not the OpenC++ Core. I have 
also seen that you have done some renaming in Ptree hierarchy; I kept it 
as it was, so this is yet another difference. That's it for the quick 
glimpse.

In fact I contemplated starting from your files, however it would force 
LGPL on OpenC++ and I am not sure if this is the right thing to do.

> I'd generally much prefer to start with an assessment (i.e. documentation
> in terms of actual text, unit tests, and examples) of the existing design.

I think it is suboptimal in this case for two reasons. First, there is 
very little documentation of existing system, so almost all 
documentation would need to be reverse engineered (I already did some of 
it, as you may remember). Second, we cannot provide accurate unit tests 
post factum, because we are not sure what the code should really do.

I think that the way to go is to read and clean the code step by step, 
adding unit tests as the knowledge about the code grows.

> Some months ago I proposed a rigorous separation between 'parser' 
> (including
> ptree API) and higher level classes (Environment, TypeInfo, Class, etc.).

In fact parser separation started several months before you joined in 
April. (http://sourceforge.net/mailarchive/message.php?msg_id=6971830)

> Now that my understanding of occ internals is a little better, I would
> revise my suggestions I then made. 

Go ahead.

> Still, I believe that I'm not the only
> one whose understanding is limitted, 

You are not the only one. Count myself.

> and so I believe in order to be able
> to make 'educated suggestions' we should start by discussing the existing
> design.
> In particular, how does parsing work in detail ? 

More detailed question solicited.

> What is the Walker's role ?

I believe that:

Walker -- does preety nothing, just traverses the subtrees. It is a 
"skeletal implementation".

ClassWalker --- traverses the AST and creates metaobjects (objects 
representing semantic entities of C++, e.g. objects of classes Class, 
Member etc.)

ClassBodyWalekr --- traverses the AST of classes and also creates 
metaobjects (perhaps difference between ClassWalker and ClassBodyWalker 
stems from the fact that in C++ binding in class scopes works 
differently than in other scopes).

> How are Encoding and TypeInfo related ?

It seems to me that Encoding is always an "expanded" type, while 
TypeInfo is "lazy" type, e.g. it can buffer referencing/dereferencing
and also it can keep typedefs unexpanded (that's why it needs to have a 
link to Environment).

There also another point to it: if I am not mistaken Encoding can encode 
not only type, but also scoped identifier.

> Without better understanding these and other questions I'm unable to 
> suggest where to draw the line between low level and high level.

I am afraid there is no single person who understand (or remembers :-) 
this all clearly today.

> Regards,
>         Stefan
> 
> PS: I consider the redesign I'm doing in Synopsis as an iterative process.
>     I work my way up starting from the lowest layers, and while I get a 
> better
>     understanding on the individual classes I'm able to improve their 
> design.

How does it relate to OpenC++? What would you suggest for OpenC++?

BR
Grzegorz

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:

 >> I'd generally much prefer to start with an assessment (i.e. documentation
 >> in terms of actual text, unit tests, and examples) of the existing design.
 >
 >
 > I think it is suboptimal in this case for two reasons. First, there is
 > very little documentation of existing system, so almost all documentation
 > would need to be reverse engineered (I already did some of it, as you may
 > remember).

Yes. But don't we need to reverse engineer if we want to apply changes in a
controlled manner ?
I can tell you how much head-aches I have these days trying to understand
why a given method is implemented the way it is. The fact that the old code
wasn't const-correct obviously doesn't help, i.e. I have to guess whether a
given argument is going to be modified in-place or whether it's only
an 'in' parameter.

So reverse-engineering is what takes 90% of my time. Once I understand a
given design it's quite easy to see whether it can be enhanced.

 > Second, we cannot provide accurate unit tests post factum, because we are
 > not sure what the code should really do.

granted. Though they would serve as regression tests, i.e. we could measure
what kind of changes we incure to the data handling while we work on the
redesign. In fact, there are, as we discussed, a couple of cases that are known
to fail. Those known failures should be accompagnied by (annotated) unit
tests, if only to document that we know about them.

 > I think that the way to go is to read and clean the code step by step,
 > adding unit tests as the knowledge about the code grows.
 >
 >> Some months ago I proposed a rigorous separation between 'parser'
 >> (including ptree API) and higher level classes (Environment, TypeInfo,
 >> Class, etc.).
 >
 >
 > In fact parser separation started several months before you joined in April.
 > (http://sourceforge.net/mailarchive/message.php?msg_id=6971830)
 >
 >> Now that my understanding of occ internals is a little better, I would
 >> revise my suggestions I then made.
 >
 >
 > Go ahead.

nothing dramatic :-) I simply concluded that the boundary between 'low' and
'high' level is a little blury, or not where I originally expected it to be.
That's mainly because I didn't realize at the time that what we called 'parsing'
was only the first half, the second half including the use of 'ClassWalker' and co.

As we now agree to redesign the parser such that it does the declarator
recording (Environment...) itself instead of letting it done by a Walker, it
could mean that the Walker could be part of the high level.
Right now it can't because parsing isn't complete (i.e. self-contained)
without ClassWalker.

 >> and so I believe in order to be able
 >> to make 'educated suggestions' we should start by discussing the existing
 >> design.
 >> In particular, how does parsing work in detail ?
 >
 >
 > More detailed question solicited.

details follow:

 >> What is the Walker's role ?
 >
 >
 > I believe that:
 >
 > Walker -- does preety nothing, just traverses the subtrees. It is a
 > "skeletal implementation".
 >
 > ClassWalker --- traverses the AST and creates metaobjects (objects
 > representing semantic entities of C++, e.g. objects of classes Class, Member
 > etc.)

not quite. That's precisely one of the spots I have problems with: the
ClassWalker plays a role as well in the low level (parsing) as well as the
high level (mop).
It records declarations to the Environments, and these are then necessary to
complete the parsing, no ? Remember when we discussed ambiguous C++ statements, i.e.
statements that are ambiguous as long as you can't check whether a given
symbol refers to a type or not...

 > ClassBodyWalekr --- traverses the AST of classes and also creates
 > metaobjects (perhaps difference between ClassWalker and ClassBodyWalker
 > stems from the fact that in C++ binding in class scopes works differently
 > than in other scopes).
 >
 >> How are Encoding and TypeInfo related ?
 >
 >
 > It seems to me that Encoding is always an "expanded" type, while TypeInfo
 > is "lazy" type, e.g. it can buffer referencing/dereferencing
 > and also it can keep typedefs unexpanded (that's why it needs to have a
 > link to Environment).
 >
 > There also another point to it: if I am not mistaken Encoding can encode
 > not only type, but also scoped identifier.

yes, it encodes names (function names, in particular, which include the
signature, i.e. parameter types !), as well as types.
I think that your separation of Encoding and EncodingUtil is a good start.
'Encoding' is a 'passive' data holder (similar to std::string). In this role
I placed it into PTree nodes, to make them more type safe.
All the active parts (name lookup, code generation, etc.) should be kept
outside. By the way, why isn't the EncodingUtil part of Environment or TypeInfo ?
Wouldn't that make things slightly simpler ?

 >> PS: I consider the redesign I'm doing in Synopsis as an iterative
 >>     process.
 >>     I work my way up starting from the lowest layers, and while I get a
 >>     better understanding on the individual classes I'm able to improve their
 >>     design.
 >
 >
 > How does it relate to OpenC++? What would you suggest for OpenC++?

oh, the process is simply the only way for me to understand OpenC++. The
design itself is what I would suggest OpenC++ to use for its core. But as I said,
it's an iterative process, so I'm far from done with it, so I expect more structural
changes to the repository (thank svn this is easy ;-)

Regards,
		Stefan

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Grzegorz J. <ja...@he...>]

Hi,

On Mon, 13 Sep 2004, Stefan Seefeld wrote:

> Grzegorz Jakacki wrote:
>
>  >> I'd generally much prefer to start with an assessment (i.e. documentation
>  >> in terms of actual text, unit tests, and examples) of the existing design.
>  >
>  >
>  > I think it is suboptimal in this case for two reasons. First, there is
>  > very little documentation of existing system, so almost all documentation
>  > would need to be reverse engineered (I already did some of it, as you may
>  > remember).
>
> Yes. But don't we need to reverse engineer if we want to apply changes in a
> controlled manner ?
>
> I can tell you how much head-aches I have these days trying to understand
> why a given method is implemented the way it is. The fact that the old code
> wasn't const-correct obviously doesn't help, i.e. I have to guess whether a
> given argument is going to be modified in-place or whether it's only
> an 'in' parameter.
>
> So reverse-engineering is what takes 90% of my time. Once I understand a
> given design it's quite easy to see whether it can be enhanced.

My take is that 100% understanding in these circumstances is not
precondition to introducing changes. But this is MHO.

>  > Second, we cannot provide accurate unit tests post factum, because we are
>  > not sure what the code should really do.
>
> granted. Though they would serve as regression tests, i.e. we could measure
> what kind of changes we incure to the data handling while we work on the
> redesign.

I was trying to take this patch for the past 3 years and for me it did not
work out with OpenC++. Writting reasonable regression tests is lots of
hard work and is not very exciting for prospective developers. I took such
road at work where I inherited undocumented system without tests, but with
open-source volunteers-based project things look differently.

Thus my proposal to focus on the core functionality and put "Greater
OpenC++" aside for some time. In particular I no longer think that it
makes sense to add regression tests to "Greater OpenC++" now.

> In fact, there are, as we discussed, a couple of cases that are known
> to fail. Those known failures should be accompagnied by (annotated) unit
> tests, if only to document that we know about them.

Agreed, but I vote for mounting those unit tests as low as possible ---
not on top of 'occ' interface, but e.g. on top of parser library.

>  > I think that the way to go is to read and clean the code step by step,
>  > adding unit tests as the knowledge about the code grows.
>  >
>  >> Some months ago I proposed a rigorous separation between 'parser'
>  >> (including ptree API) and higher level classes (Environment, TypeInfo,
>  >> Class, etc.).
>  >
>  >
>  > In fact parser separation started several months before you joined in April.
>  > (http://sourceforge.net/mailarchive/message.php?msg_id=6971830)
>  >
>  >> Now that my understanding of occ internals is a little better, I would
>  >> revise my suggestions I then made.
>  >
>  >
>  > Go ahead.
>
> nothing dramatic :-) I simply concluded that the boundary between 'low' and
> 'high' level is a little blury, or not where I originally expected it to be.
> That's mainly because I didn't realize at the time that what we called 'parsing'
> was only the first half, the second half including the use of 'ClassWalker' and co.
>
> As we now agree to redesign the parser such that it does the declarator
> recording (Environment...) itself instead of letting it done by a Walker, it
> could mean that the Walker could be part of the high level.
> Right now it can't because parsing isn't complete (i.e. self-contained)
> without ClassWalker.

Wait a moment. I don't think it is *that* easy with C++. As I already
mentioned here several weeks ago, I don't see a way to perform type
elaboration in one pass. This is because

    The potential scope of a name declared in a class consists not only of
    the declarative region following the name s declarator, but also of all
    function bodies, default arguments, and constructor ctor-initializers in
    that class. [3.6.6p1]

which in particular makes this code legal:

    class A
    {
        void f() { g(); };
        void g() {};
    };

The more tricky example:

    class A
    {
        void f() { B < 1 > ( 2 ) ; }
        template <int n> void B(int);
    };

Here we just cannot rely just on one pass. It seems to me that a reasonable
solution is that parser stores the function bodies, default arguments and
ctor-initializers and parses them after the class is parsed. At that
time the symbol table can meaningfully answer if a given name represents
a template.

>  >> and so I believe in order to be able
>  >> to make 'educated suggestions' we should start by discussing the existing
>  >> design.
>  >> In particular, how does parsing work in detail ?
>  >
>  >
>  > More detailed question solicited.
>
> details follow:
>
>  >> What is the Walker's role ?
>  >
>  >
>  > I believe that:
>  >
>  > Walker -- does preety nothing, just traverses the subtrees. It is a
>  > "skeletal implementation".
>  >
>  > ClassWalker --- traverses the AST and creates metaobjects (objects
>  > representing semantic entities of C++, e.g. objects of classes Class, Member
>  > etc.)
>
> not quite. That's precisely one of the spots I have problems with: the
> ClassWalker plays a role as well in the low level (parsing) as well as the
> high level (mop).
> It records declarations to the Environments, and these are then necessary to
> complete the parsing, no ?

I don't think OpenC++ does it now in such manner. AFAIK parser does not use
type information to resolve any syntax ambiguities (ie. at template
instantiations).

> Remember when we discussed ambiguous C++ statements, i.e.
> statements that are ambiguous as long as you can't check whether a given
> symbol refers to a type or not...
>
>  > ClassBodyWalekr --- traverses the AST of classes and also creates
>  > metaobjects (perhaps difference between ClassWalker and ClassBodyWalker
>  > stems from the fact that in C++ binding in class scopes works differently
>  > than in other scopes).
>  >
>  >> How are Encoding and TypeInfo related ?
>  >
>  >
>  > It seems to me that Encoding is always an "expanded" type, while TypeInfo
>  > is "lazy" type, e.g. it can buffer referencing/dereferencing
>  > and also it can keep typedefs unexpanded (that's why it needs to have a
>  > link to Environment).
>  >
>  > There also another point to it: if I am not mistaken Encoding can encode
>  > not only type, but also scoped identifier.
>
> yes, it encodes names (function names, in particular, which include the
> signature, i.e. parameter types !), as well as types.
> I think that your separation of Encoding and EncodingUtil is a good start.

It is original design, I have not touched it.

> 'Encoding' is a 'passive' data holder (similar to std::string). In this role
> I placed it into PTree nodes, to make them more type safe.

Personally I don't like it, because it forces clients data structure on
parser library.

> All the active parts (name lookup, code generation, etc.) should be kept
> outside. By the way, why isn't the EncodingUtil part of Environment or TypeInfo ?
> Wouldn't that make things slightly simpler ?

Putting functions from EncodingUtil into Encoding makes parser library
depend on Environment, and Walker, and Bind, and TypeInfo, and Class,
so all the parser isolation collapses.

Putting functions from EncodingUtil into Environment makes sense only if
they need private access to Environment. They don't.

If you feel that 4 functions in EncodingUtil namespace does not warrant its
existence, then I would suggest putting them in Environment.h (but not as
member functions), according to Sutter's "interface principle".

>  >> PS: I consider the redesign I'm doing in Synopsis as an iterative
>  >>     process.
>  >>     I work my way up starting from the lowest layers, and while I get a
>  >>     better understanding on the individual classes I'm able to improve their
>  >>     design.
>  >
>  >
>  > How does it relate to OpenC++? What would you suggest for OpenC++?
>
> oh, the process is simply the only way for me to understand OpenC++. The
> design itself is what I would suggest OpenC++ to use for its core. But as I said,
> it's an iterative process, so I'm far from done with it, so I expect more structural
> changes to the repository (thank svn this is easy ;-)

I think that your contributions to original OpenC++ that currently
constitute C++ submodule of Synopsis are very valuable and I would be glad
to leverage them in OpenC++ Core. As I understand the only changes that were
made under LGPL to OpenC++ 2.5.12 under Synopsis are yours, right? If so,
would you consider donating them to OpenC++ under OpenC++ license?

BR
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2004 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:

 >>So reverse-engineering is what takes 90% of my time. Once I understand a
 >>given design it's quite easy to see whether it can be enhanced.
 >
 >
 >My take is that 100% understanding in these circumstances is not
 >precondition to introducing changes. But this is MHO.

I fully agree, and I have never said anything else. It's an iterative
process...


 >>As we now agree to redesign the parser such that it does the declarator
 >>recording (Environment...) itself instead of letting it done by a Walker, it
 >>could mean that the Walker could be part of the high level.
 >>Right now it can't because parsing isn't complete (i.e. self-contained)
 >>without ClassWalker.
 >
 >
 >Wait a moment. I don't think it is *that* easy with C++. As I already
 >mentioned here several weeks ago, I don't see a way to perform type
 >elaboration in one pass. This is because
 >
 >    The potential scope of a name declared in a class consists not only of
 >    the declarative region following the name s declarator, but also of all
 >    function bodies, default arguments, and constructor ctor-initializers
 >    in that class. [3.6.6p1]

yes, yes, I acknowledge. My wording was poor. I understand that parsing
can't be a linear sequential process. The design I have a problem with is
the fact that in the Walker framework multiple things seem to be lumped
together: on the one hand it appears to operate on the meta object level
(user keywords), and on the other it seems to participate in the parsing
(the 'second stage'). Is that not so ?

The construction of scopes (i.e. symbol lookup tables) can be done during
the parsing, right ? There is just no reason to do this stuff in the same
object that looks for user keywords !

Walkers *should* only care about the mop stuff, and thus all the lower level
classes (ptree, parser, encoding, etc.) should be free of any reference
to Walkers.


 >Here we just cannot rely just on one pass. It seems to me that a reasonable
 >solution is that parser stores the function bodies, default arguments and
 >ctor-initializers and parses them after the class is parsed. At that
 >time the symbol table can meaningfully answer if a given name represents
 >a template.

yes, sounds good. What I'm not clear about is whether this would involve
high level types such as 'Class'. Put differently: should 'Class' be part
of this parse process ? Where do you draw the line between 'core' and 'high
level' ?

 >>It records declarations to the Environments, and these are then necessary
 >>to complete the parsing, no ?
 >
 >
 >I don't think OpenC++ does it now in such manner. AFAIK parser does not use
 >type information to resolve any syntax ambiguities (ie. at template
 >instantiations).

that may answer my question why I don't see any ptree changes between
before and after the translation. If I understand you right, the translation
only changes the ptree if there are user keywords ('metaclass' and co.) in
the
input, which may be expanded. Can you confirm this ?

 >>'Encoding' is a 'passive' data holder (similar to std::string). In this
 >>role I placed it into PTree nodes, to make them more type safe.
 >
 >
 >Personally I don't like it, because it forces clients data structure on
 >parser library.

not in my case, where 'Encoding' has become part of the 'parser library'.

 >Putting functions from EncodingUtil into Environment makes sense only if
 >they need private access to Environment. They don't.

hmm, from my point of view a function belongs to a class if it is
conceptually
related, not if it needs access to the class' private members. Things like
'GetBaseName' operate on both, Encodings as well as Environments. You could
argue for both, associating them with Encodings as well as Environments.
But as since without it Encoding wouldn't depend on Environment, I think it
is more natural to put it into the Environment...

 >If you feel that 4 functions in EncodingUtil namespace does not warrant its
 >existence, then I would suggest putting them in Environment.h (but not as
 >member functions), according to Sutter's "interface principle".

yeah, they may be static.
[while I think 'principles' reflect 'common sense' and generally accepted
'good practice',
  I wouldn't want to be dogmatic enforcing them. They are just principles
after all...]


 >I think that your contributions to original OpenC++ that currently
 >constitute C++ submodule of Synopsis are very valuable and I would be glad
 >to leverage them in OpenC++ Core. As I understand the only changes that
 >were made under LGPL to OpenC++ 2.5.12 under Synopsis are yours, right?

No, they are mine as well as from Stephen Davies (whom I'v lost track of,
unfortunately).

 >If so,
 >would you consider donating them to OpenC++ under OpenC++ license?

what is the 'OpenC++ license' ? How is it different from LPGL ?
What makes you not want to use LGPL ?
And, without wanting to get into a flame war, a question to all developers
on this list:
Does anybody *not* want the code to be distributed under LGPL ? If so, why ?

Regards,
		Stefan

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Grzegorz J. <ja...@he...>]

On Tue, 14 Sep 2004, Stefan Seefeld wrote:

> Grzegorz Jakacki wrote:
[...]

>  >>As we now agree to redesign the parser such that it does the declarator
>  >>recording (Environment...) itself instead of letting it done by a Walker, it
>  >>could mean that the Walker could be part of the high level.
>  >>Right now it can't because parsing isn't complete (i.e. self-contained)
>  >>without ClassWalker.
>  >
>  >
>  >Wait a moment. I don't think it is *that* easy with C++. As I already
>  >mentioned here several weeks ago, I don't see a way to perform type
>  >elaboration in one pass. This is because
>  >
>  >    The potential scope of a name declared in a class consists not only of
>  >    the declarative region following the name s declarator, but also of all
>  >    function bodies, default arguments, and constructor ctor-initializers
>  >    in that class. [3.6.6p1]
>
> yes, yes, I acknowledge. My wording was poor. I understand that parsing
> can't be a linear sequential process. The design I have a problem with is
> the fact that in the Walker framework multiple things seem to be lumped
> together: on the one hand it appears to operate on the meta object level
> (user keywords), and on the other it seems to participate in the parsing
> (the 'second stage'). Is that not so ?

Yes.

> The construction of scopes (i.e. symbol lookup tables) can be done during
> the parsing, right ?

Yes.

> There is just no reason to do this stuff in the same
> object that looks for user keywords !

Yes.

> Walkers *should* only care about the mop stuff, and thus all the lower level
> classes (ptree, parser, encoding, etc.) should be free of any reference
> to Walkers.

Yes with exceptions:

* ptree has to know about AbstractWalker (Visitor pattern)

* putting encoding in ptree is in my view wrong, because ptree should
  not be concerned with how the higher levels represent types

>  >Here we just cannot rely just on one pass. It seems to me that a reasonable
>  >solution is that parser stores the function bodies, default arguments and
>  >ctor-initializers and parses them after the class is parsed. At that
>  >time the symbol table can meaningfully answer if a given name represents
>  >a template.
>
> yes, sounds good. What I'm not clear about is whether this would involve
> high level types such as 'Class'. Put differently: should 'Class' be part
> of this parse process ?

Yes and no.

I think parser/ptree should not know about Class. However, parser should
provide callbacks, so that the higher levels can construct semantic
objects, if they wish (and how they wish, and whatever objects they wish).
I see callbacks implemented as calls to members of some interface class.
Clients of parser can provide implementation of this class and register it
with parser.

> Where do you draw the line between 'core' and 'high
> level' ?

I can see several modules of the Core:

(1) elaboration (Class, Member, TypeInfo, Encoding, concrete walkers)

(2) Ptree hierarchy, abstract walkers

(3) parser

(4) Lexer

(5) Program/Buffer

      ,------------------->|ptree|----------.
      |             ,----->|     |          |
      |             |                       v
|elaboration |--> |parser| --> |lexer| --> program/buffer


Class above is in fact only part of the original Class, with all things
petraining to reflexivity and source-to-source translation ripped out
(e.g. all Translate... functions, statics such as SetDefaultMetaclass()
etc.).

And that's it for Core.

On top of that goes "Greater OpenC++", ie. classes that constitute the
reflexivity model (things ripped out from the class above). These are the
classes that should be visible to the end-user of Greater OpenC++ after
including "mop.h", all stuff related to user-defined metaclasses etc.

Yet on top of that goes simple driver, which is an executable that reads
preprocessed source from stdin and writes translated source to stdout.

And on very top of that goes full-blown driver, that calls preprocessor,
linker, compiler etc.

>  >>It records declarations to the Environments, and these are then necessary
>  >>to complete the parsing, no ?
>  >
>  >
>  >I don't think OpenC++ does it now in such manner. AFAIK parser does not use
>  >type information to resolve any syntax ambiguities (ie. at template
>  >instantiations).
>
> that may answer my question why I don't see any ptree changes between
> before and after the translation. If I understand you right, the translation
> only changes the ptree if there are user keywords ('metaclass' and co.) in
> the
> input, which may be expanded. Can you confirm this ?

Translation changes anything only if you specify nontrivial metaclass for
some class. You can do it by with "metaclass NontrivialMetaclass MyClass;"
declaration in translated source code, or with "SetDefaultMetaclass()"
somewhere in 'compiler' code (most likely in
'NontrivialMetaclass::Initialize()').

>  >>'Encoding' is a 'passive' data holder (similar to std::string). In this
>  >>role I placed it into PTree nodes, to make them more type safe.
>  >
>  >
>  >Personally I don't like it, because it forces clients data structure on
>  >parser library.
>
> not in my case, where 'Encoding' has become part of the 'parser library'.

In my view yes, because parser should not care how its client represent
something (types). IMHO parser should provide means for clients to store
something with AST nodes, but the type of stored data should be opaque
to the parser. Yes, it is type unsafe, but it cannot be otherwise without
loss of reusability or without parametrizing all AST with client's storage
type. If clients cares for type safety, it can use AST via wrapper
(normally it is costly, because wrapper is very much sensitive to changes
in AST, but it looks like it can be made cheaper with some templates
tricks.)

>  >Putting functions from EncodingUtil into Environment makes sense only if
>  >they need private access to Environment. They don't.
>
> hmm, from my point of view a function belongs to a class if it is
> conceptually
> related, not if it needs access to the class' private members.
>
> Things like
> 'GetBaseName' operate on both, Encodings as well as Environments. You could
> argue for both, associating them with Encodings as well as Environments.
> But as since without it Encoding wouldn't depend on Environment, I think it
> is more natural to put it into the Environment...

Putting such functions into class increases coupling. If you modify
private details of implementation, then you have to go through all member
functions, to see if they need to be updated. Making free functions
members adds more work.

Another argument is that member functions that need only public interface
obfuscate the class, because by looking at class definition you assume,
that these functions need to access implementation details, while they
don't.

For more arguments see Herb Sutter "Exceptional C++", Addison-Wesley 2000.

>  >If you feel that 4 functions in EncodingUtil namespace does not warrant its
>  >existence, then I would suggest putting them in Environment.h (but not as
>  >member functions), according to Sutter's "interface principle".
>
> yeah, they may be static.

I did not mean this. I meant to add them as *free* function to
Environment.h. See Sutter.

> [while I think 'principles' reflect 'common sense' and generally accepted
> 'good practice',
>   I wouldn't want to be dogmatic enforcing them. They are just principles
> after all...]

See above for arguments why I think my point makes sense.

>  >I think that your contributions to original OpenC++ that currently
>  >constitute C++ submodule of Synopsis are very valuable and I would be glad
>  >to leverage them in OpenC++ Core. As I understand the only changes that
>  >were made under LGPL to OpenC++ 2.5.12 under Synopsis are yours, right?
>
> No, they are mine as well as from Stephen Davies (whom I'v lost track of,
> unfortunately).
>
>  >If so,
>  >would you consider donating them to OpenC++ under OpenC++ license?
>
> what is the 'OpenC++ license' ?

See file COPYING.

> How is it different from LPGL ?

Does not require redistribution of linkable binary code.

> What makes you not want to use LGPL ?

(1) LGPL is more restrictive than OpenC++ license.

(2) Part of OpenC++ license is a license from Xerox. I am not sure if
    and how should we distribute files under mixed license (LGPL + Xerox)
    and what implications it has.

> And, without wanting to get into a flame war, a question to all developers
> on this list:
> Does anybody *not* want the code to be distributed under LGPL ? If so, why ?

Myself. Putting apart the downsides listed above, the only benefit of LGPL
I can see is that it is better known and better tested. However, if I was
to get down to relicensing OpenC++ I would first think about other
licenses (MIT? Boost?). Every such move, however, most likely involves
negotiations with Xerox and I don't think that such time investment will
pay off.

BR
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2004 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:

> Yes with exceptions:
> 
> * ptree has to know about AbstractWalker (Visitor pattern)

No :-)
Yes, we need a Visitor, but this Visitor can be very minimal. Have a look
into my code: Walker with all its 'Translate' methods is just an implementation
of the Visitor.
Your AbstractWalker's 'Translate' methods are not nearly as neutral and generic
as my Visitor's 'visit' methods. Just imagine you wanted to implement a ptree
serializer with it...

> * putting encoding in ptree is in my view wrong, because ptree should
>   not be concerned with how the higher levels represent types

I don't quite understand what you have in mind. Do you suggest to remove
the PtreeDeclarator's 'name' and 'type' attributes ? I don't think that
would be a good idea, as the parser is the best place to generate the
encoding, so if you don't store it inside the ptree you either have to
store it elsewhere (but still build it from within the parser), or you
have to regenerate it in a second pass (aren't we trying to get rid of
the second pass ?).

> I think parser/ptree should not know about Class. However, parser should
> provide callbacks, so that the higher levels can construct semantic
> objects, if they wish (and how they wish, and whatever objects they wish).

hmm, the parser is a *very* rich class, so adding callbacks there appears
to me to be quite heavy.

> I see callbacks implemented as calls to members of some interface class.
> Clients of parser can provide implementation of this class and register it
> with parser.

I believe this kind of flexibility is best provided by ptree visitors.

>>Where do you draw the line between 'core' and 'high
>>level' ?
> 
> 
> I can see several modules of the Core:
> 
> (1) elaboration (Class, Member, TypeInfo, Encoding, concrete walkers)
> 
> (2) Ptree hierarchy, abstract walkers
> 
> (3) parser
> 
> (4) Lexer
> 
> (5) Program/Buffer
> 
>       ,------------------->|ptree|----------.
>       |             ,----->|     |          |
>       |             |                       v
> |elaboration |--> |parser| --> |lexer| --> program/buffer
> 
> 
> Class above is in fact only part of the original Class, with all things
> petraining to reflexivity and source-to-source translation ripped out
> (e.g. all Translate... functions, statics such as SetDefaultMetaclass()
> etc.).

interesting idea. Wouldn't a 'Class' type that is stripped down as you suggest
be a good candidate for the AST API as we discussed some months ago ?
In other words, couldn't that be the basis for the second layer on top
of the PTree API ?

> And that's it for Core.
> 
> On top of that goes "Greater OpenC++", ie. classes that constitute the
> reflexivity model (things ripped out from the class above). These are the

now I'm a little confused again. What do you refer to as 'reflexivity' ?
If I strip off the 'Translate' methods, i.e. all the generating methods,
I'm still left with accessors that I would call 'reflexive'.

> classes that should be visible to the end-user of Greater OpenC++ after
> including "mop.h", all stuff related to user-defined metaclasses etc.

metaclasses are concerned about code generation, not strictly reflection,
right ? So high level reflection (AST) would be the layer on top of PTree
which would be the basis for metaclasses and code generation.

> Yet on top of that goes simple driver, which is an executable that reads
> preprocessed source from stdin and writes translated source to stdout.

yep.

> And on very top of that goes full-blown driver, that calls preprocessor,
> linker, compiler etc.

yep.

And at each of these levels we could offer both an C++ as well as a python
API, so users can choose the API that fits there needs.

>> >>'Encoding' is a 'passive' data holder (similar to std::string). In this
>> >>role I placed it into PTree nodes, to make them more type safe.
>> >
>> >
>> >Personally I don't like it, because it forces clients data structure on
>> >parser library.
>>
>>not in my case, where 'Encoding' has become part of the 'parser library'.
> 
> 
> In my view yes, because parser should not care how its client represent
> something (types). IMHO parser should provide means for clients to store

I wouldn't call 'Encoding' a 'representation'. It's very minimal and
just enough to identify types and look them up. They can then be used as
keys into a user provided table to store 'client data', though.

>>what is the 'OpenC++ license' ?
> 
> 
> See file COPYING.

that file doesn't contain a license, nor a copyright. It contains
a bunch of different licenses. That's precisely why I suggested
some months ago to reconsider this issue: it's quite a mess, and
you'll have a hard time to convince any layer about what this 'OpenC++
license' is all about. If, in contrast, you choose a well known
license, lawyers may already know them. At least there exist various
text explaining the issues, so it's much more easy.

>>How is it different from LPGL ?
> 
> 
> Does not require redistribution of linkable binary code.

right, that much I understand. Why is that a problem ? IMO it just
improves the rights of end-users: they have free access to OpenC++
even if used inside a proprietary program.

>>What makes you not want to use LGPL ?
> 
> 
> (1) LGPL is more restrictive than OpenC++ license.

It doesn't restrict *my* right as a developer at all.
It gives more freedom to the developer and the end-user.
But well, I don't want this to become a license flame war, so
let's just agree to disagree then.

> (2) Part of OpenC++ license is a license from Xerox. I am not sure if
>     and how should we distribute files under mixed license (LGPL + Xerox)
>     and what implications it has.

while I don't see any problem with the current distribution modus, I'm
working to replace all the code that Xerox has a copyright on. If you
look into the synopsis repository you'll notice that two of the main offenders
are already gone (the 'Program' and the 'Lexer' are rewritten from scratch).

Regards,
		Stefan

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Grzegorz J. <ja...@he...>]

On Wed, 15 Sep 2004, Stefan Seefeld wrote:

> Grzegorz Jakacki wrote:
>
> > Yes with exceptions:
> >
> > * ptree has to know about AbstractWalker (Visitor pattern)
>
> No :-)
> Yes, we need a Visitor, but this Visitor can be very minimal. Have a look
> into my code: Walker with all its 'Translate' methods is just an implementation
> of the Visitor.
> Your AbstractWalker's 'Translate' methods are not nearly as neutral and generic
> as my Visitor's 'visit' methods. Just imagine you wanted to implement a ptree
> serializer with it...

I don't get your point. AbstractWalker is deprived of any implementaion.
Are you refering to the fact that its method are called "Translate..." not
"Visit..." ?

> > * putting encoding in ptree is in my view wrong, because ptree should
> >   not be concerned with how the higher levels represent types
>
> I don't quite understand what you have in mind. Do you suggest to remove
> the PtreeDeclarator's 'name' and 'type' attributes ? I don't think that
> would be a good idea, as the
> parser is the best place to generate the encoding,

Why?

Parsing is the best time to generate them. But parser is not the best
place to generate them. Let the higher layer hook up to parser and
generate whatever encodings it likes.

> so if you don't store it inside the ptree you either have to
> store it elsewhere
> (but still build it from within the parser)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Why?

> , or you
> have to regenerate it in a second pass (aren't we trying to get rid of
> the second pass ?).

It does not matter when they are generated. My point is that dependence of
parser on one particular encoding used by one particular client is
improper.

> > I think parser/ptree should not know about Class. However, parser should
> > provide callbacks, so that the higher levels can construct semantic
> > objects, if they wish (and how they wish, and whatever objects they wish).
>
> hmm, the parser is a *very* rich class, so adding callbacks there appears
> to me to be quite heavy.

Look at this design:


 +----------------+                                          |
 | Parser         |                                          |
 +----------------+       +------------------------------+   |
 | ParseFoo...    |------>| AbstractParserClient         |   |
 | ParseBar...    |       +------------------------------+   |
 | RegisterClient |       | ParsedFunctionHeader(Ptree*) |   |
 | ...            |       | ...                          |   ;
 +----------------+       +------------------------------+  /
                                         A                 /
                                         |                /
        LIBRARY    .-------------------------------------'
        .---------'                      |
 .-----'  CLIENT          +--------------------------------+
/                         | ParsedFunctionHeader(Ptree* p) |
                          |                  \             |
                          +-------------------\------------+
                                               \
                                                +---
                                                | annotates p node
                                                | with client-specific
                                                | encoding

How does it make Parser class "richer"?

Besides that, Parser is "rich" only because all parsing functions are there.
It hardly provides any other functionality.

My personal view on implementing parsers is that it does not make sense to
lump Parse... methods in one big class. Dependency hell, especially when
parser state is private data and you need to add some helper functions in
parser implementation. My opinion is that is is much better to define parser
state as an object, and have Parse... functions enclosed in namespace. (But
this is just a side-note, I don't think that this refactoring of parser
would benefit the project at the current stage.)

> > I see callbacks implemented as calls to members of some interface class.
> > Clients of parser can provide implementation of this class and register it
> > with parser.
>
> I believe this kind of flexibility is best provided by ptree visitors.

I don't get you again. You said yourself that it is suboptimal to generate
encodings by visitors.

> >>Where do you draw the line between 'core' and 'high
> >>level' ?
> >
> >
> > I can see several modules of the Core:
> >
> > (1) elaboration (Class, Member, TypeInfo, Encoding, concrete walkers)
> >
> > (2) Ptree hierarchy, abstract walkers
> >
> > (3) parser
> >
> > (4) Lexer
> >
> > (5) Program/Buffer
> >
> >       ,------------------->|ptree|----------.
> >       |             ,----->|     |          |
> >       |             |                       v
> > |elaboration |--> |parser| --> |lexer| --> program/buffer
> >
> >
> > Class above is in fact only part of the original Class, with all things
> > petraining to reflexivity and source-to-source translation ripped out
> > (e.g. all Translate... functions, statics such as SetDefaultMetaclass()
> > etc.).
>
> interesting idea. Wouldn't a 'Class' type that is stripped down as you suggest
> be a good candidate for the AST API as we discussed some months ago ?
> In other words, couldn't that be the basis for the second layer on top
> of the PTree API ?

Maybe. I must give it a thought.

> > And that's it for Core.
> >
> > On top of that goes "Greater OpenC++", ie. classes that constitute the
> > reflexivity model (things ripped out from the class above). These are the
>
> now I'm a little confused again. What do you refer to as 'reflexivity' ?

Chiba invented some object model of C++ programs, where classes are
instances of a class called 'Metaclass'. This object allows for inspecting
and changing C++ programs. I call this model "reflexive", because it "models
itself", in a sense that Metaclass is treated as one of program's classes
(thus it is an instance of itself).

On the other hand, the interface provided by parser and, in my vision,
elaboration, are slightly different in nature. They provide "introspection",
i.e. means to manipulate an object model of C++ program "from the outside",
not "from the inside".

My point is that reflexivity should be implemented on top of
introspection. In current design those two things are tangled.

> If I strip off the 'Translate' methods, i.e. all the generating methods,
> I'm still left with accessors that I would call 'reflexive'.

This is what I would like to call "introspection".

> > classes that should be visible to the end-user of Greater OpenC++ after
> > including "mop.h", all stuff related to user-defined metaclasses etc.
>
> metaclasses are concerned about code generation, not strictly reflection,
> right ?

Metaclasses provide both a mechanism for specifying code transformations
*and* mechanism of introspection. I would like to untangle those two things.

Apart of that they have lots of small things related to yet higher levels,
e.g. some parts of an interface of the high-level driver are placed as
statics in Class.

> So high level reflection (AST) would be the layer on top of PTree
> which would be the basis for metaclasses and code generation.

What is "reflection (AST)" ?

> > Yet on top of that goes simple driver, which is an executable that reads
> > preprocessed source from stdin and writes translated source to stdout.
>
> yep.
>
> > And on very top of that goes full-blown driver, that calls preprocessor,
> > linker, compiler etc.
>
> yep.
>
> And at each of these levels we could offer both an C++ as well as a python
> API, so users can choose the API that fits there needs.

As for those two top-level modules, I think that scripting implementation is
the best. Why do you think C++ API is imporant (at this level)?

> >> >>'Encoding' is a 'passive' data holder (similar to std::string). In this
> >> >>role I placed it into PTree nodes, to make them more type safe.
> >> >
> >> >
> >> >Personally I don't like it, because it forces clients data structure on
> >> >parser library.
> >>
> >>not in my case, where 'Encoding' has become part of the 'parser library'.
> >
> >
> > In my view yes, because parser should not care how its client represent
> > something (types). IMHO parser should provide means for clients to store
>
> I wouldn't call 'Encoding' a 'representation'. It's very minimal and
> just enough to identify types and look them up. They can then be used as
> keys into a user provided table to store 'client data', though.

Is it specific to client or not?

> >>what is the 'OpenC++ license' ?
> >
> >
> > See file COPYING.
>
> that file doesn't contain a license,

Sorry, but this statement is clearly false. (Especially that in the next
para you write, that it contains many licenses.) What do you mean?

> nor a copyright.

Again, what do you mean?

Copyrigh is a right ("the exclusive legal right to reproduce, publish, and
sell the matter and form"), so how can this file contain "copyright" ?

> It contains a bunch of different licenses.

It contains licenses that describe how you are allowed to use the code. What
is your point?

> That's precisely why I suggested
> some months ago to reconsider this issue:

As far as I remember you reaised licensing issue in Nov 2001. I don't
remember you bringing this issue to this forum several months ago. If I am
not mistaken you were discussing it with Chiba around that time, apparently
off-line. I got one or two e-mails from this conversation kindly CC-ed by
Chiba, but, frankly, I don't reacall any discussion in this forum.

> it's quite a mess,

You are using quite derogative term in connection with collective work of
volunteers working on OpenC++. Please do not do this. Pointing out specific
flaws would be more constructive.

> and
> you'll have a hard time to convince any layer about what this 'OpenC++
> license' is all about.

Could you be more specific?

> If, in contrast, you choose a well known
> license, lawyers may already know them. At least there exist various
> text explaining the issues, so it's much more easy.

Yes, I agree.

> >>How is it different from LPGL ?
> >
> >
> > Does not require redistribution of linkable binary code.
>
> right, that much I understand. Why is that a problem ? IMO it just
> improves the rights of end-users: they have free access to OpenC++
> even if used inside a proprietary program.

It also limits the usage of OpenC++ in proprietary software (much the same
way GPL does, but to smaller extent). If OpenC++ was under LGPL I would have
not been able to use it in my project 3 years ago.

> >>What makes you not want to use LGPL ?
> >
> >
> > (1) LGPL is more restrictive than OpenC++ license.
>
> It doesn't restrict *my* right as a developer at all.

It can prevent you from using OpenC++ in proprietary software.

> It gives more freedom to the developer and the end-user.

In Stallman's sense, yes.

> But well, I don't want this to become a license flame war, so
> let's just agree to disagree then.
>
> > (2) Part of OpenC++ license is a license from Xerox. I am not sure if
> >     and how should we distribute files under mixed license (LGPL + Xerox)
> >     and what implications it has.
>
> while I don't see any problem with the current distribution modus, I'm
> working to replace all the code that Xerox has a copyright on. If you
> look into the synopsis repository you'll notice that two of the main offenders
> are already gone (the 'Program' and the 'Lexer' are rewritten from scratch).

Anyway, I believe that what you are doing is illegal. First of all, your
COPYING file does not include statements required by Xerox's part of
the original OpenC++ license, but you are still using Xerox's files.

Second, even if you phase them out, your work will still be a derivative
work of the original sources, I believe.

The risk for yourself is minimal. The risk for somebody who leverages
Synopsis in a commercial product may be substantial --- if they make a
profitable bussines out of it, Xerox can sue them for damages.

I think it is important to keep licensing issues clean. As for OpenC++
licensing let me try to find legal advice on that issue, as well as learn
more about opinions of this forum.

I hope the licensing issues can be solved, however this will most likely
take some time. Before it happens, maybe we can continue under respective
licensing terms of Synopsis and OpenC++. In particular, it would help if
you could answer the question from my last e-mail, namely would you
relicense your changes to OpenC++ code that are now part of Synopsis, so
that they can be merged into OpenC++ and distributed under terms described
in opencxx/COPYING?

Best regards
Grzegorz


##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2004 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:

>>Your AbstractWalker's 'Translate' methods are not nearly as neutral and generic
>>as my Visitor's 'visit' methods. Just imagine you wanted to implement a ptree
>>serializer with it...
> 
> 
> I don't get your point. AbstractWalker is deprived of any implementaion.
> Are you refering to the fact that its method are called "Translate..." not
> "Visit..." ?

not really. More about the signature. The methods take a ptree and return
one. Why should a generic visitor return something ? As I said, just imagine
you'd implement a Visitor (Walker) that traverses the ptree to print it.
Or one that you use to find the type of a given ptree...
The 'Translate' method is just not sufficiently generic.

>>>* putting encoding in ptree is in my view wrong, because ptree should
>>>  not be concerned with how the higher levels represent types
>>
>>I don't quite understand what you have in mind. Do you suggest to remove
>>the PtreeDeclarator's 'name' and 'type' attributes ? I don't think that
>>would be a good idea, as the
>>parser is the best place to generate the encoding,
> 
> 
> Why?
> 
> Parsing is the best time to generate them. But parser is not the best
> place to generate them. Let the higher layer hook up to parser and
> generate whatever encodings it likes.

sounds very complex.

>>so if you don't store it inside the ptree you either have to
>>store it elsewhere
>>(but still build it from within the parser)
> 
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> Why?

yeah, I get your point, but I don't agree about the usefulness of
a split there. I don't think parser is the right place to hook up
user code. As I said, Visitors to introspect the generated ptree
seems to me a much more elegant way.

>>, or you
>>have to regenerate it in a second pass (aren't we trying to get rid of
>>the second pass ?).
> 
> 
> It does not matter when they are generated. My point is that dependence of
> parser on one particular encoding used by one particular client is
> improper.

As I said, I don't see Encoding as 'one particular' representation. It's
OpenC++'s internal representation. clients can certainly hook up their
own with it.

>>hmm, the parser is a *very* rich class, so adding callbacks there appears
>>to me to be quite heavy.
> 
> 
> Look at this design:
> 
> 
>  +----------------+                                          |
>  | Parser         |                                          |
>  +----------------+       +------------------------------+   |
>  | ParseFoo...    |------>| AbstractParserClient         |   |
>  | ParseBar...    |       +------------------------------+   |
>  | RegisterClient |       | ParsedFunctionHeader(Ptree*) |   |
>  | ...            |       | ...                          |   ;
>  +----------------+       +------------------------------+  /
>                                          A                 /
>                                          |                /
>         LIBRARY    .-------------------------------------'
>         .---------'                      |
>  .-----'  CLIENT          +--------------------------------+
> /                         | ParsedFunctionHeader(Ptree* p) |
>                           |                  \             |
>                           +-------------------\------------+
>                                                \
>                                                 +---
>                                                 | annotates p node
>                                                 | with client-specific
>                                                 | encoding
> 
> How does it make Parser class "richer"?

well, you want to be able to hook up callbacks for individual ParseSomething methods,
i.e. you require the Parser to collaborate with AbstractParserClient so you can
implement it in the client.
I stand to what I said earlier: I believe you can get the same flexibility by running
a client visitor over the generated ptree and keep the parser as simple as it is now.

> Besides that, Parser is "rich" only because all parsing functions are there.
> It hardly provides any other functionality.

right, and that's quite enough.

> My personal view on implementing parsers is that it does not make sense to
> lump Parse... methods in one big class. Dependency hell, especially when
> parser state is private data and you need to add some helper functions in
> parser implementation. My opinion is that is is much better to define parser
> state as an object, and have Parse... functions enclosed in namespace. (But
> this is just a side-note, I don't think that this refactoring of parser
> would benefit the project at the current stage.)

agreed.

>>I believe this kind of flexibility is best provided by ptree visitors.
> 
> 
> I don't get you again. You said yourself that it is suboptimal to generate
> encodings by visitors.

right. I suggest to use the Visitor to glue client-side functionality to the
ptree. The Parser class should not be 'public' to clients. It should be
considered as a 'ptree factory' without much customizability. (client keywords
may be an exception, because there's no way around this.)

> As for those two top-level modules, I think that scripting implementation is
> the best. Why do you think C++ API is imporant (at this level)?

I don't think it's important, I just consider the possibility. If everybody
is happy with a python interface I'm quite happy.


>>I wouldn't call 'Encoding' a 'representation'. It's very minimal and
>>just enough to identify types and look them up. They can then be used as
>>keys into a user provided table to store 'client data', though.
> 
> 
> Is it specific to client or not?

The Encoding ? Definitely not. The client data you store in (client side) tables
for which encodings act as keys ? Definitely yes.

[...]

>>>>what is the 'OpenC++ license' ?
>>>
>>>
>>>See file COPYING.
>>
>>that file doesn't contain a license,
> 
> 
> Sorry, but this statement is clearly false. (Especially that in the next
> para you write, that it contains many licenses.) What do you mean?

sorry for the misunderstanding. I obviously didn't express myself clearly.
I didn't mean to hurt anybody's feelings with my description of the situation.
Sorry. Let me try to rephrase:

The COPYING file doesn't contain a single license (something I thought you
referred to as the 'OpenC++ license'). Instead, it contains five licenses
(sections starting with 'Permission to use, copy, ...'). Now please try to
view this from a lawyer's perspective who needs to assess the risks associated
with the use and distribution of this package. (Yes, I'v had to deal with
IP lawyers in various occasions, and I found it to be easiest when I could
*name* the license and show how it was already used elsewhere.)

In this sense I do believe the current situation is a mess. License proliferation
is a Bad Thing !

Anyways, I really want to avoid this to become a long thread, these topics
are being discussed elsewhere to death. Suffice it to say that I did choose
the LGPL for the reasons I outlined, notably what I consider to be developer
and end-user freedom. I'm not forcing this on anybody, but whoever wants to
use my code obviously has to comply with it (or else the license would be
meaningless).

Kind regards,
		Stefan

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Grzegorz J. <ja...@he...>]

On Fri, 17 Sep 2004, Stefan Seefeld wrote:

> Grzegorz Jakacki wrote:
>
> >>Your AbstractWalker's 'Translate' methods are not nearly as neutral and generic
> >>as my Visitor's 'visit' methods. Just imagine you wanted to implement a ptree
> >>serializer with it...
> >
> >
> > I don't get your point. AbstractWalker is deprived of any implementaion.
> > Are you refering to the fact that its method are called "Translate..." not
> > "Visit..." ?
>
> not really. More about the signature. The methods take a ptree and return
> one. Why should a generic visitor return something ? As I said, just imagine
> you'd implement a Visitor (Walker) that traverses the ptree to print it.
> Or one that you use to find the type of a given ptree...
> The 'Translate' method is just not sufficiently generic.

I agree.

> >>>* putting encoding in ptree is in my view wrong, because ptree should
> >>>  not be concerned with how the higher levels represent types
> >>
> >>I don't quite understand what you have in mind. Do you suggest to remove
> >>the PtreeDeclarator's 'name' and 'type' attributes ? I don't think that
> >>would be a good idea, as the
> >>parser is the best place to generate the encoding,
> >
> >
> > Why?
> >
> > Parsing is the best time to generate them. But parser is not the best
> > place to generate them. Let the higher layer hook up to parser and
> > generate whatever encodings it likes.
>
> sounds very complex.

Any specific issues?

> >>so if you don't store it inside the ptree you either have to
> >>store it elsewhere
> >>(but still build it from within the parser)
> >
> >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > Why?
>
> yeah, I get your point, but I don't agree about the usefulness of
> a split there. I don't think parser is the right place to hook up
> user code. As I said, Visitors to introspect the generated ptree
> seems to me a much more elegant way.

In general I agree. However, how should parser know if given identifier is
a template (it has to know to resolve 'a < 1 > ( 0 )' ) ? I don't want to
put whole symbol table in parser, to prevent e.g. parser depending on
Class (which would introduce cyclic dependency between, say, Class and
Ptree).

> >>, or you
> >>have to regenerate it in a second pass (aren't we trying to get rid of
> >>the second pass ?).

What do you mean by "second pass" exactly? What is the first pass then? My
impression was that the first pass is parsing, second pass is anything
that goes after that. If so, then I kind of don't understand if you are
aiming and minimizing or maximizing second pass.

> > It does not matter when they are generated. My point is that dependence of
> > parser on one particular encoding used by one particular client is
> > improper.
>
> As I said, I don't see Encoding as 'one particular' representation. It's
> OpenC++'s  internal representation. clients can certainly hook up their
> own with it.

The whole point of my effort is to make different parts of OpenC++
reusable in separation. Encoding is a class with operations specific to
its semantics, which is representing a type. This is not a generic data,
this is data specific to one of the parser clients. I am thinking of the
parser reusable also outside of the context of OpenC++ and Synopsis.

> >>hmm, the parser is a *very* rich class, so adding callbacks there appears
> >>to me to be quite heavy.
> >
> >
> > Look at this design:
> >
> >
> >  +----------------+                                          |
> >  | Parser         |                                          |
> >  +----------------+       +------------------------------+   |
> >  | ParseFoo...    |------>| AbstractParserClient         |   |
> >  | ParseBar...    |       +------------------------------+   |
> >  | RegisterClient |       | ParsedFunctionHeader(Ptree*) |   |
> >  | ...            |       | ...                          |   ;
> >  +----------------+       +------------------------------+  /
> >                                          A                 /
> >                                          |                /
> >         LIBRARY    .-------------------------------------'
> >         .---------'                      |
> >  .-----'  CLIENT          +--------------------------------+
> > /                         | ParsedFunctionHeader(Ptree* p) |
> >                           |                  \             |
> >                           +-------------------\------------+
> >                                                \
> >                                                 +---
> >                                                 | annotates p node
> >                                                 | with client-specific
> >                                                 | encoding
> >
> > How does it make Parser class "richer"?
>
> well, you want to be able to hook up callbacks for individual ParseSomething methods,
> i.e. you require the Parser to collaborate with AbstractParserClient so you can
> implement it in the client.
> I stand to what I said earlier: I believe you can get the same flexibility by running
> a client visitor over the generated ptree and keep the parser as simple as it is now.

That would be great. If you know how can parser resolve templates in such
set-up, I am all for it.

> > Besides that, Parser is "rich" only because all parsing functions are there.
> > It hardly provides any other functionality.
>
> right, and that's quite enough.

I think not, because of templates resolution. I would be glad to be proved
wrong on that.

[...]
> >>I believe this kind of flexibility is best provided by ptree visitors.
> >
> >
> > I don't get you again. You said yourself that it is suboptimal to generate
> > encodings by visitors.
>
> right. I suggest to use the Visitor to glue client-side functionality to the
> ptree. The Parser class should not be 'public' to clients. It should be
> considered as a 'ptree factory' without much customizability. (client keywords
> may be an exception, because there's no way around this.)

What is "ptree factory"? I view a parser as a black-box that gets:

  * a generator of token (Lex interface),
  * a facility to report errors (ErrorLog interface),

and outputs:

  * a Ptree (as a return value of a member function),
  * errors (as a sequence of calls to member functions of ErrorLog object).

This is how it works today. In such form it is usable for external
clients. I don't understand what you mean by "not 'public' for clients".
Could you explain?

> > As for those two top-level modules, I think that scripting implementation is
> > the best. Why do you think C++ API is imporant (at this level)?
>
> I don't think it's important, I just consider the possibility. If everybody
> is happy with a python interface I'm quite happy.

I see.

> >>I wouldn't call 'Encoding' a 'representation'. It's very minimal and
> >>just enough to identify types and look them up. They can then be used as
> >>keys into a user provided table to store 'client data', though.
> >
> >
> > Is it specific to client or not?
>
> The Encoding ? Definitely not. The client data you store in (client side) tables
> for which encodings act as keys ? Definitely yes.

Sorry, but I don't buy it. Have a look at the list of methods of Encoding
class. Parser should not be concerned about types, while Encoding is all
about types.

> [...]
>
> >>>>what is the 'OpenC++ license' ?
> >>>
> >>>
> >>>See file COPYING.
> >>
> >>that file doesn't contain a license,
> >
> >
> > Sorry, but this statement is clearly false. (Especially that in the next
> > para you write, that it contains many licenses.) What do you mean?
>
> sorry for the misunderstanding. I obviously didn't express myself clearly.
> I didn't mean to hurt anybody's feelings with my description of the situation.
> Sorry. Let me try to rephrase:
>
> The COPYING file doesn't contain a single license (something I thought you
> referred to as the 'OpenC++ license'). Instead, it contains five licenses
> (sections starting with 'Permission to use, copy, ...'). Now please try to
> view this from a lawyer's perspective who needs to assess the risks associated
> with the use and distribution of this package. (Yes, I'v had to deal with
> IP lawyers in various occasions, and I found it to be easiest when I could
> *name* the license and show how it was already used elsewhere.)
>
> In this sense I do believe the current situation is a mess. License proliferation
> is a Bad Thing !

I agree, but OpenC++ license is not just something that can be changed at
will, at once and just by myself.

> Anyways, I really want to avoid this to become a long thread, these topics
> are being discussed elsewhere to death. Suffice it to say that I did choose
> the LGPL for the reasons I outlined, notably what I consider to be developer
> and end-user freedom. I'm not forcing this on anybody, but whoever wants to
> use my code obviously has to comply with it (or else the license would be
> meaningless).

As a copyright holder you have the right to relicense under another
licenses if the license does not forbid it. You also have the right to
give the user choice of licensing terms from two (or more) licenses.

My question was if you would agree to contribute the Synopsis code that
you hold copyrights on to OpenC++ project under terms described in
opencxx/COPYING. (Especially that LGPL puts more restrictions on
redistribution than opencxx/COPYING.)

Let me research the possibilities of converting OpenC++ to some well-known
license and get back to this subject late next week.

BR
Grzegorz

##################################################################
# Grzegorz Jakacki                       Huada Electronic Design #
# Senior Engineer, CAD Dept.              1 Gaojiayuan, Chaoyang #
# tel. +86-10-64365577 x2074               Beijing 100015, China #
# Copyright (C) 2004 Grzegorz Jakacki, HED. All Rights Reserved. #
##################################################################

Re: [Opencxx-users] OpenC++ Core Lib -- prototype

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:

>>yeah, I get your point, but I don't agree about the usefulness of
>>a split there. I don't think parser is the right place to hook up
>>user code. As I said, Visitors to introspect the generated ptree
>>seems to me a much more elegant way.
> 
> 
> In general I agree. However, how should parser know if given identifier is
> a template (it has to know to resolve 'a < 1 > ( 0 )' ) ? I don't want to
> put whole symbol table in parser, to prevent e.g. parser depending on
> Class (which would introduce cyclic dependency between, say, Class and
> Ptree).

the symbol table I have in mind doesn't depend on 'Class' at all. It's
something the parser adds declarations (types, variables, consts) to
while it encounters declarators.
Then, while it parses code, it can look up the symbol table to find
out whether the symbol is known, and if so whether it refers to a type.

As you mentioned, special precautions have to be taken because method
implementations that are given in the class body references could appear
to symbols not yet defined. I believe this issue can be dealt with by
doing a first pass over a class body ignoring all function implementations
that only collects member declarations, and then a second pass (still
*inside* the same class body parse) that parses functions implementations.

That's about it !

This symbol table could be something as simple as a hierarchical map
from encodings to ptrees (declarations). This map can either be thrown
away once the parser has finished, or it can be linked into the ptree
(i.e. those ptree elements that correspond to scopes such as 'class spec',
'brace', 'for statement', etc., etc.) to ease the AST API that can be
defined on top.

>>>>, or you
>>>>have to regenerate it in a second pass (aren't we trying to get rid of
>>>>the second pass ?).
> 
> 
> What do you mean by "second pass" exactly? What is the first pass then? My

Right now the first pass is whatever happens inside Parser::rProgram().
The second pass is ClassWalker::Translate(), which constructs the symbol
lookup tables ('Environments').

> impression was that the first pass is parsing, second pass is anything
> that goes after that. If so, then I kind of don't understand if you are
> aiming and minimizing or maximizing second pass.

You are right. The problem is precisely that the second pass is a mix
of things that (IMO) don't belong together. As I said above, I acknowledge
the parsing stage can't be fully linear / sequential, but it can't simply
be a matter of passing twice over the code (once over the code and once
over a preliminary ptree).

I suggest we get rid of 'first pass' and 'second pass', but instead talk
about the 'parsing', resulting in a *correct* and fully typed ptree,
and any number of ptree transformations that belong entirely into user
(client) code.

Again, I do believe that the parser can be considered a ptree factory,
and it can work without any need for extensibility. It creates a parse
tree, which may contain some special markup elements corresponding to
user defined tokens such as 'metaclass'.
Extensibility is then achieved by letting users provide Visitors that
will run over this parse tree and hook up these markup elements with
user code (such as metaclasses).

>>>It does not matter when they are generated. My point is that dependence of
>>>parser on one particular encoding used by one particular client is
>>>improper.
>>
>>As I said, I don't see Encoding as 'one particular' representation. It's
>>OpenC++'s  internal representation. clients can certainly hook up their
>>own with it.
> 
> 
> The whole point of my effort is to make different parts of OpenC++
> reusable in separation. Encoding is a class with operations specific to
> its semantics, which is representing a type. This is not a generic data,
> this is data specific to one of the parser clients.

I don't agree. 'Type' is a concept that is part of the parse tree layer,
there's nothing client specific.

> I am thinking of the
> parser reusable also outside of the context of OpenC++ and Synopsis.

fine. Can you come up with an example where the Encoding as you see it
would be client specific ? May be we are just talking about different
things here.


> What is "ptree factory"? I view a parser as a black-box that gets:
> 
>   * a generator of token (Lex interface),
>   * a facility to report errors (ErrorLog interface),
> 
> and outputs:
> 
>   * a Ptree (as a return value of a member function),
>   * errors (as a sequence of calls to member functions of ErrorLog object).

fine with me.

> This is how it works today. In such form it is usable for external
> clients. I don't understand what you mean by "not 'public' for clients".
> Could you explain?

all but the Parser::rProgram() methods are private. All a user does is
call Parser::rProgram() to get back a parse tree. That's just what you
said above (let's ignore the error handling for now).

My impression was that you wanted to somehow 'open up' this interface
to make the Parser class extensible, such as change the behavior of
individual 'ParseSomething()' methods (even if this just means to call
into user code at appropriate places). That's what I objected to, because

* it breaks encapsulation (i.e. destabilizes the API)
* it's very complex (i.e. easy to use wrongly)
* not necessary

Again, may be there is a fundamental misunderstanding, so may be a concrete
example (use case etc.) would help.

Regards,
		Stefan

License issues (Was: Re: [Opencxx-users] OpenC++ Core Lib -- prototype)

[Stefan S. <se...@sy...>]

Grzegorz Jakacki wrote:

> As a copyright holder you have the right to relicense under another
> licenses if the license does not forbid it. You also have the right to
> give the user choice of licensing terms from two (or more) licenses.
> 
> My question was if you would agree to contribute the Synopsis code that
> you hold copyrights on to OpenC++ project under terms described in
> opencxx/COPYING. (Especially that LGPL puts more restrictions on
> redistribution than opencxx/COPYING.)

As I said in another mail, my choice of using LGPL was deliberate,
and dual-licensing with licenses similar to MIT would just dilute
the whole idea behind LGPL. That's why my answer is no.

Hoping that this doesn't get into the way of further collaboration,

		Stefan

[Opencxx-users] What's fair (Was: License issues)

[Grzegorz J. <ja...@ac...>]

Stefan Seefeld wrote:
> Grzegorz Jakacki wrote:
> 
>> As a copyright holder you have the right to relicense under another
>> licenses if the license does not forbid it. You also have the right to
>> give the user choice of licensing terms from two (or more) licenses.
>>
>> My question was if you would agree to contribute the Synopsis code that
>> you hold copyrights on to OpenC++ project under terms described in
>> opencxx/COPYING. (Especially that LGPL puts more restrictions on
>> redistribution than opencxx/COPYING.)
> 
> 
> As I said in another mail, my choice of using LGPL was deliberate,
> and dual-licensing with licenses similar to MIT would just dilute
> the whole idea behind LGPL. 

You don't need to relicense the whole Synopsis.

 > That's why my answer is no.

You used OpenC++ code for the benefit of your project, however now you 
are effectively blocking OpenC++ from using its own derivatives present 
in your project's code.

You certainly realize that OpenC++ cannot change its license overnight, 
especially that it involves negotations with Xerox Co. If you were 
really thinking about sharing code between OpenC++ and Synopsis both 
ways, and at the same time were indeed concerned about OpenC++ license, 
you could have raised the issue several months ago, when you were 
talking to Chiba about distributing his changes under LGPL. For some 
reason you decided to keep this issue off the mailing list.

I understand that you have the legal right not to relicense your changes 
under OpenC++ license and thus block their incorporation back into 
OpenC++. However in my opinion this is unfair, counterproductive and 
certainly not in the spirit of open source community. And, above all, it 
can be hardly called a "cooperation" between the projects.

Best regards
Grzegorz