ctool-develop

[Ctool-develop] little fixes

[Stefan S. <se...@sy...>]

hi there,

trying to compile the ctool code with gcc 3.2, I run into two
minor issues with missing 'std::' prefixes and a redundant
default initializer. The patches are appended. (the diff is
against the latest release (2.11), as sf.net's cvs service
isn't working properly (sigh).

Regards,
		Stefan

Re: [Ctool-develop] little fixes

[Steven S. <ste...@cs...>]

Stefan Seefeld wrote:
> +++ src/decl.cpp	2003-07-31 11:09:43.000000000 -0400
> @@ -128,7 +128,7 @@
[...]
> -			cout << " ";
> +			std::cout << " ";

Nope, should be 'out' not 'cout' or 'std::cout' otherwise when you're
outputting to a file the spaces go to the wrong place.

This bug has been noted before and has been fixed. It's ID 610027.
Go to the bugs list on SourceForge and select Status: Any instead of
the default status Open.

I can't comment on your other fixes. I think my version of gcc is so
old it lets me get away without the std::.

While we're on the subject of patches ... For a project I was writing
I needed a way of inserting a statement before another given
statement. The only function I could see for insertion inserted after
a given statement, so I wrote a function to do what I wanted. I've left
the linked list singly linked so the insertion time is fairly poor, but
it was fine for my implementation.

Anyway, isn't writing your own linked list code a bit passé nowadays,
shouldn't we be using the STL. (No, I'm not volunteering).

I've attached the patch to this e-mail rather than putting it inline
as something in my mail system appears to be messing around with
spaces at the start of lines. This is kind of unfortunate for diffs.

	- Steven
-- 


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: [Ctool-develop] little fixes

[Shaun F. <fli...@so...>]

Steven,

I've committed your insertBefore() method to CVS.

As for your comment about vectors, cTool was originally written in C=20
(as the poorly named "CTree", which I now know is also a database=20
product).  With version 2.0, I changed over to C++ so a class hierarchy=20=

could be used to clean up the resulting AST - I was not looking to=20
rewrite from scratch using C++.

Thanks,
Shaun

On Thursday, July 31, 2003, at 11:29  AM, Steven Singer wrote:

> Stefan Seefeld wrote:
>> +++ src/decl.cpp	2003-07-31 11:09:43.000000000 -0400
>> @@ -128,7 +128,7 @@
> [...]
>> -			cout << " ";
>> +			std::cout << " ";
>
> Nope, should be 'out' not 'cout' or 'std::cout' otherwise when you're
> outputting to a file the spaces go to the wrong place.
>
> This bug has been noted before and has been fixed. It's ID 610027.
> Go to the bugs list on SourceForge and select Status: Any instead of
> the default status Open.
>
> I can't comment on your other fixes. I think my version of gcc is so
> old it lets me get away without the std::.
>
> While we're on the subject of patches ... For a project I was writing
> I needed a way of inserting a statement before another given
> statement. The only function I could see for insertion inserted after
> a given statement, so I wrote a function to do what I wanted. I've =
left
> the linked list singly linked so the insertion time is fairly poor, =
but
> it was fine for my implementation.
>
> Anyway, isn't writing your own linked list code a bit pass=E9 =
nowadays,
> shouldn't we be using the STL. (No, I'm not volunteering).
>
> I've attached the patch to this e-mail rather than putting it inline
> as something in my mail system appears to be messing around with
> spaces at the start of lines. This is kind of unfortunate for diffs.
>
> 	- Steven
> --=20
>

Re: [Ctool-develop] little fixes

[Steven S. <ste...@cs...>]

Shaun Flisakowski wrote:
> As for your comment about vectors, cTool was originally written in C (as 
> the poorly named "CTree", which I now know is also a database product).  
> With version 2.0, I changed over to C++ so a class hierarchy could be 
> used to clean up the resulting AST - I was not looking to rewrite from 
> scratch using C++.

I know about CTree. I've used that too.

CTool is a huge improvement in usability over CTree, the interfaces are
much cleaner. The latest program I tried to write I started with CTree
because that's what I had handy and got into a real mess. When I
changed to CTool, all the complexities disappeared and I could spend
my time concentrating on deciding what I wanted to do instead of how to
do it.

However, CTool is still showing its C roots. Too much of the internal
workings of the data structure are exposed. This makes it very
difficult to change any detail of the implementation.

For example, suppose I had needed a fast insertBefore method. This
would have required updating the Statement data structure to be
doubly linked. Since the next pointers are exposed, I'd have no control
over what external code is using them to manipulate the data structure.
If the manipulation did not take into account the new pointers I would
have needed to add, then the result would have been a complete mess.

Hiding all the implementation details will allow future changes to
the internal workings. At the moment you don't even have to change
the implementation, just hide all the variables and work out what
actions are needed on the objects.

Hiding the variables, however, will break any existing program that
relies on the current implementation. It would have to be reserved
for the next major version.

Do you have a good idea what sort of tools people are building with
CTool amd hence what manipulation options they need?

	- Steven
-- 



**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: [Ctool-develop] little fixes

[Shaun F. <fli...@so...>]

On Thursday, July 31, 2003, at 02:44  PM, Steven Singer wrote:

> Shaun Flisakowski wrote:
>> As for your comment about vectors, cTool was originally written in C 
>> (as the poorly named "CTree", which I now know is also a database 
>> product).  With version 2.0, I changed over to C++ so a class 
>> hierarchy could be used to clean up the resulting AST - I was not 
>> looking to rewrite from scratch using C++.
>
> I know about CTree. I've used that too.
>
> CTool is a huge improvement in usability over CTree, the interfaces are
> much cleaner. The latest program I tried to write I started with CTree
> because that's what I had handy and got into a real mess. When I
> changed to CTool, all the complexities disappeared and I could spend
> my time concentrating on deciding what I wanted to do instead of how to
> do it.

I'm glad to hear that.  I lost a few users in the switch, some just 
stuck with their CTree code, one person said they wouldn't use it now 
that it could no longer parse itself.

> However, CTool is still showing its C roots. Too much of the internal
> workings of the data structure are exposed. This makes it very
> difficult to change any detail of the implementation.

While I agree these things should be hidden, I'd want to do that very 
slowly - I'd be loathe to break people's existing solutions.  The first 
step would have to be accessors for all the classes, then after a few 
more releases pass by, start making the members private.  Where I work, 
we have a define:
	#define	privileged		public

that we use to indicate this is the direction this class (which was 
once C, which was once pascal) is headed.

My preference would be change a particular implementation as needed (to 
be vector or list), perhaps making it private at that point.  This way, 
only users modifying that particular item would be affected.

> For example, suppose I had needed a fast insertBefore method. This
> would have required updating the Statement data structure to be
> doubly linked. Since the next pointers are exposed, I'd have no control
> over what external code is using them to manipulate the data structure.
> If the manipulation did not take into account the new pointers I would
> have needed to add, then the result would have been a complete mess.
>
> Hiding all the implementation details will allow future changes to
> the internal workings. At the moment you don't even have to change
> the implementation, just hide all the variables and work out what
> actions are needed on the objects.
>
> Hiding the variables, however, will break any existing program that
> relies on the current implementation. It would have to be reserved
> for the next major version.

I agree, but as I mentioned I think API changes need to come in phases; 
with a plan that allows for slow migration among the user base - I'm 
not looking to make any new enemies, I'm full up.

> Do you have a good idea what sort of tools people are building with
> CTool amd hence what manipulation options they need?
>
> 	- Steven
> -- 

At the point I switched over to using sourceforge, there were about 50 
people on my mailing list.  Of the ones that asked me questions, many 
were graduate students using ctool in some project.  These are the 
low-end solutions, consisting of: parse, modify AST, and print modified 
output.  One example that comes to mind was someone inserting a 
function call at the beginning of each for/while loop block.

High-end use is using ctool as a front-end to your own compiler (for 
some embedded system, etc).  For example, Cisco was using ctool in 
their testing dept to do something or other. I went over to the company 
(a few blocks from where I work) to talk to them about it.  I met with 
a group of people, one who had already used ctool in a prototype of 
some new testing system - the specifics are their business alone, but 
they were willing to sink some serious resources into their project.  
Assuming they currently have 100K LOC or more in their project, I would 
imagine some unhappiness with a large change to the API unless backward 
compatibility is kept in mind.

Thanks,
Shaun

Re: [Ctool-develop] little fixes

[Steven S. <ste...@cs...>]

Shaun Flisakowski wrote:
 > I'm glad to hear that.  I lost a few users in the switch, some just
 > stuck with their CTree code, one person said they wouldn't use it now
 > that it could no longer parse itself.

I can imagine that if you're trying to write a compiler then it's
important that it be able to compile itself.

My old code written with CTree I've left alone. After all, it works
so there's no need to break it.

 > I agree, but as I mentioned I think API changes need to come in phases;
 > with a plan that allows for slow migration among the user base - I'm not
 > looking to make any new enemies, I'm full up.

I agree. Pissing people off is not a good plan.

However, slowly migrating an API is probably not the best solution.
The people who are using the old API will complain as soon as it moves
at all, the people who want a new API will complain that things aren't
moving quickly enough and everybody else will be coding to a moving
target.

Probably a better approach is to take some time to decide on a new,
clean, API - the one you would have had if you'd started from scratch
knowing all you do now. Then, make the new API available as a wrapper
to the existing API. This would allow existing users to keep using the
old API and new users to immediately write to the new API. The old API
could initially be marked as deprecated but still available.

Then, only as internal changes rendered the old API invalid would
people be forced to move.

This method has the advantage of delaying pissing people off until
absolutely necessary and also providing a nice clean API not
compromised by legacy interfaces.

Another approach would be to have a new set of data structures with
a new API and provide import and export facilities to the old data
structures. New features need be added only to the new data
structures but people could avoid the conversion step if they relied
on implementation details of the old structures.

However, this is a big job and should only be undertaken if the
resources are available to see it through to the end. Since I'm not
offering you any of my time, feel free to tell me to go away.

	- Steven
-- 




**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: [Ctool-develop] little fixes

[Shaun F. <fli...@so...>]

Steven,

It sounds to me like we're basically in agreement here.  My plan going 
forward is to, for example, take the time to add accessors to all 
classes I work on, without making the members private.  So, the old API 
continues to work while a better one is available.

Yes, I've been a bit short on time and the necessary motivation lately 
to do cleanup like this.  I was also having some difficultly with the 
early (Mac) versions of gcc 3.0 that led me to spend more time on my 
Metreowerks-using projects. :^)

I think the next thing needing doing has to be improved handling of gcc 
extensions, which is what I was working on during my last major ctool 
development push.  Not being able to cleanly use standard include files 
is quite the bummer.

Thanks,
Shaun


On Friday, August 1, 2003, at 12:03  PM, Steven Singer wrote:

> Shaun Flisakowski wrote:
> > I'm glad to hear that.  I lost a few users in the switch, some just
> > stuck with their CTree code, one person said they wouldn't use it now
> > that it could no longer parse itself.
>
> I can imagine that if you're trying to write a compiler then it's
> important that it be able to compile itself.
>
> My old code written with CTree I've left alone. After all, it works
> so there's no need to break it.
>
> > I agree, but as I mentioned I think API changes need to come in 
> phases;
> > with a plan that allows for slow migration among the user base - I'm 
> not
> > looking to make any new enemies, I'm full up.
>
> I agree. Pissing people off is not a good plan.
>
> However, slowly migrating an API is probably not the best solution.
> The people who are using the old API will complain as soon as it moves
> at all, the people who want a new API will complain that things aren't
> moving quickly enough and everybody else will be coding to a moving
> target.
>
> Probably a better approach is to take some time to decide on a new,
> clean, API - the one you would have had if you'd started from scratch
> knowing all you do now. Then, make the new API available as a wrapper
> to the existing API. This would allow existing users to keep using the
> old API and new users to immediately write to the new API. The old API
> could initially be marked as deprecated but still available.
>
> Then, only as internal changes rendered the old API invalid would
> people be forced to move.
>
> This method has the advantage of delaying pissing people off until
> absolutely necessary and also providing a nice clean API not
> compromised by legacy interfaces.
>
> Another approach would be to have a new set of data structures with
> a new API and provide import and export facilities to the old data
> structures. New features need be added only to the new data
> structures but people could avoid the conversion step if they relied
> on implementation details of the old structures.
>
> However, this is a big job and should only be undertaken if the
> resources are available to see it through to the end. Since I'm not
> offering you any of my time, feel free to tell me to go away.
>
> 	- Steven
> -- 
>
>
>
>
> **********************************************************************
> The information transmitted is intended only for the person or
> entity to which it is addressed and may contain confidential and/or 
> privileged material. Any review, retransmission, dissemination or 
> other use of, or
> taking of any action in reliance upon, this information by persons or 
> entities other than the intended recipient is prohibited.  If you 
> received this in error, please contact the sender and delete the 
> material from any computer.
> **********************************************************************
>

Re: [Ctool-develop] little fixes

[Steven S. <ste...@cs...>]

Shaun Flisakowski wrote:
> It sounds to me like we're basically in agreement here.  My plan going 
> forward is to, for example, take the time to add accessors to all 
> classes I work on, without making the members private.  So, the old API 
> continues to work while a better one is available.

I'm not convinced that accessors gain us much. For example, I'm not sure
that the Statement class should include a next pointer at all. I think
that a Statement should be just a single statement and that there should
be a StatementList class. That sort of change will break the API in a
manner that can't be fixed by accessor functions (or would leave a
permanent wart on the side of the code).

The more I think about this, the more I think we should just design an
API from scratch and intially offer that as a second clean interface.

This is an improvement over making minor changes to the existing API as
it would get rid of all the intermediate stages where we had APIs that
were neither the new or the old API. We could make a little playpen
where we're free to add and remove parameters, functions or even
entire classes wholesale without annoying anyone. We can also do minor
bits of tidying up which make the new API neater but would be
completely unjustified if backward compatability needed to be
maintained. For example, spelling could be made consistent (InclStemnt
could become IncludeStatement).

When we finish playing and have a design we're happy with, we can then
worry about migrating users.

I think we should think about this some more before making any
changes. We can let the design go through several iterations before
we even think about coding it (or coding any more than example
framgents).

I have some ideas I need to think through. I suspect I'll reject a
lot of them.

> Yes, I've been a bit short on time and the necessary motivation lately 
> to do cleanup like this.  I was also having some difficultly with the 
> early (Mac) versions of gcc 3.0 that led me to spend more time on my 
> Metreowerks-using projects. :^)

I know the feeling. When things work, there isn't a real incentive to
tidy them. There are always more important projects to work on (like
the work I'm being paid for).

> I think the next thing needing doing has to be improved handling of gcc 
> extensions, which is what I was working on during my last major ctool 
> development push.  Not being able to cleanly use standard include files 
> is quite the bummer.

Agreed. This is more important. Also, this work is independent of
the API changes.

	- Steven
-- 



**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: [Ctool-develop] little fixes

[Stefan S. <se...@sy...>]

Steven Singer wrote:
> Shaun Flisakowski wrote:
> 
>> It sounds to me like we're basically in agreement here.  My plan going 
>> forward is to, for example, take the time to add accessors to all 
>> classes I work on, without making the members private.  So, the old 
>> API continues to work while a better one is available.
> 
> 
> I'm not convinced that accessors gain us much.

The synopsis AST API (http://synopsis.sf.net) uses accessors for all
types to access the members. Yet I'm pondering to replace them all with
simple (public) variables. Of course, we are talking python here, so
the object model isn't quite the same as in C++, but the basic idea is
similar:

The AST nodes seem to me to be basically data, with no or minimal behavior.
Especially by means of the visitor pattern all behavior could be factored out
into external classes, so the only bit of polymorphism that has to remain
in the AST nodes is the dispatching mechanism, i.e. the 'accept()' implementation.

By the way, my main interest into ctool is to hook the AST up with synopsis.
Synopsis was originally developed as a source code documentation tool similar
to doxygen, but it is quite a bit more powerful. For example another developer
added cross reference support to it similar to lxr (http://lxr.linux.no/) so
you can inspect not only declarations, but any expressions. Right now this
tool (we call it sxr) is only available with the C++ parser, but I'd like to
provide the same with C (using ctool). For examples, have a look at
http://synopsis.sourceforge.net/demo/index.html...

I wonder what you think of pushing that idea even further, i.e. making synopsis
(python) modules that are basically scripting frontends to the ctool library
as a whole, i.e. also stuff like call graph inspection.

Regards,
		Stefan