Menu
▾
▴
cpptool-develop
|
From: Andre B. <and...@gm...> - 2003-12-17 21:59:29
|
Hello Baptiste, i was checking the codemodel/coderewriter sources and i hope i do understand now the general idea of it. this is enough to do the changes needed to add support for the new macro-replacement feature. However a general idea arises from looking through the sources of code model - what is the main reason to have this parallel implementation of all the statement/expression/declaration elements ? Please correct me if i'm wrong, but I have the feeling that this code makes extensions and modifications of the ast harder, since the code model needs to be synchronized manually (error prone...) wouldn't it be possible to integrate the change tracking functionality which looks very clever :-) ... into the ast ? and reuse the ast - visiting to collect the changes ? Well, my changes to the code model do not affect this ... so in next days there'll be the extension for code model working with preprocessor. bye bye André |
|
From: Baptiste L. <gai...@fr...> - 2003-12-20 12:49:31
|
The goal of the CodeModel was to provide an easy way to modify the source= . I think it partially succeed: it easier to modify the source, but 'explorin= g' the source structure is tricky (dealing with compound statement is not easy). And, as you point out, it's not easy to maintain. Having written a nearly complete c++ parser, it is clear for me that this approach is not realistic in face of the huge amount of diversity of the = c++ grammar. Those last months, I've being thinking about a new approach to this. My current idea is as follow: - have the lexer generate token for all source elements (this includes spaces and comments) - have the parser generate an AST that contains all those elements - have some kind of generic listener/tracker tracks change to the ast, an= d impacting the source code when requested. Refactoring would be implemented using highly reusable micro ast transformation (instead of a code model). For example, you would have 'insert local variable declaration', 'replace local variable initializer'= ... I'm currently focusing on getting the parser done (at this time lexer spa= ces & comments are discarded). An issue that is not solved by the above idea is macro handling. It's cle= ar that macro should be expanded for correct parsing, but we also need to be able to track the parameters passed to macro. For example, if a type is passed to a macro as a parameter, we need to be able to rename the parame= ter passed to the macro instead of its expanded instance. Also, some refactor= ing may operate on macro (rename macro...), though in this case, it might be easier to make a specific parser... Baptiste. ----- Original Message -----=20 From: "Andre Baresel" <and...@gm...> To: "CppTool Mailing List" <Cpp...@li...> Sent: Wednesday, December 17, 2003 10:59 PM Subject: [Cpptool-develop] Question for CodeModel... > Hello Baptiste, > > i was checking the codemodel/coderewriter sources and i hope i do > understand now the general > idea of it. this is enough to do the changes needed to add support for > the new macro-replacement > feature. However a general idea arises from looking through the sources > of code model - what is > the main reason to have this parallel implementation of all the > statement/expression/declaration > elements ? Please correct me if i'm wrong, but I have the feeling that > this code makes extensions > and modifications of the ast harder, since the code model needs to be > synchronized manually > (error prone...) wouldn't it be possible to integrate the change > tracking functionality which looks > very clever :-) ... into the ast ? and reuse the ast - visiting to > collect the changes ? > > Well, my changes to the code model do not affect this ... so in next > days there'll be the extension > for code model working with preprocessor. > > bye bye > Andr=E9 |
|
From: Andre B. <and...@gm...> - 2003-12-20 23:43:45
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <title></title> </head> <body> I'm fine with these ideas - this tracker concept will be the thing that can be taken from<br> the codemodel. The "micro ast transformation" is something to have in background when<br> starting with the real big refactoring operations, i agree.<br> <br> About that macro / preprocessing stuff.<br> <pre wrap="">> An issue that is not solved by the above idea is macro handling. It's clear > that macro should be expanded for correct parsing, but we also need to be > able to track the parameters passed to macro. For example, if a type is > passed to a macro as a parameter, we need to be able to rename the parameter > passed to the macro instead of its expanded instance. Also, some refactoring > may operate on macro (rename macro...), though in this case, it might be > easier to make a specific parser... Well, I think there's no real chance to get arround to do this preprocessing before parsing. However, a possible solution could be - and is implemented by the current preprocessor class - to save all changes done during preprocessing and use this information for later textual transformation of the refactoring. So that the needed transformations recognized in the preprocessed code can be mapped to the locations in the original code. e.g. A macro replacement changes the offset of all characters coming after the macro replacement area. At the moment only simple replacements can be handled. However, I see the chance to get this working even with the '#include' directive and parameterized macros - so that the preprocessor really includes the given header file (transformations are than mapped to the different files by using preprocess information). One hard point would be 'conflicts' where a transformation hits a macro area - here a some clever checking of preprocess information is needed. I have a good feeling that this can work even to track the parameters passed to macro. I'm currently working on this "include" thing. And of course refactoring of the current preprocess code ... and after this parameterized macros. I have added open points in the sourceforge RFE list just for your information. -- have a nice day André </pre> <blockquote type="cite" cite="mid02d001c3c6f9$00230fc0$67c2c3d4@gaia"> <pre wrap=""><!----></pre> </blockquote> <br> </body> </html> |
|
From: Baptiste L. <gai...@fr...> - 2004-02-09 22:14:02
|
Some news about the C++ parser I'm working on...
I've ported the core parser framework to C++. I've also decided to =
abstract the grammar away in a text file instead of executable code. =
This greatly improve grammar readability and avoid some mistakes that =
can easily be made when writing grammar in C++ (or python for the =
matter). This will also make the grammar much less sensible to change of =
the actual implementation of the parser (which I can see that evolving a =
lot).
I'll tackle the actual port the C++ grammar written in python to C++ =
soon. I'm still undecided on weither I should also abstract the unit =
test into text file or having them in code...
I have implemented a proof of concept of 'update the AST and get the =
modified source'. It handles insertion and removal of node (which can be =
composite node). It currently assumes the a depth first traversal of the =
ast tree will process the token node is the 'source' order. Giving a =
quick look at the python unit test, I didn't see this as being a false =
assertion (It can be worked around if need be, but it would complicate =
things a lot).
If a preprocessor was postprocessing the lexer tokens, it could tag =
macros parameter occurrence as special 'token'. That way, when renaming =
that token we could detect that it is an actual macro parameter =
occurrence and renaming should be done at the macro call. This would =
allow dealing with a lot of macros occurence (for example renaming a =
method present in a CPPUNIT_TEST macro).
=20
A snapshot of the sources is available at:
http://gaiacrtn.free.fr/backup/CppParser-20040209-22h30.rar
It includes VC++ 2003 projects, cppunit 2, boost and precompiled =
regex librairies for VC++ 2003.
Baptiste.
|
|
From: Baptiste L. <gai...@fr...> - 2004-02-14 10:15:27
|
Well, I've ported the expression parsing. After a rough start (some bug = that went unoticed before in the grammar builder), I'm now progressing = quickly (about 25% of the port is done). The most recent version can be found in: http://gaiacrtn.free.fr/backup/ just peek the most recent version of cppparser. Interesting stuff to look at are in examples/parser. cpp_grammar.txt: the grammar cppparsertest.cpp: the unit test for the grammar nodetracker.cpp/nodetrackertest.cpp: proof of concept for ast = modification that automatically update the source parser.h/parsercontext.h: parsing framework commandstream.cpp/commandstreamtest.cpp: converts the parser production = into an ast. Baptiste. ----- Original Message -----=20 From: Baptiste Lepilleur=20 To: CppTool Mailing List=20 Sent: Monday, February 09, 2004 11:13 PM Subject: [Cpptool-develop] C++ parser progress... Some news about the C++ parser I'm working on... I've ported the core parser framework to C++. I've also decided to = abstract the grammar away in a text file instead of executable code. = This greatly improve grammar readability and avoid some mistakes that = can easily be made when writing grammar in C++ (or python for the = matter). This will also make the grammar much less sensible to change of = the actual implementation of the parser (which I can see that evolving a = lot). I'll tackle the actual port the C++ grammar written in python to = C++ soon. I'm still undecided on weither I should also abstract the unit = test into text file or having them in code... I have implemented a proof of concept of 'update the AST and get = the modified source'. It handles insertion and removal of node (which = can be composite node). It currently assumes the a depth first traversal = of the ast tree will process the token node is the 'source' order. = Giving a quick look at the python unit test, I didn't see this as being = a false assertion (It can be worked around if need be, but it would = complicate things a lot). If a preprocessor was postprocessing the lexer tokens, it could = tag macros parameter occurrence as special 'token'. That way, when = renaming that token we could detect that it is an actual macro parameter = occurrence and renaming should be done at the macro call. This would = allow dealing with a lot of macros occurence (for example renaming a = method present in a CPPUNIT_TEST macro). =20 A snapshot of the sources is available at: http://gaiacrtn.free.fr/backup/CppParser-20040209-22h30.rar It includes VC++ 2003 projects, cppunit 2, boost and precompiled = regex librairies for VC++ 2003. Baptiste. |
|
From: Baptiste L. <gai...@fr...> - 2004-02-14 20:56:25
|
Port finished and error recovery added. Uploaded at the same place. I'll try to enhance the template support before moving back to the = nodetracker. Baptiste. ----- Original Message -----=20 From: Baptiste Lepilleur=20 To: CppTool Mailing List=20 Sent: Saturday, February 14, 2004 11:13 AM Subject: Re: [Cpptool-develop] C++ parser progress... Well, I've ported the expression parsing. After a rough start (some = bug that went unoticed before in the grammar builder), I'm now = progressing quickly (about 25% of the port is done). The most recent version can be found in: http://gaiacrtn.free.fr/backup/ just peek the most recent version of cppparser. Interesting stuff to look at are in examples/parser. cpp_grammar.txt: the grammar cppparsertest.cpp: the unit test for the grammar nodetracker.cpp/nodetrackertest.cpp: proof of concept for ast = modification that automatically update the source parser.h/parsercontext.h: parsing framework commandstream.cpp/commandstreamtest.cpp: converts the parser = production into an ast. Baptiste. ----- Original Message -----=20 From: Baptiste Lepilleur=20 To: CppTool Mailing List=20 Sent: Monday, February 09, 2004 11:13 PM Subject: [Cpptool-develop] C++ parser progress... Some news about the C++ parser I'm working on... I've ported the core parser framework to C++. I've also decided = to abstract the grammar away in a text file instead of executable code. = This greatly improve grammar readability and avoid some mistakes that = can easily be made when writing grammar in C++ (or python for the = matter). This will also make the grammar much less sensible to change of = the actual implementation of the parser (which I can see that evolving a = lot). I'll tackle the actual port the C++ grammar written in python to = C++ soon. I'm still undecided on weither I should also abstract the unit = test into text file or having them in code... I have implemented a proof of concept of 'update the AST and get = the modified source'. It handles insertion and removal of node (which = can be composite node). It currently assumes the a depth first traversal = of the ast tree will process the token node is the 'source' order. = Giving a quick look at the python unit test, I didn't see this as being = a false assertion (It can be worked around if need be, but it would = complicate things a lot). If a preprocessor was postprocessing the lexer tokens, it could = tag macros parameter occurrence as special 'token'. That way, when = renaming that token we could detect that it is an actual macro parameter = occurrence and renaming should be done at the macro call. This would = allow dealing with a lot of macros occurence (for example renaming a = method present in a CPPUNIT_TEST macro). =20 A snapshot of the sources is available at: http://gaiacrtn.free.fr/backup/CppParser-20040209-22h30.rar It includes VC++ 2003 projects, cppunit 2, boost and precompiled = regex librairies for VC++ 2003. Baptiste. |
|
From: Baptiste L. <gai...@fr...> - 2004-02-15 21:44:59
|
Added some basic template support and fixed a serious bug (pointer were =
not allowed as function return type). It is now possible to have =
template declaration (class, function).
The CppParserTest::testTranslationUnit() test is a good place to start =
playing with the parser. It contains a few code snippet. If you want to =
see the produced ast, change the assertion macro to =
CPPPARSER_ASSERT_TREE and add a dummy testNode("").
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Saturday, February 14, 2004 9:53 PM
Subject: Re: [Cpptool-develop] C++ parser progress...
Port finished and error recovery added. Uploaded at the same place.
I'll try to enhance the template support before moving back to the =
nodetracker.
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Saturday, February 14, 2004 11:13 AM
Subject: Re: [Cpptool-develop] C++ parser progress...
Well, I've ported the expression parsing. After a rough start (some =
bug that went unoticed before in the grammar builder), I'm now =
progressing quickly (about 25% of the port is done).
The most recent version can be found in:
http://gaiacrtn.free.fr/backup/
just peek the most recent version of cppparser.
Interesting stuff to look at are in examples/parser.
cpp_grammar.txt: the grammar
cppparsertest.cpp: the unit test for the grammar
nodetracker.cpp/nodetrackertest.cpp: proof of concept for ast =
modification that automatically update the source
parser.h/parsercontext.h: parsing framework
commandstream.cpp/commandstreamtest.cpp: converts the parser =
production into an ast.
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Monday, February 09, 2004 11:13 PM
Subject: [Cpptool-develop] C++ parser progress...
Some news about the C++ parser I'm working on...
I've ported the core parser framework to C++. I've also =
decided to abstract the grammar away in a text file instead of =
executable code. This greatly improve grammar readability and avoid some =
mistakes that can easily be made when writing grammar in C++ (or python =
for the matter). This will also make the grammar much less sensible to =
change of the actual implementation of the parser (which I can see that =
evolving a lot).
I'll tackle the actual port the C++ grammar written in python =
to C++ soon. I'm still undecided on weither I should also abstract the =
unit test into text file or having them in code...
I have implemented a proof of concept of 'update the AST and =
get the modified source'. It handles insertion and removal of node =
(which can be composite node). It currently assumes the a depth first =
traversal of the ast tree will process the token node is the 'source' =
order. Giving a quick look at the python unit test, I didn't see this as =
being a false assertion (It can be worked around if need be, but it =
would complicate things a lot).
If a preprocessor was postprocessing the lexer tokens, it =
could tag macros parameter occurrence as special 'token'. That way, when =
renaming that token we could detect that it is an actual macro parameter =
occurrence and renaming should be done at the macro call. This would =
allow dealing with a lot of macros occurence (for example renaming a =
method present in a CPPUNIT_TEST macro).
=20
A snapshot of the sources is available at:
http://gaiacrtn.free.fr/backup/CppParser-20040209-22h30.rar
It includes VC++ 2003 projects, cppunit 2, boost and =
precompiled regex librairies for VC++ 2003.
Baptiste.
|
|
From: Baptiste L. <gai...@fr...> - 2004-02-24 20:24:36
|
I added the project and depdency lib for VC++ 6.0. Did some code clean =
up and removed usage of token text inside the parser framework =
(TokenType is now a composite of token category and symbol id). I'll =
probably try to do a proof of concept for the preprocessor and renaming =
of macro parameters.
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Sunday, February 15, 2004 10:42 PM
Subject: Re: [Cpptool-develop] C++ parser progress...
Added some basic template support and fixed a serious bug (pointer =
were not allowed as function return type). It is now possible to have =
template declaration (class, function).
The CppParserTest::testTranslationUnit() test is a good place to start =
playing with the parser. It contains a few code snippet. If you want to =
see the produced ast, change the assertion macro to =
CPPPARSER_ASSERT_TREE and add a dummy testNode("").
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Saturday, February 14, 2004 9:53 PM
Subject: Re: [Cpptool-develop] C++ parser progress...
Port finished and error recovery added. Uploaded at the same place.
I'll try to enhance the template support before moving back to the =
nodetracker.
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Saturday, February 14, 2004 11:13 AM
Subject: Re: [Cpptool-develop] C++ parser progress...
Well, I've ported the expression parsing. After a rough start =
(some bug that went unoticed before in the grammar builder), I'm now =
progressing quickly (about 25% of the port is done).
The most recent version can be found in:
http://gaiacrtn.free.fr/backup/
just peek the most recent version of cppparser.
Interesting stuff to look at are in examples/parser.
cpp_grammar.txt: the grammar
cppparsertest.cpp: the unit test for the grammar
nodetracker.cpp/nodetrackertest.cpp: proof of concept for ast =
modification that automatically update the source
parser.h/parsercontext.h: parsing framework
commandstream.cpp/commandstreamtest.cpp: converts the parser =
production into an ast.
Baptiste.
----- Original Message -----=20
From: Baptiste Lepilleur=20
To: CppTool Mailing List=20
Sent: Monday, February 09, 2004 11:13 PM
Subject: [Cpptool-develop] C++ parser progress...
Some news about the C++ parser I'm working on...
I've ported the core parser framework to C++. I've also =
decided to abstract the grammar away in a text file instead of =
executable code. This greatly improve grammar readability and avoid some =
mistakes that can easily be made when writing grammar in C++ (or python =
for the matter). This will also make the grammar much less sensible to =
change of the actual implementation of the parser (which I can see that =
evolving a lot).
I'll tackle the actual port the C++ grammar written in =
python to C++ soon. I'm still undecided on weither I should also =
abstract the unit test into text file or having them in code...
I have implemented a proof of concept of 'update the AST and =
get the modified source'. It handles insertion and removal of node =
(which can be composite node). It currently assumes the a depth first =
traversal of the ast tree will process the token node is the 'source' =
order. Giving a quick look at the python unit test, I didn't see this as =
being a false assertion (It can be worked around if need be, but it =
would complicate things a lot).
If a preprocessor was postprocessing the lexer tokens, it =
could tag macros parameter occurrence as special 'token'. That way, when =
renaming that token we could detect that it is an actual macro parameter =
occurrence and renaming should be done at the macro call. This would =
allow dealing with a lot of macros occurence (for example renaming a =
method present in a CPPUNIT_TEST macro).
=20
A snapshot of the sources is available at:
http://gaiacrtn.free.fr/backup/CppParser-20040209-22h30.rar
It includes VC++ 2003 projects, cppunit 2, boost and =
precompiled regex librairies for VC++ 2003.
Baptiste.
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X