CEDET

Thank you for such a wonderful set of tools. 

I have recently started using CEDET with boost and have narrowed down a 
parsing "problem." 

The problem is that #ifdef pre-parsing is not evaluating correctly. 

An example of this type of statement is as follows:
#if defined(BOOST_NO_MEMBER_TEMPLATES) && !defined(BOOST_MSVC6_MEMBER_TEMPLATES)


I am using emacs 23:
"GNU Emacs 23.1.50.1 (x86_64-pc-linux-gnu, GTK+ Version 2.18.0)
 of 2009-09-27 on crested, modified by Debian"
with CEDET from the cvs repository at cedet.cvs.sourceforge.net)

I executed the following emacs-lisp after starting  
emacs via 'emacs --no-init'):

,----
| (if (not (featurep 'cedet))
|     (load-file "~/.elisp/cedet/common/cedet.elc"))
| (global-ede-mode t)
| (require 'ede-cpp-root)
| (require 'semantic-gcc)
| (ede-cpp-root-project "cedet-test"
| 		      :file "~/src/cedet-test/test.cpp")
| (semantic-load-enable-excessive-code-helpers)
| (require 'semantic-ia)
`----

Here is the contents of test.cpp:
,----
| #undef BOOST_NO_MEMBER_TEMPLATES
| #undef BOOST_MSVC6_MEMBER_TEMPLATES
| 
| 
| #if defined(BOOST_NO_MEMBER_TEMPLATES)
| int mymethod(int myint) {
|    return myint++;
| }
| #else
| // This is what should be recognized and it is
| int  mymethod(int myint) {
|    return myint + 2;
| }
| #endif
| 
| 
| #if !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
| // This is what should be recognized an it is
| void mysecondmethod(int) {
|   return;
| }
| #else
| void mysecondmethod(unsigned int) {
|   return;
| } 
| #endif
| 
| // So BOOST_NO_MEMBER_TEMPLATES is NOT defined
| // and BOOST_MSVC6_MEMBER_TEMPLATES is NOT defined
| 
| // so this is #if false && true --> 
| //      should yield false, but it yields true
| #if defined(BOOST_NO_MEMBER_TEMPLATES) && !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
| // This should not be the value that is parsed but it is...
| void definedBOOST_NO_MEMBER_TEMPLATES_AND_bang_BOOST_MSVC6()
| {
|    return;
| }
| #else
| // This is what should be parsed but it is not
| void not_definedBOOST_NO_MEMBER_TEMPLATES_AND_bang_BOOST_MSVC6()
| {
|    return;
| }
| #endif
|
| // (senator-force-refresh)
`----


At first glance (and I mean glance), it looks like the parser is evaluating 
if "BOOST_MSVC6_MEMBER_TEMPLATES && !definedBOOST_NO_MEMBER_TEMPLATES_AND_bang_BOOST_MSVC6)" is defined
rather than seeing this as two separate conditions. Or the lexer is not dealing with the && or || in the 
#if xxxxxxxxxx && xxxxxxxx type of pre-processing directive. 

Any help with this would be greatly appreciated. 

Thanks, 
;-Brian

-- 
Brian J. Carlson

Hi Brian,

   Your assessment is correct.  The preprocessor in CEDET for C doesn't 
do much for complex expressions.  It's been in there for a couple years, 
and you're the first person to notice. :)

   The reason is that it isn't really a very simple problem.  We are 
deep in the lexical analyzer, and suddenly we need a lexical analyzer to 
solve this problem.  Right now we just have a bit of a short cut for the 
simple case of a single expression.

   Ideally, we'd use the lexical analyzer and a parser to solve the 
problem, except we are trying to do a single pass lexical analysis, 
instead of the tradition 2 pass that gcc and most other compilers employ.

   If you'd like to take a crack at this problem in 
semantic-c-do-lex-if, I expect that the solution is to change the regex 
in semantic-lex-c-if to exclude the expression, and to extract it 'by 
hand' in semantic-c-do-lex-if where it could then be evaluated.  If you 
don't have time, hopefully I can examine this more after Thanksgiving.

   If your patch is over 10 lines long or so, I'll also need a release 
so your changes can be included in Emacs.

   Also, if it helps, you can disable 
semantic-c-obey-conditional-section-parsing-flag to just go with the 
'whatever' approach. :)

Eric

Brian J. Carlson wrote:
> Thank you for such a wonderful set of tools. 
> 
> I have recently started using CEDET with boost and have narrowed down a 
> parsing "problem." 
> 
> The problem is that #ifdef pre-parsing is not evaluating correctly. 
> 
> An example of this type of statement is as follows:
> #if defined(BOOST_NO_MEMBER_TEMPLATES) && !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
> 
> 
> I am using emacs 23:
> "GNU Emacs 23.1.50.1 (x86_64-pc-linux-gnu, GTK+ Version 2.18.0)
>  of 2009-09-27 on crested, modified by Debian"
> with CEDET from the cvs repository at cedet.cvs.sourceforge.net)
> 
> I executed the following emacs-lisp after starting  
> emacs via 'emacs --no-init'):
> 
> ,----
> | (if (not (featurep 'cedet))
> |     (load-file "~/.elisp/cedet/common/cedet.elc"))
> | (global-ede-mode t)
> | (require 'ede-cpp-root)
> | (require 'semantic-gcc)
> | (ede-cpp-root-project "cedet-test"
> | 		      :file "~/src/cedet-test/test.cpp")
> | (semantic-load-enable-excessive-code-helpers)
> | (require 'semantic-ia)
> `----
> 
> Here is the contents of test.cpp:
> ,----
> | #undef BOOST_NO_MEMBER_TEMPLATES
> | #undef BOOST_MSVC6_MEMBER_TEMPLATES
> | 
> | 
> | #if defined(BOOST_NO_MEMBER_TEMPLATES)
> | int mymethod(int myint) {
> |    return myint++;
> | }
> | #else
> | // This is what should be recognized and it is
> | int  mymethod(int myint) {
> |    return myint + 2;
> | }
> | #endif
> | 
> | 
> | #if !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
> | // This is what should be recognized an it is
> | void mysecondmethod(int) {
> |   return;
> | }
> | #else
> | void mysecondmethod(unsigned int) {
> |   return;
> | } 
> | #endif
> | 
> | // So BOOST_NO_MEMBER_TEMPLATES is NOT defined
> | // and BOOST_MSVC6_MEMBER_TEMPLATES is NOT defined
> | 
> | // so this is #if false && true --> 
> | //      should yield false, but it yields true
> | #if defined(BOOST_NO_MEMBER_TEMPLATES) && !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
> | // This should not be the value that is parsed but it is...
> | void definedBOOST_NO_MEMBER_TEMPLATES_AND_bang_BOOST_MSVC6()
> | {
> |    return;
> | }
> | #else
> | // This is what should be parsed but it is not
> | void not_definedBOOST_NO_MEMBER_TEMPLATES_AND_bang_BOOST_MSVC6()
> | {
> |    return;
> | }
> | #endif
> |
> | // (senator-force-refresh)
> `----
> 
> 
> At first glance (and I mean glance), it looks like the parser is evaluating 
> if "BOOST_MSVC6_MEMBER_TEMPLATES && !definedBOOST_NO_MEMBER_TEMPLATES_AND_bang_BOOST_MSVC6)" is defined
> rather than seeing this as two separate conditions. Or the lexer is not dealing with the && or || in the 
> #if xxxxxxxxxx && xxxxxxxx type of pre-processing directive. 
> 
> Any help with this would be greatly appreciated. 
> 
> Thanks, 
> ;-Brian
>

"Eric M. Ludlam" <eric@...> 
on Wed, 18 November 2009 (18:59 EST)] wrote:

>> 
>> I have recently started using CEDET with boost and have narrowed down a 
>> parsing "problem." 
>> 
>> The problem is that #ifdef pre-parsing is not evaluating correctly. 
>> 
>> An example of this type of statement is as follows:
>> #if defined(BOOST_NO_MEMBER_TEMPLATES) && !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
>> 

> Hi Brian,
>
>    Your assessment is correct.  The preprocessor in CEDET for C doesn't 
> do much for complex expressions.  It's been in there for a couple years, 
> and you're the first person to notice. :)

Yeah. Not often that I'm the first to notice stuff. 


>    The reason is that it isn't really a very simple problem.  We are 
> deep in the lexical analyzer, and suddenly we need a lexical analyzer to 
> solve this problem.  Right now we just have a bit of a short cut for the 
> simple case of a single expression.

Indeed it is not a very simple problem.

>    Ideally, we'd use the lexical analyzer and a parser to solve the 
> problem, except we are trying to do a single pass lexical analysis, 
> instead of the tradition 2 pass that gcc and most other compilers employ.
>
>    If you'd like to take a crack at this problem in 
> semantic-c-do-lex-if, I expect that the solution is to change the regex 
> in semantic-lex-c-if to exclude the expression, and to extract it 'by 
> hand' in semantic-c-do-lex-if where it could then be evaluated.  If you 
> don't have time, hopefully I can examine this more after Thanksgiving.

I started by looking at that function into how to handle this and realized that
it was anything but trivial. I'm not exactly a fluent (e)lisp developer but
I've been looking for something to sink my teeth into for some time.  

I also looked at hideif.el (in emacs/lisp/progmodes) which gave another good
start, but then started noticing that it would take some time to deal with
anything besides the #if (defined(X) && defined(Y)) || defined(Z). But even
that required some parsing. I'll give it a go and let you know if I find
anything that will work. (of course, if I _actually_ do (which may be a
stretch) you'll be welcome to use/throw away as you see fit.

>
>
>    If your patch is over 10 lines long or so, I'll also need a release 
> so your changes can be included in Emacs.

No doubt the solution that I was thinking about would have been way over 10
lines. But maybe I'm not thinking about the correct approach. If I get
something working, than I'll send you what I do and we'll figure out the
paperwork if/when I do.

>
>
>    Also, if it helps, you can disable 
> semantic-c-obey-conditional-section-parsing-flag to just go with the 
> 'whatever' approach. :)

I may have to do just that in the short term.

Of course I also noticed that the following is also not quite right, but this
is a result of my looking at the first problem and is not really that much of a
problem for me right now:

#define A
#define B
#if defined(A)
void a();
#elif defined(B)
void b();
#else
void c();
#endif

In this case both a(), and b() are parsed. Again not an issue for me and is
obviously a sibling of the first issue. 

I'll keep working in my spare time to see if I can find a way to work with more
complex preprocessor statements. 

Thanks, 
;-b

PS: Thanks again for your awesome set of tools. 

PPS: How'd the pumpkin toss go?

Thanks, 
;-Brian

Brian J. Carlson wrote:
> 
> PPS: How'd the pumpkin toss go?
> 

We threw a 9 lb pumpkin 693 feet.  Woot!

That was great for us, but 1st place in our division went 2087 ft, so we 
have room for improvement.

http://www.siege-engine.com

I'll dig into the ifdef part of the message perhaps this weekend.

Eric

>
> Eric, 
>
> I did take a quick crack at this. 

Sorry about the huge file attachment. I meant to send the cvs diff output.

-- 
Thanks, 
 ;-Brian J. Carlson
 Leominster, MA

Hi,

Thanks for the patch.

You are right, this is a pretty big patch.

Your comments indicate that you borrowed a lot from hideif.  Since 
hideif is a part of Emacs, perhaps that code can be used directly?  If 
not, since CEDET is also becoming a part of Emacs, the shared pieces 
could be extracted.

If this level of complexity is really needed and there is not shared 
part with hideif, perhaps it is time to buckle down and write a real 
lexical pre-processor for Semantic.  Though I tend to think that every 
time I have to pile a new hack on top of the existing preprocessor.

As for your need of a release, I'm guessing from the comments that the 
code you wrote is relatively small vs what was borrowed from hideif?  I 
counted 13 lines, plus a few lines in the original lexical fcn you 
modified.  That might be ok, though if you enjoy hacking Emacs, it might 
be wise for you to provide them a release anyway.  That would let you 
implement changes in hideif as necessary.

Thanks
Eric

Brian J. Carlson wrote:
> "Brian J. Carlson" <gmane.briancarlson@...> [Thu, 19 November 2009 (14:32 EST)] wrote:
> 
>>>> I have recently started using CEDET with boost and have narrowed down a 
>>>> parsing "problem." 
>>>>
>>>> The problem is that #ifdef pre-parsing is not evaluating correctly. 
>>>>
>>>> An example of this type of statement is as follows:
>>>> #if defined(BOOST_NO_MEMBER_TEMPLATES) && !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
>>>>
>>> Hi Brian,
>>>
>>>    Your assessment is correct.  The preprocessor in CEDET for C doesn't 
>>> do much for complex expressions.  
> 
>>>    If you'd like to take a crack at this problem in 
>>> semantic-c-do-lex-if, I expect that the solution is to change the regex 
>>> in semantic-lex-c-if to exclude the expression, and to extract it 'by 
>>> hand' in semantic-c-do-lex-if where it could then be evaluated.  If you 
>>> don't have time, hopefully I can examine this more after Thanksgiving.
> 
> Eric, 
> 
> I did take a quick crack at this. I borrowed from hideif.el and do some
> parsing. It's not perfect (in that it doesn't evaluate C-Macros in an
> #if pre-processor directive among other things), but it does a very
> large number of the things I needed. It's extremely "hackish" and
> probably doesn't follow the paradigm (ooooooh BS Bingo word) you were
> thinking about.  
> 
> I'd be interested in working with this some more if you'd like. 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> 
> 
>>>
>>>    If your patch is over 10 lines long or so, I'll also need a release 
>>> so your changes can be included in Emacs.
> 
> Ummm it is well over 10 lines, so if you decide to include my changes,
> let me know and I'll fill out the FSF (I'm assuming it's FSF since I
> noticed that CEDET is going into the next emacs YEAH!!!)  paperwork for
> your changes. 
> 
>> Of course I also noticed that the following is also not quite right, but this
>> is a result of my looking at the first problem and is not really that much of a
>> problem for me right now:
>>
>> #define A
>> #define B
>> #if defined(A)
>> void a();
>> #elif defined(B)
>> void b();
>> #else
>> void c();
>> #endif
> 
> My changes still doesn't address this, but this was a very tricky piece,
> to evaluate the first #if then know to skip over all the next
> #elif statement. (i.e. difficult to know whether the first #if was
> evaluated as true and the #elif at (point) was to be tested or not. 
> At least in the very short term.
> 
> 
> Thanks, 
> 
> ;-Brian J. Carlson
> Leominster, MA
> 
> 
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Cedet-devel mailing list
> Cedet-devel@...
> https://lists.sourceforge.net/lists/listinfo/cedet-devel

Hi Brian,

It's been a while, but I finally got to integrating your patch.

I've been able to figure out what is going on, and can now use the 
hideif code directly without copying it.  I also have a better idea of 
how many lines of code you wrote vs copied from hideif.  If you can get 
a release to the FSF that would be great.

You can just write to copyright-clerk@....  Tell them you have 
contributions to both EMACS and CEDET, and you will be sent the forms by 
email which you will need to send back via paper.


Despite all that, I ran all my test suites and they pass, and I've 
checked this code into CVS.  I can pull it out later if you have issues 
getting paper work signed and ready for the FSF.

For all those watching at home, I don't know how the hideif API 
stability has been over the past few releases of Emacs/XEmacs.  It would 
be great if you could test this on various Emacs flavors and let me know 
if there are issues.  Thanks!

Lastly, there is now a new side-effect feature!  If you use 
CEDET/Semantic, and HIDEIF, you can set this variable:

(setq semantic-c-takeover-hideif t)

and hideif will now automatically use all the Semantic information for 
evaluating what to hide.  Nifty!  Except that I haven't tested it as I'm 
not a hideif user.  If you are, try it at let me know how it goes, and 
we can add it to the manual as another augmented Emacs feature.

Thanks!
Eric


On 11/25/2009 12:15 PM, Brian J. Carlson wrote:
> "Brian J. Carlson"<gmane.briancarlson@...>  [Thu, 19 November 2009 (14:32 EST)] wrote:
>
>>>>
>>>> I have recently started using CEDET with boost and have narrowed down a
>>>> parsing "problem."
>>>>
>>>> The problem is that #ifdef pre-parsing is not evaluating correctly.
>>>>
>>>> An example of this type of statement is as follows:
>>>> #if defined(BOOST_NO_MEMBER_TEMPLATES)&&  !defined(BOOST_MSVC6_MEMBER_TEMPLATES)
>>>>
>>
>>> Hi Brian,
>>>
>>>     Your assessment is correct.  The preprocessor in CEDET for C doesn't
>>> do much for complex expressions.
>
>>>     If you'd like to take a crack at this problem in
>>> semantic-c-do-lex-if, I expect that the solution is to change the regex
>>> in semantic-lex-c-if to exclude the expression, and to extract it 'by
>>> hand' in semantic-c-do-lex-if where it could then be evaluated.  If you
>>> don't have time, hopefully I can examine this more after Thanksgiving.
>
> Eric,
>
> I did take a quick crack at this. I borrowed from hideif.el and do some
> parsing. It's not perfect (in that it doesn't evaluate C-Macros in an
> #if pre-processor directive among other things), but it does a very
> large number of the things I needed. It's extremely "hackish" and
> probably doesn't follow the paradigm (ooooooh BS Bingo word) you were
> thinking about.
>
> I'd be interested in working with this some more if you'd like.
>
>
>
>
>
>
>>>
>>>
>>>     If your patch is over 10 lines long or so, I'll also need a release
>>> so your changes can be included in Emacs.
>
> Ummm it is well over 10 lines, so if you decide to include my changes,
> let me know and I'll fill out the FSF (I'm assuming it's FSF since I
> noticed that CEDET is going into the next emacs YEAH!!!)  paperwork for
> your changes.
>
>> Of course I also noticed that the following is also not quite right, but this
>> is a result of my looking at the first problem and is not really that much of a
>> problem for me right now:
>>
>> #define A
>> #define B
>> #if defined(A)
>> void a();
>> #elif defined(B)
>> void b();
>> #else
>> void c();
>> #endif
>
> My changes still doesn't address this, but this was a very tricky piece,
> to evaluate the first #if then know to skip over all the next
> #elif statement. (i.e. difficult to know whether the first #if was
> evaluated as true and the #elif at (point) was to be tested or not.
> At least in the very short term.
>
>
> Thanks,
>
> ;-Brian J. Carlson
> Leominster, MA
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
>
>
>
> _______________________________________________
> Cedet-devel mailing list
> Cedet-devel@...
> https://lists.sourceforge.net/lists/listinfo/cedet-devel

To reply to my own message...

I found some bugs in hideif, so I've updated the parse to be robust to 
hideif catastrophically failing.

There is also a rather complex new test that exercises some of those 
nifty new features.

Enjoy.
Eric

On 01/19/2010 11:21 PM, Eric M. Ludlam wrote:
> Hi Brian,
>
> It's been a while, but I finally got to integrating your patch.
>
> I've been able to figure out what is going on, and can now use the
> hideif code directly without copying it.  I also have a better idea of
> how many lines of code you wrote vs copied from hideif.  If you can get
> a release to the FSF that would be great.
>
> You can just write to copyright-clerk@....  Tell them you have
> contributions to both EMACS and CEDET, and you will be sent the forms by
> email which you will need to send back via paper.
>
>
> Despite all that, I ran all my test suites and they pass, and I've
> checked this code into CVS.  I can pull it out later if you have issues
> getting paper work signed and ready for the FSF.
>
> For all those watching at home, I don't know how the hideif API
> stability has been over the past few releases of Emacs/XEmacs.  It would
> be great if you could test this on various Emacs flavors and let me know
> if there are issues.  Thanks!
>
> Lastly, there is now a new side-effect feature!  If you use
> CEDET/Semantic, and HIDEIF, you can set this variable:
>
> (setq semantic-c-takeover-hideif t)
>
> and hideif will now automatically use all the Semantic information for
> evaluating what to hide.  Nifty!  Except that I haven't tested it as I'm
> not a hideif user.  If you are, try it at let me know how it goes, and
> we can add it to the manual as another augmented Emacs feature.
>
> Thanks!
> Eric
>
>
>

"Eric M. Ludlam" <eric@...> [Tue, 19 January 2010 (23:21 EST)] wrote:

> Hi Brian,
>
> It's been a while, but I finally got to integrating your patch.

I wouldn't dare make any comments about taking a while. I've been away
for a bit (and I like to turn procrastination into an art form).

> I've been able to figure out what is going on, and can now use the 
> hideif code directly without copying it.  I also have a better idea of 
> how many lines of code you wrote vs copied from hideif.  If you can get 
> a release to the FSF that would be great.
> You can just write to copyright-clerk@....  Tell them you have 
> contributions to both EMACS and CEDET, and you will be sent the forms by 
> email which you will need to send back via paper.

I will be sending them an email in the very near future.

> Despite all that, I ran all my test suites and they pass, and I've 
> checked this code into CVS.  I can pull it out later if you have issues 
> getting paper work signed and ready for the FSF.

I found a small omission in your code around using (hif-mathify). 
Since the values taken from 'cpp -E -dM -x c++ /dev/null' in (semantic-gcc-setup)
are added to semantic-lex-c-preprocessor-symbol-map as strings,
hif-mathify needs to deal with such items:
examples;
__SIZEOF_LONG__	"8" ==> 8
__GNUC__	"4" ==> 8
__DEC32_MIN_EXP__	"(-94)" ==> "(-94)"
__cplusplus	"1" ==> 1
__DBL_HAS_DENORM__	"1" ==> 1
__DEC64_MIN_EXP__	"(-382)" ==> "(-382)"
__FLT_DENORM_MIN__	"1.40129846e-45F" ==> 1.40129846e-45

Here is the advice I have added to deal with these types of strings that
are added to SPP when using GCC:

(defadvice hif-mathify (around hif-mathify-around-advice activate)
     (let* ((value (ad-get-arg 0)))
       (if (and (stringp value) 
                (string-match "^\\(-?[0-9][.0-9]*\\([-+eE0-9]+\\)?\\)\\([FDL]+\\)?$" value))
           (setq ad-return-value (string-to-number (substring value (match-beginning 1) (match-end 1))))
         ad-do-it)))


Originally I had overwritten hif-mathify with the following, but I think
the advice follows your changes, and is a little more robust:
;;
;; this is just the code that I used originally
;; (defun semantic-c-lex-mathify (val-in)
;;  "Treat VAL as a number: if it's t or nil, use 1 or 0."
;;   (let ((val (cond ((stringp val-in) (string-to-number val-in))
;;                   (t val-in))))
;;  (cond ((eq val t) 1)
;;	((null val) 0)
;;	(t val))))

I thought about adding them to the
semantic-lex-c-preprocessor-symbol-map as numbers, but wasn't certain
if that was the preferred way.

> (setq semantic-c-takeover-hideif t)

This feature totally rocks, I'm glad you did it, because I was trying to
figure out how to do it and you beat me to the punch.

Thanks!
-- 
;-brian

 0 
  0
000