Re: [CEDET-devel] semantic lexer for python
Brought to you by:
zappo
From: <pon...@ne...> - 2002-05-31 09:06:44
|
Hi Eric & Richard, Attached you will find the latest version of `semantic-flex' including Richard's newline enhancement plus the following: - newline are also correctly handled when `semantic-ignore-comments' is disabled. - I moved BOL detection before processing `semantic-flex-extensions' because BOL tokens must be always inserted before any other tokens found on the same line. Maybe a more general approach to keep the consistency of tokens ordering would be to replace the last (nreverse ts) by (sort ts #'(lambda (t1 t2) (<= (cadr t1) (cadr t2)))) Or even better, to also merge here whitespace and comment tokens too? Anyway I think this new version is more consistent in the way it handle the various `semantic-flex-enable-...' options. Here is a small example of different results from `semantic-flex' depending on what options are enabled. The input is the following small piece of Java code. /** * Describe variable <code>x</code> here. */ if (ok) int[] x = {1,2,3}; // end 1- Want all (let ((semantic-ignore-comments nil) (semantic-flex-enable-bol t) (semantic-flex-enable-newlines t) (semantic-flex-enable-whitespace t)) (semantic-flex-buffer)) ((bol 1 . 1) (comment 1 . 50) (newline 50 . 51) (bol 51 . 51) (IF 51 . 53) (whitespace 53 . 54) (semantic-list 54 . 58) (newline 58 . 59) (bol 59 . 59) (whitespace 59 . 61) (INT 61 . 64) (semantic-list 64 . 66) (whitespace 66 . 67) (symbol 67 . 68) (whitespace 68 . 69) (punctuation 69 . 70) (whitespace 70 . 71) (semantic-list 71 . 78) (punctuation 78 . 79) (newline 79 . 80) (bol 80 . 80) (comment 80 . 86) (newline 86 . 87) (bol 87 . 87)) 2- Ignore comments (returns them as whitespaces) (let ((semantic-flex-enable-bol t) (semantic-flex-enable-newlines t) (semantic-flex-enable-whitespace t)) (semantic-flex-buffer)) ((bol 1 . 1) (whitespace 1 . 50) (newline 50 . 51) (bol 51 . 51) (IF 51 . 53) (whitespace 53 . 54) (semantic-list 54 . 58) (newline 58 . 59) (bol 59 . 59) (whitespace 59 . 61) (INT 61 . 64) (semantic-list 64 . 66) (whitespace 66 . 67) (symbol 67 . 68) (whitespace 68 . 69) (punctuation 69 . 70) (whitespace 70 . 71) (semantic-list 71 . 78) (punctuation 78 . 79) (newline 79 . 80) (bol 80 . 80) (whitespace 80 . 86) (newline 86 . 87) (bol 87 . 87)) 3- Want newline & whitespace (let ((semantic-flex-enable-newlines t) (semantic-flex-enable-whitespace t)) (semantic-flex-buffer)) ((whitespace 1 . 50) (newline 50 . 51) (IF 51 . 53) (whitespace 53 . 54) (semantic-list 54 . 58) (newline 58 . 59) (whitespace 59 . 61) (INT 61 . 64) (semantic-list 64 . 66) (whitespace 66 . 67) (symbol 67 . 68) (whitespace 68 . 69) (punctuation 69 . 70) (whitespace 70 . 71) (semantic-list 71 . 78) (punctuation 78 . 79) (newline 79 . 80) (whitespace 80 . 86) (newline 86 . 87)) 4- Just whitespace (let ((semantic-flex-enable-whitespace t)) (semantic-flex-buffer)) ((whitespace 1 . 50) (IF 51 . 53) (whitespace 53 . 54) (semantic-list 54 . 58) (whitespace 59 . 61) (INT 61 . 64) (semantic-list 64 . 66) (whitespace 66 . 67) (symbol 67 . 68) (whitespace 68 . 69) (punctuation 69 . 70) (whitespace 70 . 71) (semantic-list 71 . 78) (punctuation 78 . 79) (whitespace 80 . 87)) 5- The default (semantic-flex-buffer) ((IF 51 . 53) (semantic-list 54 . 58) (INT 61 . 64) (semantic-list 64 . 66) (symbol 67 . 68) (punctuation 69 . 70) (semantic-list 71 . 78) (punctuation 78 . 79)) 6- etc.! Eric, if you agree maybe could we check it in so Richard could use it for his semantic-python stuff? Sincerely, David __________________________________________________________________ Your favorite stores, helpful shopping tools and great gift ideas. Experience the convenience of buying online with Shop@Netscape! http://shopnow.netscape.com/ Get your own FREE, personal Netscape Mail account today at http://webmail.netscape.com/ |