[Jython-checkins] SF.net SVN: jython: [4934] branches/asm/src/org/python/antlr/ PythonTokenSource.java

[<fwi...@us...>]

Revision: 4934
          http://jython.svn.sourceforge.net/jython/?rev=4934&view=rev
Author:   fwierzbicki
Date:     2008-07-14 12:27:40 -0700 (Mon, 14 Jul 2008)

Log Message:
-----------
generate newlines before the dedents again.  Although the reverse
was solving some problems, it was creating more.

Modified Paths:
--------------
    branches/asm/src/org/python/antlr/PythonTokenSource.java

Modified: branches/asm/src/org/python/antlr/PythonTokenSource.java
===================================================================
--- branches/asm/src/org/python/antlr/PythonTokenSource.java	2008-07-14 19:26:58 UTC (rev 4933)
+++ branches/asm/src/org/python/antlr/PythonTokenSource.java	2008-07-14 19:27:40 UTC (rev 4934)
@@ -174,6 +174,8 @@
             newline = new CommonToken(PythonLexer.NEWLINE, "\n");
             newline.setLine(t.getLine());
             newline.setCharPositionInLine(t.getCharPositionInLine());
+            //XXX: should be moved to after DEDENTS are generated, but doesn't work yet.
+            tokens.addElement(newline);
         }
 
         // compute cpos as the char pos of next non-WS token in line
@@ -216,9 +218,13 @@
             }
             sp = prevIndex; // pop those off indent level
         }
+        /*
+        //XXX: I think this is really where this needs to be, but causes more problems than it
+        //     solves at the moment.
         if (t.getType() == PythonLexer.EOF) {
             tokens.addElement(newline);
         }
+        */
 
         if (t.getType() != PythonLexer.LEADING_WS) { // discard WS
             tokens.addElement(t);


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.