[JEDI.NET-commits] main/run Jedi.Text.Tokenization.TokenizerBase.pas,1.2,1.3

[Marcel B. <jed...@us...>]

Update of /cvsroot/jedidotnet/main/run
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv2596/main/run

Modified Files:
	Jedi.Text.Tokenization.TokenizerBase.pas 
Log Message:
UnknownCharacterToken resulted in an endless stream of the same token and content

Index: Jedi.Text.Tokenization.TokenizerBase.pas
===================================================================
RCS file: /cvsroot/jedidotnet/main/run/Jedi.Text.Tokenization.TokenizerBase.pas,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** Jedi.Text.Tokenization.TokenizerBase.pas	24 Dec 2005 12:25:13 -0000	1.2
--- Jedi.Text.Tokenization.TokenizerBase.pas	31 Dec 2005 11:43:26 -0000	1.3
***************
*** 208,211 ****
--- 208,212 ----
          Result := True;
          FCurrent := UnknownCharacterToken.Create(FContext, Input.Sequence, Input.Line, Input.Column);
+         Input.Next;
        end;
        if Input.Sequence = '' then