Corrected two bugs in tokenize module, substrnig bounds exceeded, as reported
by Ragunath Sankaranarayanan.
Authored by: arjenmarkus 2008-09-03
Parent: [r207]
Child: [r209]
