Updated the tokenize module (added delimiters and proper documentation)
Authored by: arjenmarkus 2008-04-02
Parent: [r78]
Child: [r80]
