Thank you! Building a simple token cache and re-using the list of tokens in the overridden tokenize method did the trick.
Allow reusing tokens