#21 Allow reusing tokens

[Daniel Obermeier]

In a scenario where the same expression should be evaluated multiple times using different variables, the evaluator will alwas perform the tokenization of the given expression. This seems to be a rather hard hit to performance, as the regex splitting can take a bit of time. In my case this equals to about 64% of the total time spent in the evaluate() method.

As in this case the expression stays constant, could you implement a way of "precompiling" the given expression so that the evaluation part will be performed quicker without the need to first tokenize the given string?

In the simplest form, this could be another method accepting the tokens that can already be extracted today by calling tokenize(). This should be a relatively easy implementation that could crucially benefit this use case already.

[Fathzer]

Hello,
Sorry for answering late, I was very busy these last days ...
I agree it should have been better to provide a way to reuse the tokens. But ...
The bad news is the tokenize method returns an Iterator, which can't be traversed more than one time. Changing that would introduced a breaking change :-(
The good news is it is very simple to subclass your Evaluator class, add attributes to the sub-class to store the tokens in a reusable form (a list for instance), and the expression that corresponds to the saved tokens. Then override the tokenize method to rebuild the list (using super.tokenize method and populating the list with the returned iterator) if the expression as changed and return an iterator on the list. It should do the trick.
Best regards.
Jean-Marc

[Daniel Obermeier]

Thank you! Building a simple token cache and re-using the list of tokens in the overridden tokenize method did the trick.