#36 Introduce context specific tokenization

[Kevin Rogers]

Following on from tracker item 1105231, we should update the tokenizer to handle context in a slightly more friendly way.

The current method of using the setWhere() and checkWhere() methods feels like a bodge, rather than a proper solution. Despite the potential of having to duplicate some code, I think the end result will be easier to understand and maintain.

I see two options:

1) Introduce a abstract tokenizer with three concrete sub-classes.
2) Introduce new methods to handle specific cases.