#278 Tokenization Errors in Fields with TokenizeTagContent Set to False

[Lemur Project]

In cases where a document field has a tokenizetagcontent attribute set to false and there are spaces between the begin tag and first term, and spaces between the last term and end tag, the spaces are included as part of the normalized token going into the index.

For example:

<AREA tokenizetagcontent=\"false\"> 2003.145  </AREA>

produces two tokens (the single term split at the decimal point) of
" 2003" and "145 ".

The tokenizer needs to squeeze out such spaces.

This extends to whitespace as well.

2003.145 \n
123.0055

This produces tokens
" 2003"
"145 \n"
" 123"
"0055 "

This ONLY appears to happen for field tags with the tokenizeTagContent attribute defined as false.

QUESTION: What is real purpose of tokenizeTagContent attribute? If it means accept the token exactly as is, then leading/trailing spaces might be acceptable, although it might also mean take multiple terms in the field as a single term (no tokenization).

Since the terms in the field are still processed as multiple terms, I believe any leading/trailing whitespace needs to be removed.