Unicode/HTML entities not indexed correctly for search
Status: Beta
Brought to you by:
sirlark
Menu
▾
▴
#22 Unicode/HTML entities not indexed correctly for search
When indexing a record for search HTML entities and non-ascii unicode characters are treated like punctuation and stripped out (partially in the case of HTML entities). When storing into the database, HTML entities should be converted to unicode (except for html(short) formtypes), and unicode characters should not be stripped treated as punctuation during indexing.
Also, in the light of Bug #3053050 (search term synonym feature request), unicode characters should have their English names as synonyms to facilitate searching on a standard US keyboard with ease, e.g. searching for "alpha one" or "α one" should yield identical results
It seems the right way to do this is to html_entity_decode the incoming string before splitting using mb_split("\W"). NB: the html_entity_decode is for search indexing only, it isn't used to process data before being committed to the database