Re: [VuFind-Tech] Is there a way to ignore punctuation in title sort?

[Demian K. <dem...@vi...>]

The problem is that your stringTitleSort field type affects the INDEXED value of the text, but the STORED value of the text is unaffected.  Solr always stores unmanipulated strings.  Since sorting is based on the stored value rather than the indexed value, your field type has no actual effect on sorting.

This is one of several things about Solr that is not immediately intuitive!

- Demian

From: Sean C Purcell [mailto:sea...@ug...]
Sent: Friday, December 03, 2010 4:00 PM
To: vuf...@li...
Subject: Re: [VuFind-Tech] Is there a way to ignore punctuation in title sort?

Update: I added the following  field type and then changed the 'title_sort' field from "string" to this new type, "stringTitleSort".  The strange thing is that the field appears to be doing exactly what it's supposed to be doing (stripping all punctuation) while running tests in Solr field Analysis, but when I look at live records in the Solr index, the title_sort field still has all punctuation present.   And of course, when I sort by title in VuFind, it's not ignoring punctuation.

   <fieldType name="stringTitleSort" class="solr.StrField" sortMissingLast="true" omitNorms="true">
      <analyzer>
        <tokenizer class="solr.KeywordTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.TrimFilterFactory"/>
        <!-- strip all punctuation -->
        <filter class="solr.PatternReplaceFilterFactory" pattern="[^\p{L}\p{N} ]" replacement="" replace="all" />
      </analyzer>
    </fieldType>

From: Sean C Purcell [mailto:sea...@ug...]
Sent: Friday, December 03, 2010 12:17 PM
To: vuf...@li...
Subject: [VuFind-Tech] Is there a way to ignore punctuation in title sort?

Hi All,

Is there a way to have Solr ignore punctuation when sorting by title?  The way things are currently set, "Food Webs at the landscape level" sorts before "Food, agriculture, and development...".  See the following example:
http://vufind.org/demo/Search/Results?lookfor=food&type=Title&sort=title&page=36

We've had a few librarians ask if this can be changed.  I've done some research and it looks like Solr 1.5 offers a filter that allows for this: solr.CollationKeyFilterFactory, but this isn't available in version 1.4.  Any ideas?

Thanks,
Sean