Re: [VuFind-Tech] callnumber browse - dropping into list based on sort instead of value

[Tod O. <to...@uc...>]

Oh yes, SolrMarc uses OneJar, which complicates things for using SolrMarc classes outside of SolrMarc itself. If you do jar -tf SolrMarc.jar, you’ll see that it contains a number of jar files. Java doesn’t load nested jar files on it’s own, OneJar provides a custom class loader that is used when executing the jar file. It’s great for single-file jar distribution, but vexing for reusing components.

Pull MarcImporter.jar out of SolrMarc.jar and put that in your path.

-Tod

On Oct 17, 2014, at 1:40 PM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:

I thought I was having classpath problems so I copied SolrMarc.jar directly into vufind-browse-handler/libs. Still the browse handler won't build with my normalizer; error is "package org.solrmarc.tools does not exist".

That seems impossible; I use it successfully in my beanshell script.

Thanks,
Anna



On Fri, Oct 17, 2014 at 12:03 PM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:
Yes, okay I meant how to use the normalizer when I index the record into solr. But maybe I don't need to do that since I'm just using org.solrmarc.tools.CallNumUtils.getLCShelfkey. I can make my normalizer a similar wrapper around that method.

Thanks!
Anna




On Fri, Oct 17, 2014 at 12:00 PM, Tod Olson <to...@uc...<mailto:to...@uc...>> wrote:
Right you are on contradictory. I’ll have to dig through some code and some change history and get back to you.

For the normalizer at index time, set the browse.normalizer property to the desired class name. Here’s the line I have in my index script:

build_browse "lcc" "callnumber_browse" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=callnumber_browse_sort -Dvaluefield=callnumber_browse -Dbrowse.normalizer=org.vufind.util.UChicagoLCCallNormalizer”

-Toe


On Oct 17, 2014, at 10:52 AM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:

> I took a first pass at documenting some of this at https://vufind.org/wiki/alphabetical_heading_browse
>
> You'll see some TODOs, mostly areas where I'm still not clear exactly how things work. I will try to fill in as this discussion progresses, but if anyone else wants to edit please feel free.
>
> Thanks,
> Anna
>
> On Fri, Oct 17, 2014 at 11:51 AM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:
> So if I write a custom normalizer I should build the browse handler locally and maintain my own version?
>
> And then how would I invoke the custom normalizer from my beanshell script or my marc_local.properties?
>
> Thanks,
> Anna
>
>
>
>
> On Fri, Oct 17, 2014 at 11:40 AM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:
> Tod, the code you gave above uses the valuefield -- is that how y'all are doing it? Seems contradictory to what you just wrote.
>
> index-alphabetic-browse:
>
> build_browse "lcc" "callnumber_browse" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=callnumber_browse_
> sort -Dvaluefield=callnumber_browse -Dbrowse.normalizer=org.vufind.util.UChicagoLCCallNormalizer”
>
> solr/biblio/conf/solrconfig.xml:
>
>     <lst name="lcc">
>       <str name="DBpath">${solr.solr.home:./solr}/alphabetical_browse/lcc_browse.db</str>
>       <str name="field">callnumber_browse</str>
>       <str name="normalizer">org.vufind.util.UChicagoLCCallNormalizer</str>
>     </lst>
>
>
> (And Demian, Tod showed how to configure the normalizer in solrconfig when he gave this example.)
>
> Thanks!
> Anna
>
>
>
>
> On Fri, Oct 17, 2014 at 11:34 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote:
> How do you turn on the normalizer? Is there a setting for this in the solrconfig.xml as well?
>
> - Demian
> From: Tod Olson [to...@uc...<mailto:to...@uc...>]
> Sent: Friday, October 17, 2014 11:32 AM
>
> To: Anna Headley
> Cc: Tod Olson; Demian Katz; vuf...@li...<mailto:vuf...@li...>
> Subject: Re: [VuFind-Tech] callnumber browse - dropping into list based on sort instead of value
>
> That should tell the BrowseRequestHandler which field to search, but it should be the sort field. And have the search term normalized the same way as at indexing time. Depending on the terms and the normalization, if the user does not type the term exactly it can drop the user into the browse list at a place that is a bit misleading.
>
> -Tod
>
>
> On Oct 17, 2014, at 10:20 AM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:
>
>> So I thought that configuring the field in the sorlconfig BrowseRequestHandler was to tell the handler which field to search. If that's not the case (and, as Demian says, it always searches the sort field), what does that field configuration do?
>>
>> Thanks!
>> Anna
>>
>>
>>
>> On Fri, Oct 17, 2014 at 11:13 AM, Tod Olson <to...@uc...<mailto:to...@uc...>> wrote:
>> Implementing a new normalizer would just be implementing the Normalizer interface. It’s pretty minimal:
>>
>> package org.vufind.util;
>>
>> /**
>>  * An interface for alphabetical browse normalizers.
>>  *
>>  * @author Tod Olson <to...@uc...<mailto:to...@uc...>>
>>  *
>>  */
>>
>> public interface Normalizer {
>>
>> /**
>> * Normalizes the input string to a collation key, which will be used for the
>> * sort order of the browse index.
>> *
>> * @param s
>> the string to be normalized, e.g. a term for the browse index
>> * @return
>> collation key for determining browse order
>> */
>> public byte[] normalize (String s);
>>
>> }
>>
>> I keep wondering if there should be more to the interface than that, so other eyes or implementors would be great.
>>
>> -Tod
>>
>> On Oct 17, 2014, at 8:11 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote:
>>
>>> Sounds good to me. Thanks for the wiki help! Let me know if you need more from me.
>>>
>>> - Demian
>>> From: Anna Headley [an...@gm...<mailto:an...@gm...>]
>>> Sent: Friday, October 17, 2014 8:50 AM
>>> To: Demian Katz
>>> Cc: Tod Olson; vuf...@li...<mailto:vuf...@li...>
>>> Subject: Re: [VuFind-Tech] callnumber browse - dropping into list based on sort instead of value
>>>
>>> Tod, thanks very much I will try that later today.
>>>
>>> Demian, I don't see any results on searching the wiki for 'normalizer.' Any explicit instructions would definitely be welcome.
>>>
>>> Actually the page at https://vufind.org/wiki/alphabetical_heading_browse is pretty bare bones; I will try to add more there based on my work so far. That might also be a good place to put documentation on building a custom normalizer.
>>>
>>> Thanks,
>>> Anna
>>>
>>>
>>>
>>>
>>> On Fri, Oct 17, 2014 at 8:07 AM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote:
>>> If memory serves, the per-index normalizer code is included in the latest VuFind release, but no actual normalizers are bundled in since we're waiting on all the SolrMarc refactoring work to be completed.
>>>
>>> I imagine there isn't any documentation yet on how to build a custom normalizer. Is this something we should add to the wiki someplace? Or does it actually exist, and I'm just forgetting?
>>>
>>> - Demian
>>> ________________________________________
>>> From: Tod Olson [to...@uc...<mailto:to...@uc...>]
>>> Sent: Thursday, October 16, 2014 7:29 PM
>>> To: Anna Headley
>>> Cc: Tod Olson; Demian Katz; vuf...@li...<mailto:vuf...@li...>
>>> Subject: Re: [VuFind-Tech] callnumber browse - dropping into list based on sort instead of value
>>>
>>> This is our production configuration for LC call number browse:
>>>
>>> index-alphabetic-browse:
>>>
>>> build_browse "lcc" "callnumber_browse" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=callnumber_browse_sort -Dvaluefield=callnumber_browse -Dbrowse.normalizer=org.vufind.util.UChicagoLCCallNormalizer”
>>>
>>> solr/biblio/conf/solrconfig.xml:
>>>
>>>     <lst name="lcc">
>>>       <str name="DBpath">${solr.solr.home:./solr}/alphabetical_browse/lcc_browse.db</str>
>>>       <str name="field">callnumber_browse</str>
>>>       <str name="normalizer">org.vufind.util.UChicagoLCCallNormalizer</str>
>>>     </lst>
>>>
>>> If that class is not present in your VuFind distro, you could try grabbing my in-progress work from https://github.com/todolson/vufind-browse-handler.
>>>
>>> See if that at least gets you closer to what you’re after.
>>>
>>> best,
>>>
>>> -tod
>>>
>>>
>>>
>>> On Oct 16, 2014, at 6:17 PM, Tod Olson <to...@uc...<mailto:to...@uc...>> wrote:
>>>
>>> > I’ve done some work on per-index normalizers, which allows you to configure, for any index, a normalizer to be applied and index time and at search time. This definitely belongs in the Java code, otherwise you must implement one normalizer in Java for indexing and another in PHP for the browse query.
>>> >
>>> > I think that functionality is in the current vufind-browse-handler source, though I don’t really have a number of normalizers there yet.
>>> >
>>> > -Tod
>>> >
>>> > On Oct 16, 2014, at 3:30 PM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:
>>> >
>>> >> Demian,
>>> >>
>>> >> Bummer.
>>> >>
>>> >> I'm not sure I have a firm enough understanding of the path an alphabrowse query takes through the code to have any opinions right now. I'll look at it and try to have an idea. Meantime I would love to hear anyone else's ideas.
>>> >>
>>> >> Thanks,
>>> >> Anna
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Oct 16, 2014 at 3:07 PM, Demian Katz <dem...@vi...<mailto:dem...@vi...>> wrote:
>>> >> Anna,
>>> >>
>>> >>
>>> >>
>>> >> I think the behavior of mapping the user query to the sort field is expected.
>>> >>
>>> >>
>>> >>
>>> >> The original purpose of the separate sort field was for titles – the value field gets the full title, the sort field gets the title with leading articles stripped based on 245 indicator 2. Obviously, in this scenario, it makes the most sense to map the user query directly to the sort field (though there are existing complications if users type articles into their search… a search for “the” will only show titles that have “the” in the sort field – which, incidentally, is a good way to find records with incorrect indicator 2 values).
>>> >>
>>> >>
>>> >>
>>> >> Obviously, this is not ideal for normalized call numbers.
>>> >>
>>> >>
>>> >>
>>> >> The missing piece is having some code to normalize user queries for consistency with sort values. This would be useful across the board – even for titles, it might be nice to have some kind of article stripping to increase the chances of correct match-ups. The question is where to put this functionality. I think Al Rykhus wrote some on the PHP side for the MnPals instance a while back, but it didn’t end up getting merged back to trunk. I’m not sure if PHP is the best place… but if we don’t put it in PHP, it’s more work to customize.
>>> >>
>>> >>
>>> >>
>>> >> Thoughts, anyone?
>>> >>
>>> >>
>>> >>
>>> >> - Demian
>>> >>
>>> >>
>>> >>
>>> >> From: Anna Headley [mailto:an...@gm...<mailto:an...@gm...>]
>>> >> Sent: Thursday, October 16, 2014 1:56 PM
>>> >> To: Tod Olson
>>> >> Cc: vuf...@li...<mailto:vuf...@li...>
>>> >> Subject: Re: [VuFind-Tech] callnumber browse - dropping into list based on sort instead of value
>>> >>
>>> >>
>>> >>
>>> >> Thanks, Tod.
>>> >>
>>> >> Unfortunately that didn't work for me. It seems that field has to match whatever I put as the leech valuefield.
>>> >>
>>> >> Here is a little bit more about the fields I have with examples from the record in question:
>>> >>
>>> >> callnumber-lc-search: used for call number searches (multivalued)
>>> >>
>>> >> callnumber-lc-sort: used for sorting callnumber search results (single value)
>>> >>
>>> >> callnumber-lc-browse: used for sorting browse results (multivalued)
>>> >>
>>> >> callnumber-lc-display: normalized for display in browse results (multivalued)
>>> >>
>>> >> "callnumber-lc-search": [
>>> >>
>>> >>           "JA 41 Z4",
>>> >>
>>> >>           "JA 41 Z7",
>>> >>
>>> >>           "JA 41 Z68"
>>> >>
>>> >>         ],
>>> >> "callnumber-lc-sort": "JA  0041.000000 Z0.400000",
>>> >>
>>> >> "callnumber-lc-browse": [
>>> >>
>>> >>           "JA  0041.000000 Z0.400000",
>>> >>
>>> >>           "JA  0041.000000 Z0.700000",
>>> >>
>>> >>           "JA  0041.000000 Z0.680000"
>>> >>
>>> >>         ],
>>> >> "callnumber-lc-display": [
>>> >>
>>> >>           "JA41.Z4",
>>> >>
>>> >>           "JA41.Z7",
>>> >>
>>> >>           "JA41.Z68"
>>> >>
>>> >>         ],
>>> >>
>>> >>
>>> >>
>>> >> Thanks!!
>>> >> Anna
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Oct 16, 2014 at 1:04 PM, Tod Olson <to...@uc...<mailto:to...@uc...>> wrote:
>>> >>
>>> >> In solrconfig.xml, try callnumber-lc-sort as the field value.
>>> >>
>>> >> -Tod
>>> >>
>>> >>
>>> >>
>>> >> On Oct 16, 2014, at 11:06 AM, Anna Headley <an...@gm...<mailto:an...@gm...>> wrote:
>>> >>
>>> >> > Hi all,
>>> >> >
>>> >> > I'm making changes to my callnumber browse setup and I'm finding that it's dropping me into the list by matching my query against the sortfield instead of the valuefield.
>>> >> >
>>> >> > e.g.,
>>> >> >
>>> >> > I enter JA41 where (in my small index) the list should show JA41.ZA<http://ja41.za/>. Instead it shows PC33something. (like I said my index is tiny.)
>>> >> >
>>> >> > if I enter JA  0041 I get the desired result.
>>> >> >
>>> >> > I'm going to paste much of my relevant diff in case that would be helpful. I hope I have simply messed up the configuration somehow, not that the handler does the search against the sortfield.
>>> >> >
>>> >> > Thanks!!
>>> >> > Anna
>>> >> >
>>> >> >
>>> >> > --- a/local/import/marc_local.properties
>>> >> > +++ b/local/import/marc_local.properties
>>> >> > @@ -52,6 +52,8 @@ callnumber-label = custom, getCallNumberLabel(090a:050a:949i)
>>> >> >  # new trico callnumber fields for searching, sorting
>>> >> >  callnumber-lc-sort = script(tricolccn.bsh), getSortLccn(090)
>>> >> >  callnumber-lc-search = script(tricolccn.bsh), getSearchLccns(090)
>>> >> > +callnumber-lc-display = script(tricolccn.bsh), getBrowseLccns(090)
>>> >> > +callnumber-lc-browse = script(tricolccn.bsh), getSortLccns(090)
>>> >> >
>>> >> > --- a/index-alphabetic-browse.sh
>>> >> > +++ b/index-alphabetic-browse.sh
>>> >> > @@ -72,5 +72,5 @@ function build_browse
>>> >> >  build_browse "title" "title_fullStr" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=title_sort -Dvaluefield=title_fullStr"
>>> >> >  build_browse "topic" "topic_browse"
>>> >> >  build_browse "author" "author_browse"
>>> >> > -#build_browse "lcc" "callnumber-a" 1
>>> >> > +build_browse "lcc" "callnumber-lc-display" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=callnumber-lc-browse -Dvaluefield=callnumber-lc-display"
>>> >> >
>>> >> > --- a/local/config/vufind/searchspecs.yaml
>>> >> > +++ b/local/config/vufind/searchspecs.yaml
>>> >> > @@ -523,7 +523,7 @@ DeweyBrowse:
>>> >> >
>>> >> >  LccBrowse:
>>> >> >    QueryFields:
>>> >> > -    callnumber-a:
>>> >> > +    callnumber-lc-display:
>>> >> >        - [onephrase, ~]
>>> >> >
>>> >> > --- a/solr/biblio/conf/schema.xml
>>> >> > +++ b/solr/biblio/conf/schema.xml
>>> >> > @@ -177,6 +177,8 @@
>>> >> >     <field name="callnumber-label" type="string" indexed="true" stored="true"/>
>>> >> >     <field name="callnumber-lc-sort" type="string" indexed="true" stored="true"/>
>>> >> >     <field name="callnumber-lc-search" type="string" indexed="true" stored="true" multiValued="true"/>
>>> >> > +   <field name="callnumber-lc-browse" type="string" indexed="true" stored="true" multiValued="true"/>
>>> >> > +   <field name="callnumber-lc-display" type="string" indexed="true" stored="true" multiValued="true"/>
>>> >> >     <field name="dewey-hundreds" type="string" indexed="true" stored="true" multiValued="true"/>
>>> >> >     <field name="dewey-tens" type="string" indexed="true" stored="true" multiValued="true"/>
>>> >> >     <field name="dewey-ones" type="string" indexed="true" stored="true" multiValued="true"/>
>>> >> >
>>> >> > --- a/solr/biblio/conf/solrconfig.xml
>>> >> > +++ b/solr/biblio/conf/solrconfig.xml
>>> >> > @@ -484,7 +484,7 @@
>>> >> >
>>> >> >      <lst name="lcc">
>>> >> >        <str name="DBpath">${solr.solr.home:./solr}/alphabetical_browse/lcc_browse.db</str>
>>> >> > -      <str name="field">callnumber-a</str>
>>> >> > +      <str name="field">callnumber-lc-display</str>
>>> >> >      </lst>
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >>
>>> >> > ------------------------------------------------------------------------------
>>> >> > Comprehensive Server Monitoring with Site24x7.
>>> >> > Monitor 10 servers for $9/Month.
>>> >> > Get alerted through email, SMS, voice calls or mobile push notifications.
>>> >> > Take corrective actions from your mobile device.
>>> >> > http://p.sf.net/sfu/Zoho_______________________________________________
>>> >> > Vufind-tech mailing list
>>> >> > Vuf...@li...<mailto:Vuf...@li...>
>>> >> > https://lists.sourceforge.net/lists/listinfo/vufind-tech
>>> >>
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>
>>
>
>
>
>