Re: [VuFind-Tech] Dewey with 1.1

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Those anomalies are likely related to records with multiple call numbers -- since the Browse feature is built on faceting, it just says "give me all the tens-level call numbers from records that also contain this hundreds-level call number."  If everything is single-valued, that works really well, but when you have different call numbers on a single record, you'll see some anomalies.  If that doesn't turn out to be the cause of the problem, please let me know and I can investigate further!

- Demian

> -----Original Message-----
> From: Matthew Hooper [mailto:mat...@fl...]
> Sent: Wednesday, March 30, 2011 10:21 PM
> To: vuf...@li...
> Subject: Re: [VuFind-Tech] Dewey with 1.1
>
> Hi Demian,
>
> Call number browse appears to be working fine now after a reindex and
> stop/start of the services. There does seem to still be some anomalies
> when you drill down to a series like the 800's with some 7xx and 9xx
> call numbers appearing in the list, but I can't remember whether that
> was happening before the upgrade or not.
>
> Thanks,
>
>         Matt.
>
> --
> Matthew Hooper
> Systems Officer,
> Flinders University Library
> G.P.O. Box 2100
> ADELAIDE, South Australia 5001
> Phone:    +618 8201 2068
> Fax:      +618 8201 2508
> Email:    Mat...@fl...
>
>
> -----Original Message-----
> From: Demian Katz [mailto:dem...@vi...]
> Sent: Wednesday, 30 March 2011 9:39 PM
> To: Matthew Hooper; vuf...@li...
> Subject: RE: [VuFind-Tech] Dewey with 1.1
>
> The fixed SolrMarc was released this morning (thanks, Bob!).  You can
> download it here:
>
> http://code.google.com/p/solrmarc/downloads/list
>
> Just grab the GenericVuFind version and copy the .jar files from the
> archive into your vufind/import directory.
>
> Please let me know if you continue to have problems after this!
>
> - Demian
> ________________________________________
> From: Matthew Hooper [mat...@fl...]
> Sent: Tuesday, March 29, 2011 9:19 PM
> To: vuf...@li...
> Subject: Re: [VuFind-Tech] Dewey with 1.1
>
> Thanks Demian,
>
> I’ve turned off dewey call number browse for the moment since that’s
> where the issue is more apparent, but if you could let the list know
> once the solr update has been committed then I’ll grab a fresh copy and
> give it a whirl.
>
> Cheers,
>
>                Matt.
>
> --
> Matthew Hooper
> Systems Officer,
> Flinders University Library
> G.P.O. Box 2100
> ADELAIDE, South Australia 5001
> Phone:  +618 8201 2068
> Fax:       +618 8201 2508
> Email:
> Mat...@fl...<mailto:Mat...@fl...>
>
> From: Demian Katz [mailto:dem...@vi...]
> Sent: Tuesday, 29 March 2011 11:46 PM
> To: Matthew Hooper; vuf...@li...
> Subject: RE: [VuFind-Tech] Dewey with 1.1
>
> It sounds like there should be a new SolrMarc release in the near
> future to fix this problem; if you can wait a week or two, the easiest
> solution may be to wait for that and then simply replace the SolrMarc
> .jar files with the new version (in fact, if you’re impatient, you
> could build it from source now to achieve the same effect – it looks
> like the fix was committed to the SolrMarc trunk yesterday).
>
> If you don’t want to deal with SolrMarc right now, the other option is
> to uncomment the dewey.bsh-related lines in
> import/marc_local.properties and edit the BeanShell code in
> import/index_scripts/dewey.bsh to change all the “if
> (CallNumUtils.isValidDewey(current))” lines to “if (true)”.  Of course,
> if you have bad data in the Dewey fields, this may let it through and
> cause a different set of problems…  but it’s the quickest patch that
> doesn’t involve upgrading SolrMarc.
>
> Apologies for the inconvenience, and please let me know if you still
> have questions.
>
> - Demian
>
> From: Matthew Hooper [mailto:mat...@fl...]
> Sent: Tuesday, March 29, 2011 3:05 AM
> To: vuf...@li...
> Subject: Re: [VuFind-Tech] Dewey with 1.1
>
> Hi Demian,
>
> I’ve just run up 1.1 with Dewey support, and can confirm that there’s
> something going on with how the records are imported. Our call numbers
> are stored in the $984c), and if you take a look at the call number
> browse screen at http://library.flinders.edu.au/vufind/ you can see
> there are large chunks missing from the indexed call number ranges. Did
> you have those instructions handy on how to change the dewey parsing
> routines to be more lenient? That way I can do a reload and see if it
> brings back those records.
>
> Thanks,
>
>                 Matt.
>
> --
> Matthew Hooper
> Systems Officer,
> Flinders University Library
> G.P.O. Box 2100
> ADELAIDE, South Australia 5001
> Phone:  +618 8201 2068
> Fax:       +618 8201 2508
> Email:
> Mat...@fl...<mailto:Mat...@fl...>
>
> From: Demian Katz [mailto:dem...@vi...]
> Sent: Wednesday, 23 March 2011 6:30 AM
> To: ho...@hs...
> Cc: vuf...@li...
> Subject: Re: [VuFind-Tech] Dewey with 1.1
>
> Okay, I see the problem now – SolrMarc includes a Dewey validation
> function which checks call numbers against a regular expression before
> indexing them.  It expects at the very minimum, three digits followed
> by a letter.  Your MARC data only has three digits – no letter.
>
> Two possible solutions:
>
>
> 1.)    Is there data missing from your MARC export?  Can you get more
> detailed Dewey numbers?
>
> 2.)    If you can’t change the data, you can revert to using the
> dewey.bsh script in place of the built-in SolrMarc custom function
> (restoring the lines I just told you to remove from
> marc_local.properties), and you can change the
> CallNumUtils.isValidDewey() calls in dewey.bsh to something else that
> allows more forgiving validation of the Dewey numbers.
>
> It appears that the Dewey validation logic was made more strict with
> SolrMarc release 2.2, which explains why the problem has just cropped
> up for you.  I’ll send a note to the solrmarc-tech list to see what the
> reasoning was behind this change, and perhaps the next version of
> SolrMarc can restore the more lenient analysis.
>
> - Demian
>
> From: John C. Houser [mailto:ho...@pa...]
> Sent: Tuesday, March 22, 2011 3:36 PM
> To: Demian Katz
> Cc: ho...@hs...; vuf...@li...
> Subject: Re: [VuFind-Tech] Dewey with 1.1
>
> Demian,
> We'll definitely close the port, when we're live. This is just a test
> server.
>
> No, I've not modified the marc.properties at all.
>
> I've attached a sample file with 10 USMARC records in it that have
> Dewey numbers in the 082a. I extracted them from the larger test file I
> loaded.
>
> J
> ________________________________
>
> [cid:image001.jpg@01CBEECD.444F86C0]
>
> Demian Katz<mailto:dem...@vi...>
> March 22, 2011 3:14 PM
>
>
> You will want to firewall off that port once you go live; mischievous
> users can do some damage if they can access Solr directly.  In the
> meantime, though, it’s definitely convenient to be able to share the
> link!
>
> It definitely looks like there is no data getting indexed into the
> dewey-tens field – that’s probably a good thing, since it at least
> narrows the problem to the import process rather than something more
> complicated buried in the guts of VuFind itself.  Is your
> import/marc.properties file customized at all, or are you using the one
> that came with 1.1?  Would it be possible for you to share a sample
> MARC record containing Dewey data?  Perhaps as a sanity check it would
> be useful for me to try indexing it here to be sure it works on my end.
>
> - Demian
>
> -----------------------------------------------------------------------
> -------
> Enable your software for Intel(R) Active Management Technology to meet
> the
> growing manageability and security demands of your customers.
> Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your
> software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...<mailto:Vufind-
> te...@li...>
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
> ________________________________
>
> [cid:image001.jpg@01CBEECD.444F86C0]
>
> John C. Houser<mailto:ho...@pa...>
> March 22, 2011 3:07 PM
>
>
> Demian,
>
> This URL worked: http://vufind101.accesspa.org:8080/solr/biblio/admin/
>
> It's open to the public, so you can try it yourself.
>
> The query you suggested, returned this:
> −
> <response>
> −
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">1</int>
> −
> <lst name="params">
> <str name="indent">on</str>
> <str name="start">0</str>
> <str name="q">dewey-tens:[* TO *]</str>
> <str name="version">2.2</str>
> <str name="rows">10</str>
> </lst>
> </lst>
> <result name="response" numFound="0" start="0"/>
> </response>
> -----------------------------------------------------------------------
> -------
> Enable your software for Intel(R) Active Management Technology to meet
> the
> growing manageability and security demands of your customers.
> Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your
> software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...<mailto:Vufind-
> te...@li...>
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
> ________________________________
>
> [cid:image001.jpg@01CBEECD.444F86C0]
>
> Demian Katz<mailto:dem...@vi...>
> March 22, 2011 2:30 PM
>
>
> I doubt this is the cause of your problem, but you can actually remove
> all of those dewey lines from marc_local.properties.  The Dewey
> functionality is now part of the SolrMarc core, so it is no longer
> necessary to use the external dewey.bsh file.  The default Dewey
> behavior in marc.properties does the same thing as your BeanShell-based
> configuration in marc_local.properties… only it should be somewhat
> faster.
>
> I also recommend getting rid of all the strip_trailing_dot stuff at the
> bottom of the file – the punctuation cleanup is now handled internally
> by Solr, so you don’t need to do it in SolrMarc.  Also, the era field
> has been renamed to era_facet, so that may be causing unexpected
> behavior.  Again, probably not the cause of your current specific
> problem, but worth cleaning up and reindexing.
>
> Is it possible that a problem was introduced when you customized the
> Solr schema?  I see that you have a custom series_facet field.  Did you
> manually re-add that in 1.1?
>
> Have you tried looking at the index in the Solr admin panel (Probably
> available at http://your_vufind_server:8080/solr/biblio)?  If you do a
> search for
>
> dewey-tens:[* TO *]
>
> …what do you get?
>
> - Demian
>
> -----------------------------------------------------------------------
> -------
> Enable your software for Intel(R) Active Management Technology to meet
> the
> growing manageability and security demands of your customers.
> Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your
> software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...<mailto:Vufind-
> te...@li...>
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
> ________________________________
>
> [cid:image001.jpg@01CBEECD.444F86C0]
>
> John C. Houser<mailto:ho...@pa...>
> March 22, 2011 2:21 PM
>
>
> Demian,
> I migrated the import rules. I also tried using the default
> marc.properties settings. The config.ini has dewey = "1" in the
> [Browse] section. The facets.ini settings appear to have migrated okay.
> I've pasted in my marc_local.properties below. What other details would
> help?
>
> J
>
> ***
>
> # Uncomment the following settings to insert appropriate values for
> your site:
> collection = "HSLC Evergreen Group"
> institution = 852a, evergreen_institution.properties(institution)
> building = 852b, evergreen_location.properties(location)
>
> # Uncomment the following settings to use the .bsh scripts in
> import/scripts/
> # instead of the built-in SolrMarc functionality found in the .jar
> file.
> # (by default, the scripts have the same behavior as the built-in
> functions,
> # but the external scripts are easier to customize to your needs).
> #format = script(format.bsh), getFormat, format_map.properties
> #callnumber = script(callnumber.bsh), getFullCallNumber
> #callnumber-subject = script(callnumber.bsh), getCallNumberSubject,
> callnumber_subject_map.properties
> #callnumber-subject-code = script(callnumber.bsh), getCallNumberSubject
> #callnumber-label = script(callnumber.bsh), getCallNumberLabel
>
> # Uncomment the following settings to index Dewey call numbers:
> dewey-hundreds = script(dewey.bsh), getDeweyNumber(082a:083a, 100),
> ddc22_map.properties(hundreds)
> dewey-tens = script(dewey.bsh), getDeweyNumber(082a:083a, 10),
> ddc22_map.properties(tens)
> dewey-ones = script(dewey.bsh), getDeweyNumber(082a:083a, 1),
> ddc22_map.properties(ones)
> dewey-full = script(dewey.bsh), getDeweySearchable(082a:083a)
> dewey-sort = script(dewey.bsh), getDeweySortable(082a:083a)
>
> # HSLC Evergreen Series facet
> series_facet = custom, removeTrailingPunct(490a)
>
> # Strip dots only
> topic_facet = 600x:610x:611x:630x:648x:650a:650x:651x:655x,
> (pattern_map.strip_trailing_dot)
> genre_facet = 600v:610v:611v:630v:648v:650v:651v:655a:655v,
> (pattern_map.strip_trailing_dot)
> geographic_facet = 600z:610z:611z:630z:648z:650z:651a:651z:655z,
> (pattern_map.strip_trailing_dot)
> era = 600d:610y:611y:630y:648a:648y:650y:651y:655y,
> (pattern_map.strip_trailing_dot)
> pattern_map.strip_trailing_dot.pattern_0 = (.*)\\.=>$1
> pattern_map.strip_trailing_dot.pattern_1 = (.*[^\\.]*)=>$1
> -----------------------------------------------------------------------
> -------
> Enable your software for Intel(R) Active Management Technology to meet
> the
> growing manageability and security demands of your customers.
> Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your
> software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...<mailto:Vufind-
> te...@li...>
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
> ________________________________
>
> [cid:image001.jpg@01CBEECD.444F86C0]
>
> Demian Katz<mailto:dem...@vi...>
> March 22, 2011 2:05 PM
>
>
> Did you migrate your SolrMarc import rules?  If you customized
> import/marc.properties or import/marc_local.properties in your RC2
> installation, you may have to copy those changes into the new
> installation.  By default, VuFind 1.1 should be populating the Dewey
> fields using 082a and 083a; if you rely on any other MARC fields, that
> might explain your problem.
>
> Also, did you double check that the settings in facets.ini,
> searches.ini and config.ini migrated over correctly?  Dewey options are
> off by default in 1.1; if you ran the upgrade script, that should have
> brought your old settings over, but if something went wrong, it’s
> possible that some configurations defaulted back to being disabled.
>
> Let me know if you still need help after checking these things – I did
> check Dewey support before making the release, so I’m pretty sure
> nothing is fundamentally broken…  but any additional details would be
> appreciated if none of these ideas turn out to provide a simple fix!
>
> - Demian
>
> From: John C. Houser [mailto:ho...@pa...]
> Sent: Tuesday, March 22, 2011 1:22 PM
> To: vuf...@li...<mailto:vufind-
> te...@li...>
> Subject: [VuFind-Tech] Dewey with 1.1
>
> Guys,
>
> I just installed VuFind 1.1, which I migrated from an older RC2
> installation. Now that I've migrated our theme and the few very minor
> code changes we've made, everything works except the Dewey call number-
> related stuff. We were using the dewey-tens index for a facet and also
> as a limit in the advanced search and in the browsing. The dewey index
> has disappeared from everything.
>
> Any ideas what I should be checking to fix this?
>
> J
> --
> John Houser
> Technology Coordinator
> HSLC
> 215-534-6820
> ho...@hs...<mailto:ho...@hs...>
>
> -----------------------------------------------------------------------
> -------
> Enable your software for Intel(R) Active Management Technology to meet
> the
> growing manageability and security demands of your customers.
> Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your
> software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...<mailto:Vufind-
> te...@li...>
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
> -----------------------------------------------------------------------
> -------
> Create and publish websites with WebMatrix
> Use the most popular FREE web apps or write code yourself;
> WebMatrix provides all the features you need to develop and
> publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...
> https://lists.sourceforge.net/lists/listinfo/vufind-tech