From: Mosior, B. <BEM...@sh...> - 2013-01-17 19:18:46
|
We are curious about how other institutions and consortia handle high-availability/redundancy of the ILS backend with respect to VuFind. While we maintain a secondary read-only ILS backend instance, we were pondering the pros and cons of writing a VuFind caching mechanism so that item statuses and locations would be stored in the Solr index and regularly updated in order to be available if the ILS backend was down. Does that seem like a reasonable approach, or is the secondary ILS backend instance approach better? For us, it seems like a Solr performance vs cost for Oracle licensing (we currently use Voyager) kind of issue. Benjamin Mosior Keystone Library Network |
From: Demian K. <dem...@vi...> - 2013-01-17 19:28:27
|
Have you looked at the "NoILS" driver yet? You can configure VuFind to load NoILS if the main driver throws an exception (or you can manually make it the primary ILS driver), and NoILS can be configured to load location and status information from holdings fields in your stored MARC records in the Solr index. That might prove to be a fairly simple Solr-based solution - just export bib + holdings from Voyager, use the SolrMarc feature that merges the holdings into the bibs (I can share documentation if you're not familiar with it), and configure NoILS to use the 852 fields. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 2:19 PM To: vuf...@li...; vuf...@li... Subject: [VuFind-General] ILS Backend Availability and VuFind (2.x) We are curious about how other institutions and consortia handle high-availability/redundancy of the ILS backend with respect to VuFind. While we maintain a secondary read-only ILS backend instance, we were pondering the pros and cons of writing a VuFind caching mechanism so that item statuses and locations would be stored in the Solr index and regularly updated in order to be available if the ILS backend was down. Does that seem like a reasonable approach, or is the secondary ILS backend instance approach better? For us, it seems like a Solr performance vs cost for Oracle licensing (we currently use Voyager) kind of issue. Benjamin Mosior Keystone Library Network |
From: Mosior, B. <BEM...@sh...> - 2013-01-17 20:04:03
|
Demian, That sounds awesome, but the question then is about keeping those fields updated. Leaning towards the side of simplicity (vs efficiency), would it make sense then to have a script that runs regularly to compare each Solr record's location/status against the ILS backend, triggering a re-export of records that have changed? I'm trying to think it through in order to reduce the number of queries necessary to perform an update, especially since we're talking 4+ million records in our case. Jay Roos replied off-list regarding the usefulness of this feature for the sake of faceting. I hadn't realized that location-based searching would be more feasible, since the query would be against the Solr index instead of the ILS. Shiny! Benjamin Mosior Keystone Library Network From: Demian Katz [mailto:dem...@vi...] Sent: Thursday, January 17, 2013 2:28 PM To: Mosior, Benjamin; vuf...@li...; vuf...@li... Subject: RE: ILS Backend Availability and VuFind (2.x) Have you looked at the "NoILS" driver yet? You can configure VuFind to load NoILS if the main driver throws an exception (or you can manually make it the primary ILS driver), and NoILS can be configured to load location and status information from holdings fields in your stored MARC records in the Solr index. That might prove to be a fairly simple Solr-based solution - just export bib + holdings from Voyager, use the SolrMarc feature that merges the holdings into the bibs (I can share documentation if you're not familiar with it), and configure NoILS to use the 852 fields. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 2:19 PM To: vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: [VuFind-General] ILS Backend Availability and VuFind (2.x) We are curious about how other institutions and consortia handle high-availability/redundancy of the ILS backend with respect to VuFind. While we maintain a secondary read-only ILS backend instance, we were pondering the pros and cons of writing a VuFind caching mechanism so that item statuses and locations would be stored in the Solr index and regularly updated in order to be available if the ILS backend was down. Does that seem like a reasonable approach, or is the secondary ILS backend instance approach better? For us, it seems like a Solr performance vs cost for Oracle licensing (we currently use Voyager) kind of issue. Benjamin Mosior Keystone Library Network |
From: Demian K. <dem...@vi...> - 2013-01-17 21:27:17
|
The question is whether Voyager's marcexport program is smart enough to export records based on a change to the availability status. If it is, then it's easy. Otherwise, as you say, some kind of scripting may be necessary. With 4 million records, continually reindexing everything is probably not entirely practical. The observation about faceting is a good point. Here at Villanova, we actually just rolled out faceting based on 852 location codes last week! There are some issues in Voyager (we haven't yet found a way to prevent the marcexport output from including suppressed holdings, and some of the records autogenerated by our ILL system cause problems), but it's at least a start. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 3:04 PM To: Demian Katz; vuf...@li...; vuf...@li... Subject: RE: ILS Backend Availability and VuFind (2.x) Demian, That sounds awesome, but the question then is about keeping those fields updated. Leaning towards the side of simplicity (vs efficiency), would it make sense then to have a script that runs regularly to compare each Solr record's location/status against the ILS backend, triggering a re-export of records that have changed? I'm trying to think it through in order to reduce the number of queries necessary to perform an update, especially since we're talking 4+ million records in our case. Jay Roos replied off-list regarding the usefulness of this feature for the sake of faceting. I hadn't realized that location-based searching would be more feasible, since the query would be against the Solr index instead of the ILS. Shiny! Benjamin Mosior Keystone Library Network From: Demian Katz [mailto:dem...@vi...] Sent: Thursday, January 17, 2013 2:28 PM To: Mosior, Benjamin; vuf...@li...; vuf...@li... Subject: RE: ILS Backend Availability and VuFind (2.x) Have you looked at the "NoILS" driver yet? You can configure VuFind to load NoILS if the main driver throws an exception (or you can manually make it the primary ILS driver), and NoILS can be configured to load location and status information from holdings fields in your stored MARC records in the Solr index. That might prove to be a fairly simple Solr-based solution - just export bib + holdings from Voyager, use the SolrMarc feature that merges the holdings into the bibs (I can share documentation if you're not familiar with it), and configure NoILS to use the 852 fields. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 2:19 PM To: vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: [VuFind-General] ILS Backend Availability and VuFind (2.x) We are curious about how other institutions and consortia handle high-availability/redundancy of the ILS backend with respect to VuFind. While we maintain a secondary read-only ILS backend instance, we were pondering the pros and cons of writing a VuFind caching mechanism so that item statuses and locations would be stored in the Solr index and regularly updated in order to be available if the ILS backend was down. Does that seem like a reasonable approach, or is the secondary ILS backend instance approach better? For us, it seems like a Solr performance vs cost for Oracle licensing (we currently use Voyager) kind of issue. Benjamin Mosior Keystone Library Network |
From: Mosior, B. <BEM...@sh...> - 2013-01-18 14:00:59
|
Demian, I think the marcexport program is capable of exporting records that have changed in the last X days. If so, I'm hopeful that we can go that route instead of an additional script. As far as faceting and preventing suppressed items, in other applications we've found that we can control those sorts of details by selecting Voyager BIB IDs based on SQL Queries, then using marcexport to export based on the set of BIB IDs: my $query = qq { SELECT Bib_Master.BIB_ID FROM Location, Bib_Master, Bib_Mfhd, Mfhd_Master WHERE (Bib_Mfhd.Bib_Id=Bib_Master.Bib_Id AND Mfhd_Master.Mfhd_Id=Bib_Mfhd.Mfhd_Id AND Mfhd_Master.Location_Id=Location.Location_Id) AND ((NOT Bib_Master.Suppress_In_Opac='Y') AND (NOT Mfhd_Master.Suppress_In_Opac='Y') AND Location.Location_Id NOT IN ($excluded_locations)) }; ... where $excluded_locations contains a comma-separated list of location IDs. Let me know if you have questions. Thank you again for the information. I'm generally encouraged that we might be able to do record status/location "caching" in Solr as an alternative to a secondary ILS DB server. I will update as we make progress. Hopefully we can generate some VuFind 2.x documentation for the process. Benjamin Mosior Keystone Library Network From: Demian Katz [mailto:dem...@vi...] Sent: Thursday, January 17, 2013 4:27 PM To: Mosior, Benjamin; vuf...@li...; vuf...@li... Subject: RE: ILS Backend Availability and VuFind (2.x) The question is whether Voyager's marcexport program is smart enough to export records based on a change to the availability status. If it is, then it's easy. Otherwise, as you say, some kind of scripting may be necessary. With 4 million records, continually reindexing everything is probably not entirely practical. The observation about faceting is a good point. Here at Villanova, we actually just rolled out faceting based on 852 location codes last week! There are some issues in Voyager (we haven't yet found a way to prevent the marcexport output from including suppressed holdings, and some of the records autogenerated by our ILL system cause problems), but it's at least a start. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 3:04 PM To: Demian Katz; vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: RE: ILS Backend Availability and VuFind (2.x) Demian, That sounds awesome, but the question then is about keeping those fields updated. Leaning towards the side of simplicity (vs efficiency), would it make sense then to have a script that runs regularly to compare each Solr record's location/status against the ILS backend, triggering a re-export of records that have changed? I'm trying to think it through in order to reduce the number of queries necessary to perform an update, especially since we're talking 4+ million records in our case. Jay Roos replied off-list regarding the usefulness of this feature for the sake of faceting. I hadn't realized that location-based searching would be more feasible, since the query would be against the Solr index instead of the ILS. Shiny! Benjamin Mosior Keystone Library Network From: Demian Katz [mailto:dem...@vi...] Sent: Thursday, January 17, 2013 2:28 PM To: Mosior, Benjamin; vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: RE: ILS Backend Availability and VuFind (2.x) Have you looked at the "NoILS" driver yet? You can configure VuFind to load NoILS if the main driver throws an exception (or you can manually make it the primary ILS driver), and NoILS can be configured to load location and status information from holdings fields in your stored MARC records in the Solr index. That might prove to be a fairly simple Solr-based solution - just export bib + holdings from Voyager, use the SolrMarc feature that merges the holdings into the bibs (I can share documentation if you're not familiar with it), and configure NoILS to use the 852 fields. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 2:19 PM To: vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: [VuFind-General] ILS Backend Availability and VuFind (2.x) We are curious about how other institutions and consortia handle high-availability/redundancy of the ILS backend with respect to VuFind. While we maintain a secondary read-only ILS backend instance, we were pondering the pros and cons of writing a VuFind caching mechanism so that item statuses and locations would be stored in the Solr index and regularly updated in order to be available if the ILS backend was down. Does that seem like a reasonable approach, or is the secondary ILS backend instance approach better? For us, it seems like a Solr performance vs cost for Oracle licensing (we currently use Voyager) kind of issue. Benjamin Mosior Keystone Library Network |
From: Demian K. <dem...@vi...> - 2013-01-18 14:11:10
|
marcexport can definitely export records that have changed in the last X days -- that is how we do nightly updates at Villanova. The question I have is what counts as a change? Is it a bib change? A MFHD change? A circulation status change? I haven't investigated this in any detail, so I'm just speculating here -- but I wonder if the change detection may be limited to bibs... in which case you would need to go to the SQL query/bib ID approach you described. Regarding suppressed MFHDs, the problem we have is that a single bib may have one suppressed MFHD and one non-suppressed MFHD. When we ask marcexport to give us the bib with holdings, we get both MFHDs. I'm not sure that the SQL query approach will help there -- though maybe I'm missing something. - Demian ________________________________ From: Mosior, Benjamin [BEM...@sh...] Sent: Friday, January 18, 2013 9:00 AM To: Demian Katz; vuf...@li...; vuf...@li... Subject: RE: ILS Backend Availability and VuFind (2.x) Demian, I think the marcexport program is capable of exporting records that have changed in the last X days. If so, I’m hopeful that we can go that route instead of an additional script. As far as faceting and preventing suppressed items, in other applications we’ve found that we can control those sorts of details by selecting Voyager BIB IDs based on SQL Queries, then using marcexport to export based on the set of BIB IDs: my $query = qq { SELECT Bib_Master.BIB_ID FROM Location, Bib_Master, Bib_Mfhd, Mfhd_Master WHERE (Bib_Mfhd.Bib_Id=Bib_Master.Bib_Id AND Mfhd_Master.Mfhd_Id=Bib_Mfhd.Mfhd_Id AND Mfhd_Master.Location_Id=Location.Location_Id) AND ((NOT Bib_Master.Suppress_In_Opac='Y') AND (NOT Mfhd_Master.Suppress_In_Opac='Y') AND Location.Location_Id NOT IN ($excluded_locations)) }; … where $excluded_locations contains a comma-separated list of location IDs. Let me know if you have questions. Thank you again for the information. I’m generally encouraged that we might be able to do record status/location “caching” in Solr as an alternative to a secondary ILS DB server. I will update as we make progress. Hopefully we can generate some VuFind 2.x documentation for the process. Benjamin Mosior Keystone Library Network From: Demian Katz [mailto:dem...@vi...] Sent: Thursday, January 17, 2013 4:27 PM To: Mosior, Benjamin; vuf...@li...; vuf...@li... Subject: RE: ILS Backend Availability and VuFind (2.x) The question is whether Voyager’s marcexport program is smart enough to export records based on a change to the availability status. If it is, then it’s easy. Otherwise, as you say, some kind of scripting may be necessary. With 4 million records, continually reindexing everything is probably not entirely practical. The observation about faceting is a good point. Here at Villanova, we actually just rolled out faceting based on 852 location codes last week! There are some issues in Voyager (we haven’t yet found a way to prevent the marcexport output from including suppressed holdings, and some of the records autogenerated by our ILL system cause problems), but it’s at least a start. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 3:04 PM To: Demian Katz; vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: RE: ILS Backend Availability and VuFind (2.x) Demian, That sounds awesome, but the question then is about keeping those fields updated. Leaning towards the side of simplicity (vs efficiency), would it make sense then to have a script that runs regularly to compare each Solr record’s location/status against the ILS backend, triggering a re-export of records that have changed? I’m trying to think it through in order to reduce the number of queries necessary to perform an update, especially since we’re talking 4+ million records in our case. Jay Roos replied off-list regarding the usefulness of this feature for the sake of faceting. I hadn’t realized that location-based searching would be more feasible, since the query would be against the Solr index instead of the ILS. Shiny! Benjamin Mosior Keystone Library Network From: Demian Katz [mailto:dem...@vi...] Sent: Thursday, January 17, 2013 2:28 PM To: Mosior, Benjamin; vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: RE: ILS Backend Availability and VuFind (2.x) Have you looked at the “NoILS” driver yet? You can configure VuFind to load NoILS if the main driver throws an exception (or you can manually make it the primary ILS driver), and NoILS can be configured to load location and status information from holdings fields in your stored MARC records in the Solr index. That might prove to be a fairly simple Solr-based solution – just export bib + holdings from Voyager, use the SolrMarc feature that merges the holdings into the bibs (I can share documentation if you’re not familiar with it), and configure NoILS to use the 852 fields. - Demian From: Mosior, Benjamin [mailto:BEM...@sh...] Sent: Thursday, January 17, 2013 2:19 PM To: vuf...@li...<mailto:vuf...@li...>; vuf...@li...<mailto:vuf...@li...> Subject: [VuFind-General] ILS Backend Availability and VuFind (2.x) We are curious about how other institutions and consortia handle high-availability/redundancy of the ILS backend with respect to VuFind. While we maintain a secondary read-only ILS backend instance, we were pondering the pros and cons of writing a VuFind caching mechanism so that item statuses and locations would be stored in the Solr index and regularly updated in order to be available if the ILS backend was down. Does that seem like a reasonable approach, or is the secondary ILS backend instance approach better? For us, it seems like a Solr performance vs cost for Oracle licensing (we currently use Voyager) kind of issue. Benjamin Mosior Keystone Library Network |