Re: [VuFind-Tech] Solr long-term "caching" of ILS holdings in VF2

[Jay R. <ja...@gm...>]

That's what the database is already doing except it's tracking other
changes (such as status) as well.


On Wed, Apr 3, 2013 at 10:20 AM, Demian Katz <dem...@vi...>wrote:

>  Do you have the ability to detect changes to the location table?  If so,
> could you set things up to reindex only the changed records on a continuous
> basis?****
>
> ** **
>
> - Demian****
>
> ** **
>
> *From:* Jay Roos [mailto:ja...@gm...]
> *Sent:* Wednesday, April 03, 2013 11:03 AM
> *To:* Demian Katz
> *Cc:* Mosior, Benjamin; vuf...@li...
>
> *Subject:* Re: [VuFind-Tech] Solr long-term "caching" of ILS holdings in
> VF2****
>
>  ** **
>
> Actually, in our environment location changes frequently because our
> collection floats. What's pretty static for us is collection.****
>
> ** **
>
> This is a very sticky problem. For Horizon libraries the database keeps
> track of records that change in a table. That table is used by the opac to
> constantly reindex as things change. I've considered piggybacking on to
> that table for VuFind, but the Solr reindexing takes so long that it would
> never be able to keep up with the rate of change to our items.****
>
> ** **
>
> On Wed, Apr 3, 2013 at 9:45 AM, Demian Katz <dem...@vi...>
> wrote:****
>
> I think this is easier if you treat “facet by location” and “availability
> status” as two separate matters.****
>
>  ****
>
> As Jay says, location information doesn’t change that often – I think it
> makes sense to simply include locations in the main biblio index and use
> them for faceting in the usual way.  With Voyager, this is just a matter of
> exporting the bib records with accompanying holdings, configuring SolrMarc
> to merge the holdings into the bibs, and creating a translation map to turn
> the location codes into human-readable strings.  (Okay, maybe that’s not
> quite a “just” – it’s a lot of work – but it can be done; I can share the
> Villanova configurations if you need them).****
>
>  ****
>
> As you say, putting availability status into the biblio index is less
> practical because it’s difficult to update.  If you have the resources,
> perhaps you could get away with setting up a dedicated box that simply
> spends all of its time reindexing all of your records (with availability
> status included) over and over again, and make it a Solr master that
> replicates to a slave server used by your VuFind instance.  Depending on
> how long each cycle takes, this might get you reasonable results… and it
> would allow faceting by availability.****
>
>  ****
>
> But if that’s too expensive or too slow, I think the next best thing is a
> secondary core for availability.  I don’t think it would take too much to
> populate this – literally just dump out the appropriate database tables to
> create an index that associates bib IDs with availability statuses.  I
> don’t think you need smarts to carefully select only relevant bib IDs – it
> doesn’t hurt if you index extra data in the name of simplicity – dumping
> everything seems reasonable to me.****
>
>  ****
>
> Assuming you go this route, the easiest way to retrieve the data from the
> secondary core would be through a custom ILS driver (as I suggested on the
> call, this might make sense as a new standard configurable feature of the
> existing NoILS driver).  The disadvantage to that approach is that you
> don’t eliminate the “Loading…” message like you wanted to (that’s much
> easier to achieve if you have the data in the biblio core rather than a
> secondary one).  You end up with a good backup system, but you don’t really
> get the caching benefits.****
>
>  ****
>
> Whichever approach you go with, I wouldn’t bother trying to update the
> index based on differences between stored values and ILS driver responses –
> I think that’s too complicated, and it has your user-facing front-end
> manipulating your central index, which strikes me as potentially
> dangerous.  I think the best approach is to simply work toward getting the
> data completely refreshed as quickly as possible.****
>
>  ****
>
> - Demian****
>
>  ****
>
> *From:* Mosior, Benjamin [mailto:BEM...@sh...]
> *Sent:* Tuesday, April 02, 2013 7:02 PM
> *To:* Jay Roos****
>
>
> *Cc:* vuf...@li...
> *Subject:* Re: [VuFind-Tech] Solr long-term "caching" of ILS holdings in
> VF2****
>
>  ****
>
> Our institutions aren't standardized as far as what they put in the
> location field. It appears that some are using physical buildings as
> locations, while others have collections in that field. Obviously any
> possible solution would need to work for everyone, not just us. ****
>
>  ****
>
> The facet question is a good point. If faceting were to be performed with
> the location, all the results would almost certainly have to be updated
> against the ILS prior to final display in order to prevent recently moved
> items from being incorrectly displayed. I was considering the possibility
> of simply removing or greying out results that were incorrectly included,
> but that's just a poor user experience, especially if a large number of
> items are modified. ****
>
>  ****
>
> Benjamin Mosior****
>  ------------------------------
>
> *From:* Jay Roos [ja...@gm...]
> *Sent:* Tuesday, April 02, 2013 16:44
> *To:* Mosior, Benjamin
> *Cc:* vuf...@li...
> *Subject:* Re: [VuFind-Tech] Solr long-term "caching" of ILS holdings in
> VF2****
>
> Would it need to contain the collection information? It seems like that
> should be static enough to be included in the main Solr index without
> frequent changes. At least that's how it is in our environment. ****
>
>  ****
>
> I'd like to see this kind of solution with the addition of tracking
> availability or status. I think a solution for faceting transient
> attributes of individual items would be a significant enhancement.****
>
>  ****
>
> I do wonder if this will be responsive enough since the "on the fly"
> updates will have to take place before the location facet could be built.
> It would have to update all of the items included in the search result
> before building the facet to be useful.****
>
>  ****
>
> On Tue, Apr 2, 2013 at 3:34 PM, Mosior, Benjamin <BEM...@sh...>
> wrote:****
>
> We had a short discussion about long-term “caching” ILS holdings
> information in a Solr index during today’s developer’s call.****
>
>  ****
>
> For context, we (the KLN) are looking to accomplish the following:****
>
> 1.       Faceting of Location/Collection information****
>
> 2.       A fallback for ILS outages.****
>
>  ****
>
> Here’s one possibility I’ve been considering after taking in some of the
> initial thoughts this morning:****
>
>  ****
>
> Storage****
>
> ·         The Location/Status data could be stored as (an) additional
> field(s) for each record in the existing Solr Index ****
>
> o   There could be an issue with updating records, since a Solr “update”
> would really involve a deletion followed by an addition of the entire
> record, require index optimization.****
>
> ·         Demian suggested the possibility of maintaining a second Solr
> index, where the recordID could be used to match statuses to records.****
>
>  ****
>
> Display****
>
> ·         A script would have to populate the data initially, performing
> a query for each record against the ILS. This would be a method for regular
> updates as well.****
>
> ·         When a results or record page loads, the Solr-cached holdings
> value could be displayed (instead of “Loading…”) while the value is checked
> against the ILS for changes. If there is a change, the new value replaces
> the old, both on the page and in the Solr index.****
>
> ·         If the above ILS lookup fails, the initial value loaded from
> the Solr index remains unchanged.****
>
>  ****
>
>  ****
>
> I’m definitely missing some possibilities, so I look forward to everyone’s
> additions to this discussion.****
>
>  ****
>
> Thanks,****
>
> Benjamin Mosior****
>
> Keystone Library Network****
>
>
>
> ------------------------------------------------------------------------------
> Minimize network downtime and maximize team effectiveness.
> Reduce network management and security costs.Learn how to hire
> the most talented Cisco Certified professionals. Visit the
> Employer Resources Portal
> http://www.cisco.com/web/learning/employer_resources/index.html
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...
> https://lists.sourceforge.net/lists/listinfo/vufind-tech****
>
>
>
> ****
>
>  ****
>
> --
> Jay Roos
> Information Technology Coordinator
> Great River Regional Library****
>
>
>
> ****
>
> ** **
>
> --
> Jay Roos
> Information Technology Coordinator
> Great River Regional Library****
>



-- 
Jay Roos
Information Technology Coordinator
Great River Regional Library