Re: [VuFind-Tech] SearchObject update (and important Advanced Searching question)

[Demian K. <dem...@vi...>]

I'm glad we're on the same page regarding the YAML search configuration.  I have limited time today, so I'll probably just do my Search Object testing this afternoon and start investigating where the YAML stuff needs to plug into the code tomorrow.  Do you think it's worth committing some or all of your Search Object progress so far in the interest of making this collaboration easier?  At the very least, it probably wouldn't hurt to check in your changes to the Search Object itself (since it isn't used in the trunk yet anyway) and possibly the Solr class (though I can't tell at a glance if your advanced search additions are likely to break anything without the rest of the supporting code).  If that's impractical, we could also create a development branch...  or I could just send you patches against your patch.  Whatever's easiest!

- Demian

From: Greg Pendlebury [mailto:Gre...@us...]
Sent: Wednesday, October 14, 2009 7:06 PM
To: Demian Katz; vuf...@li...
Subject: RE: SearchObject update (and important Advanced Searching question)


1)      True, I should have considered that. We even have an internal ticket to stop storing them before we integrate with the campus SSO system :)

2)      That would work. I'll note down the need to create something.

3)      Your thinking mirrors my own. I still have Bill's old example of the search specs in an external YAML file, and sometimes dream of having time to do something with them :). I think something like this is a good way.

Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841
From: Demian Katz [mailto:dem...@vi...]
Sent: Wednesday, 14 October 2009 10:50 PM
To: Greg Pendlebury; vuf...@li...
Subject: RE: SearchObject update (and important Advanced Searching question)

Thanks for all this work -- I'll try to start playing with the patch shortly.  In the meantime, a few first thoughts to keep things moving along:

- Regarding your question about changed passwords breaking authentication, I notice that the revised LDAP method from Franck Borel no longer saves the password in the database at all...  so at least for that method, a changed password shouldn't matter (though I haven't actually tested it).  Can't speak for the others, though.

- Regarding cleanup of the saved search table, do you think the quickest solution (for now at least) would be to add a cleanup tool to the Admin area that removes unsaved searches older than a certain date in order to clean up anything that garbage collection missed?  This probably wouldn't be needed very often, but it would be nice to have a tool to help with it.  Something in the util folder sharing the same behavior might also be nice for cron job purposes.

- Now the big one:  for advanced search, it seems to me that we still need all the old search logic in order to translate a field from the advanced search form into the actual Solr equivalent (i.e. an author search is actually author or author2, all fields is a whole bunch of different fields, etc., etc.).  This once again leads me to believe that it's probably worth implementing VUFIND-23 (external file for configuring standard Lucene searches).  As previously discussed, this might also be necessary for defining specialized things like call number searching, but regardless of that, I don't see any other way to achieve advanced searches.  Would you agree, or am I missing an easier solution?  If others also feel that this is necessary, I'm willing to take it on as my next major task after I finish cleaning up list-related problems.  That will probably just be another day or two, so if there's a better way, please speak up now before I get in too deep!

thanks,
Demian

From: Greg Pendlebury [mailto:Gre...@us...]
Sent: Wednesday, October 14, 2009 1:05 AM
To: vuf...@li...
Subject: [VuFind-Tech] SearchObject update

I've put a bit of work into this over the last week to try and progress closer to submitting to the trunk. Attached is a patch against r1606 of the trunk that includes all current work. Some basic bullet point notes are below.

As always, any feedback from testing is greatly appreciated. I was about 2-300 patches away from trunk in the usq branch, so I rebuilt the laptop codebase today and discovered I had missed maybe one or two of those patches. I'm going to recreate the usq branch as a copy of current trunk to make patch generation easier again for a while.

Database

 *   The search history is now kept in a modified 'Search' table. Added 'saved', 'session_id' and 'search_object' columns and removed 'lookfor', 'type' and 'limitto'.
 *   'last_used' is now indexed in the 'Session' table.
 *   'session_id' is now an index in the 'Search' table.
 *   In passing... I noticed that you get a key constraint on the 'User' table using the Demo driver if you don't use the same password each time (because it lets any password log in). Has anyone experienced problems with users that change their passwords via other auth methods?
 *   Schema and sql file both updated.

Saved Searches

 *   Searches now go into the 'Search' table instead of a cookie for the search history. The session object checks for searches (that aren't saved) from this session and removes them as part of it's destructor. Anecdotal (but not comprehensive) testing of session garbage collection suggests this works too.
 *   The 'saved' flag can be turned on from the top of a results list or the from the search history screen. Users are prompted to login if they aren't already.
 *   Saved searches store the 'user_id' in the table and are retrievable from anywhere the user logs in... currently with no expiry.
 *   Management of this area bears consideration... expiry dates? Limits on the number of saved searches?
 *   Possibility of orphan searches evading the garbage collector. 1) User creates the search. 2) User saves the search. 3) Garbage collector for the session fires, but ignores the search because it is saved. 4) User 'un-saves' the search and it return to their 'un-saved' search history. 5) Search is now an orphan the garbage collector won't find because it's session has already been cleaned. Easy solution would be to delete entirely instead of 'un-save'.
 *   Saved searches can be passed into the results page and retrieved. No facility for making public yet (shouldn't be too hard). Results screen ensures users can only view searches from their session or user_id.
 *   Saved searches can be handed to the advanced search screen for editing.

Login Followups

 *   Some initial work in improving the current code allowing for followup actions on a login screen.
 *   This is currently quite rough and could be overhauled in general.
 *   Used at the moment to challenge the user for a login before saving their search.

Advanced Search

 *   Completely overhauled (has been demo'd previously) to replace 'Search Within' functionality.
 *   Allows editing of old searches.
 *   Old searches will retain the filters they had applied by default. New interface elements allow the filters to be removed.
 *   Still needs <noscript> functionality.
 *   Still needs old index fields to be updated post-dismax changes.

Layout Search Form

 *   The basic search form is hidden on the advanced search screen.
 *   'Search Within' is gone, in it's place is a checkbox to retain the currently active filter(s) when starting a new search. Default to 'checked'.
 *   The results screen of an advanced search does not display a basic search form. Instead it displays links to start new searches or edit the current advanced search.

Spell checking

 *   The search object now processes the spelling suggestions provided by results. When the array is retrieved it will provide urls for using the new terms as both replacements and expansions on the current search.
 *   Replacements are a simple find/replace on the suggested term, default usage and what the interface currently presents as the link underlying the term on screen.
 *   Expansion allow you to add the new term into the search as an OR where the old term is found. Currently it's the '(E)' beside each term. Our staff here are arguing over an icon (as always).
 *   I've added them into the interface on the assumption that people who don't want them will just disable them.
 *   My dev laptop returns multiple suggestions. Testing the patch today against a fresh install it was only returning one, so I need to look at this. Probably index related.
 *   Still have concerns over 'onlyMorePopular' but I don't think there's a solution short of modifying solr/lucene.
 *   Using a recent solr nightly was required to get the most out of spell checking.

Language File

 *   I've started using more meaningful indexes in the language file so when reading the language file you would know when/why the string is being used.
 *   It's an idea and an example, please let me know if you greatly object to this change.

Search Object

 *   Fixes all url mangling issues (that I've looked for) relating to facets, pages, sorting etc on the results screen.
 *   My old RSS fix is currently in place. I haven't yet looked at Andrew's recent work on this to integrate.
 *   Still doesn't integrate with statistics, this is the last area in Results.php to change (from memory).


Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841


________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government (CRICOS Institution Code No's. QLD 00244B / NSW 02225M)
________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government (CRICOS Institution Code No's. QLD 00244B / NSW 02225M)