Re: [VuFind-Tech] New SearchObject

[Demian K. <dem...@vi...>]

Sorry, I really should have read the original email in more detail before replying -- I see now that you're already talking about moving search history to the database.  My comments still stand, though -- I think using the session is a viable option.  Perhaps this should be a configurable setting -- use the session if you want short-lived history, use the database if you want it to be remembered forever.  Different institutions may have different preferences on this behavior.   Either way, I think a "clear history" function would be helpful if we actually do decide to store it long-term...  and maybe some kind of limit setting to prevent it from growing indefinitely in the database.

- Demian

From: Demian Katz [mailto:dem...@vi...]
Sent: Thursday, September 10, 2009 8:56 AM
To: Greg Pendlebury; 'Bill Dueber'
Cc: vuf...@li...
Subject: Re: [VuFind-Tech] New SearchObject

First of all, I haven't had a chance to look at your SearchObject work in great detail yet, but I like the ideas behind it.  Hopefully I'll have some comments on the gory details once I have time to study it more closely.

Just one thought for now -- why are we using cookies to store search history?  We have a session in place now to maintain login, so why not use that for search history as well?  It's still a good idea to keep sessions from getting too bloated (loading giant sessions can significantly impact performance), but you can go well beyond 4k without any trouble.  I think we have nothing to lose by switching from cookie to session, and a lot of flexibility to gain.  Does anyone disagree?

- Demian

From: Greg Pendlebury [mailto:Gre...@us...]
Sent: Wednesday, September 09, 2009 11:03 PM
To: 'Bill Dueber'
Cc: vuf...@li...
Subject: Re: [VuFind-Tech] New SearchObject

Sounds like what we had in mind Bill, I think once the SearchObject stands on it's own two feet I'll extend it (or the suite of object like it and Solr.php) in that direction... if someone doesn't beat me to it :)



Anyway, I wanted to mention the work I did Yesterday on using the SearchObject in search history.

My first try simply serialised the object and jammed it into the cookie... it failed. Seems like those pesky 4kb cookies don't like having massive objects put into them (the Solr object, complete with response from solr was part of SearchObject).

Second try, trimming out the bigger objects resulted in a serialised string of 1.3kb (still ouch). I should note though that this was an advanced search with three filters applied.

Finally I implemented some complete 'minification' code to trim the object down as small as I could whilst retaining functional data. Now it's 452 bytes. By comparison, the url we used to store for the same search was 300 bytes, so it's not so bad.

Attached is a screenshot of the search history screen on my laptop. You'll note there's extra data inside the object too, time, results and query speed (not shown in the screen shot).

For further comparison I trimmed the extra data from the minified object to look at the old 'basic' data that was previously stored as a url:
300b (url) => 393b (object) => 452b (object + extras)

The same measurements on a super simple search (no terms, all fields).
64b (url) => 146b (object) => 208b (object + extras)

I'm not overly concerned about the growth, even if every search the user did was a bloated search we are going from 13 in a cookie to 9.

And beyond all of that our internal plans are to move the search history into the database. If the user isn't logged in it will be tied to the session table, once they log in we can move the history to be attached to their user table entry so it moves with them from computer to computer.

I see no reason why we can't eventually also save on overheads by simply using the stats table as a search history instead. We just sanitise the user data when sessions expire, time limits elapse or the user purges their history.



Usage I've tried to keep reasonably simple...

// Storage
private function addToHistory() {
    ...
    $searchHistory[] = serialize($this->minify());
    ...
}

private function minify() {
    $newObject = new minSO($this);
    return $newObject;
}



// Usage
$searchObject->deminify(unserialize($search));

$links[] = array(
    'time'        => date("g:ia, jS M y", $searchObject->getStartTime()),
    'url'         => $searchObject->renderSearchUrl(),
    'description' => $searchObject->displayQuery(),
    'filters'     => $searchObject->getFilterList(),
    'hits'        => number_format($searchObject->getResultTotal()),
    'speed'       => round($searchObject->getQuerySpeed(), 2)."s",
    // Size is purely for debugging. Not currently displayed in the template.
    // It's the size of the serialized, minified search in the cookie.
    'size'        => round($size/1024, 2)."kb"
);

Greg Pendlebury
Electronic Services Officer (Systems Team)
Division of Academic Information Services
University of Southern Queensland
Phone: +61 7 4631 1501
Fax: +61 7 4631 1841


________________________________
From: bil...@gm... [mailto:bil...@gm...] On Behalf Of Bill Dueber
Sent: Wednesday, 9 September 2009 11:30 PM
To: Greg Pendlebury
Cc: vuf...@li...
Subject: Re: [VuFind-Tech] New SearchObject

On Tue, Sep 8, 2009 at 8:32 PM, Greg Pendlebury <Gre...@us...<mailto:Gre...@us...>> wrote:

Some of the staff here aren't too fussed on the DisMax stuff, they're too in love with their old tightly defined search algorithms. I think that area can be re-integrated to the code base in a more modular way (like your yaml stuff) to allow configurations to swap between DisMax and the older style of searching at will. Ideally all of that can be done without bloating the Solr object too much. yay :(


I've thought a bit about this. The SimpleSearcher object could remain essentially the same, pulling in types, lookfors, and booleans. In the degenerate case when there's only one type and no booleans, SimpleSearch could set the search action to "dismax" and Solr.php could dispatch to dismaxSearchComponents instead of standardSearchComponents.

dismaxSearchComponents would read dismax configuration from a config file and set the appropriate key=value pairs.

I think the rest of it could remain untouched (although god knows it needs some touching).
________________________________
This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email.

The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt.

The University of Southern Queensland is a registered provider of education with the Australian Government (CRICOS Institution Code No's. QLD 00244B / NSW 02225M)