Re: [VuFind-Tech] More on JSON (was New code updates)

[Andrew N. <as...@gm...>]

Bill - where were you last night! :)  I found this arrar option after i had
written a quick loop to clean up the facet counts in the json response.  I
will look into this and also trying to make all fields an array.  One way
for this is just to make each field in Solr a multi-valued field - not sure
as to the reprecussions of this.

Andrew

On Fri, Sep 4, 2009 at 1:06 PM, Bill Dueber <bi...@du...> wrote:

> Another quick note -- it looks like the current trunk isn't using the '
> json.nl=arrarr' argument to Solr. It makes the returned data structures
> much easier to use, e.g.
>
> Facets without arrarr:
>  - subject 1
>  - count 1
>  - subject 2
>  - count 2
>
> Facets with arrarr
>  - [subject 1, count 1]
>  - [subject 2, count 2]
>
> It would also be nice if all potentially-multivalued fields came back in
> arrays (even if it were an array of one). Solr won't do this automatically,
> but using configuration to list all your multi-valued fields and then just
> walking through the returned data and doing if(!is_array)... would be easy
> and make the templates a lot easier to deal with (starting with alwaqys
> making sure you get an array of records back, a change we already made
> here).
>
> Even something braindead like the following would work, I think.
>
> if (isset($results['record']['id'])) { // it's a single record
>   $results['record'] = array($results['record']);
> }
> foreach ($results['record'] as $r) {
>   foreach ($listOfMultiValuedFields as $f) {
>     if (isset($r[$f] && !is_array($r[$f])) {
>       $r[$f] = array($r[$f]);
>    }
>   }
> }
>
>
>
>
>
> On Fri, Sep 4, 2009 at 12:22 PM, Bill Dueber <bi...@du...> wrote:
>
>> I just did a quickie test of my own, only looking at the Solr end. I I
>> grabbed 4681 queries from our live system (yeseterday's logfile) and did the
>> following, using JMeter with four threads and a 4-second warmup time.
>>
>>  - Run all queries against Solr (not vufind; directly at Solr) with XML
>> output
>>  - Run all queries with JSON output
>>  - Repeat XML, but measure this time
>>  - Repeat JSON, but measure this time.
>>
>> Basically, I'm looking at how long it takes Solr to generate XML vs to
>> generate JSON.
>>
>> My off-the-cuff results:
>>
>>  JSON: 272 requests/second
>>  XML:  181 requests/second
>>
>> The other end of it (how long it takes PHP to deal with XML vs JSON) is a
>> different test.
>> http://dailybraindump.com/php-unserialize-vs-json_decode-vs-simplexml/39/seems to indicate unserializing JSON is also a lot faster.
>>
>> The lesson here is not that Solr outputting JSON is 1/3 faster than XML.
>> The lesson is that SOLR IS NOT THE BOTTLENECK. I'm pretty sure I'm not
>> getting anywhere near 180 requests/second out of vufind :-)
>>
>>
>>
>> On Fri, Sep 4, 2009 at 9:51 AM, Till Kinstler <kin...@gb...> wrote:
>>
>>> Andrew Nagy schrieb:
>>>
>>> > If you would like to test these new changes, please grab the latest
>>> from
>>> > I would especially like someone with a large data
>>> > set to see how the new facets are performing.
>>>
>>> OK, I just did a quick test using the same setup as described in
>>> http://www.nabble.com/testing-tp24491925p24606102.html but with a newer
>>> Solr nightly build (2009-08-16).
>>> I ran the test only with 1 and 10 concurrent request threads, I haven't
>>> got the time to run the setup with 50 threads now. If needed/wanted, I
>>> can do that, but not before Monday, 14th, because I am going to be out
>>> of office the next week.
>>>
>>> I have included the results from the earlier test runs in the table
>>> below. r1212 was with facets pulled through separate AJAX requests,
>>> r1239 with "inline facets" pulled through the Solr XML interface, r1370
>>> is the latest trunk code with "inline facets" pulled through JSON.
>>>
>>> Results (see with fixed width font):
>>>
>>>        | Avg. response time (ms) |   Avg. throughput (1/s)
>>> threads|   1   |   10   |   50   |   1   |   10   |   50
>>> -------+-------+--------+--------+-------+--------+--------
>>> r1370  |  188  |   821  |    %   |  5,2  |   12,1 |    %
>>> r1239  |  212  |   824  |  4294  |  4,7  |   12,1 |   11,6
>>> r1212  |  144  |   823  |  4130  |  6,8  |   12,1 |   12,1
>>>
>>> I'd say: Either using JSON output or the more recent Solr nightly build
>>> or both speed up search: With 1 search thread ("user") response time
>>> drops from 212 ms to 188 ms.
>>> With 10 concurrent threads/"users" the results don't differ
>>> significantly again, because the HTTP server running Vufind is the
>>> limiting factor here, not Solr in the backend (on a bigger machine). So
>>> that isn't a number telling anything about Solr performance, but about
>>> that bottleneck web server on a virtual machine I use... I should really
>>> switch to more potent hardware for those tests (VuFind used for these
>>> tests is running on a VMWare ESX virtual machine, Solr on real
>>> hardware). Maybe after September, 14th, if I find the time and someone
>>> thinks, that is useful...
>>> But as a short, general conclusion: It seems, response times have
>>> improved with the current code and clearly not got worse...
>>>
>>> Till
>>>
>>> --
>>> Till Kinstler
>>> Verbundzentrale des Gemeinsamen Bibliotheksverbundes (VZG)
>>> Platz der Göttinger Sieben 1, D 37073 Göttingen
>>> kin...@gb..., +49 (0) 551 39-13431, http://www.gbv.de
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
>>> 30-Day
>>> trial. Simplify your report design, integration and deployment - and
>>> focus on
>>> what you do best, core application coding. Discover what's new with
>>> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
>>> _______________________________________________
>>> Vufind-tech mailing list
>>> Vuf...@li...
>>> https://lists.sourceforge.net/lists/listinfo/vufind-tech
>>>
>>
>>
>>
>> --
>> Bill Dueber
>> Library Systems Programmer
>> University of Michigan Library
>>
>
>
>
> --
> Bill Dueber
> Library Systems Programmer
> University of Michigan Library
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Vufind-tech mailing list
> Vuf...@li...
> https://lists.sourceforge.net/lists/listinfo/vufind-tech
>
>