From: Andrew N. <as...@gm...> - 2009-09-04 20:45:37
|
Bill - I just checked in the change for the json.nl enhancement On Fri, Sep 4, 2009 at 1:06 PM, Bill Dueber <bi...@du...> wrote: > Another quick note -- it looks like the current trunk isn't using the ' > json.nl=arrarr' argument to Solr. It makes the returned data structures > much easier to use, e.g. > > Facets without arrarr: > - subject 1 > - count 1 > - subject 2 > - count 2 > > Facets with arrarr > - [subject 1, count 1] > - [subject 2, count 2] > > It would also be nice if all potentially-multivalued fields came back in > arrays (even if it were an array of one). Solr won't do this automatically, > but using configuration to list all your multi-valued fields and then just > walking through the returned data and doing if(!is_array)... would be easy > and make the templates a lot easier to deal with (starting with alwaqys > making sure you get an array of records back, a change we already made > here). > > Even something braindead like the following would work, I think. > > if (isset($results['record']['id'])) { // it's a single record > $results['record'] = array($results['record']); > } > foreach ($results['record'] as $r) { > foreach ($listOfMultiValuedFields as $f) { > if (isset($r[$f] && !is_array($r[$f])) { > $r[$f] = array($r[$f]); > } > } > } > > > > > > On Fri, Sep 4, 2009 at 12:22 PM, Bill Dueber <bi...@du...> wrote: > >> I just did a quickie test of my own, only looking at the Solr end. I I >> grabbed 4681 queries from our live system (yeseterday's logfile) and did the >> following, using JMeter with four threads and a 4-second warmup time. >> >> - Run all queries against Solr (not vufind; directly at Solr) with XML >> output >> - Run all queries with JSON output >> - Repeat XML, but measure this time >> - Repeat JSON, but measure this time. >> >> Basically, I'm looking at how long it takes Solr to generate XML vs to >> generate JSON. >> >> My off-the-cuff results: >> >> JSON: 272 requests/second >> XML: 181 requests/second >> >> The other end of it (how long it takes PHP to deal with XML vs JSON) is a >> different test. >> http://dailybraindump.com/php-unserialize-vs-json_decode-vs-simplexml/39/seems to indicate unserializing JSON is also a lot faster. >> >> The lesson here is not that Solr outputting JSON is 1/3 faster than XML. >> The lesson is that SOLR IS NOT THE BOTTLENECK. I'm pretty sure I'm not >> getting anywhere near 180 requests/second out of vufind :-) >> >> >> >> On Fri, Sep 4, 2009 at 9:51 AM, Till Kinstler <kin...@gb...> wrote: >> >>> Andrew Nagy schrieb: >>> >>> > If you would like to test these new changes, please grab the latest >>> from >>> > I would especially like someone with a large data >>> > set to see how the new facets are performing. >>> >>> OK, I just did a quick test using the same setup as described in >>> http://www.nabble.com/testing-tp24491925p24606102.html but with a newer >>> Solr nightly build (2009-08-16). >>> I ran the test only with 1 and 10 concurrent request threads, I haven't >>> got the time to run the setup with 50 threads now. If needed/wanted, I >>> can do that, but not before Monday, 14th, because I am going to be out >>> of office the next week. >>> >>> I have included the results from the earlier test runs in the table >>> below. r1212 was with facets pulled through separate AJAX requests, >>> r1239 with "inline facets" pulled through the Solr XML interface, r1370 >>> is the latest trunk code with "inline facets" pulled through JSON. >>> >>> Results (see with fixed width font): >>> >>> | Avg. response time (ms) | Avg. throughput (1/s) >>> threads| 1 | 10 | 50 | 1 | 10 | 50 >>> -------+-------+--------+--------+-------+--------+-------- >>> r1370 | 188 | 821 | % | 5,2 | 12,1 | % >>> r1239 | 212 | 824 | 4294 | 4,7 | 12,1 | 11,6 >>> r1212 | 144 | 823 | 4130 | 6,8 | 12,1 | 12,1 >>> >>> I'd say: Either using JSON output or the more recent Solr nightly build >>> or both speed up search: With 1 search thread ("user") response time >>> drops from 212 ms to 188 ms. >>> With 10 concurrent threads/"users" the results don't differ >>> significantly again, because the HTTP server running Vufind is the >>> limiting factor here, not Solr in the backend (on a bigger machine). So >>> that isn't a number telling anything about Solr performance, but about >>> that bottleneck web server on a virtual machine I use... I should really >>> switch to more potent hardware for those tests (VuFind used for these >>> tests is running on a VMWare ESX virtual machine, Solr on real >>> hardware). Maybe after September, 14th, if I find the time and someone >>> thinks, that is useful... >>> But as a short, general conclusion: It seems, response times have >>> improved with the current code and clearly not got worse... >>> >>> Till >>> >>> -- >>> Till Kinstler >>> Verbundzentrale des Gemeinsamen Bibliotheksverbundes (VZG) >>> Platz der Göttinger Sieben 1, D 37073 Göttingen >>> kin...@gb..., +49 (0) 551 39-13431, http://www.gbv.de >>> >>> >>> ------------------------------------------------------------------------------ >>> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 >>> 30-Day >>> trial. Simplify your report design, integration and deployment - and >>> focus on >>> what you do best, core application coding. Discover what's new with >>> Crystal Reports now. http://p.sf.net/sfu/bobj-july >>> _______________________________________________ >>> Vufind-tech mailing list >>> Vuf...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>> >> >> >> >> -- >> Bill Dueber >> Library Systems Programmer >> University of Michigan Library >> > > > > -- > Bill Dueber > Library Systems Programmer > University of Michigan Library > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus > on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > |