Re: [VuFind-General] Sending lots of data to Vufind 2RC1

[Demian K. <dem...@vi...>]

I just did some benchmarking on my Windows box using the latest VuFind/SolrMarc code, and things seem to be quite slow now!  Over HTTP, I get 47 records/sec. and with direct writing I get the slightly faster 55 records/sec.
 
Some of the slowdown is related to the fact that the current code is using custom BeanShell scripts for a few things (commenting those out boosts me up to 60 records/sec. with direct writing), but I'm not sure if anything more can be done to further boost performance....

- Demian

> -----Original Message-----
> From: Michael Lackhoff [mailto:mi...@la...]
> Sent: Monday, June 10, 2013 12:21 PM
> To: vuf...@li...
> Subject: Re: [VuFind-General] Sending lots of data to Vufind 2RC1
> 
> On 09.06.2013 21:55 Demian Katz wrote:
> 
> > Before digging to deeply into any of this, I would strongly recommend
> > upgrading to the latest master code from Git (which is extremely
> > close to what will be released as VuFind 2.0 final in a couple of
> > weeks).  This has been upgraded to Solr 4.2.1, which has
> > significantly different performance characteristics than Solr 3
> > (almost certainly for the better).  You'll probably still be able to
> 
> I got the master from Git but it shows only minor speed improvements:
> 
> INFO [main] (MarcImporter.java:636) - Indexed 78136 at a rate of about
> 67.0 per sec
> Better than 58/sec but still far away from the 1000 to 1400/sec I got
> with direct indexing in RC1.
> 
> > do some work to make things run better, but I would be interested to
> > hear how your Solr 4 experience differs from your Solr 3 experience.
> > If you don't mind trying that and reporting back, we can provide
> > further advice.
> 
> I really could need some further advice. Did you or someone else try to
> index over HTTP under Windows? What speed do you get?
> All my tests are done like this (xy being the tested version):
> 
> unzip vufind-xy.zip
> cd vufind-xy
> set JAVA_HOME=%DRIVE%\PortableApps\CommonFiles\Java64
> set VUFIND_HOME=F:\vufind-xy
> run_vufind.bat start
> <second shell with same environment>
> import-marc.bat test.mrc
> 
> This should make it reproducable. During indexing I see only about 1-5%
> CPU and still about 16GB of free RAM
> 
> Anyone with better benchmarks or ideas?
> 
> -Michael
> 
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Windows:
> 
> Build for Windows Store.
> 
> http://p.sf.net/sfu/windows-dev2dev
> _______________________________________________
> VuFind-General mailing list
> VuF...@li...
> https://lists.sourceforge.net/lists/listinfo/vufind-general