From: Antonio B. <abarrera@Princeton.EDU> - 2007-11-20 19:35:21
|
Yes, that is actually my next issue to tackle, the performance. I'm intentionally using a non server, because 1) I'm not ordering a server just yet, and 2) I know if I work out performance in a low environment, it will only improve on the better hardware. =20 So far all I've done is set the java_options in vufind.sh to use 256, prior to doing that, the system would die basically on one query. I think using high record counts on lesser hardware, will help prove the viability of VuFind amongst small libraries. Granted, we are not one of them, but I figure I'll work in this environment, early on especially before I do much in terms of customization, cause I'm more than likely to break this system down anyway and have to reload it. Thanks, Antonio -----Original Message----- From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Andrew Nagy Sent: Tuesday, November 20, 2007 2:29 PM To: Antonio Barrera; Wayne Graham Cc: vuf...@li... Subject: Re: [VuFind-General] Java Importer Cool! It works! Hey - I did a test search at your site and noticed the incredible slowness - or complete deadness. I might suggest playing with the XMX settings in the jetty startup command as well as the cache settings for SOLR. The cache settings for solr are very important to pay attention to - they can make drastic effects on performance. VuFind comes with the cache settings already tweaked - but for 3.9 million records it will need different settings. I am trying to put together a dokuwiki to get better documentation - so please record anything you do to vufind in terms of settings, etc so we can post it to the wiki. Andrew > -----Original Message----- > From: vuf...@li... [mailto:vufind- > gen...@li...] On Behalf Of Antonio Barrera > Sent: Tuesday, November 20, 2007 1:59 PM > To: Wayne Graham > Cc: vuf...@li... > Subject: Re: [VuFind-General] Java Importer > > Wayne, > > I was able to import 3.9 million records in roughly 36 hours. I wrote > a > short simple php script to change the import.properties file, and run > the script every hour to go through my 30-40 record files. > > It took roughly 30-36 hours to run through all of them. Consider > though, the machine is a Dell GX620, I believe with 3.6ghz and 2 gig > ram. Also, tracking back further in the story of this particular > machine, this one had a 2-3 times slower import time with the PHP > script > then other Dell desktops I had tried. However, there were no errors in > the java import, and it certainly increased import speed significantly. > But take my numbers as probably the low end in import speed. > > Thanks, > Antonio > > -----Original Message----- > From: Antonio Barrera [mailto:abarrera@Princeton.EDU] > Sent: Wednesday, November 07, 2007 4:05 PM > To: Wayne Graham; Antonio Barrera > Cc: vuf...@li... > Subject: RE: [VuFind-General] Java Importer > > Perfect. First import of 90k records took about 40 minutes. I'll be > working my way up to 4 mill, using this. This is my slower importing > machine, but nonetheless it is a big improvement, considering it also > does the marc conversion. > > I'll report errors to you as always. > > To those of you who are idiots like me and couldn't figure this out: > > >From the vufind directory: > > ant compile > cp -R $VUFIND_HOME/java/lib $VUFIND_HOME/import > ant jar > Edit the $VUFIND_HOME/import/import.properties file "make sure there's > a > trailing slash at the end of the output.path variable" > cd import > java -jar vufind-0.7-dev....jar > > Antonio > > -----Original Message----- > From: Wayne Graham [mailto:ws...@wm...] > Sent: Wednesday, November 07, 2007 3:22 PM > To: Antonio Barrera > Cc: vuf...@li... > Subject: Re: [VuFind-General] Java Importer > > The ant script _should_ be putting the compiled jar file in the import > folder. Since it's still in development, you'll get a really long dev > string with a timestamp on it (e.g > vufind-0.7-dev.2007.11.07.20.18.17.jar). After you run "ant jar" in the > $VUFIND_HOME folder, do you see a bunch of output about making > directory > and copying? > > Antonio Barrera wrote: > > No, errors, but where is the vufind-0.7-dev supposed to be built? I > > don't see it in import. > > > > -----Original Message----- > > From: Wayne Graham [mailto:ws...@wm...] > > Sent: Wednesday, November 07, 2007 1:10 PM > > To: Antonio Barrera > > Cc: vuf...@li... > > Subject: Re: [VuFind-General] Java Importer > > > > Any errors being reported? I just looked at this and it looks like I > > forgot to include copying the lib folder in the correct place in the > ant > > script. Try this: > > > > cp -R $VUFIND_HOME/java/lib $VUFIND_HOME/import > > > > Then, > > > > java -jar vufind-0.7-dev....jar > > > > Wayne > > > > Antonio Barrera wrote: > >> Wayne, > >> > >> Ok, I built it, no errors, and configured the > import/import.properties > >> file. But how do I run it, I don't see anything else in import > other > >> than the .php files? > >> > >> Antonio > >> > >> -----Original Message----- > >> From: Wayne Graham [mailto:ws...@wm...] > >> Sent: Monday, November 05, 2007 12:01 PM > >> To: Antonio Barrera > >> Cc: vuf...@li... > >> Subject: Re: [VuFind-General] Java Importer > >> > >> Sorry for not getting this out to the list...it's been a bit busy > here > >> the last few weeks. > >> > >> Anyway, the Java importer is in the SVN repository. There are some > > XSLT > >> updates that need to be made (I have them somewhere, but haven't > >> committed them yet) and some more cleanup and documenting to be > done. > >> > >> If you're brave, you can try this out, but you'll need Ant > installed. > > If > >> you got to the main directory and do > >> > >> ant compile > >> > >> The system should build the files you need in the import file. > You'll > >> need to edit the file import.properties with your specific > > information. > >> One VERY import note...make sure there's a trailing slash at the end > > of > >> the output.path variable!!!! This is a bug that I haven't fixed yet. > > If > >> you don't put the trailing slash, it'll put the XML files in your > > vufind > >> directory with a prefix of "data" on it. Again...put a trailing > slash > > on > >> the variable! > >> > >> Another note, right now this doesn't update your real > index...rather, > > it > >> creates a folder in your import folder for the index that you have > to > >> manually move to the correct location. This is intentional until > I've > >> finished the code, but you can overwrite this in the > >> solr/config/solrconfig.xml file by putting the path into the dataDir > >> element: > >> > >> > <dataDir>${solr.data.dir:/usr/local/vufind/solr:./solr/data}<dataDir> > >> > >> There will (hopefully) be more on this later this week. > >> > >> Wayne > >> > >> Antonio Barrera wrote: > >>> Wayne, > >>> > >>> > >>> > >>> Any updates on the Java Importer? > >>> > >>> > >>> > >>> Thanks, > >>> > >>> Antonio Barrera > >>> > >>> Princeton University Library > >>> > >>> > >>> > > > ----------------------------------------------------------------------- > - > > > ----------------------------------------------------------------------- > - > >> - > >>> This SF.net email is sponsored by: Splunk Inc. > >>> Still grepping through log files to find problems? Stop. > >>> Now Search log events and configuration files using AJAX and a > >> browser. > >>> Download your FREE copy of Splunk now >> http://get.splunk.com/ > >>> > >>> > >>> > > > ----------------------------------------------------------------------- > - > >>> _______________________________________________ > >>> VuFind-General mailing list > >>> VuF...@li... > >>> https://lists.sourceforge.net/lists/listinfo/vufind-general > > > > -- > /** > * Wayne Graham > * Earl Gregg Swem Library > * PO Box 8794 > * Williamsburg, VA 23188 > * 757.221.3112 > * http://swem.wm.edu/blogs/waynegraham/ > */ > > > ----------------------------------------------------------------------- > -- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2005. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > VuFind-General mailing list > VuF...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-general ------------------------------------------------------------------------ - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2005. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ VuFind-General mailing list VuF...@li... https://lists.sourceforge.net/lists/listinfo/vufind-general |