From: Barnett, J. <jef...@ya...> - 2008-08-04 20:56:59
|
Ever since we got our full 7.8million records loaded we've experienced significant delays "Loading Narrow Options". Following discussion here and in solrmarc, I think we have the solr tuning options set to reasonable values ... queries use less than 5% cpu. The one thing that made some improvement was when we decided to drop one of the facet categories that was confusing test users. That leads me to think the problem might be in the javascript and/or AJAX that is building the facet panel. Are there places that that might be tweaked or monitored? Could we omit counting facets after the top 5? |
From: Jessie K. <jk...@st...> - 2008-08-04 21:02:12
|
Hi Jeffrey, I believe that NLA has made some improvements with their own code, maybe they can say something to that affect when they come on-line. -Jessie Barnett, Jeffrey wrote: > Ever since we got our full 7.8million records loaded we've experienced significant delays "Loading Narrow Options". Following discussion here and in solrmarc, I think we have the solr tuning options set to reasonable values ... queries use less than 5% cpu. The one thing that made some improvement was when we decided to drop one of the facet categories that was confusing test users. That leads me to think the problem might be in the javascript and/or AJAX that is building the facet panel. Are there places that that might be tweaked or monitored? Could we omit counting facets after the top 5? > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > |
From: Steven M. <smc...@nl...> - 2008-08-06 00:34:48
|
Hi All, Mark and I have been out of the office recently, so apologies for coming in late with news that probably isn't all that useful. I don't have much more to add to the discussion apart from re-iterating the importance of cache sizes for facet performance. At the NLA, we were lucky enough to have a developer who had written an alternate solr facet implementation for another project. For each facetted field, this implementation builds up a document hash once at initialisation time and holds onto it until the indexes are changed. This means we endure a once-off cost at startup (we're talking minutes here), but subsequent facet requests are very fast, under a second. The downside with this approach is the hash needs to be rebuilt every time the indexes are changed, which means we couldn't do realtime updates to solr getting hit with our multi-minute facet build time on every commit. We have something like 4 million solr documents running in a 6Gb JVM. If people are interested in learning more about how our faceting works, we're happy to share the code. I think the NLA is in the process of releasing it through this other project, but we've given out the faceting code to other VuFinders in the past, from memory. Steve On 05/08/2008, at 7:01 AM, Jessie Keck wrote: > Hi Jeffrey, > I believe that NLA has made some improvements with their own code, > maybe > they can say something to that affect when they come on-line. > > -Jessie > ---- Steven McPhillips <smc...@nl...> IT Business Systems National Library of Australia Try our new catalogue - http://catalogue.nla.gov.au |
From: Chris D. <ce...@ui...> - 2008-09-17 13:31:54
|
Hi, Steve, et al., I was wondering if your team has been playing with the later versions of VuFind (based on SOLR 1.3). Soon after VU releases version 1.0 we plan on migrating to the new code. However, we could only do so if your invaluable FastFacets plugin will continue to work. Do you foresee any issues getting FastFacets to work with the newer versions of SOLR? Much thanks! Chris On Tue, Aug 05, 2008 at 07:33:56PM -0500, Steven McPhillips wrote: > Hi All, > > Mark and I have been out of the office recently, so apologies for > coming in late with news that probably isn't all that useful. I don't > have much more to add to the discussion apart from re-iterating the > importance of cache sizes for facet performance. > > At the NLA, we were lucky enough to have a developer who had written > an alternate solr facet implementation for another project. For each > facetted field, this implementation builds up a document hash once at > initialisation time and holds onto it until the indexes are changed. > This means we endure a once-off cost at startup (we're talking > minutes here), but subsequent facet requests are very fast, under a > second. The downside with this approach is the hash needs to be > rebuilt every time the indexes are changed, which means we couldn't > do realtime updates to solr getting hit with our multi-minute facet > build time on every commit. > > > We have something like 4 million solr documents running in a 6Gb JVM. > If people are interested in learning more about how our faceting > works, we're happy to share the code. I think the NLA is in the > process of releasing it through this other project, but we've given > out the faceting code to other VuFinders in the past, from memory. > > Steve > > On 05/08/2008, at 7:01 AM, Jessie Keck wrote: > > > Hi Jeffrey, > > I believe that NLA has made some improvements with their own code, > > maybe > > they can say something to that affect when they come on-line. > > > > -Jessie > > > > > > ---- > Steven McPhillips <smc...@nl...> > IT Business Systems > National Library of Australia > Try our new catalogue - http://catalogue.nla.gov.au > > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech |
From: Steven M. <smc...@nl...> - 2008-09-18 00:02:58
|
Hi Chris, We haven't looked at solr 1.3 as yet, well not the stable release, anyway. Internally we're running a 1.3 nightly from way back in March. I haven't kept up to date with the changes made since then, so I suppose there's a chance the fast facets code might need some alteration. I'd be kind of surprised though, because the code doesn't do anything overly tricky or underhanded :) When I've got a spare moment I'll fire up solr 1.3 and see how our extensions play, then report back. Steve On 17/09/2008, at 11:31 PM, Chris Delis wrote: > Hi, Steve, et al., > > I was wondering if your team has been playing with the later versions > of VuFind (based on SOLR 1.3). Soon after VU releases version 1.0 we > plan on migrating to the new code. However, we could only do so if > your invaluable FastFacets plugin will continue to work. Do you > foresee any issues getting FastFacets to work with the newer versions > of SOLR? > > Much thanks! > > Chris > > On Tue, Aug 05, 2008 at 07:33:56PM -0500, Steven McPhillips wrote: >> Hi All, >> >> Mark and I have been out of the office recently, so apologies for >> coming in late with news that probably isn't all that useful. I don't >> have much more to add to the discussion apart from re-iterating the >> importance of cache sizes for facet performance. >> >> At the NLA, we were lucky enough to have a developer who had written >> an alternate solr facet implementation for another project. For each >> facetted field, this implementation builds up a document hash once at >> initialisation time and holds onto it until the indexes are changed. >> This means we endure a once-off cost at startup (we're talking >> minutes here), but subsequent facet requests are very fast, under a >> second. The downside with this approach is the hash needs to be >> rebuilt every time the indexes are changed, which means we couldn't >> do realtime updates to solr getting hit with our multi-minute facet >> build time on every commit. >> >> >> We have something like 4 million solr documents running in a 6Gb JVM. >> If people are interested in learning more about how our faceting >> works, we're happy to share the code. I think the NLA is in the >> process of releasing it through this other project, but we've given >> out the faceting code to other VuFinders in the past, from memory. >> >> Steve >> >> On 05/08/2008, at 7:01 AM, Jessie Keck wrote: >> >>> Hi Jeffrey, >>> I believe that NLA has made some improvements with their own code, >>> maybe >>> they can say something to that affect when they come on-line. >>> >>> -Jessie >>> >> >> >> >> ---- >> Steven McPhillips <smc...@nl...> >> IT Business Systems >> National Library of Australia >> Try our new catalogue - http://catalogue.nla.gov.au >> >> >> >> >> --------------------------------------------------------------------- >> ---- >> This SF.Net email is sponsored by the Moblin Your Move Developer's >> challenge >> Build the coolest Linux based applications with Moblin SDK & win >> great prizes >> Grand prize is a trip for two to an Open Source event anywhere in >> the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Vufind-tech mailing list >> Vuf...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-tech ---- Steven McPhillips <smc...@nl...> IT Business Systems National Library of Australia Try our new catalogue - http://catalogue.nla.gov.au |
From: Bill D. <bi...@du...> - 2008-09-18 00:42:35
|
Could someone provide a ridiculously explicit pointer to the fast facets code? I can't seem to locate it. On Wed, Sep 17, 2008 at 8:02 PM, Steven McPhillips <smc...@nl...> wrote: > Hi Chris, > > We haven't looked at solr 1.3 as yet, well not the stable release, > anyway. Internally we're running a 1.3 nightly from way back in March. > > I haven't kept up to date with the changes made since then, so I > suppose there's a chance the fast facets code might need some > alteration. I'd be kind of surprised though, because the code doesn't > do anything overly tricky or underhanded :) > > When I've got a spare moment I'll fire up solr 1.3 and see how our > extensions play, then report back. > > Steve > > > On 17/09/2008, at 11:31 PM, Chris Delis wrote: > >> Hi, Steve, et al., >> >> I was wondering if your team has been playing with the later versions >> of VuFind (based on SOLR 1.3). Soon after VU releases version 1.0 we >> plan on migrating to the new code. However, we could only do so if >> your invaluable FastFacets plugin will continue to work. Do you >> foresee any issues getting FastFacets to work with the newer versions >> of SOLR? >> >> Much thanks! >> >> Chris >> >> On Tue, Aug 05, 2008 at 07:33:56PM -0500, Steven McPhillips wrote: >>> Hi All, >>> >>> Mark and I have been out of the office recently, so apologies for >>> coming in late with news that probably isn't all that useful. I don't >>> have much more to add to the discussion apart from re-iterating the >>> importance of cache sizes for facet performance. >>> >>> At the NLA, we were lucky enough to have a developer who had written >>> an alternate solr facet implementation for another project. For each >>> facetted field, this implementation builds up a document hash once at >>> initialisation time and holds onto it until the indexes are changed. >>> This means we endure a once-off cost at startup (we're talking >>> minutes here), but subsequent facet requests are very fast, under a >>> second. The downside with this approach is the hash needs to be >>> rebuilt every time the indexes are changed, which means we couldn't >>> do realtime updates to solr getting hit with our multi-minute facet >>> build time on every commit. >>> >>> >>> We have something like 4 million solr documents running in a 6Gb JVM. >>> If people are interested in learning more about how our faceting >>> works, we're happy to share the code. I think the NLA is in the >>> process of releasing it through this other project, but we've given >>> out the faceting code to other VuFinders in the past, from memory. >>> >>> Steve >>> >>> On 05/08/2008, at 7:01 AM, Jessie Keck wrote: >>> >>>> Hi Jeffrey, >>>> I believe that NLA has made some improvements with their own code, >>>> maybe >>>> they can say something to that affect when they come on-line. >>>> >>>> -Jessie >>>> >>> >>> >>> >>> ---- >>> Steven McPhillips <smc...@nl...> >>> IT Business Systems >>> National Library of Australia >>> Try our new catalogue - http://catalogue.nla.gov.au >>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> ---- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>> challenge >>> Build the coolest Linux based applications with Moblin SDK & win >>> great prizes >>> Grand prize is a trip for two to an Open Source event anywhere in >>> the world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Vufind-tech mailing list >>> Vuf...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-tech > > > > > ---- > Steven McPhillips <smc...@nl...> > IT Business Systems > National Library of Australia > Try our new catalogue - http://catalogue.nla.gov.au > > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > -- Bill Dueber Library Systems Programmer University of Michigan Library |
From: Steven M. <smc...@nl...> - 2008-09-18 00:59:12
|
On 18/09/2008, at 10:42 AM, Bill Dueber wrote: > Could someone provide a ridiculously explicit pointer to the fast > facets code? I can't seem to locate it. Apologies - I don't think we've every made a blanket announcement. You can download the package from ftp://ftp.nla.gov.au/pub/outgoing/ vufind/nla-fastfacets.tar.bz2 - I've included a README that gives basic instruction on its use and some gotchas. I'd like to re-iterate that this code was last tested and compiled against a solr 1.3 nightly from march 2008. I'm looking to do a test run with the recently released solr 1.3 soon. You've been warned :) |
From: Steven M. <smc...@nl...> - 2008-09-18 02:14:44
|
Just a quick update for those interested, The "fast facets" package available at ftp://ftp.nla.gov.au/pub/ outgoing/vufind/nla-fastfacets.tar.bz2 does not work with the public SOLR 1.3 release. I've made the requisite changes locally (small stuff), but haven't set up SOLR 1.3 indexes to test the functionality. When I have, I'll let people know via the list. Just a reminder the package above does ship with a nightly release of SOLR 1.3 that is compatible, so you can still grab it and have a play to see what all the fuss is about. Steve ---- Steven McPhillips <smc...@nl...> IT Business Systems National Library of Australia Try our new catalogue - http://catalogue.nla.gov.au |
From: Steven M. <smc...@nl...> - 2008-09-18 23:47:54
|
I've done a rough job of rebuilding the fast facets code against SOLR 1.3.0. I haven't done a huge amount of testing, but it seems to work. Available from ftp://ftp.nla.gov.au/pub/vufind/nla-fastfacets- solr1.3.0.tar.bz2 This package does not include solr 1.3.0 - this can be downloaded separately. Installation instructions included in the tarball. feedback highly sought after :) ---- Steven McPhillips <smc...@nl...> IT Business Systems National Library of Australia Try our new catalogue - http://catalogue.nla.gov.au |
From: Chris D. <ce...@ui...> - 2008-09-19 01:33:30
|
On Fri, Sep 19, 2008 at 04:47:49PM +1000, Steven McPhillips wrote: > I've done a rough job of rebuilding the fast facets code against SOLR > 1.3.0. I haven't done a huge amount of testing, but it seems to work. This is very exciting news! Thank you! > > Available from ftp://ftp.nla.gov.au/pub/vufind/nla-fastfacets- > solr1.3.0.tar.bz2 > > This package does not include solr 1.3.0 - this can be downloaded > separately. Installation instructions included in the tarball. > > feedback highly sought after :) I definitely plan to test this out and will let you know how it goes. We aren't quite yet ready to migrate to the new (Solr 1.3-based) VuFind completely (with all of our local customizations). But, for now, we can at least test things out with a newer stock version of VuFind using some of our indexed marc records. Naomi: I'd be very interested in how this code works for you. Thanks again, Steven, and to the rest of you team over at NLA! Cheers, Chris > > > ---- > Steven McPhillips <smc...@nl...> > IT Business Systems > National Library of Australia > Try our new catalogue - http://catalogue.nla.gov.au > > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech |
From: Wayne G. <ws...@wm...> - 2008-08-04 21:03:39
|
What are your tuning parameters for the JVM (e.g. JAVA_OPTIONS)? Also, with this many records, you'll most likely be spending quite a bit of time working to tweak both your JVM and Jetty. The unfortunate thing with the most recent version of Jetty is that their documentation on tuning the Jetty server has gone by the wayside and I can't seem to find it. Wayne Barnett, Jeffrey wrote: > Ever since we got our full 7.8million records loaded we've experienced significant delays "Loading Narrow Options". Following discussion here and in solrmarc, I think we have the solr tuning options set to reasonable values ... queries use less than 5% cpu. The one thing that made some improvement was when we decided to drop one of the facet categories that was confusing test users. That leads me to think the problem might be in the javascript and/or AJAX that is building the facet panel. Are there places that that might be tweaked or monitored? Could we omit counting facets after the top 5? > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > -- /** * Wayne Graham * Earl Gregg Swem Library * PO Box 8794 * Williamsburg, VA 23188 * 757.221.3112 * http://swem.wm.edu/blogs/waynegraham/ * http://www.liquidfoot.com */ |
From: Andrew N. <and...@vi...> - 2008-08-04 21:21:30
|
Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. Hope this helps. Andrew > -----Original Message----- > From: vuf...@li... [mailto:vufind-tech- > bo...@li...] On Behalf Of Barnett, Jeffrey > Sent: Monday, August 04, 2008 4:57 PM > To: vuf...@li... > Subject: [VuFind-Tech] Facets -- where are the bottlenecks? > > Ever since we got our full 7.8million records loaded we've experienced > significant delays "Loading Narrow Options". Following discussion here > and in solrmarc, I think we have the solr tuning options set to > reasonable values ... queries use less than 5% cpu. The one thing that > made some improvement was when we decided to drop one of the facet > categories that was confusing test users. That leads me to think the > problem might be in the javascript and/or AJAX that is building the > facet panel. Are there places that that might be tweaked or monitored? > Could we omit counting facets after the top 5? > > ----------------------------------------------------------------------- > -- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the > world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech |
From: Barnett, J. <jef...@ya...> - 2008-08-05 15:16:59
|
Nope, the secret is "-d64" to make the jvm use the "64 bit model" -----Original Message----- From: Wayne Graham [mailto:ws...@wm...] Sent: Tuesday, August 05, 2008 11:15 AM To: Barnett, Jeffrey Cc: Alan Rykhus; vuf...@li... Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? Are you setting the maximum heap size (-Xmx) before the minimum (-Xms)? The default maximum is a weird algorithm of a percentage of the memory space (probably around 4 Gb on your system), and if you're not explicitly setting this, it's probably trying to make the minimum heap space larger than the maximum heap space. Take a look at http://java.sun.com/performance/reference/whitepapers/tuning.html#section4.1.2 Wayne Barnett, Jeffrey wrote: > I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: > > "Invalid initial heap size: -Xms8192M > The specified size exceeds the maximum representable size. > Could not create the Java virtual machine." > > The version is > $ java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) > > and the OS is Solaris 10 > > Any suggestion where the 3GB limit is coming from? > > -----Original Message----- > From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Alan Rykhus > Sent: Tuesday, August 05, 2008 10:07 AM > To: vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Hello Jeff, > > I made this change to my system as a trial the week before last because > of the same thing you reported. Through testing I determined that it was > the topic facet that was causing the delay. If I comment out the topic > facet things run pretty good. > > What I read about was that SOLR builds bitsets for all facets even > though it doesn't always need them. By setting this value high it only > builds the bitsets when the amount is surpassed. > > As for the version of vufing, I am most of the way through yesterday's > updates and I try to update to the latest every other day or so. I did > have problems with the solr.war and built my own. You have to have the > pieces of lucene in the solr.war in the solr/jetty/webapps directory the > same as the lucene libraries in the libraries used for the importing of > the marc records. None of this effects performance as far as I know. I > am using lucene 2.3.1. > > Additional information. I have a linux box with 32 GB of memory. > > I am giving vufind 16 GB of memory( -Xmx16384M ). > > I have the following settings for filterCache: > <filterCache > class="solr.LRUCache" > size="50000000" > initialSize="10000000" > autowarmCount="10000000"/> > > I upped my settings for documentCache > > <documentCache > class="solr.LRUCache" > size="500000" > initialSize="50000" > autowarmCount="10000"/> > > I do have my own schema for solr. But it is not too far out of line with > what is distributed. > > any other questions, I'm willing to try -- al > > On Mon, 2008-08-04 at 19:42 -0500, Barnett, Jeffrey wrote: > >> I made the change, but I don't notice any difference in response time. What was the source/reason for adding the option? >> I notice you refer to "Solr.php" instead of "SOLR.php". That is a relatively recent change (which I have not yet updated to) . Is there a related recent change that this one is dependent on? >> >> What svn revision are you using? ... I'm at 792 >> ________________________________________ >> From: vuf...@li... [vuf...@li...] On Behalf Of Alan Rykhus [ala...@mn...] >> Sent: Monday, August 04, 2008 5:40 PM >> To: vuf...@li... >> Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? >> >> Hello, >> >> I had this problem too, especially with topic facets as there are so >> many. I added an option to the facet query in SOLR.php(now solr.php) >> late last week and it seems to have helped. Have not had time to report >> on it. >> >> The original section in Solr.php starting with line 448 is: >> >> // Build Facet Options >> if ($facet) { >> $options['facet'] = 'true'; >> $options['facet.mincount'] = 1; >> $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : >> null; >> $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : >> null; >> $options['facet.prefix'] = (isset($facet['prefix'])) ? >> $facet['prefix'] : null; >> $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : >> null; >> } >> >> I added the additional facet.enum.cache.minDf option after doing some >> research: >> >> // Build Facet Options >> if ($facet) { >> $options['facet'] = 'true'; >> $options['facet.mincount'] = 1; >> $options['facet.enum.cache.minDf'] = 1000; >> $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : >> null; >> $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : >> null; >> $options['facet.prefix'] = (isset($facet['prefix'])) ? >> $facet['prefix'] : null; >> $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : >> null; >> } >> >> give it a test -- al >> >> On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: >> >>> Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. >>> >>> Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. >>> >>> Hope this helps. >>> >>> Andrew >>> >>> >>>> -----Original Message----- >>>> From: vuf...@li... [mailto:vufind-tech- >>>> bo...@li...] On Behalf Of Barnett, Jeffrey >>>> Sent: Monday, August 04, 2008 4:57 PM >>>> To: vuf...@li... >>>> Subject: [VuFind-Tech] Facets -- where are the bottlenecks? >>>> >>>> Ever since we got our full 7.8million records loaded we've experienced >>>> significant delays "Loading Narrow Options". Following discussion here >>>> and in solrmarc, I think we have the solr tuning options set to >>>> reasonable values ... queries use less than 5% cpu. The one thing that >>>> made some improvement was when we decided to drop one of the facet >>>> categories that was confusing test users. That leads me to think the >>>> problem might be in the javascript and/or AJAX that is building the >>>> facet panel. Are there places that that might be tweaked or monitored? >>>> Could we omit counting facets after the top 5? >>>> >>>> ----------------------------------------------------------------------- >>>> -- >>>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>>> challenge >>>> Build the coolest Linux based applications with Moblin SDK & win great >>>> prizes >>>> Grand prize is a trip for two to an Open Source event anywhere in the >>>> world >>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>>> _______________________________________________ >>>> Vufind-tech mailing list >>>> Vuf...@li... >>>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>>> >>> ------------------------------------------------------------------------- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >>> Build the coolest Linux based applications with Moblin SDK & win great prizes >>> Grand prize is a trip for two to an Open Source event anywhere in the world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Vufind-tech mailing list >>> Vuf...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>> >> -- >> Alan Rykhus >> PALS, A Program of the Minnesota State Colleges and Universities >> (507)389-1975 >> ala...@mn... >> >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Vufind-tech mailing list >> Vuf...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-tech >> > -- > Alan Rykhus > PALS, A Program of the Minnesota State Colleges and Universities > (507)389-1975 > ala...@mn... > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > -- /** * Wayne Graham * Earl Gregg Swem Library * PO Box 8794 * Williamsburg, VA 23188 * 757.221.3112 * http://swem.wm.edu/blogs/waynegraham/ * http://www.liquidfoot.com */ |
From: Wayne G. <ws...@wm...> - 2008-08-05 15:31:25
|
This is a good tip...mind putting it on the wiki somewhere? Also, here's an exhaustive list of JVM options... http://blogs.sun.com/watt/resource/jvm-options-list.html Kind'a makes you wish you could just compile with -O3 :) Wayne Barnett, Jeffrey wrote: > Nope, the secret is "-d64" to make the jvm use the "64 bit model" > > -----Original Message----- > From: Wayne Graham [mailto:ws...@wm...] > Sent: Tuesday, August 05, 2008 11:15 AM > To: Barnett, Jeffrey > Cc: Alan Rykhus; vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Are you setting the maximum heap size (-Xmx) before the minimum (-Xms)? > The default maximum is a weird algorithm of a percentage of the memory > space (probably around 4 Gb on your system), and if you're not > explicitly setting this, it's probably trying to make the minimum heap > space larger than the maximum heap space. > > Take a look at > http://java.sun.com/performance/reference/whitepapers/tuning.html#section4.1.2 > > Wayne > > Barnett, Jeffrey wrote: > >> I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: >> >> "Invalid initial heap size: -Xms8192M >> The specified size exceeds the maximum representable size. >> Could not create the Java virtual machine." >> >> The version is >> $ java -version >> java version "1.6.0_07" >> Java(TM) SE Runtime Environment (build 1.6.0_07-b06) >> Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) >> >> and the OS is Solaris 10 >> >> Any suggestion where the 3GB limit is coming from? >> >> -----Original Message----- >> From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Alan Rykhus >> Sent: Tuesday, August 05, 2008 10:07 AM >> To: vuf...@li... >> Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? >> >> Hello Jeff, >> >> I made this change to my system as a trial the week before last because >> of the same thing you reported. Through testing I determined that it was >> the topic facet that was causing the delay. If I comment out the topic >> facet things run pretty good. >> >> What I read about was that SOLR builds bitsets for all facets even >> though it doesn't always need them. By setting this value high it only >> builds the bitsets when the amount is surpassed. >> >> As for the version of vufing, I am most of the way through yesterday's >> updates and I try to update to the latest every other day or so. I did >> have problems with the solr.war and built my own. You have to have the >> pieces of lucene in the solr.war in the solr/jetty/webapps directory the >> same as the lucene libraries in the libraries used for the importing of >> the marc records. None of this effects performance as far as I know. I >> am using lucene 2.3.1. >> >> Additional information. I have a linux box with 32 GB of memory. >> >> I am giving vufind 16 GB of memory( -Xmx16384M ). >> >> I have the following settings for filterCache: >> <filterCache >> class="solr.LRUCache" >> size="50000000" >> initialSize="10000000" >> autowarmCount="10000000"/> >> >> I upped my settings for documentCache >> >> <documentCache >> class="solr.LRUCache" >> size="500000" >> initialSize="50000" >> autowarmCount="10000"/> >> >> I do have my own schema for solr. But it is not too far out of line with >> what is distributed. >> >> any other questions, I'm willing to try -- al >> >> On Mon, 2008-08-04 at 19:42 -0500, Barnett, Jeffrey wrote: >> >> >>> I made the change, but I don't notice any difference in response time. What was the source/reason for adding the option? >>> I notice you refer to "Solr.php" instead of "SOLR.php". That is a relatively recent change (which I have not yet updated to) . Is there a related recent change that this one is dependent on? >>> >>> What svn revision are you using? ... I'm at 792 >>> ________________________________________ >>> From: vuf...@li... [vuf...@li...] On Behalf Of Alan Rykhus [ala...@mn...] >>> Sent: Monday, August 04, 2008 5:40 PM >>> To: vuf...@li... >>> Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? >>> >>> Hello, >>> >>> I had this problem too, especially with topic facets as there are so >>> many. I added an option to the facet query in SOLR.php(now solr.php) >>> late last week and it seems to have helped. Have not had time to report >>> on it. >>> >>> The original section in Solr.php starting with line 448 is: >>> >>> // Build Facet Options >>> if ($facet) { >>> $options['facet'] = 'true'; >>> $options['facet.mincount'] = 1; >>> $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : >>> null; >>> $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : >>> null; >>> $options['facet.prefix'] = (isset($facet['prefix'])) ? >>> $facet['prefix'] : null; >>> $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : >>> null; >>> } >>> >>> I added the additional facet.enum.cache.minDf option after doing some >>> research: >>> >>> // Build Facet Options >>> if ($facet) { >>> $options['facet'] = 'true'; >>> $options['facet.mincount'] = 1; >>> $options['facet.enum.cache.minDf'] = 1000; >>> $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : >>> null; >>> $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : >>> null; >>> $options['facet.prefix'] = (isset($facet['prefix'])) ? >>> $facet['prefix'] : null; >>> $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : >>> null; >>> } >>> >>> give it a test -- al >>> >>> On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: >>> >>> >>>> Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. >>>> >>>> Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. >>>> >>>> Hope this helps. >>>> >>>> Andrew >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: vuf...@li... [mailto:vufind-tech- >>>>> bo...@li...] On Behalf Of Barnett, Jeffrey >>>>> Sent: Monday, August 04, 2008 4:57 PM >>>>> To: vuf...@li... >>>>> Subject: [VuFind-Tech] Facets -- where are the bottlenecks? >>>>> >>>>> Ever since we got our full 7.8million records loaded we've experienced >>>>> significant delays "Loading Narrow Options". Following discussion here >>>>> and in solrmarc, I think we have the solr tuning options set to >>>>> reasonable values ... queries use less than 5% cpu. The one thing that >>>>> made some improvement was when we decided to drop one of the facet >>>>> categories that was confusing test users. That leads me to think the >>>>> problem might be in the javascript and/or AJAX that is building the >>>>> facet panel. Are there places that that might be tweaked or monitored? >>>>> Could we omit counting facets after the top 5? >>>>> >>>>> ----------------------------------------------------------------------- >>>>> -- >>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>>>> challenge >>>>> Build the coolest Linux based applications with Moblin SDK & win great >>>>> prizes >>>>> Grand prize is a trip for two to an Open Source event anywhere in the >>>>> world >>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>>>> _______________________________________________ >>>>> Vufind-tech mailing list >>>>> Vuf...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>>>> >>>>> >>>> ------------------------------------------------------------------------- >>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >>>> Build the coolest Linux based applications with Moblin SDK & win great prizes >>>> Grand prize is a trip for two to an Open Source event anywhere in the world >>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>>> _______________________________________________ >>>> Vufind-tech mailing list >>>> Vuf...@li... >>>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>>> >>>> >>> -- >>> Alan Rykhus >>> PALS, A Program of the Minnesota State Colleges and Universities >>> (507)389-1975 >>> ala...@mn... >>> >>> >>> ------------------------------------------------------------------------- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >>> Build the coolest Linux based applications with Moblin SDK & win great prizes >>> Grand prize is a trip for two to an Open Source event anywhere in the world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Vufind-tech mailing list >>> Vuf...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>> >>> >> -- >> Alan Rykhus >> PALS, A Program of the Minnesota State Colleges and Universities >> (507)389-1975 >> ala...@mn... >> >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Vufind-tech mailing list >> Vuf...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-tech >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Vufind-tech mailing list >> Vuf...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-tech >> >> >> > > -- > /** > * Wayne Graham > * Earl Gregg Swem Library > * PO Box 8794 > * Williamsburg, VA 23188 > * 757.221.3112 > * http://swem.wm.edu/blogs/waynegraham/ > * http://www.liquidfoot.com > */ > > > > -- /** * Wayne Graham * Earl Gregg Swem Library * PO Box 8794 * Williamsburg, VA 23188 * 757.221.3112 * http://swem.wm.edu/blogs/waynegraham/ * http://www.liquidfoot.com */ |
From: Alan R. <ala...@mn...> - 2008-08-04 21:40:17
|
Hello, I had this problem too, especially with topic facets as there are so many. I added an option to the facet query in SOLR.php(now solr.php) late last week and it seems to have helped. Have not had time to report on it. The original section in Solr.php starting with line 448 is: // Build Facet Options if ($facet) { $options['facet'] = 'true'; $options['facet.mincount'] = 1; $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : null; $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : null; $options['facet.prefix'] = (isset($facet['prefix'])) ? $facet['prefix'] : null; $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : null; } I added the additional facet.enum.cache.minDf option after doing some research: // Build Facet Options if ($facet) { $options['facet'] = 'true'; $options['facet.mincount'] = 1; $options['facet.enum.cache.minDf'] = 1000; $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : null; $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : null; $options['facet.prefix'] = (isset($facet['prefix'])) ? $facet['prefix'] : null; $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : null; } give it a test -- al On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: > Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. > > Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. > > Hope this helps. > > Andrew > > > -----Original Message----- > > From: vuf...@li... [mailto:vufind-tech- > > bo...@li...] On Behalf Of Barnett, Jeffrey > > Sent: Monday, August 04, 2008 4:57 PM > > To: vuf...@li... > > Subject: [VuFind-Tech] Facets -- where are the bottlenecks? > > > > Ever since we got our full 7.8million records loaded we've experienced > > significant delays "Loading Narrow Options". Following discussion here > > and in solrmarc, I think we have the solr tuning options set to > > reasonable values ... queries use less than 5% cpu. The one thing that > > made some improvement was when we decided to drop one of the facet > > categories that was confusing test users. That leads me to think the > > problem might be in the javascript and/or AJAX that is building the > > facet panel. Are there places that that might be tweaked or monitored? > > Could we omit counting facets after the top 5? > > > > ----------------------------------------------------------------------- > > -- > > This SF.Net email is sponsored by the Moblin Your Move Developer's > > challenge > > Build the coolest Linux based applications with Moblin SDK & win great > > prizes > > Grand prize is a trip for two to an Open Source event anywhere in the > > world > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > _______________________________________________ > > Vufind-tech mailing list > > Vuf...@li... > > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech -- Alan Rykhus PALS, A Program of the Minnesota State Colleges and Universities (507)389-1975 ala...@mn... |
From: Barnett, J. <jef...@ya...> - 2008-08-05 00:42:14
|
I made the change, but I don't notice any difference in response time. What was the source/reason for adding the option? I notice you refer to "Solr.php" instead of "SOLR.php". That is a relatively recent change (which I have not yet updated to) . Is there a related recent change that this one is dependent on? What svn revision are you using? ... I'm at 792 ________________________________________ From: vuf...@li... [vuf...@li...] On Behalf Of Alan Rykhus [ala...@mn...] Sent: Monday, August 04, 2008 5:40 PM To: vuf...@li... Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? Hello, I had this problem too, especially with topic facets as there are so many. I added an option to the facet query in SOLR.php(now solr.php) late last week and it seems to have helped. Have not had time to report on it. The original section in Solr.php starting with line 448 is: // Build Facet Options if ($facet) { $options['facet'] = 'true'; $options['facet.mincount'] = 1; $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : null; $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : null; $options['facet.prefix'] = (isset($facet['prefix'])) ? $facet['prefix'] : null; $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : null; } I added the additional facet.enum.cache.minDf option after doing some research: // Build Facet Options if ($facet) { $options['facet'] = 'true'; $options['facet.mincount'] = 1; $options['facet.enum.cache.minDf'] = 1000; $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : null; $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : null; $options['facet.prefix'] = (isset($facet['prefix'])) ? $facet['prefix'] : null; $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : null; } give it a test -- al On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: > Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. > > Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. > > Hope this helps. > > Andrew > > > -----Original Message----- > > From: vuf...@li... [mailto:vufind-tech- > > bo...@li...] On Behalf Of Barnett, Jeffrey > > Sent: Monday, August 04, 2008 4:57 PM > > To: vuf...@li... > > Subject: [VuFind-Tech] Facets -- where are the bottlenecks? > > > > Ever since we got our full 7.8million records loaded we've experienced > > significant delays "Loading Narrow Options". Following discussion here > > and in solrmarc, I think we have the solr tuning options set to > > reasonable values ... queries use less than 5% cpu. The one thing that > > made some improvement was when we decided to drop one of the facet > > categories that was confusing test users. That leads me to think the > > problem might be in the javascript and/or AJAX that is building the > > facet panel. Are there places that that might be tweaked or monitored? > > Could we omit counting facets after the top 5? > > > > ----------------------------------------------------------------------- > > -- > > This SF.Net email is sponsored by the Moblin Your Move Developer's > > challenge > > Build the coolest Linux based applications with Moblin SDK & win great > > prizes > > Grand prize is a trip for two to an Open Source event anywhere in the > > world > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > _______________________________________________ > > Vufind-tech mailing list > > Vuf...@li... > > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech -- Alan Rykhus PALS, A Program of the Minnesota State Colleges and Universities (507)389-1975 ala...@mn... ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Vufind-tech mailing list Vuf...@li... https://lists.sourceforge.net/lists/listinfo/vufind-tech |
From: Alan R. <ala...@mn...> - 2008-08-05 14:07:06
|
Hello Jeff, I made this change to my system as a trial the week before last because of the same thing you reported. Through testing I determined that it was the topic facet that was causing the delay. If I comment out the topic facet things run pretty good. What I read about was that SOLR builds bitsets for all facets even though it doesn't always need them. By setting this value high it only builds the bitsets when the amount is surpassed. As for the version of vufing, I am most of the way through yesterday's updates and I try to update to the latest every other day or so. I did have problems with the solr.war and built my own. You have to have the pieces of lucene in the solr.war in the solr/jetty/webapps directory the same as the lucene libraries in the libraries used for the importing of the marc records. None of this effects performance as far as I know. I am using lucene 2.3.1. Additional information. I have a linux box with 32 GB of memory. I am giving vufind 16 GB of memory( -Xmx16384M ). I have the following settings for filterCache: <filterCache class="solr.LRUCache" size="50000000" initialSize="10000000" autowarmCount="10000000"/> I upped my settings for documentCache <documentCache class="solr.LRUCache" size="500000" initialSize="50000" autowarmCount="10000"/> I do have my own schema for solr. But it is not too far out of line with what is distributed. any other questions, I'm willing to try -- al On Mon, 2008-08-04 at 19:42 -0500, Barnett, Jeffrey wrote: > I made the change, but I don't notice any difference in response time. What was the source/reason for adding the option? > I notice you refer to "Solr.php" instead of "SOLR.php". That is a relatively recent change (which I have not yet updated to) . Is there a related recent change that this one is dependent on? > > What svn revision are you using? ... I'm at 792 > ________________________________________ > From: vuf...@li... [vuf...@li...] On Behalf Of Alan Rykhus [ala...@mn...] > Sent: Monday, August 04, 2008 5:40 PM > To: vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Hello, > > I had this problem too, especially with topic facets as there are so > many. I added an option to the facet query in SOLR.php(now solr.php) > late last week and it seems to have helped. Have not had time to report > on it. > > The original section in Solr.php starting with line 448 is: > > // Build Facet Options > if ($facet) { > $options['facet'] = 'true'; > $options['facet.mincount'] = 1; > $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : > null; > $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : > null; > $options['facet.prefix'] = (isset($facet['prefix'])) ? > $facet['prefix'] : null; > $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : > null; > } > > I added the additional facet.enum.cache.minDf option after doing some > research: > > // Build Facet Options > if ($facet) { > $options['facet'] = 'true'; > $options['facet.mincount'] = 1; > $options['facet.enum.cache.minDf'] = 1000; > $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : > null; > $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : > null; > $options['facet.prefix'] = (isset($facet['prefix'])) ? > $facet['prefix'] : null; > $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : > null; > } > > give it a test -- al > > On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: > > Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. > > > > Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. > > > > Hope this helps. > > > > Andrew > > > > > -----Original Message----- > > > From: vuf...@li... [mailto:vufind-tech- > > > bo...@li...] On Behalf Of Barnett, Jeffrey > > > Sent: Monday, August 04, 2008 4:57 PM > > > To: vuf...@li... > > > Subject: [VuFind-Tech] Facets -- where are the bottlenecks? > > > > > > Ever since we got our full 7.8million records loaded we've experienced > > > significant delays "Loading Narrow Options". Following discussion here > > > and in solrmarc, I think we have the solr tuning options set to > > > reasonable values ... queries use less than 5% cpu. The one thing that > > > made some improvement was when we decided to drop one of the facet > > > categories that was confusing test users. That leads me to think the > > > problem might be in the javascript and/or AJAX that is building the > > > facet panel. Are there places that that might be tweaked or monitored? > > > Could we omit counting facets after the top 5? > > > > > > ----------------------------------------------------------------------- > > > -- > > > This SF.Net email is sponsored by the Moblin Your Move Developer's > > > challenge > > > Build the coolest Linux based applications with Moblin SDK & win great > > > prizes > > > Grand prize is a trip for two to an Open Source event anywhere in the > > > world > > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > > _______________________________________________ > > > Vufind-tech mailing list > > > Vuf...@li... > > > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > > > ------------------------------------------------------------------------- > > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > > Build the coolest Linux based applications with Moblin SDK & win great prizes > > Grand prize is a trip for two to an Open Source event anywhere in the world > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > _______________________________________________ > > Vufind-tech mailing list > > Vuf...@li... > > https://lists.sourceforge.net/lists/listinfo/vufind-tech > -- > Alan Rykhus > PALS, A Program of the Minnesota State Colleges and Universities > (507)389-1975 > ala...@mn... > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech -- Alan Rykhus PALS, A Program of the Minnesota State Colleges and Universities (507)389-1975 ala...@mn... |
From: Barnett, J. <jef...@ya...> - 2008-08-05 14:58:03
|
I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: "Invalid initial heap size: -Xms8192M The specified size exceeds the maximum representable size. Could not create the Java virtual machine." The version is $ java -version java version "1.6.0_07" Java(TM) SE Runtime Environment (build 1.6.0_07-b06) Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) and the OS is Solaris 10 Any suggestion where the 3GB limit is coming from? -----Original Message----- From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Alan Rykhus Sent: Tuesday, August 05, 2008 10:07 AM To: vuf...@li... Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? Hello Jeff, I made this change to my system as a trial the week before last because of the same thing you reported. Through testing I determined that it was the topic facet that was causing the delay. If I comment out the topic facet things run pretty good. What I read about was that SOLR builds bitsets for all facets even though it doesn't always need them. By setting this value high it only builds the bitsets when the amount is surpassed. As for the version of vufing, I am most of the way through yesterday's updates and I try to update to the latest every other day or so. I did have problems with the solr.war and built my own. You have to have the pieces of lucene in the solr.war in the solr/jetty/webapps directory the same as the lucene libraries in the libraries used for the importing of the marc records. None of this effects performance as far as I know. I am using lucene 2.3.1. Additional information. I have a linux box with 32 GB of memory. I am giving vufind 16 GB of memory( -Xmx16384M ). I have the following settings for filterCache: <filterCache class="solr.LRUCache" size="50000000" initialSize="10000000" autowarmCount="10000000"/> I upped my settings for documentCache <documentCache class="solr.LRUCache" size="500000" initialSize="50000" autowarmCount="10000"/> I do have my own schema for solr. But it is not too far out of line with what is distributed. any other questions, I'm willing to try -- al On Mon, 2008-08-04 at 19:42 -0500, Barnett, Jeffrey wrote: > I made the change, but I don't notice any difference in response time. What was the source/reason for adding the option? > I notice you refer to "Solr.php" instead of "SOLR.php". That is a relatively recent change (which I have not yet updated to) . Is there a related recent change that this one is dependent on? > > What svn revision are you using? ... I'm at 792 > ________________________________________ > From: vuf...@li... [vuf...@li...] On Behalf Of Alan Rykhus [ala...@mn...] > Sent: Monday, August 04, 2008 5:40 PM > To: vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Hello, > > I had this problem too, especially with topic facets as there are so > many. I added an option to the facet query in SOLR.php(now solr.php) > late last week and it seems to have helped. Have not had time to report > on it. > > The original section in Solr.php starting with line 448 is: > > // Build Facet Options > if ($facet) { > $options['facet'] = 'true'; > $options['facet.mincount'] = 1; > $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : > null; > $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : > null; > $options['facet.prefix'] = (isset($facet['prefix'])) ? > $facet['prefix'] : null; > $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : > null; > } > > I added the additional facet.enum.cache.minDf option after doing some > research: > > // Build Facet Options > if ($facet) { > $options['facet'] = 'true'; > $options['facet.mincount'] = 1; > $options['facet.enum.cache.minDf'] = 1000; > $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : > null; > $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : > null; > $options['facet.prefix'] = (isset($facet['prefix'])) ? > $facet['prefix'] : null; > $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : > null; > } > > give it a test -- al > > On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: > > Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. > > > > Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. > > > > Hope this helps. > > > > Andrew > > > > > -----Original Message----- > > > From: vuf...@li... [mailto:vufind-tech- > > > bo...@li...] On Behalf Of Barnett, Jeffrey > > > Sent: Monday, August 04, 2008 4:57 PM > > > To: vuf...@li... > > > Subject: [VuFind-Tech] Facets -- where are the bottlenecks? > > > > > > Ever since we got our full 7.8million records loaded we've experienced > > > significant delays "Loading Narrow Options". Following discussion here > > > and in solrmarc, I think we have the solr tuning options set to > > > reasonable values ... queries use less than 5% cpu. The one thing that > > > made some improvement was when we decided to drop one of the facet > > > categories that was confusing test users. That leads me to think the > > > problem might be in the javascript and/or AJAX that is building the > > > facet panel. Are there places that that might be tweaked or monitored? > > > Could we omit counting facets after the top 5? > > > > > > ----------------------------------------------------------------------- > > > -- > > > This SF.Net email is sponsored by the Moblin Your Move Developer's > > > challenge > > > Build the coolest Linux based applications with Moblin SDK & win great > > > prizes > > > Grand prize is a trip for two to an Open Source event anywhere in the > > > world > > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > > _______________________________________________ > > > Vufind-tech mailing list > > > Vuf...@li... > > > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > > > ------------------------------------------------------------------------- > > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > > Build the coolest Linux based applications with Moblin SDK & win great prizes > > Grand prize is a trip for two to an Open Source event anywhere in the world > > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > _______________________________________________ > > Vufind-tech mailing list > > Vuf...@li... > > https://lists.sourceforge.net/lists/listinfo/vufind-tech > -- > Alan Rykhus > PALS, A Program of the Minnesota State Colleges and Universities > (507)389-1975 > ala...@mn... > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech -- Alan Rykhus PALS, A Program of the Minnesota State Colleges and Universities (507)389-1975 ala...@mn... ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Vufind-tech mailing list Vuf...@li... https://lists.sourceforge.net/lists/listinfo/vufind-tech |
From: Wayne G. <ws...@wm...> - 2008-08-05 15:14:17
|
Are you setting the maximum heap size (-Xmx) before the minimum (-Xms)? The default maximum is a weird algorithm of a percentage of the memory space (probably around 4 Gb on your system), and if you're not explicitly setting this, it's probably trying to make the minimum heap space larger than the maximum heap space. Take a look at http://java.sun.com/performance/reference/whitepapers/tuning.html#section4.1.2 Wayne Barnett, Jeffrey wrote: > I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: > > "Invalid initial heap size: -Xms8192M > The specified size exceeds the maximum representable size. > Could not create the Java virtual machine." > > The version is > $ java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) > > and the OS is Solaris 10 > > Any suggestion where the 3GB limit is coming from? > > -----Original Message----- > From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Alan Rykhus > Sent: Tuesday, August 05, 2008 10:07 AM > To: vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Hello Jeff, > > I made this change to my system as a trial the week before last because > of the same thing you reported. Through testing I determined that it was > the topic facet that was causing the delay. If I comment out the topic > facet things run pretty good. > > What I read about was that SOLR builds bitsets for all facets even > though it doesn't always need them. By setting this value high it only > builds the bitsets when the amount is surpassed. > > As for the version of vufing, I am most of the way through yesterday's > updates and I try to update to the latest every other day or so. I did > have problems with the solr.war and built my own. You have to have the > pieces of lucene in the solr.war in the solr/jetty/webapps directory the > same as the lucene libraries in the libraries used for the importing of > the marc records. None of this effects performance as far as I know. I > am using lucene 2.3.1. > > Additional information. I have a linux box with 32 GB of memory. > > I am giving vufind 16 GB of memory( -Xmx16384M ). > > I have the following settings for filterCache: > <filterCache > class="solr.LRUCache" > size="50000000" > initialSize="10000000" > autowarmCount="10000000"/> > > I upped my settings for documentCache > > <documentCache > class="solr.LRUCache" > size="500000" > initialSize="50000" > autowarmCount="10000"/> > > I do have my own schema for solr. But it is not too far out of line with > what is distributed. > > any other questions, I'm willing to try -- al > > On Mon, 2008-08-04 at 19:42 -0500, Barnett, Jeffrey wrote: > >> I made the change, but I don't notice any difference in response time. What was the source/reason for adding the option? >> I notice you refer to "Solr.php" instead of "SOLR.php". That is a relatively recent change (which I have not yet updated to) . Is there a related recent change that this one is dependent on? >> >> What svn revision are you using? ... I'm at 792 >> ________________________________________ >> From: vuf...@li... [vuf...@li...] On Behalf Of Alan Rykhus [ala...@mn...] >> Sent: Monday, August 04, 2008 5:40 PM >> To: vuf...@li... >> Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? >> >> Hello, >> >> I had this problem too, especially with topic facets as there are so >> many. I added an option to the facet query in SOLR.php(now solr.php) >> late last week and it seems to have helped. Have not had time to report >> on it. >> >> The original section in Solr.php starting with line 448 is: >> >> // Build Facet Options >> if ($facet) { >> $options['facet'] = 'true'; >> $options['facet.mincount'] = 1; >> $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : >> null; >> $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : >> null; >> $options['facet.prefix'] = (isset($facet['prefix'])) ? >> $facet['prefix'] : null; >> $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : >> null; >> } >> >> I added the additional facet.enum.cache.minDf option after doing some >> research: >> >> // Build Facet Options >> if ($facet) { >> $options['facet'] = 'true'; >> $options['facet.mincount'] = 1; >> $options['facet.enum.cache.minDf'] = 1000; >> $options['facet.limit'] = (isset($facet['limit'])) ? $facet['limit'] : >> null; >> $options['facet.field'] = (isset($facet['field'])) ? $facet['field'] : >> null; >> $options['facet.prefix'] = (isset($facet['prefix'])) ? >> $facet['prefix'] : null; >> $options['facet.sort'] = (isset($facet['sort'])) ? $facet['sort'] : >> null; >> } >> >> give it a test -- al >> >> On Mon, 2008-08-04 at 16:21 -0500, Andrew Nagy wrote: >> >>> Solr processes all the work with the facets - all that VuFind does is take the data from solr and display it. All the performance should happen with Jetty and Solr - mainly with Solr. The best place to look is with the cache settings. I spoke to some of the folks at CNET and said that the cache settings can be seen as almost black magic. It's very hard to determine the best settings - but minor tweaks can have major differences. >>> >>> Plus also your hardware infrastructure like - that could also make a major impact. The more memory the faster the facets. >>> >>> Hope this helps. >>> >>> Andrew >>> >>> >>>> -----Original Message----- >>>> From: vuf...@li... [mailto:vufind-tech- >>>> bo...@li...] On Behalf Of Barnett, Jeffrey >>>> Sent: Monday, August 04, 2008 4:57 PM >>>> To: vuf...@li... >>>> Subject: [VuFind-Tech] Facets -- where are the bottlenecks? >>>> >>>> Ever since we got our full 7.8million records loaded we've experienced >>>> significant delays "Loading Narrow Options". Following discussion here >>>> and in solrmarc, I think we have the solr tuning options set to >>>> reasonable values ... queries use less than 5% cpu. The one thing that >>>> made some improvement was when we decided to drop one of the facet >>>> categories that was confusing test users. That leads me to think the >>>> problem might be in the javascript and/or AJAX that is building the >>>> facet panel. Are there places that that might be tweaked or monitored? >>>> Could we omit counting facets after the top 5? >>>> >>>> ----------------------------------------------------------------------- >>>> -- >>>> This SF.Net email is sponsored by the Moblin Your Move Developer's >>>> challenge >>>> Build the coolest Linux based applications with Moblin SDK & win great >>>> prizes >>>> Grand prize is a trip for two to an Open Source event anywhere in the >>>> world >>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>>> _______________________________________________ >>>> Vufind-tech mailing list >>>> Vuf...@li... >>>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>>> >>> ------------------------------------------------------------------------- >>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >>> Build the coolest Linux based applications with Moblin SDK & win great prizes >>> Grand prize is a trip for two to an Open Source event anywhere in the world >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >>> _______________________________________________ >>> Vufind-tech mailing list >>> Vuf...@li... >>> https://lists.sourceforge.net/lists/listinfo/vufind-tech >>> >> -- >> Alan Rykhus >> PALS, A Program of the Minnesota State Colleges and Universities >> (507)389-1975 >> ala...@mn... >> >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Vufind-tech mailing list >> Vuf...@li... >> https://lists.sourceforge.net/lists/listinfo/vufind-tech >> > -- > Alan Rykhus > PALS, A Program of the Minnesota State Colleges and Universities > (507)389-1975 > ala...@mn... > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > -- /** * Wayne Graham * Earl Gregg Swem Library * PO Box 8794 * Williamsburg, VA 23188 * 757.221.3112 * http://swem.wm.edu/blogs/waynegraham/ * http://www.liquidfoot.com */ |
From: Alan R. <ala...@mn...> - 2008-08-05 15:23:12
|
Hello Jeff, Do you have a 32 or 64 bit machine? It seems that you need a 64 bit machine for the larger memory settings. al On Tue, 2008-08-05 at 09:57 -0500, Barnett, Jeffrey wrote: > I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: > > "Invalid initial heap size: -Xms8192M > The specified size exceeds the maximum representable size. > Could not create the Java virtual machine." > > The version is > $ java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) > > and the OS is Solaris 10 > > Any suggestion where the 3GB limit is coming from? > -- Alan Rykhus PALS, A Program of the Minnesota State Colleges and Universities (507)389-1975 ala...@mn... |
From: Barnett, J. <jef...@ya...> - 2008-08-05 19:01:05
|
With the breakthough on JVM memory, and increased Cache size, I'm down to roughly 10 sec to build an respond to a facet query. We will have to do better, but I'm wondering if there is a way to prevent rebuilding the query every time a new page is displayed. With a large result set, every time you click on "next" you have to wait for the "narrowing options" to redisplay, even though the query itself hasn't changed. Even for sites with smaller catalogs and shorter reponse times, the extra cpu load means ultimately fewer concurrent users. Possibly related glitch: When you are in the n'th page of a multi-page response, and you select one of the narrowing options, you are dropped into the same n'th page of the new response set. "&page=n" gets incorporated into the narrowing url. -----Original Message----- From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Alan Rykhus Sent: Tuesday, August 05, 2008 11:23 AM To: vuf...@li... Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? Hello Jeff, Do you have a 32 or 64 bit machine? It seems that you need a 64 bit machine for the larger memory settings. al On Tue, 2008-08-05 at 09:57 -0500, Barnett, Jeffrey wrote: > I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: > > "Invalid initial heap size: -Xms8192M > The specified size exceeds the maximum representable size. > Could not create the Java virtual machine." > > The version is > $ java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) > > and the OS is Solaris 10 > > Any suggestion where the 3GB limit is coming from? > -- Alan Rykhus PALS, A Program of the Minnesota State Colleges and Universities (507)389-1975 ala...@mn... ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Vufind-tech mailing list Vuf...@li... https://lists.sourceforge.net/lists/listinfo/vufind-tech |
From: Wayne G. <ws...@wm...> - 2008-08-05 19:17:57
|
I'm not sure if the bottleneck is in the PHP construction of the query since it's just a string, or the actual searching of the index for facets. Since you have such a large index, you may need to play with the cache settings in solr (see http://wiki.apache.org/solr/SolrCaching). Also, did you play with your young generation settings? wayne Barnett, Jeffrey wrote: > With the breakthough on JVM memory, and increased Cache size, I'm down to roughly 10 sec to build an respond to a facet query. We will have to do better, but I'm wondering if there is a way to prevent rebuilding the query every time a new page is displayed. With a large result set, every time you click on "next" you have to wait for the "narrowing options" to redisplay, even though the query itself hasn't changed. Even for sites with smaller catalogs and shorter reponse times, the extra cpu load means ultimately fewer concurrent users. > > Possibly related glitch: > When you are in the n'th page of a multi-page response, and you select one of the narrowing options, you are dropped into the same n'th page of the new response set. "&page=n" gets incorporated into the narrowing url. > > -----Original Message----- > From: vuf...@li... [mailto:vuf...@li...] On Behalf Of Alan Rykhus > Sent: Tuesday, August 05, 2008 11:23 AM > To: vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Hello Jeff, > > Do you have a 32 or 64 bit machine? It seems that you need a 64 bit > machine for the larger memory settings. > > al > > On Tue, 2008-08-05 at 09:57 -0500, Barnett, Jeffrey wrote: > >> I looks like the root of my problem is too little memory. I have a 16GB machine, but I can't get the JVM to use more than 3072M. Any thing bigger gets an error message: >> >> "Invalid initial heap size: -Xms8192M >> The specified size exceeds the maximum representable size. >> Could not create the Java virtual machine." >> >> The version is >> $ java -version >> java version "1.6.0_07" >> Java(TM) SE Runtime Environment (build 1.6.0_07-b06) >> Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) >> >> and the OS is Solaris 10 >> >> Any suggestion where the 3GB limit is coming from? >> >> > > -- > Alan Rykhus > PALS, A Program of the Minnesota State Colleges and Universities > (507)389-1975 > ala...@mn... > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > -- /** * Wayne Graham * Earl Gregg Swem Library * PO Box 8794 * Williamsburg, VA 23188 * 757.221.3112 * http://swem.wm.edu/blogs/waynegraham/ * http://www.liquidfoot.com */ |
From: Andrew N. <and...@vi...> - 2008-08-11 13:10:09
|
Jeffrey - can you check the Solr statistics page and check your evictions with each cache handler. You want that number as small as possible. If the number is high - you need to increase your cache settings. My guess if you don't have enough cache allocated. > -----Original Message----- > From: vuf...@li... [mailto:vufind-tech- > bo...@li...] On Behalf Of Barnett, Jeffrey > Sent: Tuesday, August 05, 2008 3:01 PM > To: Alan Rykhus; vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- why rebuild every page? > > With the breakthough on JVM memory, and increased Cache size, I'm down > to roughly 10 sec to build an respond to a facet query. We will have > to do better, but I'm wondering if there is a way to prevent rebuilding > the query every time a new page is displayed. With a large result set, > every time you click on "next" you have to wait for the "narrowing > options" to redisplay, even though the query itself hasn't changed. > Even for sites with smaller catalogs and shorter reponse times, the > extra cpu load means ultimately fewer concurrent users. > > Possibly related glitch: > When you are in the n'th page of a multi-page response, and you select > one of the narrowing options, you are dropped into the same n'th page > of the new response set. "&page=n" gets incorporated into the > narrowing url. > > -----Original Message----- > From: vuf...@li... [mailto:vufind-tech- > bo...@li...] On Behalf Of Alan Rykhus > Sent: Tuesday, August 05, 2008 11:23 AM > To: vuf...@li... > Subject: Re: [VuFind-Tech] Facets -- where are the bottlenecks? > > Hello Jeff, > > Do you have a 32 or 64 bit machine? It seems that you need a 64 bit > machine for the larger memory settings. > > al > > On Tue, 2008-08-05 at 09:57 -0500, Barnett, Jeffrey wrote: > > I looks like the root of my problem is too little memory. I have a > 16GB machine, but I can't get the JVM to use more than 3072M. Any > thing bigger gets an error message: > > > > "Invalid initial heap size: -Xms8192M > > The specified size exceeds the maximum representable size. > > Could not create the Java virtual machine." > > > > The version is > > $ java -version > > java version "1.6.0_07" > > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > > Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode) > > > > and the OS is Solaris 10 > > > > Any suggestion where the 3GB limit is coming from? > > > > -- > Alan Rykhus > PALS, A Program of the Minnesota State Colleges and Universities > (507)389-1975 > ala...@mn... > > > ----------------------------------------------------------------------- > -- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the > world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech > > ----------------------------------------------------------------------- > -- > This SF.Net email is sponsored by the Moblin Your Move Developer's > challenge > Build the coolest Linux based applications with Moblin SDK & win great > prizes > Grand prize is a trip for two to an Open Source event anywhere in the > world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Vufind-tech mailing list > Vuf...@li... > https://lists.sourceforge.net/lists/listinfo/vufind-tech |