Re: [VuFind-Tech] Problem solved: Problems with max boolean clause and tags

[Demian K. <dem...@vi...>]

VuFind 2.0 may help with this -- I'm trying to design it so that more of the work of dealing with tags and favorites is done by the MySQL database in order to avoid these types of enormous OR'ed Solr lookups.  The disadvantage is that not all of the same sorting options will be available, but I think that is a fair trade-off for avoiding artificial ceilings on how many records can be saved/tagged.

- Demian
________________________________________
From: Oliver Goldschmidt [o.g...@tu...]
Sent: Tuesday, February 07, 2012 9:32 AM
To: vuf...@li...
Subject: [VuFind-Tech] Problem solved: Problems with max boolean clause and     tags

Hi again,

I found a solution just after posting my problem to the list. The form size can be manipulated on Vufind startup using the parameter
-Dorg.mortbay.jetty.Request.maxFormContentSize=xy

I have set xy to 1000000 (default value was 200000). That works for larger result sets. But its not very performant.

-Oliver

-------- Original-Nachricht --------
Betreff:        Problems with max boolean clause and tags
Datum:  Tue, 07 Feb 2012 14:46:23 +0100
Von:    Oliver Goldschmidt <o.g...@tu...><mailto:o.g...@tu...>
An:     vuf...@li...<mailto:vuf...@li...> <vuf...@li...><mailto:vuf...@li...>



Hi all,

we had some problems executing queries against the tag-database
concerning the maxBooleanClause on Solr and Vufind side. We have tags,
that are used for more records than configurated in maxBooleanClause in
Solr. So I raised that value for our Vufind index and for the shards.
But I think somewhere between 5000 and 5500 must be a border; if I
configure a value higher than that, I always get an exception:

Unable to process query
Solr Returned: Form too large331769>200000
java.lang.IllegalStateException: Form too large331769>200000 at
org.mortbay.jetty.Request.extractParameters(Request.java:1404) at
org.mortbay.jetty.Request.getParameterMap(Request.java:760) at
org.apache.solr.request.ServletSolrParams.<init>(ServletSolrParams.java:29)
at
org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:394)
at
org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:115)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324) at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647) at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

RequestURI=/solr/biblio/select/

I have written a patch, which breaks the one query into severals, if the
result would return more hits than configured in config.ini
maxBooleanClause. But this patch cannot get facets from the whole result
(because the whole result is splitted), so it suppresses the facets
completly. I have uploaded the patch here:
http://vufind.org/jira/browse/VUFIND-504

Does anyone have an idea how to raise the value in Solr's config file,
so that I can get a large number of tagged hits in Vufind? How can this
value get raised? How do I avoid the exception?

Any hints are appreciated!
Thanks in advance
Oliver

--
Oliver Goldschmidt
TU Hamburg-Harburg / Universitätsbibliothek / Digitale Dienste
Denickestr. 22
21071 Hamburg - Harburg
Tel.    +49 (0)40 / 428 78 - 32 91
eMail   o.g...@tu...<mailto:o.g...@tu...>
--
GPG/PGP-Schlüssel:
http://www.tub.tu-harburg.de/keys/Oliver_Marahrens_pub.asc
--
Projekt DISCUS http://discus.tu-harburg.de
Projekt TUBdok http://doku.b.tu-harburg.de