Re: [VuFind-Tech] high availability question

[Tuan N. <tu...@yo...>]

Thank you, Joe and Demian.
 
For this purpose, I think mod_proxy and nginx are quite similar. I like the idea of dynamically adjusting the round-robin weigh. I think it should be a doable with some shell scripts on the apache front end. I will try that route first.

I see your point about guaranteeing response time. We do have pretty predictable spikes, so whenever the response time is more than 2 or 3 seconds, it is usually caused by our Google search appliance hitting us hard, or it is at the end of classes, and then the load goes away within 5 minutes or so. I think monitoring and adjusting the load balancing on the fly is a good idea and will work nicely.

SolrCloud does look very promising, and will be worth looking into when we upgrade to 4.x.

On 2013-03-18, at 12:48 PM, Joe Atzberger wrote:

> I would consider using nginx to do the load balancing between SOLR instances, as described here:
> http://stackoverflow.com/questions/10971298/load-balancing-with-nginx
> 
> However, a word of warning.  If you make your server fail over after a 2 or 3 second timeout, you are essentially guaranteeing that it never takes longer than that to service a request.  That is IMHO an unrealistic expectation and you will be creating (and recirculating) additional connections and duplicate work.  In that respect you can make things worse than if you were just balancing via round-robin.  A more articulate approach might try to use monitoring/reporting on the backend SOLR servers themselves (e.g. via nagios) to dynamically adjust the balance-algorithm weights in an otherwise round-robin setup.  That scenario only duplicates actual failures and provides more of a gradual response, at the cost of some complexiy.
> 
> --joe
> 
> On Mon, Mar 18, 2013 at 12:19 PM, Tuan Nguyen <tu...@yo...> wrote:
> Hi everyone,
> 
> I am pondering the idea of have of modifying the sys/Solr.php and sys/ConnectionManager.php to allow for a list of SOLR servers to try upon failure or to switch to a different solr host if the response time is not adequate.
> 
> Has anyone done this? If so, would you please share your experience and/or code.
> 
> We currently have 2 solr instances sitting on 2 servers. We have apache mod_proxy sitting in front and pass connections to the 2 back-end solr instances. At this time we have one server set up as hot failover. This has been working quite well. However, it requires setting up apache to direct requests. One of the problem that we have been seeing is that when the main SOLR instance is busy, the requests keep piling on and failover does not kick in until that instance is dead. Ideally, I would like to have the failover happens as soon as the request takes for than 2 or 3 seconds, thus giving the users an adequate response time as well as relieving the stress on the other server.
> 
> I am thinking of implementing this functionality inside of sys/Solr.php, but I am hoping I don't have to.
> 
> Thanks,
> Tuan