Re: [Sqlgrey-users] Load balancing question, part 2

[Dan F. <da...@ha...>]

Gary Smith wrote:
>> I have setup two sqlgrey servers load balanced with ipvsadm.  Load balancing
>> is operating but I end up with a lot of orphaned ESTABLISHED connections on
>> the real servers.  In a period of 48 hours, I received ~500 requests (per real
>> server) and here were about ~250 established connections per server.
>>
>>
>>     
Ive not tried LVS'ing sqlgrey but i do use LVS (ipvsadm) alot.
I run sqlgrey on the MTA, and then LVS to the MTA's. The MTA's then 
communicate with sqlgrey on localhost. Sqlgrey reads from MySQL on 
localhost and writes to a master for replication.


When you say ESTABLISHED, i assume you mean from looking at the output 
from netstat or something like it. If so, i cant imagine that it can be 
an application level problem. If postfix deliberately closes the 
connection (eg due to a timeout), it should/would transmit either a TCP 
RST or FIN. The receiving party (your cluster node) will handle RST and 
FIN in the kernel's ip-stack, not in the application.
Upon receiving FIN, "ESTABLISHED" should have changed to "CLOSE-WAIT" in 
netstat.
Which suggests that your cluster node does not actually receive a FIN. 
Which again suggest that the connection drop before your cluster node, 
ie. the loadbalancer.

Of course, not knowing you LVS setup and config, i can only guess. But a 
typical misconfigure would be using "Direct Routing" (the "gatewaying" 
option in ipvsadm. which is default), without taking precautions against 
the cluster nodes ARP'ing the virutal ip. If this is the case, all 
cluster nodes would "battle" for the virtual ip, making the ip "hop" 
around the nodes.
That would, i imagine, leave ESTABLISHED connections behind.


Regards
- Dan Faerch