Thread: [Proxool-developer] concurrency bug in Prototyper class?

UNMAINTAINED!

Brought to you by: billhorsman

proxool-developer

[Proxool-developer] concurrency bug in Prototyper class?

From: Matt S. <msa...@gm...> - 2009-06-19 17:28:28

It seems Class Prototyper should either acquire the lock on private member
'lock' whenever it reads or modifies 'connectionCount' or make
'connectionCount' a volatile variable.

Prototyper.buildConnection( ) acquires the lock to read and modify
'connectionCount' but other methods do not: Prototyper.sweep( ),
Prototyper.connectionRemoved( ) and Prototyper.getConnectionCount().

I started poking around after my application started seeing
ProxoolExceptions indicating the maximum connection count had been reached
and discovered what appears to be a memory visibility issue.  If calls to
Prototyper.connectionRemoved( ) and Prototyper.buildConnection( ) happen in
different threads there could be unwarranted ProxoolExceptions for reaching
the maximum connection count - a 'connectionPool' decrement might not be
visible to the threads calling Prototyper.buildConnection( ).

Re: [Proxool-developer] concurrency bug in Prototyper class?

From: Bill H. <bi...@lo...> - 2009-06-19 18:03:01

Is that a problem that corrects itself? In other words, is it a temporary
problem where a connection is refused when it didn't have to be but that the
connection count does end up being correct? - Bill

2009/6/19 Matt Salafia <msa...@gm...>

> It seems Class Prototyper should either acquire the lock on private member
> 'lock' whenever it reads or modifies 'connectionCount' or make
> 'connectionCount' a volatile variable.
>
> Prototyper.buildConnection( ) acquires the lock to read and modify
> 'connectionCount' but other methods do not: Prototyper.sweep( ),
> Prototyper.connectionRemoved( ) and Prototyper.getConnectionCount().
>
> I started poking around after my application started seeing
> ProxoolExceptions indicating the maximum connection count had been reached
> and discovered what appears to be a memory visibility issue.  If calls to
> Prototyper.connectionRemoved( ) and Prototyper.buildConnection( ) happen in
> different threads there could be unwarranted ProxoolExceptions for reaching
> the maximum connection count - a 'connectionPool' decrement might not be
> visible to the threads calling Prototyper.buildConnection( ).
>
>
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
>
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> Proxool-developer mailing list
> Pro...@li...
> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>
>

Re: [Proxool-developer] concurrency bug in Prototyper class?

From: Matt S. <msa...@gm...> - 2009-06-19 18:59:08

Possibly, but I don't know - it's indeterminate.  Without the proper memory
visibility semantics (e.g. synchronization, immutability, volatile et. al.)
I think all bets are off.


On Fri, Jun 19, 2009 at 1:32 PM, Bill Horsman <bi...@lo...>wrote:

> Is that a problem that corrects itself? In other words, is it a temporary
> problem where a connection is refused when it didn't have to be but that the
> connection count does end up being correct? - Bill
>
> 2009/6/19 Matt Salafia <msa...@gm...>
>
>> It seems Class Prototyper should either acquire the lock on private member
>> 'lock' whenever it reads or modifies 'connectionCount' or make
>> 'connectionCount' a volatile variable.
>>
>> Prototyper.buildConnection( ) acquires the lock to read and modify
>> 'connectionCount' but other methods do not: Prototyper.sweep( ),
>> Prototyper.connectionRemoved( ) and Prototyper.getConnectionCount().
>>
>> I started poking around after my application started seeing
>> ProxoolExceptions indicating the maximum connection count had been reached
>> and discovered what appears to be a memory visibility issue.  If calls to
>> Prototyper.connectionRemoved( ) and Prototyper.buildConnection( ) happen in
>> different threads there could be unwarranted ProxoolExceptions for reaching
>> the maximum connection count - a 'connectionPool' decrement might not be
>> visible to the threads calling Prototyper.buildConnection( ).
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Are you an open source citizen? Join us for the Open Source Bridge
>> conference!
>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>> $250.
>> Need another reason to go? 24-hour hacker lounge. Register today!
>>
>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>> _______________________________________________
>> Proxool-developer mailing list
>> Pro...@li...
>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>
>>
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
>
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> Proxool-developer mailing list
> Pro...@li...
> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>
>

Re: [Proxool-developer] concurrency bug in Prototyper class?

From: Bill H. <bi...@lo...> - 2009-06-19 20:24:07

I see your point. We've jumped through a few hoops in Proxool to get the
balance right between concurrency and excessive blocking. As you'd expect,
our first priority is to ensure the code doesn't break in concurrent
situations, but it's closely followed by our second priority: that it must
be very fast and not do unnecessary locking. I'm not saying you're right or
wrong - just that it would be hasty to grab a lock to read the
connectionCount unless we explore the implications.

2009/6/19 Matt Salafia <msa...@gm...>

> Possibly, but I don't know - it's indeterminate.  Without the proper memory
> visibility semantics (e.g. synchronization, immutability, volatile et. al.)
> I think all bets are off.
>
>
>
> On Fri, Jun 19, 2009 at 1:32 PM, Bill Horsman <bi...@lo...>wrote:
>
>> Is that a problem that corrects itself? In other words, is it a temporary
>> problem where a connection is refused when it didn't have to be but that the
>> connection count does end up being correct? - Bill
>>
>> 2009/6/19 Matt Salafia <msa...@gm...>
>>
>>>  It seems Class Prototyper should either acquire the lock on private
>>> member 'lock' whenever it reads or modifies 'connectionCount' or make
>>> 'connectionCount' a volatile variable.
>>>
>>> Prototyper.buildConnection( ) acquires the lock to read and modify
>>> 'connectionCount' but other methods do not: Prototyper.sweep( ),
>>> Prototyper.connectionRemoved( ) and Prototyper.getConnectionCount().
>>>
>>> I started poking around after my application started seeing
>>> ProxoolExceptions indicating the maximum connection count had been reached
>>> and discovered what appears to be a memory visibility issue.  If calls to
>>> Prototyper.connectionRemoved( ) and Prototyper.buildConnection( ) happen in
>>> different threads there could be unwarranted ProxoolExceptions for reaching
>>> the maximum connection count - a 'connectionPool' decrement might not be
>>> visible to the threads calling Prototyper.buildConnection( ).
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Are you an open source citizen? Join us for the Open Source Bridge
>>> conference!
>>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>>> $250.
>>> Need another reason to go? 24-hour hacker lounge. Register today!
>>>
>>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>>> _______________________________________________
>>> Proxool-developer mailing list
>>> Pro...@li...
>>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Are you an open source citizen? Join us for the Open Source Bridge
>> conference!
>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>> $250.
>> Need another reason to go? 24-hour hacker lounge. Register today!
>>
>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>> _______________________________________________
>> Proxool-developer mailing list
>> Pro...@li...
>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>
>>
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
>
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> Proxool-developer mailing list
> Pro...@li...
> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>
>

Re: [Proxool-developer] concurrency bug in Prototyper class?

From: Matt S. <msa...@gm...> - 2009-06-19 21:30:19

In this specific case, I believe making 'connectionCount' a volatile
variable satisfies both goals: correctness and performance.  It even
establishes the 'out-of-thin-air' guarantee you don't have when reading a
non-synchronized, non-volatile 64-bit value.

I think sacrificing correctness for performance is OK in places where stale
(although correct) values won't hurt you.  I don't believe it works with the
variable holding keys to the connection pool.

My application is massively multithreaded.  There are a large number of
concurrent threads acquiring and releasing connections. The fact that these
threads will always see the 'connectionCount' increment done in
Prototyper.buildConnection( ) but might not see any decrements done in
Prototyper.connectionRemoved( ) seems serious to me.


On Fri, Jun 19, 2009 at 3:59 PM, Bill Horsman <bi...@lo...>wrote:

> I see your point. We've jumped through a few hoops in Proxool to get the
> balance right between concurrency and excessive blocking. As you'd expect,
> our first priority is to ensure the code doesn't break in concurrent
> situations, but it's closely followed by our second priority: that it must
> be very fast and not do unnecessary locking. I'm not saying you're right or
> wrong - just that it would be hasty to grab a lock to read the
> connectionCount unless we explore the implications.
>
> 2009/6/19 Matt Salafia <msa...@gm...>
>
>> Possibly, but I don't know - it's indeterminate.  Without the proper
>> memory visibility semantics (e.g. synchronization, immutability, volatile
>> et. al.) I think all bets are off.
>>
>>
>>
>> On Fri, Jun 19, 2009 at 1:32 PM, Bill Horsman <bi...@lo...>wrote:
>>
>>> Is that a problem that corrects itself? In other words, is it a temporary
>>> problem where a connection is refused when it didn't have to be but that the
>>> connection count does end up being correct? - Bill
>>>
>>> 2009/6/19 Matt Salafia <msa...@gm...>
>>>
>>>>  It seems Class Prototyper should either acquire the lock on private
>>>> member 'lock' whenever it reads or modifies 'connectionCount' or make
>>>> 'connectionCount' a volatile variable.
>>>>
>>>> Prototyper.buildConnection( ) acquires the lock to read and modify
>>>> 'connectionCount' but other methods do not: Prototyper.sweep( ),
>>>> Prototyper.connectionRemoved( ) and Prototyper.getConnectionCount().
>>>>
>>>> I started poking around after my application started seeing
>>>> ProxoolExceptions indicating the maximum connection count had been reached
>>>> and discovered what appears to be a memory visibility issue.  If calls to
>>>> Prototyper.connectionRemoved( ) and Prototyper.buildConnection( ) happen in
>>>> different threads there could be unwarranted ProxoolExceptions for reaching
>>>> the maximum connection count - a 'connectionPool' decrement might not be
>>>> visible to the threads calling Prototyper.buildConnection( ).
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Are you an open source citizen? Join us for the Open Source Bridge
>>>> conference!
>>>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>>>> $250.
>>>> Need another reason to go? 24-hour hacker lounge. Register today!
>>>>
>>>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>>>> _______________________________________________
>>>> Proxool-developer mailing list
>>>> Pro...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Are you an open source citizen? Join us for the Open Source Bridge
>>> conference!
>>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>>> $250.
>>> Need another reason to go? 24-hour hacker lounge. Register today!
>>>
>>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>>> _______________________________________________
>>> Proxool-developer mailing list
>>> Pro...@li...
>>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Are you an open source citizen? Join us for the Open Source Bridge
>> conference!
>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>> $250.
>> Need another reason to go? 24-hour hacker lounge. Register today!
>>
>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>> _______________________________________________
>> Proxool-developer mailing list
>> Pro...@li...
>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>
>>
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
>
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> Proxool-developer mailing list
> Pro...@li...
> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>
>

Re: [Proxool-developer] concurrency bug in Prototyper class?

From: Bill H. <bi...@lo...> - 2009-06-19 21:33:41

If you want to patch it and try it out in your app I'm certainly open to
including it Proxool. Anybody else got anything to say on the subject?

2009/6/19 Matt Salafia <msa...@gm...>

> In this specific case, I believe making 'connectionCount' a volatile
> variable satisfies both goals: correctness and performance.  It even
> establishes the 'out-of-thin-air' guarantee you don't have when reading a
> non-synchronized, non-volatile 64-bit value.
>
> I think sacrificing correctness for performance is OK in places where stale
> (although correct) values won't hurt you.  I don't believe it works with the
> variable holding keys to the connection pool.
>
> My application is massively multithreaded.  There are a large number of
> concurrent threads acquiring and releasing connections. The fact that these
> threads will always see the 'connectionCount' increment done in
> Prototyper.buildConnection( ) but might not see any decrements done in
> Prototyper.connectionRemoved( ) seems serious to me.
>
>
>
> On Fri, Jun 19, 2009 at 3:59 PM, Bill Horsman <bi...@lo...>wrote:
>
>> I see your point. We've jumped through a few hoops in Proxool to get the
>> balance right between concurrency and excessive blocking. As you'd expect,
>> our first priority is to ensure the code doesn't break in concurrent
>> situations, but it's closely followed by our second priority: that it must
>> be very fast and not do unnecessary locking. I'm not saying you're right or
>> wrong - just that it would be hasty to grab a lock to read the
>> connectionCount unless we explore the implications.
>>
>> 2009/6/19 Matt Salafia <msa...@gm...>
>>
>>> Possibly, but I don't know - it's indeterminate.  Without the proper
>>> memory visibility semantics (e.g. synchronization, immutability, volatile
>>> et. al.) I think all bets are off.
>>>
>>>
>>>
>>> On Fri, Jun 19, 2009 at 1:32 PM, Bill Horsman <bi...@lo...>wrote:
>>>
>>>> Is that a problem that corrects itself? In other words, is it a
>>>> temporary problem where a connection is refused when it didn't have to be
>>>> but that the connection count does end up being correct? - Bill
>>>>
>>>> 2009/6/19 Matt Salafia <msa...@gm...>
>>>>
>>>>>  It seems Class Prototyper should either acquire the lock on private
>>>>> member 'lock' whenever it reads or modifies 'connectionCount' or make
>>>>> 'connectionCount' a volatile variable.
>>>>>
>>>>> Prototyper.buildConnection( ) acquires the lock to read and modify
>>>>> 'connectionCount' but other methods do not: Prototyper.sweep( ),
>>>>> Prototyper.connectionRemoved( ) and Prototyper.getConnectionCount().
>>>>>
>>>>> I started poking around after my application started seeing
>>>>> ProxoolExceptions indicating the maximum connection count had been reached
>>>>> and discovered what appears to be a memory visibility issue.  If calls to
>>>>> Prototyper.connectionRemoved( ) and Prototyper.buildConnection( ) happen in
>>>>> different threads there could be unwarranted ProxoolExceptions for reaching
>>>>> the maximum connection count - a 'connectionPool' decrement might not be
>>>>> visible to the threads calling Prototyper.buildConnection( ).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Are you an open source citizen? Join us for the Open Source Bridge
>>>>> conference!
>>>>> Portland, OR, June 17-19. Two days of sessions, one day of
>>>>> unconference: $250.
>>>>> Need another reason to go? 24-hour hacker lounge. Register today!
>>>>>
>>>>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>>>>> _______________________________________________
>>>>> Proxool-developer mailing list
>>>>> Pro...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Are you an open source citizen? Join us for the Open Source Bridge
>>>> conference!
>>>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>>>> $250.
>>>> Need another reason to go? 24-hour hacker lounge. Register today!
>>>>
>>>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>>>> _______________________________________________
>>>> Proxool-developer mailing list
>>>> Pro...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Are you an open source citizen? Join us for the Open Source Bridge
>>> conference!
>>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>>> $250.
>>> Need another reason to go? 24-hour hacker lounge. Register today!
>>>
>>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>>> _______________________________________________
>>> Proxool-developer mailing list
>>> Pro...@li...
>>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Are you an open source citizen? Join us for the Open Source Bridge
>> conference!
>> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
>> $250.
>> Need another reason to go? 24-hour hacker lounge. Register today!
>>
>> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
>> _______________________________________________
>> Proxool-developer mailing list
>> Pro...@li...
>> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>>
>>
>
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge
> conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference:
> $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
>
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> Proxool-developer mailing list
> Pro...@li...
> https://lists.sourceforge.net/lists/listinfo/proxool-developer
>
>