Thread: Re: [C3p0-users] [c3p0 bug]
Status: Beta
Brought to you by:
swaldman
From: Steve W. <swa...@mc...> - 2006-07-23 14:20:07
|
Prishan, First, apologies that your message doesn't seem to have made it through to the list. I tried to approve it, but I think I messed up somehow. Anyway, the message you are seeing is quite fascinating. Really fascinating. Check it: > DBConnectionManager[getConnection]=javax.naming.NamingException: > Could not dereference object [Root exception is > java.lang.RuntimeException: [c3p0 bug] Only brand new > IdentityTokenized's, with their identities just set, should be > registered!!! Attempted to register > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > acquireRetryDelay -> 1000, autoCommitOnClose -> false, > automaticTestTable -> null, breakAfterAcquireFailure -> false, > checkoutTimeout -> 0, connectionTesterClassName -> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > false, identityToken -> 17a159b, idleConnectionTestPeriod -> -1, > initialPoolSize -> 3, maxIdleTime -> 0, maxPoolSize -> 15, > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > 3, nestedDataSource -> null, preferredTestQuery -> null, > propertyCycle -> 300, testConnectionOnCheckin -> false, > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > -> false; userOverrides: {} ] (with identity token 17a159b); > Coalesced to com.mchange.v2.c3p0.PoolBackedDataSource@17a159b > [ connectionPoolDataSource -> > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@630763 > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > acquireRetryDelay -> 1000, autoCommitOnClose -> true, > automaticTestTable -> dual, breakAfterAcquireFailure -> false, > checkoutTimeout -> 0, connectionTesterClassName -> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > false, identityToken -> 630763, idleConnectionTestPeriod -> -1, > initialPoolSize -> 10, maxIdleTime -> 10, maxPoolSize -> 30, > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > 5, nestedDataSource -> > com.mchange.v2.c3p0.DriverManagerDataSource@adccd6 [ description -> > null, driverClass -> oracle.jdbc.driver.OracleDriver, > factoryClassLocation -> null, identityToken -> adccd6, jdbcUrl -> > jdbc:oracle:oci8:@test , properties -> {{user=******, > password=******}} ], preferredTestQuery -> SELECT * FROM dual, > propertyCycle -> 300, testConnectionOnCheckin -> true, > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > -> false; userOverrides: {} ], factoryClassLocation -> null, > identityToken -> 17a159b, numHelperThreads -> 10 ](with identity > token 17a159b).] Note that two java objects are shown as turning up the same System.identityHasCode()! com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b and com.mchange.v2.c3p0.PoolBackedDataSource@17a159b! They are both ungarbage collected and in the same JVM at the same time! You've blown my mind here. Upon more research, it's quite good that you've blown my mind. I've presumed in the implementation of c3p0 and some its underlying libraries that System.identityHashCode() returns a unique ID for each object. It turns out that this presumption is usually true in practice in most JVM implementations, but not guaranteed to be true, particularly in 64-bit VMs. My very, very bad. (I really did think there was a uniqueness guarantee here, it's something I thought I knew. Learn something new you've screwed up every day.) Anyway, I'm going to have to work harder to check uniqueness in my canonicalization libraries and in constructing c3p0 object identity tokens in light of the sad information that there exists no facility in java for extracting a guaranteed unique object ID. (Sun's suggested workaround [ http://bugs.sun.com/bugdatabase/view_bug.do? bug_id=4990451 ] is to map user-defined 64-bit IDs to objects in a weak hash map when UIDs are required. I'll have to look into some variant of that.) In the meantime, could you provide me with a bit of information? 1) What kind of sysatem [hardware / OS / JVM] are you seeing this error on? 2) Approximately how frequently do you see this error? (Under normal usage patterns, presuming an implementation of System.identityHashCode () that tries hard to minimize collisions and infrequent construction of c3p0 PooledDataSources this should be a very, very rare event. It's a bug that it's very rare and not never, but still. Is this a once-in-five-years or a seven-times-a-day event?) 3) How does your client code use c3p0 PooledDataSources? Do clients look up the DataSource by JNDI each time the DataSource is used, or do clients do a single JNDI-lookup, caching the DataSource for future use? The latter use-pattern is the better one: from a performance perspective, depending in the implementation o your name-service, JNDI lookups can be quite slow, sometimes slow enough to obviate the performance advantage of Connection pooling entirely if you do a lookup for each Connection acquire. However, if you do lookup for each acquisition, that would lead to a higher frequency of hash collisions, as each JNDI lookup may require the reconstruction and canonicalization of a Serialized or Referenceable PooledDataShort (again, this depends on name service implementation). Anyway, thanks for this report. It's nice, even if a bit embarrassing, to learn something new every day. And please do send me a bit more information if you get a chance. smiles, Steve --- Steve Waldman Machinery For Change, Inc. |
From: Prishan M. <pr...@gm...> - 2006-07-24 10:56:08
|
Hi Steve, Thanks for the reply . This is a frequent error for us . So we need to fix this badly. Here are more info to your questions... 1) What kind of system [hardware / OS / JVM] are you seeing this error on? - Intel 3.0Ghz Xion Based Servers/ Linux RH 4.0 / 1.4.2 Blackdow= n /Jboss 2) Approximately how frequently do you see this error? (Under normal usage patterns, presuming an implementation of System.identityHashCode () that tries hard to minimize collisions and infrequent construction of c3p0 PooledDataSources this should be a very, very rare event. It's a bug that it's very rare and not never, but still. Is this a once-in-five-years or a seven-times-a-day event?) Once in a few thousand connections (under heavy load) Approx: 60 Connection Calls per minute =96 continuously throughout the day. We get about 10 such [c3p0 bug]'s per day. 3) How does your client code use c3p0 PooledDataSources? Do clients look up the DataSource by JNDI each time the DataSource is used, or do clients do a single JNDI-lookup, caching the DataSource for future use? The latter use-pattern is the better one: from a performance perspective, depending in the implementation o your name-service, JNDI lookups can be quite slow, sometimes slow enough to obviate the performance advantage of Connection pooling entirely if you do a lookup for each Connection acquire. However, if you do lookup for each acquisition, that would lead to a higher frequency of hash collisions, as each JNDI lookup may require the reconstruction and canonicalization of a Serialized or Referenceable PooledDataShort (again, this depends on name service implementation). // This is the JNDI code for the JBOSS System.out.println("In C3PO [if] connection"); String _pool_name =3D "MyDB"; if(ctx =3D=3D null) throw new Exception("Boom - No Context"); ds =3D (DataSource)ctx.lookup(JProps.getProperty("JBOSS_JNDI."+_pool_name= )); con =3D ds.getConnection(); ------------------------------------------------ server/all/deploy/c3p1-service.xml <?xml version=3D"1.0" encoding=3D"UTF-8" ?> <!DOCTYPE server (View Source for full doctype...)> - <server> - <mbean code=3D"com.mchange.v2.c3p0.mbean.C3P0PooledDataSource" name=3D"jboss:service=3DC3P0PooledDataSource1"> <attribute name=3D"JndiName">java:PooledDS1</attribute> <attribute name=3D"JdbcUrl">jdbc:oracle:oci8:@MyDBstr</attribute> <attribute name=3D"DriverClass">oracle.jdbc.driver.OracleDriver</attribut= e> <attribute name=3D"User">scott</attribute> <attribute name=3D"Password">*****</attribute> <attribute name=3D"AutoCommitOnClose">true</attribute> <attribute name=3D"AutomaticTestTable">dual</attribute> <attribute name=3D"InitialPoolSize">1</attribute> <attribute name=3D"MaxIdleTime">10</attribute> <attribute name=3D"MaxPoolSize">15</attribute> <attribute name=3D"MaxStatements">0</attribute> <attribute name=3D"MaxStatementsPerConnection">0</attribute> <attribute name=3D"MinPoolSize">3</attribute> <attribute name=3D"PreferredTestQuery">select 1 from dual</attribute> <attribute name=3D"TestConnectionOnCheckin">true</attribute> <attribute name=3D"TestConnectionOnCheckout">false</attribute> <depends>jboss:service=3DNaming</depends> </mbean> </server> Thanks and regards, ~Prishan On 7/23/06, Steve Waldman <swa...@mc...> wrote: > > Prishan, > > First, apologies that your message doesn't seem to have made it > through to the list. I tried to approve it, but I think I messed up > somehow. > > Anyway, the message you are seeing is quite fascinating. Really > fascinating. Check it: > > > DBConnectionManager[getConnection]=3Djavax.naming.NamingException: > > Could not dereference object [Root exception is > > java.lang.RuntimeException: [c3p0 bug] Only brand new > > IdentityTokenized's, with their identities just set, should be > > registered!!! Attempted to register > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b > > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > > acquireRetryDelay -> 1000, autoCommitOnClose -> false, > > automaticTestTable -> null, breakAfterAcquireFailure -> false, > > checkoutTimeout -> 0, connectionTesterClassName -> > > com.mchange.v2.c3p0.impl.DefaultConnectionTester, > > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > > false, identityToken -> 17a159b, idleConnectionTestPeriod -> -1, > > initialPoolSize -> 3, maxIdleTime -> 0, maxPoolSize -> 15, > > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > > 3, nestedDataSource -> null, preferredTestQuery -> null, > > propertyCycle -> 300, testConnectionOnCheckin -> false, > > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > > -> false; userOverrides: {} ] (with identity token 17a159b); > > Coalesced to com.mchange.v2.c3p0.PoolBackedDataSource@17a159b > > [ connectionPoolDataSource -> > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@630763 > > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > > acquireRetryDelay -> 1000, autoCommitOnClose -> true, > > automaticTestTable -> dual, breakAfterAcquireFailure -> false, > > checkoutTimeout -> 0, connectionTesterClassName -> > > com.mchange.v2.c3p0.impl.DefaultConnectionTester, > > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > > false, identityToken -> 630763, idleConnectionTestPeriod -> -1, > > initialPoolSize -> 10, maxIdleTime -> 10, maxPoolSize -> 30, > > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > > 5, nestedDataSource -> > > com.mchange.v2.c3p0.DriverManagerDataSource@adccd6 [ description -> > > null, driverClass -> oracle.jdbc.driver.OracleDriver, > > factoryClassLocation -> null, identityToken -> adccd6, jdbcUrl -> > > jdbc:oracle:oci8:@test , properties -> {{user=3D******, > > password=3D******}} ], preferredTestQuery -> SELECT * FROM dual, > > propertyCycle -> 300, testConnectionOnCheckin -> true, > > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > > -> false; userOverrides: {} ], factoryClassLocation -> null, > > identityToken -> 17a159b, numHelperThreads -> 10 ](with identity > > token 17a159b).] > > Note that two java objects are shown as turning up the same > System.identityHasCode()! > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b and > com.mchange.v2.c3p0.PoolBackedDataSource@17a159b! They are both > ungarbage collected and in the same JVM at the same time! You've > blown my mind here. > > Upon more research, it's quite good that you've blown my mind. I've > presumed in the implementation of c3p0 and some its underlying > libraries that System.identityHashCode() returns a unique ID for each > object. It turns out that this presumption is usually true in > practice in most JVM implementations, but not guaranteed to be true, > particularly in 64-bit VMs. My very, very bad. (I really did think > there was a uniqueness guarantee here, it's something I thought I > knew. Learn something new you've screwed up every day.) > > Anyway, I'm going to have to work harder to check uniqueness in my > canonicalization libraries and in constructing c3p0 object identity > tokens in light of the sad information that there exists no facility > in java for extracting a guaranteed unique object ID. (Sun's > suggested workaround [ http://bugs.sun.com/bugdatabase/view_bug.do? > bug_id=3D4990451 ] is to map user-defined 64-bit IDs to objects in a > weak hash map when UIDs are required. I'll have to look into some > variant of that.) > > In the meantime, could you provide me with a bit of information? > > 1) What kind of sysatem [hardware / OS / JVM] are you seeing this > error on? > > 2) Approximately how frequently do you see this error? (Under normal > usage patterns, presuming an implementation of System.identityHashCode > () that tries hard to minimize collisions and infrequent construction > of c3p0 PooledDataSources this should be a very, very rare event. > It's a bug that it's very rare and not never, but still. Is this a > once-in-five-years or a seven-times-a-day event?) > > 3) How does your client code use c3p0 PooledDataSources? Do clients > look up the DataSource by JNDI each time the DataSource is used, or > do clients do a single JNDI-lookup, caching the DataSource for future > use? The latter use-pattern is the better one: from a performance > perspective, depending in the implementation o your name-service, > JNDI lookups can be quite slow, sometimes slow enough to obviate the > performance advantage of Connection pooling entirely if you do a > lookup for each Connection acquire. However, if you do lookup for > each acquisition, that would lead to a higher frequency of hash > collisions, as each JNDI lookup may require the reconstruction and > canonicalization of a Serialized or Referenceable PooledDataShort > (again, this depends on name service implementation). > > Anyway, thanks for this report. It's nice, even if a bit > embarrassing, to learn something new every day. And please do send me > a bit more information if you get a chance. > > smiles, > Steve > > --- > Steve Waldman > Machinery For Change, Inc. > > > > |
From: Steve W. <swa...@mc...> - 2006-07-24 12:00:23
|
Prishan, First, thanks for the quick response to my questions. Once in every several thousand lookups seems a lot more frequent than =20= I would expect given my understanding of the issue. But observation =20 trumps theory... each lookup triggers the creation and =20 canonicalization of more than one PooledDataSource, and it's possible =20= that blackdown's 64 bit JVM, since it can't produce unique =20 identityHashCodes, uses a smaller space than a full 32-bits in =20 generating identityHashCodes. (Or it could be that I'm understanding =20 something wrong...) Anyway, if I am anywhere close to sane (maybe, maybe not...), looking =20= up your PooledDataSource only once and than caching the reference =20 should resolve the problem. I'm not sure in exactly what context or =20 sort of entity your code is being executed in, but I'm betting from =20 what you've shown that if you were to change... ds =3D (DataSource)ctx.lookup(JProps.getProperty=20 ("JBOSS_JNDI."+_pool_name)); to if (ds =3D=3D null) ds =3D (DataSource)ctx.lookup(JProps.getProperty=20 ("JBOSS_JNDI."+_pool_name)); ...your problem would go away. Plus, your application's connection =20 acquisition performance would be significantly improved, because the =20 JNDI-lookup-for-every-Connection-acquisition is slow. That said, your application's code, although not optimal, is correct =20 as written, and c3p0 should deal. I will try to fix this, undo my =20 presumption that System.identityHashCode() can be used as a unique =20 Object ID. In the meantime, please try out the workaround above (or =20 alternatively, if you want to avoid the null-check, you can just put =20 the assignment of ds in some code block that gets executed once-per-=20 app rather than once-per-Connection), and let me know if your problem =20= goes away. (When I fix things on my end, I may ask you to temporarily =20= undo the workaround top verify the fix.) Thanks again for the very interesting and informative bug report! smiles, Steve --- Steve Waldman Machinery For Change, Inc. On Jul 24, 2006, at 1:55 PM, Prishan Makandurage wrote: > Hi Steve, > Thanks for the reply . This is a frequent error for us . So we need =20= > to fix this badly. > > Here are more info to your questions... > > 1) What kind of system [hardware / OS / JVM] are you seeing this > error on? - Intel 3.0Ghz Xion Based Servers/ Linux RH 4.0 / 1.4.2 =20 > Blackdown /Jboss > > 2) Approximately how frequently do you see this error? (Under normal > usage patterns, presuming an implementation of System.identityHashCode > () that tries hard to minimize collisions and infrequent construction > of c3p0 PooledDataSources this should be a very, very rare event. > It's a bug that it's very rare and not never, but still. Is this a > once-in-five-years or a seven-times-a-day event?) > > Once in a few thousand connections (under heavy load) > Approx: 60 Connection Calls per minute =96 continuously throughout =20 > the day. > > We get about 10 such [c3p0 bug]'s per day. > > > 3) How does your client code use c3p0 PooledDataSources? Do clients > look up the DataSource by JNDI each time the DataSource is used, or > do clients do a single JNDI-lookup, caching the DataSource for future > use? The latter use-pattern is the better one: from a performance > perspective, depending in the implementation o your name-service, > JNDI lookups can be quite slow, sometimes slow enough to obviate the > performance advantage of Connection pooling entirely if you do a > lookup for each Connection acquire. However, if you do lookup for > each acquisition, that would lead to a higher frequency of hash > collisions, as each JNDI lookup may require the reconstruction and > canonicalization of a Serialized or Referenceable PooledDataShort > (again, this depends on name service implementation). > > > // This is the JNDI code for the JBOSS > > System.out.println("In C3PO [if] connection"); > > String _pool_name =3D "MyDB"; > > > if(ctx =3D=3D null) throw new Exception("Boom - No Context"); > > ds =3D (DataSource)ctx.lookup(JProps.getProperty=20 > ("JBOSS_JNDI."+_pool_name)); > > con =3D ds.getConnection(); > > ------------------------------------------------ > > server/all/deploy/c3p1-service.xml > > <?xml version=3D"1.0" encoding=3D"UTF-8" ?> > > <!DOCTYPE server (View Source for full doctype...)> > - <server> > - <mbean code=3D"com.mchange.v2.c3p0.mbean.C3P0PooledDataSource" =20 > name=3D"jboss:service=3DC3P0PooledDataSource1"> > <attribute name=3D"JndiName">java:PooledDS1</attribute> > <attribute name=3D"JdbcUrl">jdbc:oracle:oci8:@MyDBstr</attribute> > <attribute name=3D"DriverClass"> oracle.jdbc.driver.OracleDriver</=20= > attribute> > <attribute name=3D"User">scott</attribute> > <attribute name=3D"Password">*****</attribute> > <attribute name=3D"AutoCommitOnClose">true</attribute> > <attribute name=3D"AutomaticTestTable">dual</attribute> > <attribute name=3D"InitialPoolSize">1</attribute> > <attribute name=3D"MaxIdleTime">10</attribute> > <attribute name=3D"MaxPoolSize">15</attribute> > <attribute name=3D"MaxStatements">0</attribute> > <attribute name=3D"MaxStatementsPerConnection">0</attribute> > <attribute name=3D"MinPoolSize">3</attribute> > <attribute name=3D"PreferredTestQuery">select 1 from = dual</attribute> > <attribute name=3D"TestConnectionOnCheckin">true</attribute> > <attribute name=3D"TestConnectionOnCheckout">false</attribute> > <depends>jboss:service=3DNaming</depends> > </mbean> > </server> > > Thanks and regards, > > ~Prishan > > > > On 7/23/06, Steve Waldman <swa...@mc...> wrote: Prishan, > > First, apologies that your message doesn't seem to have made it > through to the list. I tried to approve it, but I think I messed up > somehow. > > Anyway, the message you are seeing is quite fascinating. Really > fascinating. Check it: > > > DBConnectionManager[getConnection]=3Djavax.naming.NamingException: > > Could not dereference object [Root exception is > > java.lang.RuntimeException: [c3p0 bug] Only brand new > > IdentityTokenized's, with their identities just set, should be > > registered!!! Attempted to register > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b > > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > > acquireRetryDelay -> 1000, autoCommitOnClose -> false, > > automaticTestTable -> null, breakAfterAcquireFailure -> false, > > checkoutTimeout -> 0, connectionTesterClassName -> > > com.mchange.v2.c3p0.impl.DefaultConnectionTester, > > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > > false, identityToken -> 17a159b, idleConnectionTestPeriod -> -1, > > initialPoolSize -> 3, maxIdleTime -> 0, maxPoolSize -> 15, > > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > > 3, nestedDataSource -> null, preferredTestQuery -> null, > > propertyCycle -> 300, testConnectionOnCheckin -> false, > > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > > -> false; userOverrides: {} ] (with identity token 17a159b); > > Coalesced to com.mchange.v2.c3p0.PoolBackedDataSource@17a159b > > [ connectionPoolDataSource -> > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@630763 > > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > > acquireRetryDelay -> 1000, autoCommitOnClose -> true, > > automaticTestTable -> dual, breakAfterAcquireFailure -> false, > > checkoutTimeout -> 0, connectionTesterClassName -> > > com.mchange.v2.c3p0.impl.DefaultConnectionTester , > > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > > false, identityToken -> 630763, idleConnectionTestPeriod -> -1, > > initialPoolSize -> 10, maxIdleTime -> 10, maxPoolSize -> 30, > > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > > 5, nestedDataSource -> > > com.mchange.v2.c3p0.DriverManagerDataSource@adccd6 [ description -> > > null, driverClass -> oracle.jdbc.driver.OracleDriver, > > factoryClassLocation -> null, identityToken -> adccd6, jdbcUrl -> > > jdbc:oracle:oci8:@test , properties -> {{user=3D******, > > password=3D******}} ], preferredTestQuery -> SELECT * FROM dual, > > propertyCycle -> 300, testConnectionOnCheckin -> true, > > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > > -> false; userOverrides: {} ], factoryClassLocation -> null, > > identityToken -> 17a159b, numHelperThreads -> 10 ](with identity > > token 17a159b).] > > Note that two java objects are shown as turning up the same > System.identityHasCode()! > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b and > com.mchange.v2.c3p0.PoolBackedDataSource@17a159b! They are both > ungarbage collected and in the same JVM at the same time! You've > blown my mind here. > > Upon more research, it's quite good that you've blown my mind. I've > presumed in the implementation of c3p0 and some its underlying > libraries that System.identityHashCode() returns a unique ID for each > object. It turns out that this presumption is usually true in > practice in most JVM implementations, but not guaranteed to be true, > particularly in 64-bit VMs. My very, very bad. (I really did think > there was a uniqueness guarantee here, it's something I thought I > knew. Learn something new you've screwed up every day.) > > Anyway, I'm going to have to work harder to check uniqueness in my > canonicalization libraries and in constructing c3p0 object identity > tokens in light of the sad information that there exists no facility > in java for extracting a guaranteed unique object ID. (Sun's > suggested workaround [ http://bugs.sun.com/bugdatabase/view_bug.do? > bug_id=3D4990451 ] is to map user-defined 64-bit IDs to objects in a > weak hash map when UIDs are required. I'll have to look into some > variant of that.) > > In the meantime, could you provide me with a bit of information? > > 1) What kind of sysatem [hardware / OS / JVM] are you seeing this > error on? > > 2) Approximately how frequently do you see this error? (Under normal > usage patterns, presuming an implementation of System.identityHashCode > () that tries hard to minimize collisions and infrequent construction > of c3p0 PooledDataSources this should be a very, very rare event. > It's a bug that it's very rare and not never, but still. Is this a > once-in-five-years or a seven-times-a-day event?) > > 3) How does your client code use c3p0 PooledDataSources? Do clients > look up the DataSource by JNDI each time the DataSource is used, or > do clients do a single JNDI-lookup, caching the DataSource for future > use? The latter use-pattern is the better one: from a performance > perspective, depending in the implementation o your name-service, > JNDI lookups can be quite slow, sometimes slow enough to obviate the > performance advantage of Connection pooling entirely if you do a > lookup for each Connection acquire. However, if you do lookup for > each acquisition, that would lead to a higher frequency of hash > collisions, as each JNDI lookup may require the reconstruction and > canonicalization of a Serialized or Referenceable PooledDataShort > (again, this depends on name service implementation). > > Anyway, thanks for this report. It's nice, even if a bit > embarrassing, to learn something new every day. And please do send me > a bit more information if you get a chance. > > smiles, > Steve > > --- > Steve Waldman > Machinery For Change, Inc. > > > > > |
From: Prishan M. <pr...@gm...> - 2006-07-25 04:00:55
|
Hi Steve, Thanks for the reply with workaround.We have implemented workaround for ou= r development server. So We need to do a QA test before we upload to our production server. But we can't simulate this error in development servers since the load is very little. So the success of your workaround can be seen only after uploading to the production server.That will take around one week. So I will let you know the status after I upload this to live server. I will revert this when you send the bug fix as you said. Thanks again for the help.... ~Prishan On 7/24/06, Steve Waldman <swa...@mc...> wrote: > > Prishan, > > First, thanks for the quick response to my questions. > > Once in every several thousand lookups seems a lot more frequent than > I would expect given my understanding of the issue. But observation > trumps theory... each lookup triggers the creation and > canonicalization of more than one PooledDataSource, and it's possible > that blackdown's 64 bit JVM, since it can't produce unique > identityHashCodes, uses a smaller space than a full 32-bits in > generating identityHashCodes. (Or it could be that I'm understanding > something wrong...) > > Anyway, if I am anywhere close to sane (maybe, maybe not...), looking > up your PooledDataSource only once and than caching the reference > should resolve the problem. I'm not sure in exactly what context or > sort of entity your code is being executed in, but I'm betting from > what you've shown that if you were to change... > > ds =3D (DataSource)ctx.lookup(JProps.getProperty > ("JBOSS_JNDI."+_pool_name)); > > to > > if (ds =3D=3D null) > ds =3D (DataSource)ctx.lookup(JProps.getProperty > ("JBOSS_JNDI."+_pool_name)); > > ...your problem would go away. Plus, your application's connection > acquisition performance would be significantly improved, because the > JNDI-lookup-for-every-Connection-acquisition is slow. > > That said, your application's code, although not optimal, is correct > as written, and c3p0 should deal. I will try to fix this, undo my > presumption that System.identityHashCode() can be used as a unique > Object ID. In the meantime, please try out the workaround above (or > alternatively, if you want to avoid the null-check, you can just put > the assignment of ds in some code block that gets executed once-per- > app rather than once-per-Connection), and let me know if your problem > goes away. (When I fix things on my end, I may ask you to temporarily > undo the workaround top verify the fix.) > > Thanks again for the very interesting and informative bug report! > > smiles, > Steve > > --- > Steve Waldman > Machinery For Change, Inc. > > > > On Jul 24, 2006, at 1:55 PM, Prishan Makandurage wrote: > > > > Hi Steve, > > Thanks for the reply . This is a frequent error for us . So we need > > to fix this badly. > > > > Here are more info to your questions... > > > > 1) What kind of system [hardware / OS / JVM] are you seeing this > > error on? - Intel 3.0Ghz Xion Based Servers/ Linux RH 4.0 / 1.4.2 > > Blackdown /Jboss > > > > 2) Approximately how frequently do you see this error? (Under normal > > usage patterns, presuming an implementation of System.identityHashCode > > () that tries hard to minimize collisions and infrequent construction > > of c3p0 PooledDataSources this should be a very, very rare event. > > It's a bug that it's very rare and not never, but still. Is this a > > once-in-five-years or a seven-times-a-day event?) > > > > Once in a few thousand connections (under heavy load) > > Approx: 60 Connection Calls per minute =96 continuously throughout > > the day. > > > > We get about 10 such [c3p0 bug]'s per day. > > > > > > 3) How does your client code use c3p0 PooledDataSources? Do clients > > look up the DataSource by JNDI each time the DataSource is used, or > > do clients do a single JNDI-lookup, caching the DataSource for future > > use? The latter use-pattern is the better one: from a performance > > perspective, depending in the implementation o your name-service, > > JNDI lookups can be quite slow, sometimes slow enough to obviate the > > performance advantage of Connection pooling entirely if you do a > > lookup for each Connection acquire. However, if you do lookup for > > each acquisition, that would lead to a higher frequency of hash > > collisions, as each JNDI lookup may require the reconstruction and > > canonicalization of a Serialized or Referenceable PooledDataShort > > (again, this depends on name service implementation). > > > > > > // This is the JNDI code for the JBOSS > > > > System.out.println("In C3PO [if] connection"); > > > > String _pool_name =3D "MyDB"; > > > > > > if(ctx =3D=3D null) throw new Exception("Boom - No Context"); > > > > ds =3D (DataSource)ctx.lookup(JProps.getProperty > > ("JBOSS_JNDI."+_pool_name)); > > > > con =3D ds.getConnection(); > > > > ------------------------------------------------ > > > > server/all/deploy/c3p1-service.xml > > > > <?xml version=3D"1.0" encoding=3D"UTF-8" ?> > > > > <!DOCTYPE server (View Source for full doctype...)> > > - <server> > > - <mbean code=3D"com.mchange.v2.c3p0.mbean.C3P0PooledDataSource" > > name=3D"jboss:service=3DC3P0PooledDataSource1"> > > <attribute name=3D"JndiName">java:PooledDS1</attribute> > > <attribute name=3D"JdbcUrl">jdbc:oracle:oci8:@MyDBstr</attribute> > > <attribute name=3D"DriverClass"> oracle.jdbc.driver.OracleDriver</ > > attribute> > > <attribute name=3D"User">scott</attribute> > > <attribute name=3D"Password">*****</attribute> > > <attribute name=3D"AutoCommitOnClose">true</attribute> > > <attribute name=3D"AutomaticTestTable">dual</attribute> > > <attribute name=3D"InitialPoolSize">1</attribute> > > <attribute name=3D"MaxIdleTime">10</attribute> > > <attribute name=3D"MaxPoolSize">15</attribute> > > <attribute name=3D"MaxStatements">0</attribute> > > <attribute name=3D"MaxStatementsPerConnection">0</attribute> > > <attribute name=3D"MinPoolSize">3</attribute> > > <attribute name=3D"PreferredTestQuery">select 1 from dual</attribute> > > <attribute name=3D"TestConnectionOnCheckin">true</attribute> > > <attribute name=3D"TestConnectionOnCheckout">false</attribute> > > <depends>jboss:service=3DNaming</depends> > > </mbean> > > </server> > > > > Thanks and regards, > > > > ~Prishan > > > > > > > > On 7/23/06, Steve Waldman <swa...@mc...> wrote: Prishan, > > > > First, apologies that your message doesn't seem to have made it > > through to the list. I tried to approve it, but I think I messed up > > somehow. > > > > Anyway, the message you are seeing is quite fascinating. Really > > fascinating. Check it: > > > > > DBConnectionManager[getConnection]=3Djavax.naming.NamingException: > > > Could not dereference object [Root exception is > > > java.lang.RuntimeException: [c3p0 bug] Only brand new > > > IdentityTokenized's, with their identities just set, should be > > > registered!!! Attempted to register > > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b > > > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > > > acquireRetryDelay -> 1000, autoCommitOnClose -> false, > > > automaticTestTable -> null, breakAfterAcquireFailure -> false, > > > checkoutTimeout -> 0, connectionTesterClassName -> > > > com.mchange.v2.c3p0.impl.DefaultConnectionTester, > > > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > > > false, identityToken -> 17a159b, idleConnectionTestPeriod -> -1, > > > initialPoolSize -> 3, maxIdleTime -> 0, maxPoolSize -> 15, > > > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > > > 3, nestedDataSource -> null, preferredTestQuery -> null, > > > propertyCycle -> 300, testConnectionOnCheckin -> false, > > > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > > > -> false; userOverrides: {} ] (with identity token 17a159b); > > > Coalesced to com.mchange.v2.c3p0.PoolBackedDataSource@17a159b > > > [ connectionPoolDataSource -> > > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@630763 > > > [ acquireIncrement -> 3, acquireRetryAttempts -> 30, > > > acquireRetryDelay -> 1000, autoCommitOnClose -> true, > > > automaticTestTable -> dual, breakAfterAcquireFailure -> false, > > > checkoutTimeout -> 0, connectionTesterClassName -> > > > com.mchange.v2.c3p0.impl.DefaultConnectionTester , > > > factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> > > > false, identityToken -> 630763, idleConnectionTestPeriod -> -1, > > > initialPoolSize -> 10, maxIdleTime -> 10, maxPoolSize -> 30, > > > maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> > > > 5, nestedDataSource -> > > > com.mchange.v2.c3p0.DriverManagerDataSource@adccd6 [ description -> > > > null, driverClass -> oracle.jdbc.driver.OracleDriver, > > > factoryClassLocation -> null, identityToken -> adccd6, jdbcUrl -> > > > jdbc:oracle:oci8:@test , properties -> {{user=3D******, > > > password=3D******}} ], preferredTestQuery -> SELECT * FROM dual, > > > propertyCycle -> 300, testConnectionOnCheckin -> true, > > > testConnectionOnCheckout -> false, usesTraditionalReflectiveProxies > > > -> false; userOverrides: {} ], factoryClassLocation -> null, > > > identityToken -> 17a159b, numHelperThreads -> 10 ](with identity > > > token 17a159b).] > > > > Note that two java objects are shown as turning up the same > > System.identityHasCode()! > > com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@17a159b and > > com.mchange.v2.c3p0.PoolBackedDataSource@17a159b! They are both > > ungarbage collected and in the same JVM at the same time! You've > > blown my mind here. > > > > Upon more research, it's quite good that you've blown my mind. I've > > presumed in the implementation of c3p0 and some its underlying > > libraries that System.identityHashCode() returns a unique ID for each > > object. It turns out that this presumption is usually true in > > practice in most JVM implementations, but not guaranteed to be true, > > particularly in 64-bit VMs. My very, very bad. (I really did think > > there was a uniqueness guarantee here, it's something I thought I > > knew. Learn something new you've screwed up every day.) > > > > Anyway, I'm going to have to work harder to check uniqueness in my > > canonicalization libraries and in constructing c3p0 object identity > > tokens in light of the sad information that there exists no facility > > in java for extracting a guaranteed unique object ID. (Sun's > > suggested workaround [ http://bugs.sun.com/bugdatabase/view_bug.do? > > bug_id=3D4990451 ] is to map user-defined 64-bit IDs to objects in a > > weak hash map when UIDs are required. I'll have to look into some > > variant of that.) > > > > In the meantime, could you provide me with a bit of information? > > > > 1) What kind of sysatem [hardware / OS / JVM] are you seeing this > > error on? > > > > 2) Approximately how frequently do you see this error? (Under normal > > usage patterns, presuming an implementation of System.identityHashCode > > () that tries hard to minimize collisions and infrequent construction > > of c3p0 PooledDataSources this should be a very, very rare event. > > It's a bug that it's very rare and not never, but still. Is this a > > once-in-five-years or a seven-times-a-day event?) > > > > 3) How does your client code use c3p0 PooledDataSources? Do clients > > look up the DataSource by JNDI each time the DataSource is used, or > > do clients do a single JNDI-lookup, caching the DataSource for future > > use? The latter use-pattern is the better one: from a performance > > perspective, depending in the implementation o your name-service, > > JNDI lookups can be quite slow, sometimes slow enough to obviate the > > performance advantage of Connection pooling entirely if you do a > > lookup for each Connection acquire. However, if you do lookup for > > each acquisition, that would lead to a higher frequency of hash > > collisions, as each JNDI lookup may require the reconstruction and > > canonicalization of a Serialized or Referenceable PooledDataShort > > (again, this depends on name service implementation). > > > > Anyway, thanks for this report. It's nice, even if a bit > > embarrassing, to learn something new every day. And please do send me > > a bit more information if you get a chance. > > > > smiles, > > Steve > > > > --- > > Steve Waldman > > Machinery For Change, Inc. > > > > > > > > > > > > > |