Can anyone help me on analyzing this thread dump [attached] ?
I have Symmetric [3.7.23,03.7.24,03.7.26 versions tested] running on a customer and since yesterday I am getting these bizarre oracle errors [below] and Symmetric starts up but does not process anything.
SYM_DATA table is getting entries from triggers, but SYM_OUTGOING_BATCH and SYM_DATA_EVENT are not getting anything. Symmetric is not routing anything and on logs nothing comes out. This Oracle errors comes from any standalone Symmetric binary that I run [dbsql, symadmin, etc].
I took a 300 seconds / 1 second interval thread dump and would like someone to help me checking on it because it seems that thread that builds connection pool is getting blocked, but I am no thread dump expert.
Chris Henson, Eric Long, or any or the other experts would care to lend me hand here ?
Oracle error [I'm pretty sure this is something customer has to deal with, but I would like to give them evidences]. Also, this dpc-ora2 is not a valid host, so I am not sure where java class is getting this hostname from [properties file for oracle points directly to oracle ip]:
An exception occurred. Please see the following for details:
java.net.UnknownHostException: dpc-ora2: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
at java.net.InetAddress.getAllByName(InetAddress.java:1171)
at java.net.InetAddress.getAllByName(InetAddress.java:1105)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:117)
at oracle.net.nt.ConnOption.connect(ConnOption.java:133)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370) [wrapped] oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:392)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:343)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320) [wrapped] java.sql.SQLException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1617)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1395) [wrapped] org.apache.commons.dbcp.SQLNestedException: Error preloading the connection pool
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1398)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.jumpmind.db.platform.JdbcDatabasePlatformFactory.determineDatabaseNameVersionSubprotocol(JdbcDatabasePlatformFactory.java:194) [wrapped] org.jumpmind.db.sql.SqlException: Error while reading the database metadata: Error preloading the connection pool
at org.jumpmind.db.platform.JdbcDatabasePlatformFactory.determineDatabaseNameVersionSubprotocol(JdbcDatabasePlatformFactory.java:255)
at org.jumpmind.db.platform.JdbcDatabasePlatformFactory.createNewPlatformInstance(JdbcDatabasePlatformFactory.java:151)
at org.jumpmind.symmetric.ClientSymmetricEngine.createDatabasePlatform(ClientSymmetricEngine.java:309)
at org.jumpmind.symmetric.AbstractCommandLauncher.getDatabasePlatform(AbstractCommandLauncher.java:408)
at org.jumpmind.symmetric.DbSqlCommand.executeWithOptions(DbSqlCommand.java:65)
at org.jumpmind.symmetric.AbstractCommandLauncher.execute(AbstractCommandLauncher.java:190)
at org.jumpmind.symmetric.DbSqlCommand.main(DbSqlCommand.java:37)
Thanks gurus.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't think a thread dump would help much in this case. Seems to be a connectivity issue, the hints being:
- java.net.UnknownHostException
- oracle.net.ns.NetException: The Network Adapter could not establish the connection
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Can anyone help me on analyzing this thread dump [attached] ?
I have Symmetric [3.7.23,03.7.24,03.7.26 versions tested] running on a customer and since yesterday I am getting these bizarre oracle errors [below] and Symmetric starts up but does not process anything.
SYM_DATA table is getting entries from triggers, but SYM_OUTGOING_BATCH and SYM_DATA_EVENT are not getting anything. Symmetric is not routing anything and on logs nothing comes out. This Oracle errors comes from any standalone Symmetric binary that I run [dbsql, symadmin, etc].
I took a 300 seconds / 1 second interval thread dump and would like someone to help me checking on it because it seems that thread that builds connection pool is getting blocked, but I am no thread dump expert.
Chris Henson, Eric Long, or any or the other experts would care to lend me hand here ?
Oracle error [I'm pretty sure this is something customer has to deal with, but I would like to give them evidences]. Also, this dpc-ora2 is not a valid host, so I am not sure where java class is getting this hostname from [properties file for oracle points directly to oracle ip]:
An exception occurred. Please see the following for details:
java.net.UnknownHostException: dpc-ora2: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
at java.net.InetAddress.getAllByName(InetAddress.java:1171)
at java.net.InetAddress.getAllByName(InetAddress.java:1105)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:117)
at oracle.net.nt.ConnOption.connect(ConnOption.java:133)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370)
[wrapped] oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:392)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:343)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)
[wrapped] java.sql.SQLException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1617)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1395)
[wrapped] org.apache.commons.dbcp.SQLNestedException: Error preloading the connection pool
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1398)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.jumpmind.db.platform.JdbcDatabasePlatformFactory.determineDatabaseNameVersionSubprotocol(JdbcDatabasePlatformFactory.java:194)
[wrapped] org.jumpmind.db.sql.SqlException: Error while reading the database metadata: Error preloading the connection pool
at org.jumpmind.db.platform.JdbcDatabasePlatformFactory.determineDatabaseNameVersionSubprotocol(JdbcDatabasePlatformFactory.java:255)
at org.jumpmind.db.platform.JdbcDatabasePlatformFactory.createNewPlatformInstance(JdbcDatabasePlatformFactory.java:151)
at org.jumpmind.symmetric.ClientSymmetricEngine.createDatabasePlatform(ClientSymmetricEngine.java:309)
at org.jumpmind.symmetric.AbstractCommandLauncher.getDatabasePlatform(AbstractCommandLauncher.java:408)
at org.jumpmind.symmetric.DbSqlCommand.executeWithOptions(DbSqlCommand.java:65)
at org.jumpmind.symmetric.AbstractCommandLauncher.execute(AbstractCommandLauncher.java:190)
at org.jumpmind.symmetric.DbSqlCommand.main(DbSqlCommand.java:37)
Thanks gurus.
Thread dump.
And this is a log file totally on DEBUG mode, during this bizarre scenario.
I don't think a thread dump would help much in this case. Seems to be a connectivity issue, the hints being:
- java.net.UnknownHostException
- oracle.net.ns.NetException: The Network Adapter could not establish the connection