Menu
▾
▴
sequoiadb-discuss — Mailing list for discussing topics related to Sequoia clustering
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
(10) |
May
(44) |
Jun
(10) |
Jul
(4) |
Aug
(3) |
Sep
(1) |
Oct
(4) |
Nov
(3) |
Dec
(2) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
|
Feb
|
Mar
(4) |
Apr
|
May
(5) |
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(7) |
Dec
|
|
From: Toni M. L. <ton...@gm...> - 2009-05-29 16:35:29
|
Emmanuel, Where I have to setup the DEBUG ? Now I have the table empty : mysql> select * from cdpdb_logtable; Empty set (0.00 sec) 2009/5/29, Emmanuel Cecchet <ma...@fr...>: > > Toni, > > I am not able to get the recovery logs, I have done this in the > > log4j.properties : > The recovery log is stored in a database that you configure in your > virtual database configuration file. You can directly look at the > content of the table by connecting to the database or use the Sequoia > admin console using the 'dump recoverylog' command (note that you have > to set the debug mode to on first). > > Hope this helps, > Emmanuel > > -- > Emmanuel Cecchet > FTO @ Frog Thinker > Open Source Development & Consulting > -- > Web: http://www.frogthinker.org > email: ma...@fr... > Skype: emmanuel_cecchet > > _______________________________________________ > Sequoia mailing list > Se...@li... > http://forge.continuent.org/mailman/listinfo/sequoia > |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-29 16:25:56
|
Toni, > I am not able to get the recovery logs, I have done this in the > log4j.properties : The recovery log is stored in a database that you configure in your virtual database configuration file. You can directly look at the content of the table by connecting to the database or use the Sequoia admin console using the 'dump recoverylog' command (note that you have to set the debug mode to on first). Hope this helps, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Toni M. L. <ton...@gm...> - 2009-05-29 16:21:00
|
Emmanuel,
I am not able to get the recovery logs, I have done this in the
log4j.properties :
log4j.logger.org.continuent.sequoia.controller.recoverylog=INFO,
Console,Filetrace
log4j.additivity.org.continuent.sequoia.controller.recoverylog=true
Where :
# Filetrace is used for Sequoia log files, this appender is removed from the
log4j system
# if the parameter setFileLogging is set to false in the controller xml
configuration file.
log4j.appender.Filetrace=org.apache.log4j.RollingFileAppender
log4j.appender.Filetrace.File=${sequoia.log}/full_cluster.log
log4j.appender.Filetrace.MaxFileSize=10MB
log4j.appender.Filetrace.MaxBackupIndex=5
log4j.appender.Filetrace.layout=org.apache.log4j.PatternLayout
log4j.appender.Filetrace.layout.ConversionPattern=%d %-5p %c{3} %m\n
log4j.appender.Filetrace.immediateFlush=true
But I am not able to get any info related to recovery log.
Is it needed any special config ?
2009/5/29, Emmanuel Cecchet <ma...@fr...>:
>
> Hi Toni,
>
> The Sequoia mailing list has moved to
> seq...@li....
> > We are doing LoadTests of our product which is based in Sequoia +
> > mysql with 2 BE´s.
> >
> > And the thing is that some SQL comands get stuck from time to time
> > spending more that 1seg to answer, I have donr a trace in MYSQL and
> > response in MYSQL is quite quickly ( 1mseg ). So, I suspect that we
> > have some problems in sequoia.
> >
> > Is there any possiblity to traduce the messages from a TCPDUMP, in
> > order to see when sequoia response the the SQL sent from hibernate ?
> >
> > We are working with jboss-4.0.3SP1, sequoia-2.10.10 and mysql 4.1.20
> > in a RHEL environment !
> The delays you are observing might be due to retransmissions in the
> group communication. Are you using JGroups?
>
> To sample the request arrival time, you can activate the following in
> log4j.properties:
> # To trace requests #
>
> log4j.logger.org.continuent.sequoia.controller.virtualdatabase.request=INFO,
> Requests
> # To trace distributed requests #
>
> log4j.logger.org.continuent.sequoia.controller.distributedvirtualdatabase.request=INFO,
> DistributedRequests
>
> This will give you the request arrival time at each controller (look for
> the results in log/request.log and log/distributed_request.log).
>
> For the request response time, the write request response times are
> logged in the recovery log. You can also use the SQLMonitoring element
> in your virtual database configuration file (look for more info in
> sequoia.dtd) to profile request execution time.
>
> Hope this helps,
> Emmanuel
>
> --
> Emmanuel Cecchet
> FTO @ Frog Thinker
> Open Source Development & Consulting
> --
> Web: http://www.frogthinker.org
> email: ma...@fr...
> Skype: emmanuel_cecchet
>
> _______________________________________________
> Sequoia mailing list
> Se...@li...
> http://forge.continuent.org/mailman/listinfo/sequoia
>
|
|
From: Emmanuel C. <ma...@fr...> - 2009-05-29 14:14:17
|
Hi Toni, The Sequoia mailing list has moved to seq...@li.... > We are doing LoadTests of our product which is based in Sequoia + > mysql with 2 BE´s. > > And the thing is that some SQL comands get stuck from time to time > spending more that 1seg to answer, I have donr a trace in MYSQL and > response in MYSQL is quite quickly ( 1mseg ). So, I suspect that we > have some problems in sequoia. > > Is there any possiblity to traduce the messages from a TCPDUMP, in > order to see when sequoia response the the SQL sent from hibernate ? > > We are working with jboss-4.0.3SP1, sequoia-2.10.10 and mysql 4.1.20 > in a RHEL environment ! The delays you are observing might be due to retransmissions in the group communication. Are you using JGroups? To sample the request arrival time, you can activate the following in log4j.properties: # To trace requests # log4j.logger.org.continuent.sequoia.controller.virtualdatabase.request=INFO, Requests # To trace distributed requests # log4j.logger.org.continuent.sequoia.controller.distributedvirtualdatabase.request=INFO, DistributedRequests This will give you the request arrival time at each controller (look for the results in log/request.log and log/distributed_request.log). For the request response time, the write request response times are logged in the recovery log. You can also use the SQLMonitoring element in your virtual database configuration file (look for more info in sequoia.dtd) to profile request execution time. Hope this helps, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-21 14:06:29
|
Hi Richard, >> Sorry for forgetting about this. The fastest quick fix would be to >> change org.continuent.sequoia.controller.backend.DriverCompliance >> line 586 > It's alright. You must be quite busy with the support of Sequoia. > > I stumbled on a page on a discussion board where someone was > describing a similar problem > (http://forge.continuent.org/pipermail/sequoia/2008-November/006339.html) > The problem corresponded to 2 open bugs: SEQUOIA-1131 and > SEQUOIA-1132. Someone suggested, as a work around, to use a previous > release of the MySQL JDBC driver (5.1.6). > So I tried it and it worked. Alfresco is now working and I was able > to run some benchmarks. The results are awesome. There is a > significant performance improvement when using 2 MySQL databases in > RAIDb Level 1. Excellent, don't hesitate to share your results on the list. > One question though: when do you plan the next release? > Also what is the future for Sequoia? It seems that Continuent dropped > its support of Sequoia. Are you on your own with this product? I guess the next release will be when I find enough time to finish the doc and get a decent test suite together. Otherwise, Sequoia 4 is almost ready to go for beta. For the future, I think that you summarized it well, I'm basically on my own with the community but Continuent's support is fading out. It looks like the early days of C-JDBC but the project is in a much better shape with a stable code base. Thanks for your interest in Sequoia, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-20 17:49:34
|
Hi Pierre, It might well be that the problem comes from an infinite retransmission from JGroups. You should just get a newer version of JGroups and replace the jar file in the lib directory. I don't remember which version shipped with Sequoia 2.10.9 but it's probably quite old. JGroups 2.6.10.GA should be compatible with Sequoia 2.10.*. Thanks for the feedback, Emmanuel > This is not reproduced every time, it's the first time I explicitly observed this behavior. > > I can see in this log that StackOverflowError is thrown for each query from user1 (vdb user=oracle user), queries for user2 work perfectly. > Time between log "Load balancing read request" and "Runtime error during command execution (null)" is always between 1s and 2s. > > 5 minutes after first error, we can see strange behavior : > - c1 see c2 quitting group (29 Apr 2009 13:00:22,684 | sequoia.controller.recoverylog | Storing checkpoint Member(address=/172.20.1.81:65201, uid=db) quit group db-172.20.1.77:65209-20090429130022678+0000 at request id 1679), but c2 is still alive and does not see other quit ... This cause c1 is in a 1 member group and c2 in a 2 members group !! > > After a restart of the 2 controllers, all seems to be fine. > > Is it possible that StackOverflow is thrown by jGroups (we see it even when a query is select on local backend) ? > After bug on c1 see c2 quit, how can it be possible to have an update on c2 ? > > I join a log which shows that c2 send update request to all members, c1 does not receive anything, but at the end the request seems to be executed (I can not check database) > > > I hope at least 1 symptom could help to find problem source ... > > > > > Pierre Besson-Deblon > > -----Message d'origine----- > De : seq...@li... > [mailto:seq...@li...]De la part de > Emmanuel Cecchet > Envoyé : 18 May 2009 22:39 > À : Sequoia general mailing list; > seq...@li... > Objet : Re: [Sequoia] Bug ? > > > Hi Pierre, > > Note that the mailing list has now moved to > seq...@li... > The java.lang.StackOverflowError error probably comes from an infinite > retry loop somewhere (hard to tell where since the exception is null). > Could you reproduce the issue with Sequoia 2.10.10? I don't have the > source code of 2.10.9 readily available and a number of fixes (as well > as a JGroups update) have been performed in 2.10.10. > Does the problem happen systematically? > > Thanks for your feedback, > Emmanuel > > >> continue ... I made an edit at the end of previous message >> >> >> -----Message d'origine----- >> De : seq...@li... >> [mailto:seq...@li...]De la part de >> BESSON-DEBLON, Pierre (SOGETI HIGH TECH) >> Envoyé : 18 May 2009 19:32 >> À : Sequoia general mailing list >> Objet : [Sequoia] Bug ? >> >> >> Hello, >> >> We use sequoia with 2 controllers on RAIDb1, sequoia is 2.10.9 with jgroups. >> >> We experienced this weird behavior : >> >> # on controller 1 (c1) backend deactivated, on c2 backend activated >> c2.log -> 29 Apr 2009 12:56:12,400 | sequoia.controller.RequestManager.db | Database backend db2 is now enabled >> >> # c1 receive a read request and forward to c2 >> c1.log -> >> 29 Apr 2009 12:56:16,271 | sequoia.controller.virtualdatabase.request.db | S 844424930131983 0 SELECT ... >> 29 Apr 2009 12:56:16,273 | sequoia.controller.RequestManager.db |Sending request SELECT ... to Member c2 >> >> c2.log -> >> 29 Apr 2009 12:56:16,353 | sequoia.controller.virtualdatabase.db | S 844424930131983 0 SELECT ... >> 29 Apr 2009 12:56:16,355 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Scheduling distributedRequest 844424930131983 from Member c1 >> 29 Apr 2009 12:56:16,355 | sequoia.controller.distributedvirtualdatabase.request.db | S 844424930131983 0 SELECT ... >> 29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | handleMessageMultiThreaded (class org.continuent.sequoia.controller.virtualdatabase.protocol.ExecRemoteStatementExecuteQuery): S 844424930131983 0 SELECT ... >> 29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Executing distributedRequest 844424930131983 from Member c1 >> 29 Apr 2009 12:56:16,356 | sequoia.controller.RequestManager.db | Database schema is dirty, refreshing it >> 29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Scheduling read Request (844424930131983): SELECT ... >> 29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Load balancing read request (844424930131983): SELECT ... >> >> and right after in c2.log file >> 29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | StatementExecuteQuery command >> 29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.request.db | S 562949953430692 0 SELECT ... >> >> c1.log >> 29 Apr 2009 12:56:21,233 | sequoia.controller.RequestManager.db | Request SELECT ... completed. >> 29 Apr 2009 12:56:21,234 | sequoia.controller.RequestManager.db | An error occured while executing remote select request 844424930131983 >> org.continuent.sequoia.common.exceptions.NoMoreBackendException >> at org.continuent.sequoia.controller.requestmanager.distributed.RAIDb1DistributedRequestManager.execRemoteStatementExecuteQuery(Unknown Source) >> at org.continuent.sequoia.controller.requestmanager.distributed.DistributedRequestManager.statementExecuteQuery(Unknown Source) >> at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(Unknown Source) >> at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(Unknown Source) >> at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(Unknown Source) >> 29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.db | Request '844424930131983' failed (null) >> 29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Error during command execution (null) >> >> >> So, How can request fail with NoMoreBackendException despite db2 on c2 is enabled ? >> Is it normal to have no more log in c2.log to indicate Exception ?, is it possible that request is hanging on c2 and a timeout raise the exception ? >> >> edit >> found this in c2.log >> 29 Apr 2009 12:56:24,553 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Runtime error during command execution (null) >> java.lang.StackOverflowError >> 29 Apr 2009 12:56:24,554 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Reset command >> I think it could help >> >> >> Thanks for any help. >> >> >> Pierre Besson-Deblon >> >> >> The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised. >> If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. >> Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. >> All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. >> >> _______________________________________________ >> Sequoia mailing list >> Se...@li... >> http://forge.continuent.org/mailman/listinfo/sequoia >> >> >> > > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > ------------------------------------------------------------------------ > > _______________________________________________ > Sequoiadb-discuss mailing list > Seq...@li... > https://lists.sourceforge.net/lists/listinfo/sequoiadb-discuss > -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-20 17:46:35
|
Hi Adam, It should not be that hard. I will be traveling to PGCon until the end of the week but should have some time to look at that. I'll keep you posted. Thanks manu > Would it take much work to move away from the crisom parser? We would > be happy to help. If not, do you have any other suggestions? > > Thanks! > > Adam > > On Fri, May 1, 2009 at 5:32 PM, Adam Denning <ada...@gm...> wrote: > >> I am testing Sequoia 4 in Ubuntu 8.10 with Java 1.6.0_10. When >> flushing preferences inside a process that starts Sequoia, the >> preferences aren't saved and I get >> >> Exception in thread "Timer-0" java.lang.AbstractMethodError: >> org.apache.crimson.tree.XmlDocument.getXmlStandalone()Z >> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.setDocumentInfo(DOM2TO.java:373) >> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.setDocumentInfo(DOM2TO.java:373) >> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:127) >> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:94) >> at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(TransformerImpl.java:662) >> at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:708) >> at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313) >> at java.util.prefs.XmlSupport.writeDoc(XmlSupport.java:259) >> at java.util.prefs.XmlSupport.exportMap(XmlSupport.java:338) >> at java.util.prefs.FileSystemPreferences$8.run(FileSystemPreferences.java:607) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.util.prefs.FileSystemPreferences.writeBackCache(FileSystemPreferences.java:600) >> at java.util.prefs.FileSystemPreferences.syncSpiPrivileged(FileSystemPreferences.java:784) >> at java.util.prefs.FileSystemPreferences.access$2300(FileSystemPreferences.java:33) >> at java.util.prefs.FileSystemPreferences$13.run(FileSystemPreferences.java:754) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.util.prefs.FileSystemPreferences.syncSpi(FileSystemPreferences.java:752) >> at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1317) >> at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) >> at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) >> at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) >> at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) >> at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) >> at java.util.prefs.AbstractPreferences.sync(AbstractPreferences.java:1308) >> at java.util.prefs.FileSystemPreferences.sync(FileSystemPreferences.java:731) >> at java.util.prefs.FileSystemPreferences.flush(FileSystemPreferences.java:807) >> at java.util.prefs.FileSystemPreferences.syncWorld(FileSystemPreferences.java:458) >> at java.util.prefs.FileSystemPreferences.access$1200(FileSystemPreferences.java:33) >> at java.util.prefs.FileSystemPreferences$4.run(FileSystemPreferences.java:419) >> at java.util.TimerThread.mainLoop(Timer.java:512) >> at java.util.TimerThread.run(Timer.java:462) >> >> If I add this jvm option, preferences work again. >> >> -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl >> >> But, then Sequoia can't start. >> >> java.lang.RuntimeException: Unable to create and start controller, >> reason: org.continuent.sequoia.common.exceptions.ControllerException: >> Error while analysing xml configuration file >> (org.xml.sax.SAXException: Controller Xml configuration file is not >> valid.). >> java.lang.RuntimeException: Unable to create and start controller, >> reason: org.continuent.sequoia.common.exceptions.ControllerException: >> Error while analysing xml configuration file >> (org.xml.sax.SAXException: Controller Xml configuration file is not >> valid.). >> Caused by: org.continuent.sequoia.common.exceptions.ControllerException: >> Error while analysing xml configuration file >> (org.xml.sax.SAXException: Controller Xml configuration file is not >> valid.). >> at org.continuent.sequoia.common.exceptions.SequoiaException.fillInStackTrace(SequoiaException.java:103) >> at java.lang.Throwable.<init>(Throwable.java:196) >> at java.lang.Exception.<init>(Exception.java:41) >> at org.continuent.sequoia.common.exceptions.SequoiaException.<init>(SequoiaException.java:57) >> at org.continuent.sequoia.common.exceptions.ControllerException.<init>(ControllerException.java:51) >> at org.continuent.sequoia.controller.core.ControllerConfiguration.setup(ControllerConfiguration.java:213) >> at org.continuent.sequoia.controller.core.ControllerConfiguration.<init>(ControllerConfiguration.java:103) >> at com.popstarnetworks.popstar_db.process.ControllerConfig.createSetupAndStartController(ControllerConfig.java:35) >> >> Regards, >> >> Adam >> >> > > ------------------------------------------------------------------------------ > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables > unlimited royalty-free distribution of the report engine > for externally facing server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Sequoiadb-discuss mailing list > Seq...@li... > https://lists.sourceforge.net/lists/listinfo/sequoiadb-discuss > > -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: BESSON-DEBLON,
Pierre \(S. H. TECH\) <PIE...@ai...> - 2009-05-20 10:03:32
|
Thanks Emmanuel, This is not reproduced every time, it's the first time I explicitly observed this behavior. I can see in this log that StackOverflowError is thrown for each query from user1 (vdb user=oracle user), queries for user2 work perfectly. Time between log "Load balancing read request" and "Runtime error during command execution (null)" is always between 1s and 2s. 5 minutes after first error, we can see strange behavior : - c1 see c2 quitting group (29 Apr 2009 13:00:22,684 | sequoia.controller.recoverylog | Storing checkpoint Member(address=/172.20.1.81:65201, uid=db) quit group db-172.20.1.77:65209-20090429130022678+0000 at request id 1679), but c2 is still alive and does not see other quit ... This cause c1 is in a 1 member group and c2 in a 2 members group !! After a restart of the 2 controllers, all seems to be fine. Is it possible that StackOverflow is thrown by jGroups (we see it even when a query is select on local backend) ? After bug on c1 see c2 quit, how can it be possible to have an update on c2 ? I join a log which shows that c2 send update request to all members, c1 does not receive anything, but at the end the request seems to be executed (I can not check database) I hope at least 1 symptom could help to find problem source ... Pierre Besson-Deblon -----Message d'origine----- De : seq...@li... [mailto:seq...@li...]De la part de Emmanuel Cecchet Envoyé : 18 May 2009 22:39 À : Sequoia general mailing list; seq...@li... Objet : Re: [Sequoia] Bug ? Hi Pierre, Note that the mailing list has now moved to seq...@li... The java.lang.StackOverflowError error probably comes from an infinite retry loop somewhere (hard to tell where since the exception is null). Could you reproduce the issue with Sequoia 2.10.10? I don't have the source code of 2.10.9 readily available and a number of fixes (as well as a JGroups update) have been performed in 2.10.10. Does the problem happen systematically? Thanks for your feedback, Emmanuel > continue ... I made an edit at the end of previous message > > > -----Message d'origine----- > De : seq...@li... > [mailto:seq...@li...]De la part de > BESSON-DEBLON, Pierre (SOGETI HIGH TECH) > Envoyé : 18 May 2009 19:32 > À : Sequoia general mailing list > Objet : [Sequoia] Bug ? > > > Hello, > > We use sequoia with 2 controllers on RAIDb1, sequoia is 2.10.9 with jgroups. > > We experienced this weird behavior : > > # on controller 1 (c1) backend deactivated, on c2 backend activated > c2.log -> 29 Apr 2009 12:56:12,400 | sequoia.controller.RequestManager.db | Database backend db2 is now enabled > > # c1 receive a read request and forward to c2 > c1.log -> > 29 Apr 2009 12:56:16,271 | sequoia.controller.virtualdatabase.request.db | S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,273 | sequoia.controller.RequestManager.db |Sending request SELECT ... to Member c2 > > c2.log -> > 29 Apr 2009 12:56:16,353 | sequoia.controller.virtualdatabase.db | S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,355 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Scheduling distributedRequest 844424930131983 from Member c1 > 29 Apr 2009 12:56:16,355 | sequoia.controller.distributedvirtualdatabase.request.db | S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | handleMessageMultiThreaded (class org.continuent.sequoia.controller.virtualdatabase.protocol.ExecRemoteStatementExecuteQuery): S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Executing distributedRequest 844424930131983 from Member c1 > 29 Apr 2009 12:56:16,356 | sequoia.controller.RequestManager.db | Database schema is dirty, refreshing it > 29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Scheduling read Request (844424930131983): SELECT ... > 29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Load balancing read request (844424930131983): SELECT ... > > and right after in c2.log file > 29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | StatementExecuteQuery command > 29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.request.db | S 562949953430692 0 SELECT ... > > c1.log > 29 Apr 2009 12:56:21,233 | sequoia.controller.RequestManager.db | Request SELECT ... completed. > 29 Apr 2009 12:56:21,234 | sequoia.controller.RequestManager.db | An error occured while executing remote select request 844424930131983 > org.continuent.sequoia.common.exceptions.NoMoreBackendException > at org.continuent.sequoia.controller.requestmanager.distributed.RAIDb1DistributedRequestManager.execRemoteStatementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.requestmanager.distributed.DistributedRequestManager.statementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(Unknown Source) > 29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.db | Request '844424930131983' failed (null) > 29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Error during command execution (null) > > > So, How can request fail with NoMoreBackendException despite db2 on c2 is enabled ? > Is it normal to have no more log in c2.log to indicate Exception ?, is it possible that request is hanging on c2 and a timeout raise the exception ? > > edit >> found this in c2.log > 29 Apr 2009 12:56:24,553 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Runtime error during command execution (null) > java.lang.StackOverflowError > 29 Apr 2009 12:56:24,554 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Reset command > I think it could help > > > Thanks for any help. > > > Pierre Besson-Deblon > > > The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised. > If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. > Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. > All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. > > _______________________________________________ > Sequoia mailing list > Se...@li... > http://forge.continuent.org/mailman/listinfo/sequoia > > -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet _______________________________________________ Sequoia mailing list Se...@li... http://forge.continuent.org/mailman/listinfo/sequoia This mail has originated outside your organization, either from an external partner or the Global Internet. Keep this in mind if you answer this message. The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. |
|
From: Adam D. <ada...@gm...> - 2009-05-20 02:51:15
|
Would it take much work to move away from the crisom parser? We would be happy to help. If not, do you have any other suggestions? Thanks! Adam On Fri, May 1, 2009 at 5:32 PM, Adam Denning <ada...@gm...> wrote: > I am testing Sequoia 4 in Ubuntu 8.10 with Java 1.6.0_10. When > flushing preferences inside a process that starts Sequoia, the > preferences aren't saved and I get > > Exception in thread "Timer-0" java.lang.AbstractMethodError: > org.apache.crimson.tree.XmlDocument.getXmlStandalone()Z > at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.setDocumentInfo(DOM2TO.java:373) > at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.setDocumentInfo(DOM2TO.java:373) > at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:127) > at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:94) > at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(TransformerImpl.java:662) > at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:708) > at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313) > at java.util.prefs.XmlSupport.writeDoc(XmlSupport.java:259) > at java.util.prefs.XmlSupport.exportMap(XmlSupport.java:338) > at java.util.prefs.FileSystemPreferences$8.run(FileSystemPreferences.java:607) > at java.security.AccessController.doPrivileged(Native Method) > at java.util.prefs.FileSystemPreferences.writeBackCache(FileSystemPreferences.java:600) > at java.util.prefs.FileSystemPreferences.syncSpiPrivileged(FileSystemPreferences.java:784) > at java.util.prefs.FileSystemPreferences.access$2300(FileSystemPreferences.java:33) > at java.util.prefs.FileSystemPreferences$13.run(FileSystemPreferences.java:754) > at java.security.AccessController.doPrivileged(Native Method) > at java.util.prefs.FileSystemPreferences.syncSpi(FileSystemPreferences.java:752) > at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1317) > at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) > at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) > at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) > at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) > at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322) > at java.util.prefs.AbstractPreferences.sync(AbstractPreferences.java:1308) > at java.util.prefs.FileSystemPreferences.sync(FileSystemPreferences.java:731) > at java.util.prefs.FileSystemPreferences.flush(FileSystemPreferences.java:807) > at java.util.prefs.FileSystemPreferences.syncWorld(FileSystemPreferences.java:458) > at java.util.prefs.FileSystemPreferences.access$1200(FileSystemPreferences.java:33) > at java.util.prefs.FileSystemPreferences$4.run(FileSystemPreferences.java:419) > at java.util.TimerThread.mainLoop(Timer.java:512) > at java.util.TimerThread.run(Timer.java:462) > > If I add this jvm option, preferences work again. > > -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl > > But, then Sequoia can't start. > > java.lang.RuntimeException: Unable to create and start controller, > reason: org.continuent.sequoia.common.exceptions.ControllerException: > Error while analysing xml configuration file > (org.xml.sax.SAXException: Controller Xml configuration file is not > valid.). > java.lang.RuntimeException: Unable to create and start controller, > reason: org.continuent.sequoia.common.exceptions.ControllerException: > Error while analysing xml configuration file > (org.xml.sax.SAXException: Controller Xml configuration file is not > valid.). > Caused by: org.continuent.sequoia.common.exceptions.ControllerException: > Error while analysing xml configuration file > (org.xml.sax.SAXException: Controller Xml configuration file is not > valid.). > at org.continuent.sequoia.common.exceptions.SequoiaException.fillInStackTrace(SequoiaException.java:103) > at java.lang.Throwable.<init>(Throwable.java:196) > at java.lang.Exception.<init>(Exception.java:41) > at org.continuent.sequoia.common.exceptions.SequoiaException.<init>(SequoiaException.java:57) > at org.continuent.sequoia.common.exceptions.ControllerException.<init>(ControllerException.java:51) > at org.continuent.sequoia.controller.core.ControllerConfiguration.setup(ControllerConfiguration.java:213) > at org.continuent.sequoia.controller.core.ControllerConfiguration.<init>(ControllerConfiguration.java:103) > at com.popstarnetworks.popstar_db.process.ControllerConfig.createSetupAndStartController(ControllerConfig.java:35) > > Regards, > > Adam > |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-20 02:39:11
|
Hi Richard,
Sorry for forgetting about this. The fastest quick fix would be to
change org.continuent.sequoia.controller.backend.DriverCompliance line 586
replace:
return supportGetGeneratedKeys;
with
return true;
That should fix it for now.
To really know what happens, using directly your MySQL driver to connect
to your MySQL database, try a code like:
Connection c = DriverManager.connect(...);
Statement s = c.createStatement();
s.execute("select 1");
s.getGeneratedKeys();
This last call should trigger an SQLException. Please post that
exception so that I can diagnose the problem.
Thanks for your feedback,
Emmanuel
> Hi Emmanuel,
>
> Is there any info I could send you in order to pinpoint the problem.
> I am running out of time and a decision will soon be made to go with
> MySQL Cluster ($$$) and I don't want that.
>
> Thanks for your help,
>
> Richard
>
>
> On May 14, 2009, at 6:48 PM, Richard Vigeant wrote:
>
>> Hi Emmanuel,
>>
>> I'm using MySQL 5.1.31 community with mysql-connector-java-5.1.7.
>>
>> Thanks,
>>
>> Richard
>>
>> -----Original Message-----
>> From: Emmanuel Cecchet [mailto:ma...@fr...]
>> Sent: Thu 5/14/2009 6:02 PM
>> To: Richard Vigeant
>> Cc: seq...@li...
>> <mailto:seq...@li...>
>> Subject: Re: [Sequoiadb-discuss] Request blocked forever in Sequoia
>>
>> Hi Richard,
>>
>> > I'm sending you the debug log of the problem.
>> > I noticed that an error that repeatedly occurs about table
>> > 'alfresco.sequoiaSABase' does not exists. In fact I don't know where
>> > this comes from because there is no reference to such a table anywhere
>> > in our modules.
>> This was a prototype implementation of stored procedure semantic in the
>> database that has been deprecated. The message was not removed but it
>> can be safely ignored.
>>
>> The problem comes from there:
>> 2009-05-14 16:26:55,338 WARN backend.DatabaseBackend.mysql1 Task
>> execution failed (java.sql.SQLException: Backend mysql1 does not support
>> auto generated keys.)
>>
>> What is strange is that the error does not get propagated back to the
>> client and get stuck.
>>
>> It might well be that your version of MySQL does not support generated
>> keys or that the JDBC driver is not compliant to the spec and when we
>> test Statement.getGeneratedKeys() it throws an exception rather than
>> returning a null ResultSet.
>>
>> Which version of MySQL are you using, with which driver?
>>
>> Thanks for the feedback,
>> Emmanuel
>>
>> --
>> Emmanuel Cecchet
>> FTO @ Frog Thinker
>> Open Source Development & Consulting
>> --
>> Web: http://www.frogthinker.org
>> email: ma...@fr... <mailto:ma...@fr...>
>> Skype: emmanuel_cecchet
>>
>>
>
--
Emmanuel Cecchet
FTO @ Frog Thinker
Open Source Development & Consulting
--
Web: http://www.frogthinker.org
email: ma...@fr...
Skype: emmanuel_cecchet
|
|
From: Richard V. <ric...@va...> - 2009-05-19 21:27:53
|
Hi Emmanuel, Is there any info I could send you in order to pinpoint the problem. I am running out of time and a decision will soon be made to go with MySQL Cluster ($$$) and I don't want that. Thanks for your help, Richard On May 14, 2009, at 6:48 PM, Richard Vigeant wrote: > Hi Emmanuel, > > I'm using MySQL 5.1.31 community with mysql-connector-java-5.1.7. > > Thanks, > > Richard > > -----Original Message----- > From: Emmanuel Cecchet [mailto:ma...@fr...] > Sent: Thu 5/14/2009 6:02 PM > To: Richard Vigeant > Cc: seq...@li... > Subject: Re: [Sequoiadb-discuss] Request blocked forever in Sequoia > > Hi Richard, > > > I'm sending you the debug log of the problem. > > I noticed that an error that repeatedly occurs about table > > 'alfresco.sequoiaSABase' does not exists. In fact I don't know where > > this comes from because there is no reference to such a table > anywhere > > in our modules. > This was a prototype implementation of stored procedure semantic in > the > database that has been deprecated. The message was not removed but it > can be safely ignored. > > The problem comes from there: > 2009-05-14 16:26:55,338 WARN backend.DatabaseBackend.mysql1 Task > execution failed (java.sql.SQLException: Backend mysql1 does not > support > auto generated keys.) > > What is strange is that the error does not get propagated back to the > client and get stuck. > > It might well be that your version of MySQL does not support generated > keys or that the JDBC driver is not compliant to the spec and when we > test Statement.getGeneratedKeys() it throws an exception rather than > returning a null ResultSet. > > Which version of MySQL are you using, with which driver? > > Thanks for the feedback, > Emmanuel > > -- > Emmanuel Cecchet > FTO @ Frog Thinker > Open Source Development & Consulting > -- > Web: http://www.frogthinker.org > email: ma...@fr... > Skype: emmanuel_cecchet > > > |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-18 20:38:47
|
Hi Pierre, Note that the mailing list has now moved to seq...@li... The java.lang.StackOverflowError error probably comes from an infinite retry loop somewhere (hard to tell where since the exception is null). Could you reproduce the issue with Sequoia 2.10.10? I don't have the source code of 2.10.9 readily available and a number of fixes (as well as a JGroups update) have been performed in 2.10.10. Does the problem happen systematically? Thanks for your feedback, Emmanuel > continue ... I made an edit at the end of previous message > > > -----Message d'origine----- > De : seq...@li... > [mailto:seq...@li...]De la part de > BESSON-DEBLON, Pierre (SOGETI HIGH TECH) > Envoyé : 18 May 2009 19:32 > À : Sequoia general mailing list > Objet : [Sequoia] Bug ? > > > Hello, > > We use sequoia with 2 controllers on RAIDb1, sequoia is 2.10.9 with jgroups. > > We experienced this weird behavior : > > # on controller 1 (c1) backend deactivated, on c2 backend activated > c2.log -> 29 Apr 2009 12:56:12,400 | sequoia.controller.RequestManager.db | Database backend db2 is now enabled > > # c1 receive a read request and forward to c2 > c1.log -> > 29 Apr 2009 12:56:16,271 | sequoia.controller.virtualdatabase.request.db | S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,273 | sequoia.controller.RequestManager.db |Sending request SELECT ... to Member c2 > > c2.log -> > 29 Apr 2009 12:56:16,353 | sequoia.controller.virtualdatabase.db | S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,355 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Scheduling distributedRequest 844424930131983 from Member c1 > 29 Apr 2009 12:56:16,355 | sequoia.controller.distributedvirtualdatabase.request.db | S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | handleMessageMultiThreaded (class org.continuent.sequoia.controller.virtualdatabase.protocol.ExecRemoteStatementExecuteQuery): S 844424930131983 0 SELECT ... > 29 Apr 2009 12:56:16,356 | sequoia.controller.virtualdatabase.db | 172.20.1.81:65209: Executing distributedRequest 844424930131983 from Member c1 > 29 Apr 2009 12:56:16,356 | sequoia.controller.RequestManager.db | Database schema is dirty, refreshing it > 29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Scheduling read Request (844424930131983): SELECT ... > 29 Apr 2009 12:56:16,359 | sequoia.controller.RequestManager.db | Load balancing read request (844424930131983): SELECT ... > > and right after in c2.log file > 29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | StatementExecuteQuery command > 29 Apr 2009 12:56:21,268 | sequoia.controller.virtualdatabase.request.db | S 562949953430692 0 SELECT ... > > c1.log > 29 Apr 2009 12:56:21,233 | sequoia.controller.RequestManager.db | Request SELECT ... completed. > 29 Apr 2009 12:56:21,234 | sequoia.controller.RequestManager.db | An error occured while executing remote select request 844424930131983 > org.continuent.sequoia.common.exceptions.NoMoreBackendException > at org.continuent.sequoia.controller.requestmanager.distributed.RAIDb1DistributedRequestManager.execRemoteStatementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.requestmanager.distributed.DistributedRequestManager.statementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(Unknown Source) > at org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(Unknown Source) > 29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.db | Request '844424930131983' failed (null) > 29 Apr 2009 12:56:21,234 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Error during command execution (null) > > > So, How can request fail with NoMoreBackendException despite db2 on c2 is enabled ? > Is it normal to have no more log in c2.log to indicate Exception ?, is it possible that request is hanging on c2 and a timeout raise the exception ? > > edit >> found this in c2.log > 29 Apr 2009 12:56:24,553 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Runtime error during command execution (null) > java.lang.StackOverflowError > 29 Apr 2009 12:56:24,554 | sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.db | Reset command > I think it could help > > > Thanks for any help. > > > Pierre Besson-Deblon > > > The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other then the addressee. Access to this e-mail by anyone else is unauthorised. > If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. > Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. > All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. > > _______________________________________________ > Sequoia mailing list > Se...@li... > http://forge.continuent.org/mailman/listinfo/sequoia > > -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-16 19:53:20
|
Hi, Note that the mailing list has moved to seq...@li... > I want to know more about RAIDb controller.How it is achieved£¿ I would recommend that you download the Sequoia documentation from http://sourceforge.net/project/showfiles.php?group_id=252061&package_id=308175&release_id=672606 You can find older documents, especially about the RAIDb concepts, at http://c-jdbc.ow2.org/doc/index.html Hope this helps, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Slawomir K. <sa...@sa...> - 2009-05-15 11:57:09
|
Hi Emmanuel, Yesterday I've lost my test environment on VMware so I've decided to move it to independent machines. I was suspecting that network connectivity on my virtual machines could cause behaviour described in last message.But now I'm sure that low level(on operating system) network configuration of my computers has nothing to do with my problems. Currently my test environment is: 2 laptops, both 4 GB of RAM and >2GHz dual core CPU's. Distribution changed to Ubuntu(32bit) and PostgreSQL 8.3, Java: 32bit 1.6.0_13. Sequoia version didn't change. Configuration is the same but I've changed values You proposed. Computers are connected through 100Mbits switch, I've measured connection. It's about 10MB/s on TCP and UDP. >I am not sure I understand the memory consumption part, is that for a >single query or for all queries in your workload? Unless you are >inserting a big BLOB, there is no reason that so much memory is used. >Could you give an example of an insert you are performing in the >database? Table I'm testing on has 3 columns, integer, integer and varchar(128). My test consist of 4500 inserts filling this table. Without Sequoia my postgres handles this insert about 0.3 second.. When sequoia is used 6-10 minutes. Those are only test queries, target queries will consist of few integers and one bytea(about 1kB). If we talk about memory consumption part I mean values shown in JConsole connected to Sequoia controller node. >The low CPU percentage is probably due to network configuration problems. >Here are some things to test: >1. Run JConsole in the same VM as Postgres and measure insert >performance directly on Postgres >2. Run JConsole in a different VM and measure insert performance >directly on Postgres in the other VM What You mean directly on Postgres? >3. Run JConsole on a single Sequoia controller (without group >communication, remove the Distribution element) After removing Distribution element on one node, and shutdown second one performance increases. Performing query I've described before takes about 8 seconds. Almost 600 queries per second. >4. Run JConsole on a single Sequoia controller with group communication >enabled (re-introduce the Distribution element in your virtual database >config file) When only one node is running and distribution element is present performance is the same as when both nodes are running and enabled. 6-10 minutes 4500 simple insert queries. -----------------CUT---------------------------- ;) Hi Slawomir, Use the new sequoiadb-discuss@li... mailing list, the mailing list at continuent.org is being discontinued. > We are currently trying to cluster our database. > I'm testing solution based on Sequoia, but performance is very poor. > About 8 database insert per second. We need reach at least 100/150 > inserts per second. You should be able to achieve 800 to 1000 inserts/sec without any fancy hardware. There is certainly a configuration problem somewhere in your stack. > We are using PostgreSQL 8.1.11 on CentOS 5.2 on 32 bit Sun JDK and > Sequoia 2.10.10 on our test environment. Both test servers are inside > vmware server and have assigned 2 cores of Intel Xeon 3.06GHz CPU's > and 2 GB RAM each. Servers have very low load. This version of Postgres is quite dated now but I think that is what ships with CentOS 5.2. > Server are named node0 and node1. Each server consist of PostgreSQL > database and Sequoia controller. > Database replication is RAIDb1. > Most of configuration details are left default from Sequoia install. > As I can see default configuration uses JGroups communication. > Each node has assigned in /etc/hosts static hostnames. > When I run any insert query on JConsole attached to each node I see > little peak, about 5-10% of CPU but nothing more. > Memory usage is mostly constant, when I run query heap size grows upto > 90MB but when query finish sequoia uses only ~20MB of Heap I am not sure I understand the memory consumption part, is that for a single query or for all queries in your workload? Unless you are inserting a big BLOB, there is no reason that so much memory is used. Could you give an example of an insert you are performing in the database? The low CPU percentage is probably due to network configuration problems. Here are some things to test: 1. Run JConsole in the same VM as Postgres and measure insert performance directly on Postgres 2. Run JConsole in a different VM and measure insert performance directly on Postgres in the other VM 3. Run JConsole on a single Sequoia controller (without group communication, remove the Distribution element) 4. Run JConsole on a single Sequoia controller with group communication enabled (re-introduce the Distribution element in your virtual database config file) Usually UDP multicast does not perform very well with virtual machines. You should try a JGroups configuration based on TCP instead, you might see a significant performance improvement. > Here's my controller.xml > > <SEQUOIA-CONTROLLER> > <Controller ipAddress="node1" port="54321"> > <JmxSettings> > <RmiJmxAdaptor port="1090"/> > </JmxSettings> > <VirtualDatabase configFile="postgres-raidb1-distribution-1.xml" > virtualDatabaseName="counter" autoEnableBackends="true"/> > </Controller> > </SEQUOIA-CONTROLLER> > > And here is my VD configuration > <SEQUOIA> > <VirtualDatabase name="counter"> > <Distribution > > <MessageTimeouts /> > </Distribution> > <Backup> > <Backuper backuperName="pgdump" > > className="org.continuent.sequoia.controller.backup.backupers.PostgreSQLBinaryBackuper" > /> > </Backup> > > <AuthenticationManager> > <Admin> > <User username="admin" password="xxx" /> > </Admin> > <VirtualUsers> > <VirtualLogin vLogin="user" > vPassword="yyyy" /> > </VirtualUsers> > </AuthenticationManager> > > <DatabaseBackend name="node1" > driver="org.postgresql.Driver" > url="jdbc:postgresql://node1/counter" > connectionTestStatement="select now()"> > <DatabaseSchema dynamicPrecision="all" > gatherSystemTables="true"> > <DefaultStoredProcedureSemantic > hasSelect="true" hasInsert="true" > hasUpdate="true" hasDelete="true" > hasDDL="false" hasTransaction="false" > isCausallyDependent="true" > isCommutative="false" /> > <StoredProcedureSemantic > procedureName="sequoia_rowcounts" > parameterCount="1" > writeTables="true" > hasSelect="true" hasInsert="true" > hasUpdate="false" > hasDelete="false" > hasDDL="false" > hasTransaction="false" > isCausallyDependent="true" > isCommutative="false" /> > </DatabaseSchema> > > <ConnectionManager vLogin="user" rLogin="postgres" > rPassword=""> > <VariablePoolConnectionManager > initPoolSize="10" > minPoolSize="5" > maxPoolSize="50" idleTimeout="30" waitTimeout="10" /> > </ConnectionManager> > </DatabaseBackend> > <RequestManager> > <RequestScheduler> > <RAIDb-1Scheduler level="passThrough" /> > </RequestScheduler> > > <RequestCache> > <MetadataCache /> > <ParsingCache /> I would recommend to remove the ResultCache element below: > <ResultCache granularity="database" /> > </RequestCache> > > <LoadBalancer> > <RAIDb-1> As you have a single database attached to your controller, use WaitForCompletion policy="all" instead, this will be faster. > <WaitForCompletion > policy="first" /> > > <RAIDb-1-LeastPendingRequestsFirst /> > </RAIDb-1> > </LoadBalancer> > <RecoveryLog driver="org.postgresql.Driver" > url="jdbc:postgresql://node1/recovery" > login="sequoia" password=""> > <RecoveryLogTable /> > <CheckpointTable /> > <BackendTable /> > <DumpTable /> > </RecoveryLog> > </RequestManager> > </VirtualDatabase> > </SEQUOIA> > sequoia_sequencer.xml [used in jgroups.properties]: > > --> > <config> > <UDP mcast_port="45566" > mcast_addr="228.8.8.9" > tos="16" > ucast_recv_buf_size="20000000" > ucast_send_buf_size="640000" > mcast_recv_buf_size="25000000" > mcast_send_buf_size="640000" > loopback="false" > discard_incompatible_packets="true" > max_bundle_size="64000" > max_bundle_timeout="30" > use_incoming_packet_handler="true" > use_outgoing_packet_handler="false" > ip_ttl="2" > down_thread="false" up_thread="false" > enable_bundling="true"/> > <PING timeout="2000" > down_thread="false" up_thread="false" num_initial_members="3"/> > <MERGE2 max_interval="10000" > down_thread="false" up_thread="false" min_interval="5000"/> > <FD_SOCK down_thread="false" up_thread="false"/> > <!--VERIFY_SUSPECT timeout="1500" down_thread="false"/--> > <pbcast.NAKACK max_xmit_size="60000" > use_mcast_xmit="false" gc_lag="0" > retransmit_timeout="100,200,300,600,1200,2400,4800" > down_thread="false" up_thread="false" > discard_delivered_msgs="true"/> > <UNICAST timeout="300,600,1200,2400,3600" > down_thread="false" up_thread="false"/> > <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" > down_thread="false" up_thread="false" > max_bytes="400000"/> > <VIEW_SYNC avg_send_interval="60000" down_thread="false" > up_thread="false" /> > <pbcast.GMS print_local_addr="true" join_timeout="3000" > down_thread="false" up_thread="false" > join_retry_timeout="2000" shun="true" > handle_concurrent_startup="true" /> > <SEQUENCER down_thread="false" up_thread="false" /> > <FC max_credits="2000000" down_thread="false" up_thread="false" > min_threshold="0.10"/> > <!-- FRAG2 frag_size="60000" down_thread="false" up_thread="true"/ --> > <!-- pbcast.STATE_TRANSFER down_thread="false" up_thread="false"/--> > </config> > > > What can be wrong. What we can do to increase performance? Keep us posted with the performance numbers with the different configurations proposed above. > I've been browsing list and I found something about group > communication protocol change and use appia but I can't find anything > about proper configuration, When I change distribution element of > configuration to use hedera_appia.properties I'm getting No Route to > host at controller output. If you use TCP, you will have to hardcode the IP addresses in the configuration files or use a gossip server. That might be the issue. Keep us posted with your progress, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: manu@fr... Skype: emmanuel_cecchet |
|
From: Richard V. <ric...@va...> - 2009-05-14 22:48:50
|
Hi Emmanuel, I'm using MySQL 5.1.31 community with mysql-connector-java-5.1.7. Thanks, Richard -----Original Message----- From: Emmanuel Cecchet [mailto:ma...@fr...] Sent: Thu 5/14/2009 6:02 PM To: Richard Vigeant Cc: seq...@li... Subject: Re: [Sequoiadb-discuss] Request blocked forever in Sequoia Hi Richard, > I'm sending you the debug log of the problem. > I noticed that an error that repeatedly occurs about table > 'alfresco.sequoiaSABase' does not exists. In fact I don't know where > this comes from because there is no reference to such a table anywhere > in our modules. This was a prototype implementation of stored procedure semantic in the database that has been deprecated. The message was not removed but it can be safely ignored. The problem comes from there: 2009-05-14 16:26:55,338 WARN backend.DatabaseBackend.mysql1 Task execution failed (java.sql.SQLException: Backend mysql1 does not support auto generated keys.) What is strange is that the error does not get propagated back to the client and get stuck. It might well be that your version of MySQL does not support generated keys or that the JDBC driver is not compliant to the spec and when we test Statement.getGeneratedKeys() it throws an exception rather than returning a null ResultSet. Which version of MySQL are you using, with which driver? Thanks for the feedback, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-14 22:02:32
|
Hi Richard, > I'm sending you the debug log of the problem. > I noticed that an error that repeatedly occurs about table > 'alfresco.sequoiaSABase' does not exists. In fact I don't know where > this comes from because there is no reference to such a table anywhere > in our modules. This was a prototype implementation of stored procedure semantic in the database that has been deprecated. The message was not removed but it can be safely ignored. The problem comes from there: 2009-05-14 16:26:55,338 WARN backend.DatabaseBackend.mysql1 Task execution failed (java.sql.SQLException: Backend mysql1 does not support auto generated keys.) What is strange is that the error does not get propagated back to the client and get stuck. It might well be that your version of MySQL does not support generated keys or that the JDBC driver is not compliant to the spec and when we test Statement.getGeneratedKeys() it throws an exception rather than returning a null ResultSet. Which version of MySQL are you using, with which driver? Thanks for the feedback, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Richard V. <ric...@va...> - 2009-05-14 21:41:32
|
Emmanuel,
I'm sending you the debug log of the problem.
I noticed that an error that repeatedly occurs about table
'alfresco.sequoiaSABase' does not exists. In fact I don't know where
this comes from because there is no reference to such a table anywhere
in our modules.
2009-05-14 16:26:40,663 DEBUG DatabaseBackend.alfresco.mysql1
Failed to get stored procedure semantics from database backend (Table
'alfresco.sequoiaSABase' doesn't exist)
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
'alfresco.sequoiaSABase' doesn't exist
Again I want to thank you for your time,
Richard
On May 14, 2009, at 3:36 PM, Emmanuel Cecchet wrote:
> Richard,
>
>> 2- The query that blocks in Sequoia's write request queue is
>> slightly different in Sequoia than in direct MySQL:
>> In mysql: insert into alf_attributes (version, acl_id,
>> type) values (0, null, 'M')
>> In Sequoia: insert into alf_attributes (version, acl_id,
>> type) values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%>
> Actually it's the same query. The %N tag is to encode null so that
> the code called is PreparedStatement.setNull(param, sqlType) with
> sqlType being equal to -5 (which corresponds to BIGINT in
> java.sql.Types). In both cases, you are exactly executing the same
> query.
>
> Otherwise your configuration seems fine so we probably have to look
> somewhere else.
> Set the scheduler, loadbalancer and DatabaseBackend loggers to DEBUG
> in log4j.properties and send the trace so that we can see what is
> going on.
>
> Thanks for your feedback,
> Emmanuel
>
> --
> Emmanuel Cecchet
> FTO @ Frog Thinker Open Source Development & Consulting
> --
> Web: http://www.frogthinker.org
> email: ma...@fr...
> Skype: emmanuel_cecchet
>
|
|
From: Richard V. <ric...@va...> - 2009-05-14 19:54:25
|
Hi, Actually I meant that the query as seen in MySQL had a value of (0, null, 'M') while in Sequoia it shows (0, -5, 'M'). I will make a debug trace and and send it to you. Thanks, Richard On May 14, 2009, at 3:36 PM, Emmanuel Cecchet wrote: > Richard, > >> 2- The query that blocks in Sequoia's write request queue is >> slightly different in Sequoia than in direct MySQL: >> In mysql: insert into alf_attributes (version, acl_id, >> type) values (0, null, 'M') >> In Sequoia: insert into alf_attributes (version, acl_id, >> type) values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%> > Actually it's the same query. The %N tag is to encode null so that > the code called is PreparedStatement.setNull(param, sqlType) with > sqlType being equal to -5 (which corresponds to BIGINT in > java.sql.Types). In both cases, you are exactly executing the same > query. > > Otherwise your configuration seems fine so we probably have to look > somewhere else. > Set the scheduler, loadbalancer and DatabaseBackend loggers to DEBUG > in log4j.properties and send the trace so that we can see what is > going on. > > Thanks for your feedback, > Emmanuel > > -- > Emmanuel Cecchet > FTO @ Frog Thinker Open Source Development & Consulting > -- > Web: http://www.frogthinker.org > email: ma...@fr... > Skype: emmanuel_cecchet > |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-14 19:36:48
|
Richard, > 2- The query that blocks in Sequoia's write request queue is > slightly different in Sequoia than in direct MySQL: > In mysql: insert into alf_attributes (version, acl_id, > type) values (0, null, 'M') > In Sequoia: insert into alf_attributes (version, acl_id, > type) values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%> Actually it's the same query. The %N tag is to encode null so that the code called is PreparedStatement.setNull(param, sqlType) with sqlType being equal to -5 (which corresponds to BIGINT in java.sql.Types). In both cases, you are exactly executing the same query. Otherwise your configuration seems fine so we probably have to look somewhere else. Set the scheduler, loadbalancer and DatabaseBackend loggers to DEBUG in log4j.properties and send the trace so that we can see what is going on. Thanks for your feedback, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Richard V. <ric...@va...> - 2009-05-14 15:50:03
|
Hi Emmanuel,
The problem occurs systematically, always at the same place with the
same query stuck in the write scheduler queue.
There is no pending request on the MySQL backends. By the way, in
order to simplify the configuration I removed one backend from the
config, so now I use only one backend, and the results are identical.
Yes, both backends (now the single backend) are enabled.
I ran some tests with Alfresco connecting directly to MySQL instead of
Sequoia. It is obviously successful. I made some traces using mysql
"general_log" option to see all the queries that were generated during
initialization. I compared with the contents of Sequoia Parsing Cache
for the same initialization sequence. Doing so I noticed 2 things:
1- The order of the queries is considerably different. The
sequence contains several create table, alter table, create index, etc
and was surprised to see that they were not executed in the same order.
2- The query that blocks in Sequoia's write request queue is
slightly different in Sequoia than in direct MySQL:
In mysql: insert into alf_attributes (version, acl_id,
type) values (0, null, 'M')
In Sequoia: insert into alf_attributes (version, acl_id,
type) values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%>
I tried to execute both queries in the SQL client in the Sequoia's
console and I got this:
jdbc:sequoia://localhost:25322/alfresco (alfresco) > insert into
alf_attributes (version, acl_id, type) values (0, null, 'M');
Affected rows: 1
Query executed in 0 s 117 ms .
jdbc:sequoia://localhost:25322/alfresco (alfresco) > insert into
alf_attributes (version, acl_id, type) values (0, -5, 'M');
An error occured while executing SQL query
(org.continuent.sequoia.common.exceptions.driver.DriverSQLException:
Message of cause: write request 322 failed:
Backend alfresco - BackendWorkerThread for backend 'mysql1' with
RAIDb level:1 failed (Cannot add or update a child row: a foreign key
constraint fails (`alfresco`.`alf_attributes`,
CONSTRAINT`fk_alf_attr_acl` FOREIGN KEY (`acl_id`) REFERENCES
`alf_access_control_list` (`id`))))
Clearly the query in Sequoia's Parsing cache is wrong.
Here are my configs:
Controller:
------------
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE SEQUOIA-CONTROLLER PUBLIC "-//Continuent//DTD SEQUOIA-
CONTROLLER 2.10.10//EN" "http://sequoia.continuent.org/dtds/sequoia-controller-2.10.10.dtd
">
<SEQUOIA-CONTROLLER>
<Controller ipAddress="0.0.0.0" port="25322">
<Report hideSensitiveData="true" generateOnShutdown="true"
generateOnFatal="true" enableFileLogging="true" />
<JmxSettings>
<RmiJmxAdaptor port="1090"/>
</JmxSettings>
<VirtualDatabase configFile="mysqltest-raidb1.xml"
virtualDatabaseName="alfresco" autoEnableBackends="false"/>
</Controller>
</SEQUOIA-CONTROLLER>
Virtual database:
--------------------
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE SEQUOIA PUBLIC "-//Continuent//DTD SEQUOIA 2.10.10//EN" "http://sequoia.continuent.org/dtds/sequoia-2.10.10.dtd
">
<SEQUOIA>
<VirtualDatabase name="alfresco">
<!-- <Distribution>
<MessageTimeouts/>
</Distribution>
-->
<Monitoring>
<SQLMonitoring defaultMonitoring="off">
<SQLMonitoringRule queryPattern="^select"
caseSensitive="false" applyToSkeleton ="false" monitoring="on"/>
</SQLMonitoring>
</Monitoring>
<Backup>
<Backuper backuperName="Octopus"
className
="org.continuent.sequoia.controller.backup.backupers.OctopusBackuper"
options="zip=true"/>
</Backup>
<AuthenticationManager>
<Admin>
<User username="admin" password=""/>
</Admin>
<VirtualUsers>
<VirtualLogin vLogin="alfresco" vPassword="alfresco"/>
</VirtualUsers>
</AuthenticationManager>
<DatabaseBackend name="mysql1" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/alfresco"
connectionTestStatement="select 1">
<ConnectionManager vLogin="alfresco" rLogin="testuser"
rPassword="testuser">
<VariablePoolConnectionManager initPoolSize="10"
minPoolSize="5"
maxPoolSize="50" idleTimeout="30" waitTimeout="10"/>
</ConnectionManager>
</DatabaseBackend>
<!-- <DatabaseBackend name="mysql2" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://172.16.131.14:3306/alfresco"
connectionTestStatement="select 1">
<ConnectionManager vLogin="alfresco" rLogin="testuser"
rPassword="testuser">
<VariablePoolConnectionManager initPoolSize="40"
minPoolSize="20"
maxPoolSize="100" idleTimeout="30" waitTimeout="10"/>
</ConnectionManager>
</DatabaseBackend>
-->
<RequestManager>
<RequestScheduler>
<!-- <SingleDBScheduler level="pessimisticTransaction"/> -->
<RAIDb-1Scheduler level="passThrough"/>
</RequestScheduler>
<RequestCache>
<MetadataCache/>
<ParsingCache/>
</RequestCache>
<LoadBalancer>
<RAIDb-1>
<WaitForCompletion policy="all"/>
<RAIDb-1-LeastPendingRequestsFirst/>
</RAIDb-1>
</LoadBalancer>
<RecoveryLog driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/sequoia?
dumpQueriesOnException=true"
login="testuser" password="testuser">
<RecoveryLogTable
tableName="RECOVERY"
logIdColumnType="BIGINT NOT NULL"
vloginColumnType="VARCHAR(26) NOT NULL"
sqlColumnName="sqlStmt"
sqlColumnType="VARCHAR(2048) NOT NULL"
sqlParamColumnType="VARCHAR(8192)"
extraStatementDefinition=",PRIMARY KEY (log_id)"/>
<CheckpointTable tableName="CHECKPOINT"
checkpointNameColumnType="VARCHAR(80) NOT NULL"/>
<BackendTable tableName="BACKEND"
databaseNameColumnType="VARCHAR(256) NOT NULL"
backendNameColumnType="VARCHAR(256) NOT NULL"
checkpointNameColumnType="VARCHAR(256) NOT NULL"/>
<DumpTable tableName="DUMP"
dumpNameColumnType="VARCHAR(256) NOT NULL"
dumpDateColumnType="VARCHAR(100) NOT NULL"
dumpPathColumnType="VARCHAR(512) NOT NULL"
dumpFormatColumnType ="VARCHAR(8192) NOT NULL"
checkpointNameColumnType="VARCHAR(8192) NOT NULL"
backendNameColumnType="VARCHAR(8192) NOT NULL"
tablesColumnType="VARCHAR(8192) NOT NULL"/>
</RecoveryLog>
</RequestManager>
</VirtualDatabase>
</SEQUOIA>
Thanks for your help,
Richard Vigeant
Senior Software Developer
Vantrix
T +1 514-866-1717
M +1 438-275-2610
F +1 514-866-6868
ric...@va...
www.vantrix.com
On May 13, 2009, at 5:42 PM, Emmanuel Cecchet wrote:
> Hi Richard,
>
> Can you reproduce the problem systematically?
> Could you post your virtual database configuration file?
> As the request is executing, it might be blocked on a lock (that
> would be a bug) or waiting for MySQL backends to complete (we would
> then have to figure out why MySQL is blocked).
>
> Do you see any pending request on any MySQL backend (when you log
> directly on MySQL)?
> Both backends show as enabled in Sequoia?
>
> Thanks for your feedback,
> Emmanuel
>
>> I encounter a strange problem when using Alfresco that uses
>> Hibernate with Sequoia.
>> Sequoia is configured with 1 virtual database using 2 MySQL backends.
>> What happens is that when Alfresco first starts, it creates its
>> tables successfully, then issues a number of SELECTS successfully
>> and then issues a "INSERT" which gets stuck in Sequoia request
>> queue forever.
>>
>> Is there anything I can do to find out what Sequoia is waiting for?
>>
>> Here are the details of the request:
>>
>> insert into alf_attributes (version, acl_id, type) values (?, ?,
>> 'M'): Id=523, query=insert into alf_attributes (version, acl_id,
>> type) values (?, ?, 'M'), params=<!%L|0|!%><!%N|-5|!%>, transaction
>> id=324, persistent connection id=0
>>
>> alfresco(admin) > dump scheduler queues Active transactions: 1
>> +-----+-------------+
>> | tid | time (in s) | +-----+-------------+
>> | 324 | 76247 | +-----+-------------+
>>
>> ----------------
>>
>> Pending read requests: 0
>> Read request id list:
>>
>> Pending write requests: 1
>> Write request id list:
>> 523
>>
>> ----------------
>>
>> alfresco(admin) > dump transaction 324
>> +--------+----------
>> +
>> -----------------------------------------------------------------------+
>> -----------------------+----------------+----------------
>> +------------+-------------+-----------+--------------+
>> | log_id | vlogin |
>> sqlStmt
>> |
>> sqlStmt_param | auto_conn_tran | transaction_id |
>> request_id | exec_status | exec_time | update_count | +--------
>> +----------
>> +
>> -----------------------------------------------------------------------+
>> -----------------------+----------------+----------------
>> +------------+-------------+-----------+--------------+
>> | 1 | alfresco |
>> begin
>> |
>> null | T | 324 |
>> 0 | S | 0 | -1 | | 2 |
>> alfresco | insert into alf_attributes (version, acl_id, type)
>> values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%> | T |
>> 324 | 523 | E | 2 |
>> 0 | +--------+----------
>> +
>> -----------------------------------------------------------------------+
>> -----------------------+----------------+----------------
>> +------------+-------------+-----------+--------------+
>>
>> ----------------
>> alfresco(admin) > dump request 523
>> Request id: 523
>> query: insert into alf_attributes (version, acl_id, type) values
>> (?, ?, 'M')
>> parameters: <!%L|0|!%><!%N|-5|!%>
>> login: alfresco
>> autocommit: false
>> transaction id: 324
>> cacheable status: CACHEABLE
>> isolation level: TRANSACTION_UNDEFINED
>> start time: 1242150743853
>> end time: 0
>> timeout in seconds: 0
>> locked tables:
>> alf_attributes
>> persistent connection id: 0
>> client ip address: /127.0.0.1
>>
>>
>> Thanks for any help.
>>
>> Richard Vigeant
>>
>> ------------------------------------------------------------------------
>>
>> ------------------------------------------------------------------------------
>> The NEW KODAK i700 Series Scanners deliver under ANY circumstances!
>> Your
>> production scanning environment may not be a perfect world - but
>> thanks to
>> Kodak, there's a perfect scanner to get the job done! With the NEW
>> KODAK i700
>> Series Scanner you'll get full speed at 300 dpi even with all image
>> processing features enabled. http://p.sf.net/sfu/kodak-com
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Sequoiadb-discuss mailing list
>> Seq...@li...
>> https://lists.sourceforge.net/lists/listinfo/sequoiadb-discuss
>>
>
>
> --
> Emmanuel Cecchet
> FTO @ Frog Thinker Open Source Development & Consulting
> --
> Web: http://www.frogthinker.org
> email: ma...@fr...
> Skype: emmanuel_cecchet
>
|
|
From: Emmanuel C. <ma...@fr...> - 2009-05-14 13:13:00
|
Hi Slawomir, Use the new seq...@li... mailing list, the mailing list at continuent.org is being discontinued. > We are currently trying to cluster our database. > I'm testing solution based on Sequoia, but performance is very poor. > About 8 database insert per second. We need reach at least 100/150 > inserts per second. You should be able to achieve 800 to 1000 inserts/sec without any fancy hardware. There is certainly a configuration problem somewhere in your stack. > We are using PostgreSQL 8.1.11 on CentOS 5.2 on 32 bit Sun JDK and > Sequoia 2.10.10 on our test environment. Both test servers are inside > vmware server and have assigned 2 cores of Intel Xeon 3.06GHz CPU's > and 2 GB RAM each. Servers have very low load. This version of Postgres is quite dated now but I think that is what ships with CentOS 5.2. > Server are named node0 and node1. Each server consist of PostgreSQL > database and Sequoia controller. > Database replication is RAIDb1. > Most of configuration details are left default from Sequoia install. > As I can see default configuration uses JGroups communication. > Each node has assigned in /etc/hosts static hostnames. > When I run any insert query on JConsole attached to each node I see > little peak, about 5-10% of CPU but nothing more. > Memory usage is mostly constant, when I run query heap size grows upto > 90MB but when query finish sequoia uses only ~20MB of Heap I am not sure I understand the memory consumption part, is that for a single query or for all queries in your workload? Unless you are inserting a big BLOB, there is no reason that so much memory is used. Could you give an example of an insert you are performing in the database? The low CPU percentage is probably due to network configuration problems. Here are some things to test: 1. Run JConsole in the same VM as Postgres and measure insert performance directly on Postgres 2. Run JConsole in a different VM and measure insert performance directly on Postgres in the other VM 3. Run JConsole on a single Sequoia controller (without group communication, remove the Distribution element) 4. Run JConsole on a single Sequoia controller with group communication enabled (re-introduce the Distribution element in your virtual database config file) Usually UDP multicast does not perform very well with virtual machines. You should try a JGroups configuration based on TCP instead, you might see a significant performance improvement. > Here's my controller.xml > > <SEQUOIA-CONTROLLER> > <Controller ipAddress="node1" port="54321"> > <JmxSettings> > <RmiJmxAdaptor port="1090"/> > </JmxSettings> > <VirtualDatabase configFile="postgres-raidb1-distribution-1.xml" > virtualDatabaseName="counter" autoEnableBackends="true"/> > </Controller> > </SEQUOIA-CONTROLLER> > > And here is my VD configuration > <SEQUOIA> > <VirtualDatabase name="counter"> > <Distribution > > <MessageTimeouts /> > </Distribution> > <Backup> > <Backuper backuperName="pgdump" > > className="org.continuent.sequoia.controller.backup.backupers.PostgreSQLBinaryBackuper" > /> > </Backup> > > <AuthenticationManager> > <Admin> > <User username="admin" password="xxx" /> > </Admin> > <VirtualUsers> > <VirtualLogin vLogin="user" > vPassword="yyyy" /> > </VirtualUsers> > </AuthenticationManager> > > <DatabaseBackend name="node1" > driver="org.postgresql.Driver" > url="jdbc:postgresql://node1/counter" > connectionTestStatement="select now()"> > <DatabaseSchema dynamicPrecision="all" > gatherSystemTables="true"> > <DefaultStoredProcedureSemantic > hasSelect="true" hasInsert="true" > hasUpdate="true" hasDelete="true" > hasDDL="false" hasTransaction="false" > isCausallyDependent="true" > isCommutative="false" /> > <StoredProcedureSemantic > procedureName="sequoia_rowcounts" > parameterCount="1" > writeTables="true" > hasSelect="true" hasInsert="true" > hasUpdate="false" > hasDelete="false" > hasDDL="false" > hasTransaction="false" > isCausallyDependent="true" > isCommutative="false" /> > </DatabaseSchema> > > <ConnectionManager vLogin="user" rLogin="postgres" > rPassword=""> > <VariablePoolConnectionManager > initPoolSize="10" > minPoolSize="5" > maxPoolSize="50" idleTimeout="30" waitTimeout="10" /> > </ConnectionManager> > </DatabaseBackend> > <RequestManager> > <RequestScheduler> > <RAIDb-1Scheduler level="passThrough" /> > </RequestScheduler> > > <RequestCache> > <MetadataCache /> > <ParsingCache /> I would recommend to remove the ResultCache element below: > <ResultCache granularity="database" /> > </RequestCache> > > <LoadBalancer> > <RAIDb-1> As you have a single database attached to your controller, use WaitForCompletion policy="all" instead, this will be faster. > <WaitForCompletion > policy="first" /> > > <RAIDb-1-LeastPendingRequestsFirst /> > </RAIDb-1> > </LoadBalancer> > <RecoveryLog driver="org.postgresql.Driver" > url="jdbc:postgresql://node1/recovery" > login="sequoia" password=""> > <RecoveryLogTable /> > <CheckpointTable /> > <BackendTable /> > <DumpTable /> > </RecoveryLog> > </RequestManager> > </VirtualDatabase> > </SEQUOIA> > sequoia_sequencer.xml [used in jgroups.properties]: > > --> > <config> > <UDP mcast_port="45566" > mcast_addr="228.8.8.9" > tos="16" > ucast_recv_buf_size="20000000" > ucast_send_buf_size="640000" > mcast_recv_buf_size="25000000" > mcast_send_buf_size="640000" > loopback="false" > discard_incompatible_packets="true" > max_bundle_size="64000" > max_bundle_timeout="30" > use_incoming_packet_handler="true" > use_outgoing_packet_handler="false" > ip_ttl="2" > down_thread="false" up_thread="false" > enable_bundling="true"/> > <PING timeout="2000" > down_thread="false" up_thread="false" num_initial_members="3"/> > <MERGE2 max_interval="10000" > down_thread="false" up_thread="false" min_interval="5000"/> > <FD_SOCK down_thread="false" up_thread="false"/> > <!--VERIFY_SUSPECT timeout="1500" down_thread="false"/--> > <pbcast.NAKACK max_xmit_size="60000" > use_mcast_xmit="false" gc_lag="0" > retransmit_timeout="100,200,300,600,1200,2400,4800" > down_thread="false" up_thread="false" > discard_delivered_msgs="true"/> > <UNICAST timeout="300,600,1200,2400,3600" > down_thread="false" up_thread="false"/> > <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" > down_thread="false" up_thread="false" > max_bytes="400000"/> > <VIEW_SYNC avg_send_interval="60000" down_thread="false" > up_thread="false" /> > <pbcast.GMS print_local_addr="true" join_timeout="3000" > down_thread="false" up_thread="false" > join_retry_timeout="2000" shun="true" > handle_concurrent_startup="true" /> > <SEQUENCER down_thread="false" up_thread="false" /> > <FC max_credits="2000000" down_thread="false" up_thread="false" > min_threshold="0.10"/> > <!-- FRAG2 frag_size="60000" down_thread="false" up_thread="true"/ --> > <!-- pbcast.STATE_TRANSFER down_thread="false" up_thread="false"/--> > </config> > > > What can be wrong. What we can do to increase performance? Keep us posted with the performance numbers with the different configurations proposed above. > I've been browsing list and I found something about group > communication protocol change and use appia but I can't find anything > about proper configuration, When I change distribution element of > configuration to use hedera_appia.properties I'm getting No Route to > host at controller output. If you use TCP, you will have to hardcode the IP addresses in the configuration files or use a gossip server. That might be the issue. Keep us posted with your progress, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-13 22:09:41
|
Hi Richard, Can you reproduce the problem systematically? Could you post your virtual database configuration file? As the request is executing, it might be blocked on a lock (that would be a bug) or waiting for MySQL backends to complete (we would then have to figure out why MySQL is blocked). Do you see any pending request on any MySQL backend (when you log directly on MySQL)? Both backends show as enabled in Sequoia? Thanks for your feedback, Emmanuel > I encounter a strange problem when using Alfresco that uses Hibernate > with Sequoia. > Sequoia is configured with 1 virtual database using 2 MySQL backends. > What happens is that when Alfresco first starts, it creates its > tables successfully, then issues a number of SELECTS successfully and > then issues a "INSERT" which gets stuck in Sequoia request queue forever. > > Is there anything I can do to find out what Sequoia is waiting for? > > Here are the details of the request: > > insert into alf_attributes (version, acl_id, type) values (?, ?, 'M'): > Id=523, query=insert into alf_attributes (version, acl_id, type) > values (?, ?, 'M'), params=<!%L|0|!%><!%N|-5|!%>, transaction id=324, > persistent connection id=0 > > alfresco(admin) > dump scheduler queues > Active transactions: 1 > +-----+-------------+ > | tid | time (in s) | > +-----+-------------+ > | 324 | 76247 | > +-----+-------------+ > > ---------------- > > Pending read requests: 0 > Read request id list: > > Pending write requests: 1 > Write request id list: > 523 > > ---------------- > > alfresco(admin) > dump transaction 324 > +--------+----------+-----------------------------------------------------------------------+-----------------------+----------------+----------------+------------+-------------+-----------+--------------+ > | log_id | vlogin | sqlStmt > | sqlStmt_param | auto_conn_tran | > transaction_id | request_id | exec_status | exec_time | update_count | > +--------+----------+-----------------------------------------------------------------------+-----------------------+----------------+----------------+------------+-------------+-----------+--------------+ > | 1 | alfresco | begin > | null | T | 324 > | 0 | S | 0 | -1 | > | 2 | alfresco | insert into alf_attributes (version, acl_id, > type) values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%> | T | > 324 | 523 | E | 2 | 0 | > +--------+----------+-----------------------------------------------------------------------+-----------------------+----------------+----------------+------------+-------------+-----------+--------------+ > > ---------------- > alfresco(admin) > dump request 523 > Request id: 523 > query: insert into alf_attributes (version, acl_id, type) values > (?, ?, 'M') > parameters: <!%L|0|!%><!%N|-5|!%> > login: alfresco > autocommit: false > transaction id: 324 > cacheable status: CACHEABLE > isolation level: TRANSACTION_UNDEFINED > start time: 1242150743853 > end time: 0 > timeout in seconds: 0 > locked tables: > alf_attributes > persistent connection id: 0 > client ip address: /127.0.0.1 > > > Thanks for any help. > > Richard Vigeant > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your > production scanning environment may not be a perfect world - but thanks to > Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 > Series Scanner you'll get full speed at 300 dpi even with all image > processing features enabled. http://p.sf.net/sfu/kodak-com > ------------------------------------------------------------------------ > > _______________________________________________ > Sequoiadb-discuss mailing list > Seq...@li... > https://lists.sourceforge.net/lists/listinfo/sequoiadb-discuss > -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
|
From: Richard V. <ric...@va...> - 2009-05-13 16:46:43
|
Hi,
I encounter a strange problem when using Alfresco that uses Hibernate
with Sequoia.
Sequoia is configured with 1 virtual database using 2 MySQL backends.
What happens is that when Alfresco first starts, it creates its tables
successfully, then issues a number of SELECTS successfully and then
issues a "INSERT" which gets stuck in Sequoia request queue forever.
Is there anything I can do to find out what Sequoia is waiting for?
Here are the details of the request:
insert into alf_attributes (version, acl_id, type) values (?, ?, 'M'):
Id=523, query=insert into alf_attributes (version, acl_id, type)
values (?, ?, 'M'), params=<!%L|0|!%><!%N|-5|!%>, transaction id=324,
persistent connection id=0
alfresco(admin) > dump scheduler queues
Active transactions: 1
+-----+-------------+
| tid | time (in s) |
+-----+-------------+
| 324 | 76247 |
+-----+-------------+
----------------
Pending read requests: 0
Read request id list:
Pending write requests: 1
Write request id list:
523
----------------
alfresco(admin) > dump transaction 324
+--------+----------
+
-----------------------------------------------------------------------
+-----------------------+----------------+----------------+------------
+-------------+-----------+--------------+
| log_id | vlogin |
sqlStmt
| sqlStmt_param | auto_conn_tran | transaction_id | request_id
| exec_status | exec_time | update_count |
+--------+----------
+
-----------------------------------------------------------------------
+-----------------------+----------------+----------------+------------
+-------------+-----------+--------------+
| 1 | alfresco |
begin
| null | T | 324 | 0
| S | 0 | -1 |
| 2 | alfresco | insert into alf_attributes (version, acl_id,
type) values (?, ?, 'M') | <!%L|0|!%><!%N|-5|!%> | T |
324 | 523 | E | 2 | 0 |
+--------+----------
+
-----------------------------------------------------------------------
+-----------------------+----------------+----------------+------------
+-------------+-----------+--------------+
----------------
alfresco(admin) > dump request 523
Request id: 523
query: insert into alf_attributes (version, acl_id, type) values
(?, ?, 'M')
parameters: <!%L|0|!%><!%N|-5|!%>
login: alfresco
autocommit: false
transaction id: 324
cacheable status: CACHEABLE
isolation level: TRANSACTION_UNDEFINED
start time: 1242150743853
end time: 0
timeout in seconds: 0
locked tables:
alf_attributes
persistent connection id: 0
client ip address: /127.0.0.1
Thanks for any help.
Richard Vigeant
|
|
From: Heshel R. <hs...@gm...> - 2009-05-08 22:48:02
|
Hi Emmanuel, On Thu, May 7, 2009 at 6:21 PM, Emmanuel Cecchet <ma...@fr...>wrote: > Hi Heshel, > >> I have a couple of questions concerning the replication of Sequoia. Some >> background info, we have two locations that we are in the process of setting >> up. In each location will be two controllers, currently sitting on the same >> box as the application server. These controllers then have one or more >> PostgreSQL database boxes for backends. We will also be setting up a pair of >> switches for the internal communications in each location. Hopefully the >> duplication will allow me to sleep soundly if we ever have a piece of >> hardware fail.. ;) We are running FreeBSD for the under lying OS, and the >> binary of Sequoia 2.10.10. >> >> First, I am attempting to use a dedicated link between the controllers in >> the same physical location, however I cannot see to get the controllers to >> use the IP of this interface for replication. I will be removing my current >> configuration and starting fresh in hopes that it may just be an error in >> them, but I thought I would also ask here if can it be done while I work >> that out. >> > You can force which IP address you want the controller to bind in the > controller configuration file. Sorry, I guess I laid out the points in a bad order, as this "first" question did not come up until after the second one.. I have set the controller IPs via the configuration file, and I have set another IP from the second interface for the replication. I did this based on the hope that the clients would continue to connect to the public interface, while the controllers would move the replication communication to the dedicated crossover between them. I have not had a chance to redo the configuration to attempt this again. > > Second, I did have a working configuration for using a single nic for all >> communication, however, there was some issues with it. The controllers would >> disconnect shortly after forming the cluster, upon checking my logs they >> were using ports outside of what I had configured the hedra/jgroups >> configuration files for. I had configured the controllers to use 7900 as the >> communication port, which they did use, but none of the other ports were >> defined by myself to connect to. (This is a reason I am looking at the >> separate interface for the controller communication, as opening pf right up >> on the main interface(s), or turning it off is not an option. However, >> having a crossconnect between the two servers that is unhindered is >> acceptable.) >> > Are you sure that the controller is using your configuration file? It will > use the first one it finds in the classpath. > I am sure that the configuration file is being loaded. I had stumbled upon the naming issue with the default sequencer.xml, I changed my configuration file to be called jgroups_tcp.xml, and made sure the hedra_jgroups.properties files was updated to reflect that. > > Third I am required to setup replication between 2 physical locations. >> Since the plan is to use Sequoia for replication in each location, there >> have been talks of using it for the wan replication as well. I am curious >> about peoples experience with such a setup. As another possibility, I have >> been looking into the use of the tungsten connector between the controllers >> at each site. >> > For Tungsten, I guess you will have to ask on the Tungsten mailing list. I > have not been following their development closely (shame on me) and I am not > sure how this would work. > You can try to setup a tunnel for the group communication between the 2 > sites but latency is probably going to be a penalty. Another option is to > use the remote controller as an additional backend of the local controller. Latency is the one reason I have been trying to convince the designer to fold everything into one site, that has proper redundancy, but that is neither here nor there. The remote controller as a backend is an interesting idea, which would limit the traffic to a known port for each remote controller. It would be best in that case to add all remote controllers as backends for redundancy, however I see an issue with bandwidth there.. Would there be a way to have a cluster to cluster communication as a single stream, that would be able to failover between controllers? So the primary controller in each cluster would have a channel open for passing the data, most likely async, that if one of the primaries failed, the secondary would resume the role? They would use the primary's recovery log that is still up to ensure consistancy? As I continue to think about this, having the sequoia controllers share an IP via CARP or other type of ip clustering could allow the single stream possibly.. > > Some additonal questions; >> >> FreeBSD rc scripts? I did some digging in the past to see if this was >> possible, found some talk of it but nothing more. >> > There are some scripts for Linux but nothing for FreeBSD as far as I know. Doh. Maybe I might have to look into the creation of one, as it would help ease my management steps of the servers if I could ensure Sequoia is up before Tomcat, and the databases are sync'd before a shutdown/reboot with minimal effort. > > Probably a very common question, new releases of Sequoia? I know working >> is being done on 4.x, and possibly 2.10.11? Just curious if there is any >> timelimes? >> > Sequoia 2.10.11 if it ever comes out will be provided by Continuent. Not > sure if this will ever happen. > Sequoia 4 relies on me. I have been very busy with my 2 other jobs recently > and I have had no time to make progress. I have to make a new release > (probably a beta), if possible this month, otherwise in June. The problem is > that I have to redo the documentation and that will take time. Also I have > no access to the test suite that was developed at Continuent so the > debugging phase might be more painful. Thanks for the info! An additonal question, has Sequoia 4 been relatively stable? Stable enough that it is used in a production environment even though it would be classed as alpha/beta? > > > Keep us posted with your progress. > Emmanuel > > -- > Emmanuel Cecchet > FTO @ Frog Thinker Open Source Development & Consulting > -- > Web: http://www.frogthinker.org > email: ma...@fr... > Skype: emmanuel_cecchet > > |
|
From: Emmanuel C. <ma...@fr...> - 2009-05-08 02:34:01
|
Stefano, The mailing list has now moved to SourceForge. You can post to seq...@li... > Hi everybody, and thanks for your effort in open source projects like Sequoia. > > My company colleagues and I studied C-JDBC for one of our project, but > we was not able to use it (the project was based on stored > procedures). > Now, after some years, we are facing a new project from scratch, and > I'm investigating the use of Sequoia to guarantee high availability of > our system. > > I'm a bit confused about current state of Sequoia. > > I installed v. 2.10.10 but it seems to have evident problems, I wonder > if there is someone working on it (2.10.10 has been released one year > ago, 2.10.9 two years ago). The most evident problem is the restore > option in the console, which duplicates the path of the backup file (i > had to manually hack the config files to get the restore working, > which is not a brilliant start). > I did not get what your problem was. What backuper are you using? How did you configure it? What command line did you use in the console? What do you mean by backing the config files? What database engine are you using? > So I searched for the sequoia 4, but it seems to be not available in > any way. No cvs working on Continuent forge. No branch on sourceforge. > Sequoia 4 is the CVS HEAD on SourceForge. There has been no release so far, I had not find the time yet to release a beta version of it. I have to write the documentation and work on a new test suite, and as I am the only one working on it, that will definitely take time. The CVS has moved to SourceForge, so there is no CVS anymore on Continuent. You can access it at http://sourceforge.net/scm/?type=cvs&group_id=252061 > Can you confirm that Sequoia is a dying tree, or it is my fault, and I > was not able to find the right stuff? > Sequoia 2.10 is still probably supported by Continuent but their effort is now focused on Tungsten and I guess there is not much interest from them to continue much further with Sequoia. I am still working on Sequoia 4 on my free time but I currently have 2 other jobs that keep me quite busy. So Sequoia 4 is moving at a very slow pace. Thanks for your interest in Sequoia, Emmanuel -- Emmanuel Cecchet FTO @ Frog Thinker Open Source Development & Consulting -- Web: http://www.frogthinker.org email: ma...@fr... Skype: emmanuel_cecchet |
