DBMS doesn't disable after shutdown

[Jean-Philippe MINETTI]

Hi Paul,

I created a test application to test HA-JDBC with:
- PostgreSQL;
- Hibernate;
- Spring + Spring Data JPA;

When I stop my second DBMS (databaseB), it is disabled and updates are done on my first DBMS.

But if I stop my first DBMS (databaseA), it doesn't disable and each operation to the database throws exceptions.

For this case, I observed:
- in step mode (debug): it works;
- if I set a Thread.sleep of 20 seconds between each database operation: it works;
- with 1 database operation per second: it doesn't work;

My ha-jdbc-mycluster.xml:

<?xml version="1.0" encoding="UTF-8"?>
<ha-jdbc xmlns="urn:ha-jdbc:cluster:3.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:ha-jdbc:cluster:3.0
                             http://ha-jdbc.github.com/xsd/ha-jdbc-3.0.3.xsd">

    <distributable id="jgroups">
    </distributable>

    <sync id="full">
    <property name="fetchSize">1000</property>
    </sync>

    <cluster dialect="postgresql" default-sync="full" balancer="simple" durability="fine" auto-activate-schedule="*/10 * * ? * *">
        <database id="databaseA" location="jdbc:postgresql://192.168.0.23/baseB" weight="1">
            <user>myusername</user>
            <password>*********</password>
        </database>
        <database id="databaseB" location="jdbc:postgresql://192.168.0.32/baseB" weight="1">
            <user>myusername</user>
            <password>*********</password>
        </database>
    </cluster>

</ha-jdbc>

My entityManagerFactory and datasource configuration (Spring):

<!-- entityManagerFactory -->
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="dataSource" ref="clusterDataSource"/>
    <property name="persistenceUnitName" value="cluster"/>
    <property name="jpaVendorAdapter">
        <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
    </property>
</bean>

    <!-- clusterDataSource -->
<bean id="clusterDataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value="net.sf.hajdbc.sql.Driver"/>
    <property name="url" value="jdbc:ha-jdbc:mycluster"/>
    <property name="username" value="myusername"/>
    <property name="password" value="*********"/>
</bean>

My META-INF/persistence.xml:

<persistence xmlns="http://java.sun.com/xml/ns/persistence"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
                                 http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
             version="2.0">

    <persistence-unit name="cluster">
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <class>com.lrd.hermes.test.object.TransactionS1</class>
        <class>com.lrd.hermes.test.object.TransactionS2</class>
        <properties>
            <property name="hibernate.connection.driver_class" value="net.sf.hajdbc.sql.Driver"/>
            <property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect"/>
            <property name="hibernate.show_sql" value="false"/>
        </properties>

    </persistence-unit>

</persistence>

Thanking you in advance,

Jean-Philippe MINETTI

[Paul Ferraro]

Did you re-activate databaseB prior to attempting to stop databaseA? HA-JDBC will persist the fact that databaseB is inactive - even after restarting your test/application. If this is the case, your cluster has only 1 remaining active database (databaseA) and HA-JDBC will never deactivate the last database node of a cluster, unless specifically configured to do so (i.e. <cluster allow-empty-cluster="true"/>)

[Jean-Philippe MINETTI]

When starting my test the two databases are active. I checked with jconsole (JMX). By done, if I stop databaseA, it's databaseB which is disabled. It's very strange because it seemed to me yesterday that the two databases were still active.

I tried the allow-empty-cluster property for see: no change.

I disabled the cache (state): it's the same.

If I execute the isAlive method on cluster MBean, I have:
databaseA = false
databaseB = true
Which is correct for a shutdown of databaseA.

If I try to call the activate method of cluster MBean for databaseB, I throw an InvocationTargetException in jconsole.

As against, if:
* I set allow-empty-cluster="true";
* I stop databaseA: it's databaseB is disabled;
* I call the activate method for databaseB: it throw an InvocationTargetException;
* I call the deactivate method for databaseA;
* and I call the activate method for databaseB: it works.

Thank you,

Jean-Philippe MINETTI

[Paul Ferraro]

Can you provide more context for the InvocationTargetException? What stacktrace do you see in your log?

[Jean-Philippe MINETTI]

Hi Paul,

Attached to this post, the logs with exception after the attempt to activation of databaseB.

The InvocationTargetException appears in jconsole, but in the logs this is a JDBC exception (Connection refused. Check that the hostname and port are correct and that the postmaster accepting TCP / IP connections).

Regarding logs, in trace mode I have nothing else.

Other thing less important, in the logs, the WARN trace appears like this:
Na pas activé la base de données {0} du faisceau {1}

With MessageFormat, single quotes is used to not format text. You must double single quotes to ensure correct message as this:
N''a pas activé la base de données {0} du faisceau {1}

Jean-Philippe MINETTI

[Jean-Philippe MINETTI]

Hi Paul,

If you are unable to reproduce the problem, I can send you my test application (16.5 MB for binary) ?

Jean-Philippe MINETTI

[Paul Ferraro]

Sorry about those error messages - they were auto-generated (and probably out-of-date) and it looks like the quotes were mangled in the process. In master, I'm using gettext for log/error message i18n, for which users can contribute translations.

The last time I saw someone with this issue was because they had 2 server instances running HA-JDBC, where the definitions for each database were not the same (i.e. one server had databaseA pointing to 192.168.0.23, and databaseB pointing to 192.168.0.32; while the other server had databaseA pointing to 192.168.0.32 and databaseB pointing to 192.168.0.23). Are you certain this isn't the case?

You say that you stopped databaseA, but databaseB was disabled. There should be a log message that indicates why databaseB was disabled. Can you post that?

[Jean-Philippe MINETTI]

No, for the last test, I used a single server instance running HA-JDBC. It can not be that.

Attached to this post, my last logs just before databaseA stopping.

A small detail that may be important, I have two threads (UPDATE and INSERT in logs).

Jean-Philippe MINETTI

[Jean-Philippe MINETTI]

Hi Paul,

Were you able to reproduce the problem ?

If not, I can send you the binaries and/or sources of my test application ?

Jean-Philippe MINETTI

[Paul Ferraro]

Can you post a test that reproduces the problem?

[Jean-Philippe MINETTI]

Hi Paul,

Attached to this post, the binary.

To run:
you need to create empty PostgreSQL databases;
unzip the tar.gz;
configure access to databases in files etc/datasources.xml and etc/ha-jdbc-mycluster.xml;
type "bin/run create" to create the structure of databases on 2 DBMS;
* type "bin/run test2 --instance=XXX --insert=2 --update=1" to start the test.

...

[Jean-Philippe MINETTI]

...

and attached to this post, the source.

Jean-Philippe MINETTI