#452 GTM failure or restart may cause data loss and inconsistency

1.0 Dev Q
closed
nobody
None
9
2013-11-27
2013-09-14
No

We discovered major XC issue, that may lead to data loss and transactional inconsistency.
The source of the problem the GTM does not persist info about transactions.

Steps to reproduce:

Start cluster and open psql session

$ psql -d postgres

Create test tables

postgres=# create table t1 (a int, b int);
CREATE TABLE
postgres=# create table t2 (a int, b int);
CREATE TABLE

Start a transaction and insert test rows

postgres=# begin;
BEGIN
postgres=# insert into t1 values (1,1);
INSERT 0 1
postgres=# insert into t2 values (2,2);
INSERT 0 1

Check the data are here

postgres=# select * from t1 union all select * from t2;
 a | b 
---+---
 1 | 1
 2 | 2
(2 rows)

Now restart GTM to simulate failure

$ gtm_ctl -D data-g -Z gtm restart
waiting for server to shut down.... done
server stopped
server starting

Select from one of the tables in a new psql session

$ psql -d postgres
postgres=# select * from t1;
 a | b 
---+---
(0 rows)

Table is empty, that is correct - the transaction is not committed yet
Return to the first session, commit the transaction and check data:

postgres=# commit;
COMMIT
postgres=# select * from t1 union all select * from t2;
 a | b 
---+---
 2 | 2
(1 row)

One row is here, other is disappeared.

What's going on behind the scene:
When GTM is restarted it "forgets" about active transaction. So transaction on second session is the only known to GTM and when it is about to read data it receives respective snapshot.
When t1 is scanned there is one newly inserted tuple. According to the snapshot, the inserting transaction has been finished already, and it is checked, if the transaction has been committed. It hasn't so visibility flag is set on the tuple, to shortcut further visibility checks of the tuple, making the tuple invisible. Second table is read for the first time after the commit, so opposite visibility flag is set on it.

Discussion

  • mason_s

    mason_s - 2013-09-18

    After some discussion internally here, I think we should lower the severity.

    The primary workaround here is that if GTM goes down, one should shutdown the other components, then restart GTM and then the other components to be safe. Just restarting GTM without stopping the other components can lead to the problems outlined above, especially if a vacuum freeze is going on in the background.

    The other option is to use GTM standby and have failover occur to that.

     
  • Koichi Suzuki

    Koichi Suzuki - 2013-11-27

    GTM slave will solve a part of this problem. On the other hand, we have the following corner case.

    When GTM fails and existing transaction tries to commit before GTM slave fails over and GTM proxy reconnects, it fails to report this to GTM and aborts.

    We can save this case as follows:

    1. If the transaction has received snapshots, it can continue with this snapshot.
    2. It can commit or abort locally but has to write this to some place (new catalog?).
    3. When GTM connection is back, coordinator/datanode reports GTM the result of such locally-commited/aborted transactions.

    This needs additional infrastructure.

    I'd like to close this and continue the above discussion/design as a new feature.

    -- Koichi

     
  • Koichi Suzuki

    Koichi Suzuki - 2013-11-27
    • status: open --> closed
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks