sym_data sym_data_event keeps growing

Help
2012-06-29
2012-09-15
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-06-29

    Hi, i am using SymmetricDS 1.7.6 to replicate PostgreSQL databases in Linux,
    at the stores the purge jobs works fine and sym_data it's correlated with
    sym_outgoing_batch but in the root node i found today that the
    min(create_time) of sym_data it's from 2010 and i have 18926276 rows right now
    in sym_data and 35551772 in sym_data_event.

    sym_outgoing_batch it's fine, have 146245 rows and the min(create_time) it's
    from 2012-06-24, i have checked the configuration and are the same at root and
    store nodes, what could be happenning and how could i fix it? Thanks!

    Regards,

    Miguel Angel.

     
  • Chris Henson
    Chris Henson
    2012-06-29

    Does the log file have errors?

     
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-06-29

    no, i can't find any error in the logs.

     
  • Chris Henson
    Chris Henson
    2012-06-29

    Does the purge process run and delete 0 rows? Is the data in sym_data valid
    data that had not been sent?

     
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-06-29

    This is the last process, it seems to delete 0 rows at sym_data, yes.

    2012-06-29 19:32:53,737 INFO The outgoing purge process is about to run.

    2012-06-29 19:32:53,737 INFO Getting range for outgoing batch

    2012-06-29 19:32:53,861 INFO About to purge sym_data_event

    2012-06-29 19:32:53,869 INFO Done purging 0 of sym_data_event rows.

    2012-06-29 19:32:53,869 INFO About to purge sym_outgoing_batch

    2012-06-29 19:32:53,869 INFO Done purging 0 of sym_outgoing_batch rows.

    2012-06-29 19:32:53,870 INFO About to purge sym_outgoing_batch_hist

    2012-06-29 19:32:54,002 INFO Done purging 0 of sym_outgoing_batch_hist rows.

    2012-06-29 19:32:54,002 INFO Getting range for data

    2012-06-29 19:32:54,655 INFO About to purge sym_data

    2012-06-29 19:35:39,744 INFO Loading properties file from class path resource

    2012-06-29 19:35:39,745 INFO Loading properties file from class path resource

    2012-06-29 19:35:39,745 WARN Could not load properties from class path
    resource : class path resource cannot be opened because it does not exist

    2012-06-29 19:35:39,745 INFO Loading properties file from class path resource

    2012-06-29 19:35:39,745 WARN Could not load properties from class path
    resource : class path resource cannot be opened because it does not exist

    2012-06-29 19:35:39,745 INFO Loading properties file from URL

    2012-06-29 19:37:24,462 INFO Done purging 0 of sym_data rows.

    2012-06-29 19:37:24,469 INFO The outgoing purge process has completed.

    2012-06-29 19:37:24,469 INFO The incoming purge process is about to run.

    2012-06-29 19:37:24,469 INFO Getting range for incoming batch

    2012-06-29 19:37:24,524 INFO About to purge sym_incoming_batch_hist

    2012-06-29 19:37:24,708 INFO Done purging 33 of sym_incoming_batch_hist rows.

    2012-06-29 19:37:24,708 INFO About to purge sym_incoming_batch

    2012-06-29 19:37:24,812 INFO Done purging 33 of sym_incoming_batch rows.

    2012-06-29 19:37:24,813 INFO The incoming purge process has completed.

    2012-06-29 19:37:24,813 INFO The statistic purge process is about to run.

    2012-06-29 19:37:24,839 INFO Purged 4 statistic rows.

    2012-06-29 19:37:24,839 INFO The statistic purge process has completed.

     
  • Mark Hanes
    Mark Hanes
    2012-06-29

    Purge looks to be running, but purge will only purge data that has already
    been routed and thus is no longer needed. Is the "old" data you expected to be
    purged destined for a node that doesn't exist, for example?

     
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-06-29

    yeah it seems to be the case, the node that it's locking the delete of
    sym_data it's a closed store, how should i proceed? marking the node at
    sym_node with sync_enabled = 0 and update all sym_data_event batches as
    "batched = 1"?

     
  • Mark Hanes
    Mark Hanes
    2012-06-30

    That should do the trick. Be prepared for a long purge run the next time purge
    process starts due to the volume of data to be purged.

     
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-06-30

    Ok, i will try it.

    Thanks for help! :)

     
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-07-01

    This changes don't seems to be enough, still i have 0 rows deleted in sym_data
    :(

     
  • Chris Henson
    Chris Henson
    2012-07-01

    It has been 3 years since I've looked at 1.7 purge code. It looks like you'll
    just have to come up with a strategy to delete sym_data_event manually. If you
    delete sym_data_event then the purge process should clean up sym_data.

    I'd suggest deleting the sym_data_event row in chunks (not all of them at the
    same time).

     
  • Miguel Angel Rasero
    Miguel Angel Rasero
    2012-07-01

    ok, i will delete sym_data_event rows manually in chunks, thanks.