SymmetricDS low performance when the number of records in sym_outgoing_batch grows

Developers
2013-02-22
2013-02-24
  • Enrique Altuna

    Enrique Altuna - 2013-02-22

    Hello for everybody.

    I have been using SymmetricDS for a while. In my business some nodes can(and it is the normal behavior) be without network access for long time periods.

    Recently, the performance of the system has significantly dropout. When I have check the sym_outgoing_batch table I can see about 2000000 batches waiting to be syncronized to some nodes, some of then with no connections in the last month. Also about 20000000 records in sym_data_event. There are a lot of querys for sym_outgoing_batch, sym_data, sym_data_event and a lot of joins.

    My request is about a new functionality. One where you can configure a max_count of outgoing batches to be in the sym_outgoing_batch. Other batches for this node can be temporarily moved to other table.
    A new job can be created to do this work.

     
  • Chris Henson

    Chris Henson - 2013-02-24

    That definitely sounds like a scenario that we should be able to handle better. Instead of moving the data to another table, I wonder if it would be better to go ahead and extract the data to the file system so the data can be purged from the tables. I would be pretty easy to create a job that does that. Maybe we could add a field on sym_outgoing_batch that indicates that the data has already been extracted. Downside might be that the file system might be a bit more susceptible to corruption and might take up more file system space.

    Thoughts, anyone?

     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks