Synchronization Workarounds

Help
Andrew
2012-08-07
2014-05-02
1 2 > >> (Page 1 of 2)
  • Andrew

    Andrew - 2012-08-07

    A small team of us here share a single KeePass database and our changes often get out of step.  For whatever reason the synchronization tool corrupts the database when we try to synchronize two of the most recently edited database files from two team members.   For some bizarre reason it often (about 25% of the time) deletes several entries even when these entries were visible in both of the original databases that are being synchronized.

    Because the synchronization tool gives no warnings or even an output log and instantly saves all of it's changes to both original databases it seems to be far too dangerous to use unless it can be guaranteed not to do something unintended (and it can't).

    So I have had to resort to manually stepping through the whole database to merge them, making sure I am not missing anything (this takes at least 30 minutes).  So I am just wondering if there are any ways of making this process a bit easier.

    Is there such a thing as a global change log or history record for the whole database?  I have tried selecting "show all entries" and ordering by last modified date but there are many groups and subgroups and I can't seem to get all of the entries to show in one large ordered list.  They remain grouped and so I at least have to step through each group looking for the most recently altered files.

    There is surely a sensible way of doing this but I can't find any mention of anyone having similar problems!

    Thanks for any pointers.
    Andrew

     
  • Paul

    Paul - 2012-08-07

    I assume you are using V2.
    The sync is based on UUID and time stamp, with "older" entries being moved to history and deleted to the Recycle Bin folder.
    Check the times on the computers you are using.
    Are the entries being deleted or moved to history?
    You could export to HTML - without passwords - and sort by time stamp.

    cheers, Paul

     
  • wellread1

    wellread1 - 2012-08-07

    One way that entry may appear to get lost is when two users edit the same entry and at least one has moved it.  However, the entry is not actually lost, it just ends up in a location that one or the other user did not expect. See topic https://sourceforge.net/projects/keepass/forums/forum/329220/topic/4414689 for additional information.  There is no global history, but each entry maintains its own history accessible via the history tab of the Edit entry dialog.

    If you can reproduce an actual loss of data problem you should describe the exact steps and configuration that cause the issue and file a bug report along with an example database (if needed).

    Otherwise I suggest you try using a central sync database along with individual local databases as described in http://keepass.info/help/kb/kb091127_trigger_examples.html#dbsync, where the database described in step 1 is a centrally located database (e.g. on a network share).  This has the advantage that the central database is only accessed for explicit synchronization purposes.    This tends to be less confusing.

    -wellread1

     
  • Andrew

    Andrew - 2012-08-13

    Thanks very much for all of these suggestions, I'm quite sure some combination of these will do the trick.  I'll update the thread when I next get a chance to work on this!

    Andrew

     
  • Andrew

    Andrew - 2012-08-14

    I'm sorry but I will not be able to produce a series of steps to reproduce the problem (too many time contraints on me right now).

    But I have copies of two versions of the database saved by two different developers.  One is 196KB and the other is 197KB.  When I sync them, they become 55KB each.  I can see many entries have disappeared, even when these entries were in both of the original databases.  Also there are *fewer* entries in the synced Recycle Bin folder.  In fact there are only 2 entries there while there are 3 entries in both of the original databases.  Unless there is some kind of strange compression going on, this seems to prove that data is being lost.

    I was in the process of setting up a trigger based sync to a central database as suggested.  But since the sync process breaks even in the simple situation described above, presumably I wouldn't be able to rely on the central sync database option either.  - Each sync operation with the central database could end up deleting data in a similar way.

    I'll try the HTML export option and use that to merge manually.

     
  • wellread1

    wellread1 - 2012-08-14

    It sounds like you have some entries that could give you some insight when you have the time to investigate.

    A few questions and suggestions:

    1. What is the origin of the two databases?  Were they created life as completely different databases, or as a copy of each other? If they were originally different new databases the first sync will behave more like an import than a sync.

    2.  Verify that all  the database settings of the two databases are identical (File->Database Settings).  Particularly verify that "Use a recycle bin" is checked in both database and that the history settings are identical.  If history is different, set both databases to match the settings of the database with the least restrictive rules.  Save the databases, check the respective sizes, then try syncing.  Is the result different?

    3.  Make a copy of each database and sync it with its copy.  Is the synced copy different from the original in any way?  If it is, determine how it is different.

    4. In each of the databases delete all but one of the entries, leaving just one problem entry that is common to both databases. Then sync the two databases. Does the common entry disappear?  If not what happens to it?

    5.  Do the disappearing entries that are common to both databases have the same UUID?  Are they originally identical or do they contain small differences?

    -wellread1

     
  • Paul

    Paul - 2012-08-15

    History settings (File > Database Settings > Advanced) specify maximum size for history and you may be losing a large attachment which may explain the file size difference.
    It is worth exporting to HTML and checking the values of entries that seem to have disappeared.

    cheers, Paul

     
  • illovo

    illovo - 2012-08-19

    WIN 7, 64 bit, Keepass 2.19

    After several sync tests that ended in many many lost entries here is a description of what I did.

    My target is to sync step by step several kdbx files to have 1 in the end with all entries.

    1 Laptop. Local directory with original.kdbx.

    USB stick. Create new: target.kdbx. Save. Closed Keepass (just because I thought may be…).

    Opened target.kdbx. Opened original.kdbx. and checked:.

    2. Verify that all the database settings of the two databases are identical (File->Database Settings). Particularly verify that "Use a recycle bin" is checked in both database and that the history settings are identical. If history is different, set both databases to match the settings of the database with the least restrictive rules. Save the databases…

    Closed Keepass.

    Checked the sizes:
    original.kdbx 54KB
    target.kdbx 3KB

    Opened Keepass (target.kdbx).

    Synced.

    I have 2 identical copies now. (with lots of double empty groups).
    original.kdbx 54KB
    target.kdbx 54KB

    Now I tried a second sync the same way with another file: original2.kdbx

    Checked the sizes:
    original2.kdbx 110KB
    target.kdbx 54KB

    Afterwards:
    Just 2 of 24 groups were copied to target.kdbx.
    1 entry lost in 1 group.
    All older groups (origin was original.kdbx) are lost in target.kdbx.
    original2.kdbx is a copy of target.kdbx. Most entries are lost.

    original2.kdbx 29KB
    target.kdbx 29KB

     
  • develop1

    develop1 - 2012-08-19

    I have never tried a sync the way you desccribed.
    In all my usage of sync every .kdbx file was a clone of the original .kdbx file as some point in historcal time.
    I successfully have used sync the following two ways:

    case a)
    copy  my_laptop.kdbx  my_traveling.kdbx
    copy                                my_traveling.kdbx       my_pc.kdbx

    After having peformed the above copies I put triggers on the laptop and PC that says when I open the local.kdbx if the traveling.kdbx is present perform a sync with it.   By doing this I can use the USB keychain device to keep the laptop and pc kdbx current with each other.

    More  recently I have been doing this:
    case b)
    copy My_laptop.kdbx   my_dropbox.kdbx
    the my_dropbox.kdbx is of course on the dropbox folder and is pushed into the dropbox cloud within seconds.

    Like case "A" my PC has a file called "my_pc.kdbx" which is a copy of my_laptop.kdbx from some point in time.
    The pc also a "my_dropbox.kdbx" 
    Both the Laptop and the PC use the same drop box account and the dropbox service does its best to keep the most current "my_dropbox.kdbx" file present on both the PC and Laptop at all times.
    Both the laptop and PC have triggers configured to sync with their respective dropbox.kdbx file each time I open/unlock the local .kdbx.

    Once you have all the above configuration in place: when on the laptop if I make a change to the local file. the very next time I unlock/open the local .kdbx a sync will occur to the dropbox file.  Ideally, within seconds the dropbox file goes to the clould and the cloud then pushes that dropbox file onto the pc.
    sometime later when I go the PC and open/unlock the local.kdbx an sync with the dropbox .kdbx occurs and I have new entries on the PC.
    The very worse case I have ever seen was I make a change on the laptop and a different change on the PC. Which ever change occured last is the dropbox file that gets pushed by the cloud o the other side. so when the each side does their open/unlock the dropbox file being sync'd might not not not contain the change of the other side.  But even this is ok, because the other side still has it. Over time as one opens/unlocks the database on each side a sync occurs and eventually dropbox will have the most recent change pushed to the other side and eventually all .kdbx files are in sync with each other.

    So far I have never seen entries "lost".

     
  • wellread1

    wellread1 - 2012-08-19

    1 Laptop. Local directory with original.kdbx. USB stick. Create new: target.kdbx. Save. Closed Keepass (just because I thought may be…). Opened target.kdbx. Opened original.kdbx. and checked:. " 2. Verify that all the database settings of the two databases are identical (File->Database Settings). Particularly verify that "Use a recycle bin" is checked in both database and that the history settings are identical. If history is different, set both databases to match the settings of the database with the least restrictive rules. Save the databases… " Closed Keepass. Checked the sizes: original.kdbx 54KB target.kdbx 3KB

    The reason you are seeing duplicate groups is because KeePass sync does not merge groups that have the same name but that were created in different databases, it imports them as a separate group.  It also does not merge entries based on their Title.  Instead is syncs entries that have the same UUID but imports all other entries.

    To see all entries in the newly merged "target.kdbx" place your cursor in the KeePass search box and press ENTER.  I suspect they will appear.

    -wellread1

     
  • illovo

    illovo - 2012-08-19

    Instead is syncs entries that have the same UUID but imports all other entries.

    Yes, I know. I just wanted to make clear that the first sync worked correctly.

    To see all entries in the newly merged "target.kdbx" place your cursor in the KeePass search box and press ENTER. I suspect they will appear.

    They appear after step 1. After step 2 (second sync) entries are lost as described.

    See the file sizes, too:

    Before second sync:
    original2.kdbx 110KB
    target.kdbx 54KB

    After second sync:
    original2.kdbx 29KB
    target.kdbx 29KB

    And just 2 groups were copied. No other group on same level.

    I'm trying to sync for several hours now (just "because I want to know it"). Different files, different computers, older Keepass version.

    No way.

     
  • steelej

    steelej - 2012-08-19

    I use a two stage process for synchronisation. The image (assuming it works!) shows the way in which it works.

    Computer 1 has a local Keepass database.

    Computer 2 also has a local Keypass database.

    Whenever it is appropriate this local database on either computer is synchronised with the  local copy of KeePass Master database. This can be manual through a button or on database save or whatever.

    When the local KeyPass Master on either computer is changed following a local synchronisation  this new version is replicated to the cloud copy and then to all computers sharing the copy. Note that the cloud copy is further encrypted by SpiderOak both in transit across the internet and at rest on their servers. This is why I use SpiderOak rather than DropBox

    Each copy of KeyPass Master is therefore the same once replication has completed - typically a minute or two.

    If there is a clash in updates a consistent copy will still appear at all locations. Changes to the Master may appear to have been lost but the next time local synchronisation takes place the Master will again be updated  from the Local database and therefore differences due to conflicts will be re-applied.

    I think this guarantees that no updates can be lost although I have not tested it with deletes. I have a feeling that these might not happen!

    I have not yet had any issues with this process although there is an oddity in that the master file size changes radically (approximately a factor of two) depending on which computer synchronised last. Computer 1 is 32 bit Vista and Computer 2 is W7 65 bit. There are no differences in the individual records, none are lost.

     
  • wellread1

    wellread1 - 2012-08-19

    Your reports of file size changes before and after sync, and statements about varying amounts of missing groups and entries after a sync is not enough information to allow forum members to help you.  KeePass 2.x sync has been a part of KeePass since the first alpha release in (May-2007). Data loss like you describe has not been previously reported in the 5 years since.

    Please try to reproduce your problem using two newly created databases containing no more than two entries each.  Describe how you create the databases.  Unless it is required to reproduce the problem, please keep all groups on the same level, otherwise report the group hierarchy needed to create the problem. After a failed sync, report the names of the groups that are lost. If that group name existed in only one database note whether it was in the opened database (source) or the closed database (target). Also after a failed sync report the UUID of at least one entry that was lost, the name of the group that it was in originally,  and whether that group originally existed in both databases, the source database only, or the target database only.

    -wellread1

     
  • illovo

    illovo - 2012-08-20

    Please try to reproduce your problem using two newly created databases

    I don't have any problems with 2 databases. The problem occurs when I try to merge more than 2 databases. See post 1. too.
    In post 8. I described what I've done.

    Describe how you create the databases.
    

    Default settings concerning database settings. But everybody can freely create groups, change groups, group hierarchy, delete entries, create new databases… whatever.

    At the moment we are busy with copy&paste actions and comparing databases (exported as XML) with backups of the past 2 years to get our complete single database.
    Means: When I'll find the time I'll follow your instructions…
    Thanks!

     
  • Paul

    Paul - 2012-08-20

    Maybe somehow you have some duplicate UUIDs and these are marked as deleted in "original2.kdbx".
    Try a database clean up before the next sync - Tools > Database Tools > Database Maintenance > Deleted objects information.

    cheers, Paul

     
  • illovo

    illovo - 2012-08-20

    Thx! Noted!

     
  • Andrew

    Andrew - 2012-08-23

    Paul: "You could export to HTML - without passwords - and sort by time stamp." - I finally have a moment to give this a shot.

    However, am I right in believing that it is not possible to sort by timestamp, or am I just blind?  I'm about to go and write a script to sort the HTML elements for me, but having an option to sort by creation/modification date would be brilliant here (if it's not already an option somwhere)!

    Is there a proper place to submit feature requests?  I can't seem to find it if there is.

    Thanks.

     
  • Paul

    Paul - 2012-08-23

    When exporting you can only sort by the 5 main fields.
    Your script can read them into an array and sort for you, or if you open the file in a spreadsheet you can sort on any column.

    Feature requests are under Tracker.

    cheers, Paul

     
  • anqzlvtzhyrlalq

    anqzlvtzhyrlalq - 2012-08-29

    I've had some issues with items disappearing during syncing that I have a feeling may relate to the trouble you are having illovo.

    When I try syncing to my networked .kdbx prior to a local save (assuming there are changes from the last save) I lose these changes. I have solved the problem by ALWAYS doing a local save prior to any network sync.

    HTH!!

     
  • Dominik Reichl

    Dominik Reichl - 2012-08-29

    anqzlvtzhyrlalq, I cannot reproduce this. I've just tried synchronizing two databases where the currently opened one has unsaved changes, and these changes are not lost.

    If you find a reproducible way, please provide a sample database and detailed steps what you are doing.

    Thanks and best regards
    Dominik

     
  • illovo

    illovo - 2012-08-30

    Post 15.:

    Maybe somehow you have some duplicate UUIDs and these are marked as deleted in "original2.kdbx". Try a database clean up before the next sync - Tools > Database Tools > Database Maintenance > Deleted objects information.

    That's the hint we needed. Thanks!

    So, we here call the procedure "merging without double entries", not "syncing" in the future. 2 diferent things somehow or not? I don't care. It works.

    If you want to merge several databases that have the same origin or not and in some databases entries were deleted in others not but you want to have all entries afterwards (including all that aren't deleted in ALL databases) :
    - create a new database
    - make a copy of all database files that you want to merge with the new database file (keep the originals untouched)
    - open these copies one after the other
    - Tools > Database Tools > Database Maintenance > Deleted objects information => Delete
    - save the copies
    - In new database start "Synchronize with File" and select the copied files one after the other.

    Recommended:
    Keep backups of all original files!
    Set "Tools > Database Tools > Database Maintenance > Entry history" to a higher level.

     
  • illovo

    illovo - 2012-08-30

    Unclear:

    In new database start "Synchronize with File" and select the copied files one after the other.

    should be:

    In new database start "Synchronize with File" and synchronize the copied files one after the other.

     
  • Paul

    Paul - 2012-08-30

    We understood - at least I did.   ;-))

    cheers, Paul

     
  • David Dyer-Bennet

    I've just spent a couple of days trying to unravel a problem that sounds similar to this. Since I was told synchronize was "safe", I used it fairly casually when I updated to V2 last week. No triggers, just a couple of manual synchs of files all derived from the same V2 import of a V1 original over a couple of days. And I suddenly noticed my file was much smaller than it should have been and missing most of the groups.

    I had backups of the starting points, of course.

    But now I've got a manually merged file with a lot of duplicates in it. I'm frankly afraid to go near synch again after this, until I hear the problem has been found and fixed (maybe in the documentation -- maybe it's not as safe as described). I didn't write down exactly what I did, but I think that last choice in the synch menu may be more dangerous than it sounds.

     
  • Paul

    Paul - 2013-08-27

    Sync is safe, but it's relatively easy to make changes and forget what you did - I do this all the time.

    The database can shrink on systems that implement file compression differently (Win 7 / 8).

    To see how many entries you have place the cursor in the Search box and press Enter. In the bottom left corner you will see the number of records.

    cheers, Paul

     
1 2 > >> (Page 1 of 2)

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks