Synchronize problem: duplicate groups or all deleted

[Adam Toner]

I'm using KeePass 2.28 for Windows and MiniKeePass 1.5.2 for iOS. Things have been great for many months until I tried to synchronize recently.

My procedure is:
1) KeePass on Windows opens my main database from Google Drive
2) Copy the database file from my iPhone to my Google Drive (different file name)
3) Synchronize using the Windows app
Usually this works fine, resulting in my normal database (Windows) updating with any changes from my iPhone.
4) Copy the database back to the iPhone.

The last time I tried this, I ended up with all of my groups and entries duplicated. Now I face having to manually figure out which are new/updated.

The groups appear in aplhabetical order twice in the new database list. I tried making copies of the new database, and in one removing the top groups (iPhone?) and in the other removing the bottom groups (Windows?) If I try to synchronize those files, both end up empty as if all groups/entries have been deleted.

Please help. Thanks.

[wellread1]

Compare the UUIDs (universally unique identifier) of the 'original' with 'duplicated' entries (Select View>Configure Columns..., and check 'UUID' in the Standard fields section), the UUIDs should be different. If you find any entries have the same UUID you should report it here because that should never happen. See also the note below

An entry's UUID is permanently assigned when it is created and will never change over the lifetime of the database. Keepass 2.x synchronizes entries in two different database only if their UUIDs match otherwise the entries are treated as different. If a user separately creates 'identical' entries in a miniKeePass database and a KeePass database these 'identical' entries are different for synchronization purposes because they have different UUIDs. They will be 'duplicated', i.e. both entries added to the merged database. The same synchronization behavior will result if you perform an operation on a database that creates new, apparently identical entries (e.g. copy entries, or import the entries into a new database and use the default "Create New IDs"). When the modified database is synchronized with the original database (or one derived from the original) the 'identical' entries with different UUIDs will be treated as separate entries and 'duplicated'. However every entry in the database will have a different UUID.

Your easiest recovery method will be to recover a backup copy of your KeePass database that is prior to the synchronization event. If you don't have a backup copy of your original KeePass database you may be able to recover one from Google Drive file versions or Windows previous versions.

If you are able to recover pre-syncrhonization copies of the two databases carefully inspect the individual entries (especially their UUIDs) to determine how the 'identical' entries are different from one another. Update the database copy that is most nearly up-to-date manually. Once it is updated use it as the KeePass database and copy it back to miniKeePass. Discard the old miniKeePass database. At this point the UUIDs of the entries in the miniKeePass database and the KeePass database will be identical and synchronization should work properly.

---

If you must de-duplicate data manually, you may find the Delete Duplicate Entries(Tools>Database Tools) helpful.

During manual de-duplication it may also be helpful to distinguish 'duplicate' entries based on their creation date and modification dates (Select View>Configure Columns..., and select the appropriate field to display). Displaying all entries without any grouping may also be helpful (Edit>Show All Entries; or Select View>Show Entries of SubGroups plus select the top group, plus View>Grouping in Entry List (select the desired display option))

After you have de-duplicated your database, copy it back to miniKeePass and discard the original miniKeePass database or you may experience the same problem the next time you synchronize.

---

Note: MiniKeePass has been reported to violate the requirement that each entry have a universally unique identifier, but to date, not on the scale that you are describing. If you find duplicate UUIDs in your database see also http://sourceforge.net/p/keepass/discussion/329220/thread/7e3fe63d/ and https://sourceforge.net/p/keepass/discussion/329220/thread/d12ccfa0/?limit=250#a5d0.

[wellread1]

Make copies of your recovered databases before performing major operations on the data contained in them.

[Adam Toner]

Wellread1, thanks very much for your extremely thorough and prompt reply. At the risk of hyperbole, your post is probably the best reply I've ever gotten to a forum post or support request.

I pulled down old versions of the database files from Google Drive. I didn't explicitly mention in my original post that that database from MiniKeePass came from KeePass a few months ago (last time I synchronized). Because of this, the UUID's should have been the same except for the few new entries created since the last sync. However, the UUID's are all different, which explains why duplicate entries now appear on synchronization. Your suggestion to check UUID’s was critical.

I ended up manually synchronizing the two databases by showing and sorting by last modification time, as you suggested.

A few questions remain:
1) How did the UUID's change in the past few months? KeePass? MiniKeePass? There's also a KeePass 2.x for MacOS X which infrequently accesses the same Google Drive database as the Windows KeePass.
I went back to even older versions of the MiniKeePass databases. When I post them to Google Drive there’s a number at the end of the filename which gets incremented, so I can easily see old versions in Google Drive. UUID’s stayed the same going back 6 months and a few synchronizations, so I don’t think MiniKeePass is to blame.
I checked KeePass history on some entries, and there’s nothing that explains the new UUID’s.
I can only see back about a month on the database used with the Windows app. It has the same UUID’s (different than MiniKeePass) a month ago. So I’m assuming that somehow, in the past few months, the UUID’s changed on my KeePass database. I don’t recall doing any wholesale copy or import operations, but between 2 users and 2 OS’es I suppose anything is possible. I’ll pay more attention to this in the future and keep backups.

2) Back to the database which contains duplicate entries. If I make a copy, delete the “top” entries in one copy and the “bottom” entries in the other, then synchronize, why do I end up with no entries? I checked, and I’m indeed just replicating the above-mentioned situation where the UUID’s are different. I should end up with duplicates.

Thanks again for your help.

[wellread1]

Its a pleasure to be of assistance.

[The] database from MiniKeePass came from KeePass a few months ago (last time I synchronized). Because of this, the UUID's should have been the same except for the few new entries created since the last sync.

...

How did the UUID's change in the past few months? KeePass? MiniKeePass?

If the miniKeePass database originated from a KeePass database then you are correct the UUIDs of entries that were in the original database should have been the same in both databases.

The KeePass developer is well aware of the issues that can occur when UUIDs are managed improperly. I am confident as a long time user of KeePass that the UUIDs of entries don't change over the lifetime of a KeePass managed database.

I can't comment about miniKeePass except to note that there have been some reports, cited in my original reply, of duplicate UUIDs created in miniKeePass databases that have been reportedly fixed. I don't know the details about the reported fix... or how it would affect existing UUIDs.

Going forward: The first step to identifying the origin of your problem would be to inspect the entries of the databases again after some time has elapsed and before attempting synchronization to look for unexpected changes in UUIDs. A backup of copy of the current restored database would be useful for this purpose. This inspection is also warranted until your confidence in the synchronization process is restored.

I can only see back about a month on the database used with the Windows app. It has the same UUID’s (different than MiniKeePass) a month ago

Without some records about the time of the previous synchronization this doesn't demonstrate much except to confirm that KeePass won't alter entry UUIDs during normal use over a month.

Back to the database which contains duplicate entries. If I make a copy, delete the “top” entries in one copy and the “bottom” entries in the other, then synchronize, why do I end up with no entries? I checked, and I’m indeed just replicating the above-mentioned situation where the UUID’s are different. I should end up with duplicates.

Note: I will assume sort order in the two database copies was identical.

Each KeePass database keeps a record of deleted entries, otherwise if one deletes entries in one copy of a database and then synchronizes, the deleted entries will reappear in the synchronized copies because there was no deletion record. During synchronization, KeePass treats the entry that is most recently changed in either database as the current version of the entry.

What you did was delete entries (1 to n) in copy A and then entry (n+1 to last) in copy B, and these deletions were the most recent modification to each entry. The synchronization will delete entries 1 to last (the most recent modification of each entry) yielding an empty database. If however, you also modified entry 1 in copy B the synchronization result would be a database containing only entry 1. The more recent modification of 1 in Copy B overrides the older deletion in Copy A.

Tip: You may find the Delete Duplicate Entries(Tools>Database Tools) helpful for deleting the duplicates. It compares the entries content to identify duplicates and eliminate them.