#2231 Database latest touch timestamp as trigger's condition to sync

Looks like there is no essential condition to sync databases.
The logic sequence should be:

1. Local client device's db has URL for master db to sync with. Depending on its type, the following checks needed first. If that's a file, check if it has r-w-delete/rename access rights (possibly with supplied credentials). For FTP, check connection to db file in similar matter. So we have network and user access (mostly) covered already.
2. If the previous access check was okay, check for the latest timestamp of the last data record modification in both databases [do we have that in kdbx v4. without need for entire remote db file download first, maybe only headers check/comparison?], if they match, return false (no need to sync). In the simplest case, just compare kdbx files modification stamps with adjustable tolerance like +/- 20 seconds.

That way we don't need to perform sync with master copy on db opening blindly - time consuming operation, especially over slow network or when db is significant in size.

[Paul]

1. Checking the file effectively means reading, writing and renaming if you want to be really sure, which is what you do in a sync. Why do it twice?
2. You can't check anything in the database without downloading the entire file and then decrypting it. There is no way around this - just as well otherwise the security would be broken.

cheers, Paul

[Anonymous]

The goal is specifically to avoid downloading entire kdbx for sync when not necessary.
Read-write-rename access is usually directory based (FTP, network shares), so to check that we do not even need to write into kdbx file, that may be some 2 byte temp file in the same dir.
KDBX v4 format has specific header and plain unencrypted hash on some offset. You can't say it's not KDBX database. Also, after each save/sync operation, even the db file size slightly changes. So to ensure the files are the same, only small part of the kdbx file from the appropriate offset should be read and downloaded (even FTP can resume downloads, so it's possible), like a 32 bytes instead of entire kdbx.

Edit 1: Also, when syncronizing, the current default behaviour (re-downloading the entire file after written) might be significally changed thus improving twice the sync speed.

Edit 2: Moreover, I think the remote sync with master db from diff clients is so common usage scenario that I'd seriously consider built-in feature for remote sync as the KDBX database built-in feature, where you just specify the type of remote access protocol, the location/path and master file name, plus access credentials. All that instead of currently let users to mess with triggers, special sync entries and placeholders with UUIDs.

Edit 3: And as I said before, as a dirt and quickest solution, give us even a simplest means to check kdbx file modification timestamp with preset tolerance parameter (in seconds) and compare two files as 'File timestamps near' condition in a trigger. Sure that probably need performing protocol-specific calls and addind a couple of extra credentials fields in the condition.

[Paul]

There is no way to tell if a database has entries that need syncing. The sync process doesn't even check, it just processes each entry and then writes two new files.

Edit 2. KeePass remebers credentials for remote file access so your sync should work just by selecting the file from your recent sync list. There is no need to use triggers for this.

Edir 3. If the database timestamps are different then something may have changed and a sync is required.

cheers, Paul

[Anonymous]

1. That's exactly what need to be enchanced. No need to wait for unnecessary remote write operations.
2. No. These credentials are in local .net ini (settings) file so provide no security when accessing remote location. That's not feasible when you can keep all this inside encrypted container/kdbx.
3. Obviously. So please provide means for comparing timestamps with adjastable tolerance parameter to take into account network delays when writing database files. Currently, the only secure implementation requires setting triggers, so the solution should be in shape of trigger condition. When there will be (if ever) more robust built-in remote sync option, that would be another story.

[wellread1]

RE: First paragraph & Edit 3.

Keep in mind that the kdbx file timestamp and file header will change every time the database is saved, regardles of whether user data was actually changed. A consequence of this is that the only situtation where synchronization could be avoided programatically is where neither local version nor the remote (master) version of the kdbx file have been saved since the last sync.

Edit 1: Also, when syncronizing, the current default behaviour (re-downloading the entire file after written) might be significally changed thus improving twice the sync speed.

Are you certain that databases are re-download (re-opened) after writing? The target database of the sync pair is saved and closed after a synchronization. Assuming the target database is your remote (master) database, I don't see a reason to download it until the next sync.

Edit 2: ... consider built-in feature for remote sync as the KDBX database built-in feature, where you just specify the type of remote access protocol, the location/path and master file name, plus access credentials...

I am not certain because I haven't used this feature recently, but I believe the above features are all present and can be accessed from the File>File Synchronize>Synchronize with File URL... menu or using Ctrl+Shift+R

[Anonymous]

1. Yes, that's why we don't have Save button in UI enabled until there are actual changes, and also the trigger condition 'Db has changes'. So provided the implemented correctly, mere fact of rewriting a db will suggest that there is need for sync. And that costs especially in normal case the changes are much more seldom that just opening for reading from different locations.
2. Yes, there is the option to disable verification in settings. I guess file size, modification timestamp, and possibly header's unencrypted hash check would be sufficient instead.
3. See my reply to Paul (§2). Becides, the manual sync from menu or even hot key is just that and prone for user mistakes leaving the data unsynched.

[wellread1]

So as I understand your request you need a trigger condition that can compare two files and determine whether they are identical. When combined with the existing trigger condition "Database has unsaved conditions", a trigger system could be devised that would cover the two situations where synchronization is provably unnecessary:

1. If at opening a pair of database files to be synchronized are identical. OR
2. If when requesting synchronization, the open database contains no changes AND its underlying file and the synchronization target database file are identical.

Additionally, you would like to see these conditions built into the logic of the manual synchronization option.

Note: While non-identical KeePass database files may contain identical user data, it is not possible to prove identity without first decrypting the files.

[Anonymous]

Re #2. No, we don't need compare kdbx files (nor perform sync) when there wasn't a single change made locally. As such the situation will happen on closing local kdbx, we are finished work with it. Thus even if there were changes in remote kdbx, we will catch it next time on opening.

Everything is possible. No decription needed to check the kbdx identity status. Header and its checking hashes aren't encrypted. Read

http://keepass.info/help/kb/kdbx_4.html

Even if plugin for v4 kdbx may be possible, the best thing probably would be an integrated solution, maybe even kdbx v5 may be introduced with that in mind. Plus the sync mechanism should be more robust with clear transactional particals - extra encrypted files or the main kdbx file offsets metadata with just changed entries should be transferred from remote (master) location.

[wellread1]

No, we don't need compare kdbx files (nor perform sync) when there wasn't a single change made locally.

What is needed if the master database was changed (e.g. by another user synchronizing with it)?

Header and its checking hashes aren't encrypted

True but a file comparison trigger condition shouldn't assume that it will only be used on kdbx files.

the best thing probably would be an integrated solution, maybe even kdbx v5 may be introduced with that in mind

I suspect you are making this more complicated that it needs to be. Rather that focus on changing the kdbx file format I suggest you determine what will satisfy your immediate needs given the constraints of the current format.

[Anonymous]

What is needed if the master database was changed (e.g. by another user synchronizing with it)?

During session nothing. To re-sync, just re-open local kdbx.

True but a trigger condition shouldn't assume that it will only be used on kdbx files.

Why we have other DB specific conditions like Unsaved changes? That's db-specific conditions, so that trigger condition may be named as "Has remote (master db) changes". But if wanted the normal timestamp and file size equality check may be performed for all files, and additionally, for files with kdbx headers some offset read may be taken. Then name would remain "Remote file different?"

I suspect ...

Those are just suggestions for developing. The working logic of internals. Monolitic transfer of entire db over slow network is ridiculous. Now, per default settings, on sync it happens three times: remote location > client to download, client > remote location to to upload new copy, and re-download of this copy back to client to ensure the perfect match.

What most needed for now is a way to set triggers:
a. Not to blindly sync on kdbx opening when there were no changes on remote side;
b. When performing sync, utilizing quick algo implementing hash or some offset piece of remote kdbx file read to check the files are same (with size and timestamp included), intead of currently mandatory first (that should be performed only when remote file timestamp is different [with acceptable tolerance] from the local db replica BEFORE opening, so we know there are possible changes happened on remote side) and practically excessive third downloads mentioned above.

[wellread1]

What is it that you would like in as few words as possible?

[Paul]

The simplest and therefore safest sync method is to download, decrypt, sync and save. This makes no assumptions and requires no file access features that may not be available from a remote server.
It is up to you to provide connections to remote files, not for KeePass to manage your slow connections.

cheers, Paul

[Anonymous]

There are several well-known built-in protocols for remote sync already implemented in KP.
As pointed out in this thread, there is significant overhead in performance and usability regarding remote syncing. That's all my request is about. Using the well known protocol abilities to at least provide a possibility to check file stamp and size of kdbx files and implement comparison condition for triggers accordingly. Ignoration and laziness are bad friends both. Cheers!