IOProtocolExt - differing files despite sync?
A lightweight and easy-to-use password manager
Brought to you by:
dreichl
Menu
▾
▴
Hi all,
I'm using the IOProtocolExt extension to sync my database file via SFTP.
As far as I can see, the content is synced just fine, however - even immeadiately after a sync - it seems the binary file content (i.e. hashes of the files) on the server and locally does not match.
This is problematic, as both locations are also synced by a file-based tool, which always reports conflicts in my KeePass database.
Is this behavior expected, and - if yes - is there a chance KeePass/IOProtocolExt could ensure files are actually identical after a sync?
This happens when your FTP server does not respond as expected.
You need to check that the files are identical before the other sync program runs.
cheers, Paul
What's that supposed to mean?
Isn't the whole upload -> re-download -> verify procedure IOProtocolExt performs meant to ensure integrity of file that was uploaded? If that does not fail, how could my FTP server be causing any issues?
As far as I interpreted it, KeePass uploads the file properly, but either
a) uploads a file that is different from the current database, or
b) modifies the current database again after uploading.
KeePass does neither. It asks the FTP serverfor file details and confirms it matches the one in memory.
Some FTP servers do not report details as expected and KeePass complains. You need to manually confirm the details.
cheers, Paul
Thanks Paul, but have you actually used IOProtocolExt before? There's a whole section on this behavior in the IOProtocolExt ReadMe (bundled, see https://keepass.info/extensions/v2/ioprotocolext/IOProtocolExt-1.16.zip) that talks about this:
So, unless the manual (and the messages in the KeePass status bar) lie KeePass totally uploads the file, then downloads it again to verify it matches. There's absolutely not FTP "magic" involved.
Following this thread I had a thought. I am nmot fmiliar with the IOProtocolExt plugun but does IOProtocolExt syncronise the local and the FTP copy at the record level or does it just copy the keypass databse file file?
I am only familiar with the shared file scenario using TeamDrive (or OneDrive etc) where the two files WILL always be DIFFERENT at the binary level.
Could this explain the behaviour reported?
Yes, that's pretty much my current working thesis, but I don't enough of the internals of KeePass to verify it. I had hoped (and still hope) that someone on this forum has the necessary insight into the KeePass and IOProtocolExt code to explain what is happening here.
Last edit: Ede_123 2019-12-21
KeePass has only one save mechanism and this is used with all types of file systems.
Ioprotocolext is used to replace the FTP handlerin .NET
All that KeePass is doing is reporting the written file is not the same because the FTP server does not respond as expected.
cheers, Paul
Paul, I don't think you understand my issue.
KeePass does not report anything at all (the status bar does reflect what it's doing, though: "Uploading file" , then "Downloading file", then the process is complete) - as far as KeePass is concerned I think the save succeeds.
The problem is that the uploaded file does not match my local file after the synchronization on a binary level.
Last edit: Ede_123 2019-12-22
I hope we can get some insight by Dominik on this, because he developed both components and has the necessary insight to explain what's going on. I'm sure he'll understand what the problem is and if this is an issue that I can solve on my end or whether it's an issue in KeePass.
Sorry, misunderstood the question.
The difference is because KeePass uses adifferent salt for every save, including sync.
cheers, Paul
I will repeat - A file on a server that is syncronised with a local copy (I think this is what you are doing with the plugin) will NEVER be identical at a binary leval athough they contain the same KeePass record data.
A file copy is different from a KeePass file syncronisation. A binary file comparision between the local copy and the syncronised file server copy will ALWAYS show a difference.
Thanks Paul, this is the information I was looking for!
So it is indeed as steelej suspected and the files will always differ.
Is there a reason it is implemented like this (and is there a possibility to avoid it)? It would be great to have an option that ensures "sync" does not only sync entries, but also ensures the files are identical on a binary level.
KeePass is designed so that more than one device can access the same database. One example is a PC and a mobile phone. Another is for members of a family to be able to share a common database containing all of the shared family accounts.
See the example here https://keepass.info/help/kb/trigger_examples.html#dbsync
This means that two users on separate devices can open and update the database at the same time. If the only mechanism were to simply copy the files then this would guarantee data loss. The first user's changes would get written to the server but would then get overwritten by the second user's data.
One could invent complex locking mechamisms to detect such a clash but the KeePass syncronisation using a common shared database in (typically) a cloud based store guarantees that records, wherever they are changed, are never lost. It might take more than one syncronisation for the databases to be fully updated but there has never been any report I can recall of any loss of data.
The worst case scenario is two users changing the SAME KeePass record. In this case both updates as retained in all databases, One will the the latest, the older version will be recorded in the record history. It is very easy to choose which record should be regarded as the correct one.
Note that the two local databases, and the master copy, will all always differ BY DESIGN at the binary file level but will all contain the same records.
If this powerfuldatabase sharing mechanism is not what you want then you need a solution that simply copies the database file to your server on every change. This however would become a simple backup (not a syncronisation) and I am sure that a simple file copy could be initiated via a trigger without using a plugin. This however would prevent the file being shared safely by more than one user.
Using a different salt means it is not possible to compare two databases to determine the encryption. This is a basic security feature added on day one.
cheers,
So this this mean the salt changes on every save?
I really do understand how you might want to have different salts for different databases, but immediately after a sync it could absolutely be handled as the same database for al intents and purposes.
I would be absolutely fine if my local database was simply copied to the server as-is after the sync.
A dtatbase syncronise is NOT the same as a file save. A syncronise compares each record and merges changes into each file (local to remote and vice versa) and then reencrypt both files. The remote file is therfore going to change and will never be a binary match with the local file. This is consequence of how KeePass is designed to work and this works very well for most users.
A file copy is just that a binary copy.
I am not at all clear on why this is such an issue for you. It appears to be somthing associated with your file server and the way you want KeePass to work after a change to your database.
As I have suggested before - if you do not want to have the file server database copy syncronised to the local database then it should be quite easy to run a trigger function in KeePass to just copy the local database file to the file server, bypassing the KeePass plugin which is designed to syncronise the files, I envisage a simple command line function, or batch file, to copy the file to the server. KeePass can be configured to do this.
If you are able to specify exactly what you want to happen step by step we may be able to help further.
The issue at hand is really trivial:
I have one database
I have a copy on my local machine.
I have a copy on my server.
File systems on local machine and server are synchronized by a file-based synchronization utility.
I usually access and modify the local copy for obvious reasons. However occasionally (e.g. when accessing the DB remotely from my mobile phone) I access and change the server copy directly.
This is where it becomes inconvenient from a user-side: Local and server copies have diverged and need to be synchronized. However - due to how synchronization seems to be implemented in KeePass - the databases are synchronized afterwards, but differ on a binary level, which causes the file-based synchronization utility to complain about a conflict.
I don't think this is a particularly unusual set-up and it could indeed be solved by a simple entry-level sync in combination with a plain file-based copy back to the server.
Let me know if you know if this is doable with KeePass.
Now you have explained it makes some sort of sense.
KeePass is designed to support multiple uses for exacrtly the purpose you want it to do.
KeyPass however was designed to use a far more conventional ways of sharing files. You eem to have adopted a compteleyt different way of doing it. I cannot see any way of achieving what you want without doing one of three things.
Option 1: Change your method of file sharing to use a conventional "off the self" and free file sharing mechanism that has proven to be very reliable and is guaranteed out of the box to work without loss of data. No extra plugins are required and all ofther file sharing needs will be catered for. TeamDrive provides such a cloud based file sharing mechanism, at no cost for private use but there are other cloud based solutions. It does detect confilict out to the box and advises you that they exist. It retains an automatic versioning system that enable conflict to be resolved. Note that KeyPass sycronisation mechanism using triggers almost entirely eliminates the possibili[ty of a conflicted update of the datbase file losing data. You have not said why such an option is not suitable for you needs. Note that Keepass2android is fully compatible with this option.
Option 2: Persist in using some more complex mechanism (possibly home written?) that does not work with KeePass in the way you seem to want it to. In this case it would appear that you would need to write your own plugin. I would personally not want to see the excelent, reliable, and well proven database sharing mechanism changed to be capable of supporting the method you want to use when there is already a reliable method of doing built into keePass. Adding such an option (even if possible) would lead to misunderstandings and consequential numerous requests for suport for something that has never been asked for before and I can't imagine would be needed by anyone else in the future.
Option 3: choose or write an alternative password managment program
I very strongly recommend option 1
Last edit: steelej 2019-12-25
Replacing the file sync utility with a KeePass trigger seems the obvious solution.
A sync on save trigger is described in the Trigger Examples on the KeePass help page.
cheers,
@steelej Your option 1 is basically what I'm doing, but any "off the shelf" sharing mechanism will be faced with file conflicts if the database is modified on different devices in parallel. Feel free to elaborate on how to resolve those conflicts easily...
@Paul Thanks, triggers seem useful. I'll have a look at those. There is one for "Synchronized database file", so maybe I can copy the file manually after a sync to achieve binary identical copies locally and on the server. Syncing on save is not really what I want as it looses most of the advantages that working with a local copy has.
If two users (A and B) are sharing the same database file using the recommended method the following happens. [It is assumed that they have a LOCAL databse and a cloud replicated MASTER database - all changes to this copy are replicated to all users sharing this database.]
Initially both A and B have their LOCAL copy of the database open and modify their LOCAL database at the about the same time. Now assume that they then SAVE their local databse at almost exactly the same time.
On Computer A the recommended trigger will syncronise the LOCAL database to the local MASTER copy which is updated with the changes. This will be copied to the cloud server and replicated to computer B
If Computer B has saved the change but their MASTER copy has not yet been replicated from the cloud then their MASTER copy may be overwritten by the version created on Computer A.
The situation now is that the MASTER copy, via replication will have the changes from computer A or unpredictable the changes from computer B. The important thing to note is that the local copies still do contain the locally applied updates plus some of the changes
The next time aither A or B open or close their databases the triggers wull again syncronise their LOCAL copy wiht the MASTER cloud replicated copy. This time they will again synchronise any changes that have not yet been applied to their local MASTER even if the changes were made and then overwritten by the cloud replication. The new MASTER is replicated to all other users. File conflicts on this file, if reported by the cloud service, can be ignored.
There are two points to be made.
Changes are not automatically updated to their LOCAL databases instantly. The users may have to close and open thair LOCAL database (perhaps more than once) for the changes to be applied. No data is lost however and both users will then have identical database RECORD content. It is however certain that the local copies on both Computer A and Computer B nd the MASTER copy be different at the byte level and a file comparison will always show a difference..
In an extreme case if both users update the SAME Keypass record then both users will see only ONE of the changes - the other will be in the record history. The data is not lost. The preferred update will have to be selected manually.
I am not aware of anyreported losses of data when this method of syncronisation has been applied correctly.
This process is illustrated in
https://keepass.info/help/kb/trigger_examples.html#dbsync
Which describes the essential triggers.
I am aware it looks complicated but if you follow the steps described (adapting for your choice of file names and paths) it works reliably 100% of the time. You only have to set this up once. I first set this up early in 2011.
This a stable and well proven "off the shelf" solution using triggers to apply the features built into the KeePass program and using basic file replication via most cloud servers.
Last edit: steelej 2019-12-26