I had my KeePass file corrupted by Windows 10. I am now scavenging the disk in hopes of finding some useful data.
The magic number is 03 D9 A2 9A 67 FB 4B B5. Correct?
I have found these values on several locations. But how do I tell the beginning and end of the file? Is this described in the file header?
What are my chances of recovering something useful? How does the fact that the file was encrypted affect the recovery? Will the entire file have to be intact for the decryption to work properly? Can I at least find some pieces of entries from the old KeePass file? I knwo there is a recovery option inside KeePass itself.
Thanks in advance!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't think there is an easy way to determine the length of the file.
My suggestion would be to recover as much data as possible and import it in repair mode [1]. In repair mode, any additional data at the end of a KDBX file is ignored.
The reason I asked is mainly because I wanted to know if the length or size of the file is vital information for decryption to work properly. Also note that I had no key file, I only use a master password.
If decryption is not affected by the length of the file, then I guess it's not so important how long the file is. As long as it is long enough to enclose some useful data. My file was just a little bit over 1 MB. So any selection range that stretches beyond the 1 MB (plus a few KB) limit will suffice, e.g. 2 MB is good enough because any remaining data at the end, i.e. data above 1 MB (plus a few KB) will be ignored. Correct?
Here's my plan:
Scan the disk volume and find all occurances of the magic number.
Manually copy a 2 MB block for each occurance to a new file.
Create a new container KeePass kdbx file.
Import each file into the new kdbx file using recovery mode.
Is it a good plan? Anything I should change?
I am scanning the disk right now, and I have found 4 occurrances after 1 hour. But at the current speed, it will take 150 hours to complete the scan! Ay caramba! I have to find a way to do it in stages.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can hardly believe it myself but I managed to recover all of my KeePass data! Wow! I was so happy when I saw my lost entries appear in KeePass. I am no data analysis expert, but this clearly shows why having a deeper understanding of the inner workings of things is useful. But it also helps to be patient and to think outside of the box. Of course, we also need the right tools.
This incident actually happened almost 4 weeks ago, on 17 January to be exact. At around 11 PM, I started upgrading my Windows 8.1 Pro installation to Windows 10 Pro. That turned out to be a big mistake.
What happened is that Windows 10 issued a CHKDSK command on the volume where I had my KeePass file stored. I have several disk drives and partitions, and each partition has its own volume. The KeePass file was on partition J, which is another physical drive, separate from the drive that Windows 10 was being installed on. Microsoft was dumb enough to have enabled the CHKDSK repair switch by default. Instead of prompting the user for action. So it did the little "scanning and repairing" ritual on J and repaired it real good... It did this at the end of the Windows 10 upgrade process before I logged into Windows 10 for ths first time (not that I haven't been Beta testing Win 10 in the past). All files on partition J were intact, all except the most important one: the KeePass file. The KeePass file was reduced to 0 byte so I could not open it.
Now, here's the worst part: I had no backup! No backup of the user files, system files, and no backup of the KeePass file. Reason? I was dumb enough not to create a system image backup before starting the upgrade, and even more so for not creating a separate backup of the KeePass file. I know better than this. I was in the process of getting an upgrade for my Acronis True Image license, because I knew my version was not compatible with Windows 10 (it runs but can't access storage devices). So I actually uninstalled the Acronis software I had, and deleted the only backup archive I had. I wanted to do a clean install of the latest version of Acronis software. But I decided to postpone this till I have upgraded to Windows 10 first. Big mistake! Worse still... the J partition... it is the backup partition!
I managed to recover an earlier version of the original file using the Recuva software. The recovered file was complete, but it was missing about 50% worth of additions and edits I made to it since I created the original file. It's at least 2 years older version than the corrupted original. It's the only good file Recuve could find, not counting the 0 byte file it found. I also tried the deep scan but with the same result.
I wish I never touched Windows 10, but Microsoft kept nagging me to "do what 300 million users have already done". And they kept promising: "your files are right where you left them". Em... I beg to differ! Windows had no business scanning and repairing that partition, because I know it's a healthy, almost brand new WD Caviar Red drive.
I think Windows may have detected some traces of old installs on one or several disks, because I used to have a multiboot configuration, and not just a single one either. I once had one triple boot, and one quad boot, in two boot chains. As a seasoned multibooter I have noticed that the latest Windows versions have become more sensitive at picking up multiboot configurations and they start "scanning and repairing" disks every now and then. That is, if they pick up boot chains and boot orders that are too advanced for Windows to understand, and especially if the disks contain unknown filesystems. Even the most simple dual boot is sometimes too much for Windows to handle so it starts choking, and then falls back to do the only thing it knows: "scanning and repairing".
So it was a false positive really, there is nothing wrong with that partition. I saw the countdown timer before it started scanning, but it was a mere 3 second, and I was AFK. So it was all really set up to screw up data. They should have a 30 second timeout in my opinion. But the best thing is to prompt the user for action and await answer.
In my opinion, a primitive utility program like CHKDSK should never be allowed to "repair" disks as part of an automation script. It's much better to prompt the user for course of action. No matter what Microsoft and other big name companies would like us to believe, humans are still much more intelligent than any AI system and dumb automation scripts. Humans can also make better informed decisions and handle exceptions like no program in the "cloud".
I was actually building up a new KeePass file using the recovered file using Recuva. I also had a hard copy of some of the passwords and stuff on paper. So in the end I had about 80% recovered by manual work. But now I'm back at 100%, thanks to the scavanged raw data. I have to merge it all now to a single file.
I have made sure of making a backup copy of my KeePass file now, and it's stored externally on a USB drive this time. This was certainly a good lesson for me, but I don't ever want to go through this again. Also, I will be going all the way back to Windows 7. No matter what Windows 10 promoters say, Windows 7 is the best Windows ever that's still supported. If Microsoft sticks to its current "vision" for Windows, then Windows 7 might as well be titled "The Last of the Windows". It will take them at least another 2 years before Windows 10 reaches a level of quality that Windows 7 was at when it debuted.
Their so called second level support, a.k.a. account specialists totally suck. I tried to have them help me regain access to my Microsoft account. Becase Windows screwed up my KeePass file, I was unable to access the account. The password I changed to last time was generated by KeePass and stored in KeePass. So I didn't have to remember it. Additinally, my security info was not up to date. The phone number I had on the account was no longer in use. Without this, I could not get a reset code. But even though I had provided detailed and accurate info, such as previously used passwords and recent email recipients in their reset question form, they claimed I did not provide enough info.
Their account team people are like robots. So I exchanged several messages with them but they were still not convinced. One of them said I almost passed the validation process but still would not allow me to access my account. I even offered to send them physical evidence to prove my ownership of the account, but they refused because they had no such communication channel. It's like they never heard of postal service.
But that's all really for another discussion. Sorry for going off topic here! I wanted to share the story with you. I'm just glad to have my data back... Microsoft can go... do something.
Last edit: Samir 2016-02-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi!
I had my KeePass file corrupted by Windows 10. I am now scavenging the disk in hopes of finding some useful data.
The magic number is 03 D9 A2 9A 67 FB 4B B5. Correct?
I have found these values on several locations. But how do I tell the beginning and end of the file? Is this described in the file header?
What are my chances of recovering something useful? How does the fact that the file was encrypted affect the recovery? Will the entire file have to be intact for the decryption to work properly? Can I at least find some pieces of entries from the old KeePass file? I knwo there is a recovery option inside KeePass itself.
Thanks in advance!
The magic number is correct.
I don't think there is an easy way to determine the length of the file.
My suggestion would be to recover as much data as possible and import it in repair mode [1]. In repair mode, any additional data at the end of a KDBX file is ignored.
Best regards,
Dominik
[1] http://keepass.info/help/base/repair.html
Hi Dominik! Thanks for the quick reply!
The reason I asked is mainly because I wanted to know if the length or size of the file is vital information for decryption to work properly. Also note that I had no key file, I only use a master password.
If decryption is not affected by the length of the file, then I guess it's not so important how long the file is. As long as it is long enough to enclose some useful data. My file was just a little bit over 1 MB. So any selection range that stretches beyond the 1 MB (plus a few KB) limit will suffice, e.g. 2 MB is good enough because any remaining data at the end, i.e. data above 1 MB (plus a few KB) will be ignored. Correct?
Here's my plan:
Is it a good plan? Anything I should change?
I am scanning the disk right now, and I have found 4 occurrances after 1 hour. But at the current speed, it will take 150 hours to complete the scan! Ay caramba! I have to find a way to do it in stages.
Correct; sounds like a good plan.
Best regards,
Dominik
If the file was fragmented - a very good chance - a contiguous block won't have your data, but worth trying anyway.
How did W10 corrupt the file - and what happened to your backups?
cheers, Paul
Hi guys!
I can hardly believe it myself but I managed to recover all of my KeePass data! Wow! I was so happy when I saw my lost entries appear in KeePass. I am no data analysis expert, but this clearly shows why having a deeper understanding of the inner workings of things is useful. But it also helps to be patient and to think outside of the box. Of course, we also need the right tools.
This incident actually happened almost 4 weeks ago, on 17 January to be exact. At around 11 PM, I started upgrading my Windows 8.1 Pro installation to Windows 10 Pro. That turned out to be a big mistake.
What happened is that Windows 10 issued a CHKDSK command on the volume where I had my KeePass file stored. I have several disk drives and partitions, and each partition has its own volume. The KeePass file was on partition J, which is another physical drive, separate from the drive that Windows 10 was being installed on. Microsoft was dumb enough to have enabled the CHKDSK repair switch by default. Instead of prompting the user for action. So it did the little "scanning and repairing" ritual on J and repaired it real good... It did this at the end of the Windows 10 upgrade process before I logged into Windows 10 for ths first time (not that I haven't been Beta testing Win 10 in the past). All files on partition J were intact, all except the most important one: the KeePass file. The KeePass file was reduced to 0 byte so I could not open it.
Now, here's the worst part: I had no backup! No backup of the user files, system files, and no backup of the KeePass file. Reason? I was dumb enough not to create a system image backup before starting the upgrade, and even more so for not creating a separate backup of the KeePass file. I know better than this. I was in the process of getting an upgrade for my Acronis True Image license, because I knew my version was not compatible with Windows 10 (it runs but can't access storage devices). So I actually uninstalled the Acronis software I had, and deleted the only backup archive I had. I wanted to do a clean install of the latest version of Acronis software. But I decided to postpone this till I have upgraded to Windows 10 first. Big mistake! Worse still... the J partition... it is the backup partition!
I managed to recover an earlier version of the original file using the Recuva software. The recovered file was complete, but it was missing about 50% worth of additions and edits I made to it since I created the original file. It's at least 2 years older version than the corrupted original. It's the only good file Recuve could find, not counting the 0 byte file it found. I also tried the deep scan but with the same result.
I wish I never touched Windows 10, but Microsoft kept nagging me to "do what 300 million users have already done". And they kept promising: "your files are right where you left them". Em... I beg to differ! Windows had no business scanning and repairing that partition, because I know it's a healthy, almost brand new WD Caviar Red drive.
I think Windows may have detected some traces of old installs on one or several disks, because I used to have a multiboot configuration, and not just a single one either. I once had one triple boot, and one quad boot, in two boot chains. As a seasoned multibooter I have noticed that the latest Windows versions have become more sensitive at picking up multiboot configurations and they start "scanning and repairing" disks every now and then. That is, if they pick up boot chains and boot orders that are too advanced for Windows to understand, and especially if the disks contain unknown filesystems. Even the most simple dual boot is sometimes too much for Windows to handle so it starts choking, and then falls back to do the only thing it knows: "scanning and repairing".
So it was a false positive really, there is nothing wrong with that partition. I saw the countdown timer before it started scanning, but it was a mere 3 second, and I was AFK. So it was all really set up to screw up data. They should have a 30 second timeout in my opinion. But the best thing is to prompt the user for action and await answer.
In my opinion, a primitive utility program like CHKDSK should never be allowed to "repair" disks as part of an automation script. It's much better to prompt the user for course of action. No matter what Microsoft and other big name companies would like us to believe, humans are still much more intelligent than any AI system and dumb automation scripts. Humans can also make better informed decisions and handle exceptions like no program in the "cloud".
I was actually building up a new KeePass file using the recovered file using Recuva. I also had a hard copy of some of the passwords and stuff on paper. So in the end I had about 80% recovered by manual work. But now I'm back at 100%, thanks to the scavanged raw data. I have to merge it all now to a single file.
I have made sure of making a backup copy of my KeePass file now, and it's stored externally on a USB drive this time. This was certainly a good lesson for me, but I don't ever want to go through this again. Also, I will be going all the way back to Windows 7. No matter what Windows 10 promoters say, Windows 7 is the best Windows ever that's still supported. If Microsoft sticks to its current "vision" for Windows, then Windows 7 might as well be titled "The Last of the Windows". It will take them at least another 2 years before Windows 10 reaches a level of quality that Windows 7 was at when it debuted.
Their so called second level support, a.k.a. account specialists totally suck. I tried to have them help me regain access to my Microsoft account. Becase Windows screwed up my KeePass file, I was unable to access the account. The password I changed to last time was generated by KeePass and stored in KeePass. So I didn't have to remember it. Additinally, my security info was not up to date. The phone number I had on the account was no longer in use. Without this, I could not get a reset code. But even though I had provided detailed and accurate info, such as previously used passwords and recent email recipients in their reset question form, they claimed I did not provide enough info.
Their account team people are like robots. So I exchanged several messages with them but they were still not convinced. One of them said I almost passed the validation process but still would not allow me to access my account. I even offered to send them physical evidence to prove my ownership of the account, but they refused because they had no such communication channel. It's like they never heard of postal service.
But that's all really for another discussion. Sorry for going off topic here! I wanted to share the story with you. I'm just glad to have my data back... Microsoft can go... do something.
Last edit: Samir 2016-02-11
More backups please, one is never enough. :)
cheers, Paul