Help using ddrutility
LINUX based utility for use with gnuddrescue to aid with data recovery
Brought to you by:
maximus57
Menu
▾
▴
I have never used ddrutility but think it may help with my first dive recovery. Since 28th December I have run gddrescue to image a 500GB laptop drive that was dropped. So far it has recovered 135GB with approx. 5800 errors flagged. All the errors so far are in the first 35GB of the drive, the other 100GB of the drive that has been imaged is the last 100GB of the drive.
I only want to recover the files and have no need to recover the OS (Windows 7). Also I think there were only 100 - 150GB files on the drive.
Can I use ddrutility? To tell me how much of the drive has files on it and so save time imaging empty drive? To identify the size of the os partition (can I then write 0's to this partition so it is easier to mount the drive image)?
Do I use ddrutility on the dropped drive or the image? I'm nervous about using it and not sure what command to set it running. Any ideas please.
Since you are still working on the recovery and by what you say you have only recovered 135 of 500 GB, the ddrutility tool ddru_ntfsbitmap may be able to help you some. What it does is recover the bitmap file from the failing drive and use it to create a ddresuce domain file that will allow recovery of only the used space. The used space does include the operating system. If you think that there was only 100-150GB of used space out of the 500GB then it could be worth it.
First, since you are well into your recovery, you should make a copy of the current ddrescue logfile before you try this, just in case. If possible and you want to be really really safe, also make a copy of the current image.
Second, you need to read and then re-read the instructions for ddru_ntfsbitmap. Ddru_ntfsbitmap needs to be ran on the real drive you are trying to recover. I would recommend using the --mftdomain option. You must get the options correct or it may produce a domain file with the wrong offset (read the instructions again!) Once it produces a domain file, you don't run it again.
Third, use the domain file(s)s with ddrescue as per the instructions, in addition to your current ddrescue command. If you make an MFT domain file, use that first as it is most important. If at any time you wish to go back to normal, then just don't use the domain file with ddrescue and it will continue as normal.
I would just like to add that you will continue to use your current ddrescue logfile when using the domain file. That is why I recommend to back it up. The logfile contains the data of what has already been copied. Using the domain file does not replace the logfile. It only limits what is copied from this point forward. What has already been copied stays copied and in the logfile.
Thank you that's really helpful. I will read the instructions more than once and then follow your directions.
Can I download ddrutility with sudo get-apt install ddrutility? Which repository contains the file.
Ddrutility is not available in a repository. You must download it from here (sourceforge). Go to the files tab, and the latest version will be up at the top above the rest of the list of files. The instructions for how to install it are in the readme file which can be downloaded, but it is also the text below the list of files. You may also wish to download the help.html file as it opens with a web browser and the navigation works unlike the wiki page.
Hello
I have had a first attempt to run ddru_ntfsbitmap. I feel out of my depth and would appreciate further guidance please.
I instaled ddrutility to /home/ubuntu/ddrutility-2.6
Then ran ddru_ntfsbitmap and got an error:
ubuntu@ubuntu:~$ sudo ddru_ntfsbitmap /dev/sdc domain_logfile -m mft_logfile
ddru_ntfsbitmap 1.4 20141011
Reading boot sector...
GNU ddrescue 1.17
Press Ctrl-C to interrupt
rescued: 512 B, errsize: 0 B, current rate: 512 B/s
ipos: 0 B, errors: 0, average rate: 512 B/s
opos: 0 B, time since last successful read: 0 s
Finished
ERROR! This program only allows for 8 sectors per MFT record, and this partition reports 21
ERROR! Boot sector ID is not NTFS
Removing the boot sector file and log
Any ideas please.
The first thing I want to make sure to get strait is it appears you are doing a whole drive recovery. I am assuming this from your command:
ddru_ntfsbitmap /dev/sdc domain_logfile -m mft_logfile
The “/dev/sdc” is what I am referring to. So I assume that your rescue command up to this point has been something like:
Ddrescue /dev/sdc image.dd logfile.log
If this is the case, then make sure to keep using the /dev/sdc, and not a command that involves a partition like /dev/sdc2. This is very important! Never switch commands from using the whole disk to a partition in a recovery! You could compromise your image and have to start over. With that being said…
Now you need to read the help section for ddru_ntfsbitmap, and focus on the examples section for WHOLE DISK TO DISK OR IMAGE SIMPLE. Considering that your recovery is Windows 7, it will almost surely have a boot partition, likely have a recovery partition, and then the actual OS partition. You need to figure out the offset of the OS partition (it will be the big NTFS partition). When you get the offset correct it will work. So your command would look something like this, only using the offset you figured out instead of the 1048576:
sudo ddru_ntfsbitmap /dev/sdc domain_logfile -m mft_logfile -i 1048576
After success with this, your first ddrescue command to get the MFT first would be something like:
sudo ddrescue /dev/sdc image.dd logfile.log -m mft_logfile
And then after that to recover the used portion it would be something like:
sudo ddrescue /dev/sdc image.dd logfile.log -m domain_logfile
Thank you for your help. I've been away for 5 days so left ddrescue running - it has now recovered 195GB - so still going slowly. Yesterday I launched ddru_ntfsbitmap with the offset calculated from fdisk -lu:
Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7c8fd653
Device Boot Start End Blocks Id System
/dev/sdc1 * 2048 1538047 768000 de Dell Utility
/dev/sdc2 1538048 976771071 487616512 7 HPFS/NTFS/exFAT
Run command:
sudo ddru_ntfsbitmap /dev/sdc domain_logfile -m mft_logfile -i 787480576
The mft is in 2 fragments and progress is slow, currently splitting failed blocks in mft part 0. It seems to take about an hour before moving on to the next block if there is a fault. Is this normal?
The output after 12 hours running reads:
ddru_ntfsbitmap 1.5 20150111
Reading boot sector...
GNU ddrescue 1.17
Press Ctrl-C to interrupt
rescued: 512 B, errsize: 0 B, current rate: 170 B/s
ipos: 787480 kB, errors: 0, average rate: 170 B/s
opos: 0 B, time since last successful read: 0 s
Finished
Reading bitmap inode from mft...
GNU ddrescue 1.17
Press Ctrl-C to interrupt
rescued: 8192 B, errsize: 8192 B, current rate: 0 B/s
ipos: 4008 MB, errors: 1, average rate: 170 B/s
opos: 15360 B, time since last successful read: 40 s
Finished
ddrescuelog: Logfile '__mftshort.log' not done.
Creating MFT domain logfile...
total mft fragments = 2
mft part 0 offset=0xEEF00000 size=0xC820000
mft part 1 offset=0x244BCCA000 size=0x48A0000
............Reading part 0 of $Bitmap...........
GNU ddrescue 1.17
Press Ctrl-C to interrupt
rescued: 68096 B, errsize: 15173 kB, current rate: 0 B/s
ipos: 4002 MB, errors: 11, average rate: 1 B/s
opos: 8563 kB, time since last successful read: 5.5 m
Splitting failed blocks...
[end]
Do I just leave this running? Any ideas how much longer it might take? Is it doing any further damage to the disk that may prevent further recovery?
I have no experiance of using this utility so any guidance/advice you can give is appreciated. Thank you.
Current status - no change for 5.7hrs
GNU ddrescue 1.17
Press Ctrl-C to interrupt
rescued: 68096 B, errsize: 15173 kB, current rate: 0 B/s
ipos: 4004 MB, errors: 11, average rate: 1 B/s
opos: 10992 kB, time since last successful read: 5.7 h
Splitting failed blocks...
Any suggestions?
That disk seems very badly damaged. Just so you know, the more you read from it, the worse it could get. There is no reason to continue letting ddru_ntfsbitmap run. You will not get a good used space domain file from it with the error size in the bitmap file that big. It looks like you did get an MFT domain file that you can use to focus on the MFT.
If the data is worth it, you may wish to consider consulting a professional data recovery company.
Thank you.
Have just stopped it at:
............Reading part 0 of $Bitmap...........
GNU ddrescue 1.17
Press Ctrl-C to interrupt
rescued: 75776 B, errsize: 15165 kB, current rate: 0 B/s
ipos: 3998 MB, errors: 12, average rate: 1 B/s
opos: 5412 kB, time since last successful read: 45.7 m
Splitting failed blocks...^C
ddresuce was terminated by the user
full exit value= 0x0002
Am now running sudo ddrescue /dev/sdc image.dd logfile.log -m mft_logfile with current status
GNU ddrescue 1.17
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued: 36352 B, errsize: 209 MB, errors: 1
Current status
rescued: 185982 kB, errsize: 100 MB, current rate: 0 B/s
ipos: 4010 MB, errors: 3, average rate: 13998 B/s
opos: 4010 MB, time since last successful read: 2.1 h
Splitting failed blocks...
Should I give up on this too?
If the data is valuable enough that you would consider paying a professional to recover it, then you should stop all attempts now.
If you have no plan of paying for data recovery, then you can let the MFT focused recovery run. The MFT is the most important part of the recovery, so you do want to make the best attempt at it and it is best to get it first.
After the MFT recovery is done, you could go back and try ddru_ntfsbitmap again to get the used space domain file if you wished. If you did not delete anything, it should pick up where it left off. Let it run until it is done and then use the domain file for the rest of the recovery. Any errors in the bitmap file are defaulted to used space when creating the domain file, so the worst case is it that the domain file would recover some unused space.
Just understand that you are in for a very, very long recovery attempt overall, with the possibility of recovering most, some, or no data, although you never know that until it is done.
Since it looks like you are using Linux, I would suggest using the --direct (-d) option of ddrescue, as it is normally faster with read errors. It can be 3 times faster from what I have seen. And if you run ddru_ntfsbitmap again make sure to read the instructions as to how to pass options to ddrescue.
If you are running Linux and are really adventurous, you could update ddrescue to 1.19 and then apply my passthrough patch. Using the --ata-passthrough option from my patch can help speed up the recovery (possibly an additional 5 time more than --direct if you are on an older Linux kernel).
You will have to download the latest ddrescue from the ddrescue home page as it is likely not available in any repositories.
http://www.gnu.org/software/ddrescue/
My passthrough patch can be found in the files section here.
https://sourceforge.net/projects/ddrutility/files/ddrescue%20patches/passthrough%20patch/
Last edit: maximus57 2015-01-14
How do you have the drive hooked up to the computer? Is it connected directly or with a USB adapter?
It is a little concerning to see so many hours between good reads. Can you post your logfile on here?
Both source and destination drives are connected by USB.
I'm still running ddrescue, currently with sudo ddrescue -n -v /dev/sdc ~/Desktop/CMS-raw-image-link/cmsimage.dd ~/Desktop/CMS-raw-image-link/rescue.log
Here is the mtf_logfile:
Rescue MFT log file created by ntfsbitmap
Command line:
current_pos current_status
0x00000000 +
pos size status
0x00000000 0x00007E00 +
0x00007E00 0x2EEF8200 ?
0x2EF00000 0x00001000 +
0x2EF01000 0xBFFFF000 ?
0xEEF00000 0x0C820000 +
0xFB720000 0x23505AA000 ?
0x244BCCA000 0x048A0000 +
0x245056A000 0x502069C000 ?
The _part0__bitmapfile.log is attached
Does this help?
Some of the data on the disk is of personal value. It's not worth paying for professional recovery. But I am interested to see what I can recover, and this is the first time I have used your tried recovering data from a failed drieve, usinf=g linux or you software.
I feel out of my depth and I'm interested in using ddrescueview. Any ideas how I run this program? Thank you for all your help, I really appreciate it.
Last edit: simon076 2015-01-16
I was actually hoping for the full rescue.log file. It may show more about the whole condition of the drive. I did take a look at the log you attached, and can say that it doesn't look good. For the data that has been attempted the error size is much greater than the good reads. If the drive has a bad head, that head is likely trashed and barely able to read.
As for ddrescueview, are you having trouble running it in linux? I don't remember what it takes to install on linux right now. But there is also a windows version. I don't think it has to be installed either, just extracted. You just have to copy the logfile to a flash drive or something that windows can see.
It had been so long since I had installed ddrescueview that I forgot. I never had to install it on linux either. Just extract it and run the executable. It opens a gui window so you don't even have to launch it from the command line, you can click on it. But it does have to have permission to run as an executable.
I almost forgot. Since you have the drive connected via USB, you can forget what I said about using my passthrough patch. It does not work well with USB attached drives. But I would still recommend using the --direct option of ddrescue.
Here is the rescue logfile as requested. Ddrescue is still running.
I haven't tried the -d option since you suggested it. In the first few days of the recovery I tried various options including -d but it didn't seem to make much difference. I will try -d later and keep it running for much longer and see if the recovery rate doens't tail off as much.
After seeing your logfile, I would recommend not running ntfsbitmap again to get the used space domain file. The bitmap file is in a very bad spot, and the less time it spends there the better. As you said the errors all appear at the beginning of the disk. There appears to be much more good data that can be read yet as long as the drive holds up.
It looks like you are in a good run for reading right now. Did you try reading in reverse at some point in time? It looks like you recovered ¼ at the beginning, ¼ at the end, but it is still working on the middle ½ of the disk. I tell you to focus in the MFT below, but if you are not getting any errors right now then maybe you should let it run normally. When it is done with the copy phase and starts the trimming phase then I would recommend stopping it and going for the MFT. Don’t use the no-split option when getting the MFT, let it get all of it.
It looks like about the first 1/3 of the MFT is also in a bad spot, although most of it is in a big non-split spot so you won't know how damaged it is until you work on it. I would highly recommend focusing on the MFT. After that focus on the rest of the disk as a whole using the no-split option like you are currently doing. After that you could try for the rest of the bitmap file, or run ddrescue without the no-split option and let it finish.
The -d option may not help with a usb drive. One way to tell would be to time how long ddrescue takes to update the screen when reading errors. Ddrescue normally refreshes the screen every second, but a read error will take many seconds and ddrescue will not update the screen until it is finished with the current read attempt. This method only works when in a bad spot, but you could use it to see if there is any difference with and without the -d option. The -d option does not normally help with the read speed of good areas, only with the amount of time spent on read errors.
If the read speed it just slow but not increasing the errors, you may try stopping ddrescue and restarting it to see if the speed comes up. It has been noted that sometimes after a read error the speed can drop and stay that way until it is reopened. Newer versions of ddrescue have an option to reopen the drive after every read error. Version 1.19 has many improvements, including a no-trim option.
The last few days running ddrescue the average rate has been about 140kB/s (GNU ddrscue 1.17). I have downloaded and run version 1.19 for the first time. After 10mins here are the results:
ubuntu@ubuntu:~$ sudo ddrescue -d -n -v /dev/sdc ~/Desktop/CMS-raw-image-link/cmsimage.dd ~/Desktop/CMS-raw-image-link/rescue.log
GNU ddrescue 1.19
About to copy 500107 MBytes from /dev/sdc to /home/ubuntu/Desktop/CMS-raw-image-link/cmsimage.dd.
Starting positions: infile = 0 B, outfile = 0 B
Copy block size: 128 sectors Initial skip size: 128 sectors
Sector size: 512 Bytes
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued: 227046 MB, errsize: 1800 MB, errors: 5963
Current status
rescued: 227072 MB, errsize: 1800 MB, current rate: 0 B/s
ipos: 128686 MB, errors: 5963, average rate: 41752 B/s
opos: 128686 MB, run time: 10.33 m, successful read: 1 s ago
Copying non-tried blocks... Pass 1 (forwards)^C
At no time did the speed go above 66000 B/s. Why is it the new version so slow.
I will try running without -d
I have run ddrescue backwards but on both occasions I stopped it after a short while as the speed was consistently low (approx 30-50Kb/s)
Version 1.19 seems really slow. Without -d ater 5mins
ubuntu@ubuntu:~$ sudo ddrescue -v /dev/sdc ~/Desktop/CMS-raw-image-link/cmsimage.dd ~/Desktop/CMS-raw-image-link/rescue.log
GNU ddrescue 1.19
About to copy 500107 MBytes from /dev/sdc to /home/ubuntu/Desktop/CMS-raw-image-link/cmsimage.dd.
Starting positions: infile = 0 B, outfile = 0 B
Copy block size: 128 sectors Initial skip size: 128 sectors
Sector size: 512 Bytes
Press Ctrl-C to interrupt
Initial status (read from logfile)
rescued: 227077 MB, errsize: 1800 MB, errors: 5963
Current status
rescued: 227092 MB, errsize: 1800 MB, current rate: 0 B/s
ipos: 128705 MB, errors: 5963, average rate: 44231 B/s
opos: 128705 MB, run time: 5.38 m, successful read: 1 s ago
Copying non-tried blocks... Pass 1 (forwards)^C
10mins of running backwards resulted in very low speeds again:
Initial status (read from logfile)
rescued: 227092 MB, errsize: 1800 MB, errors: 5963
Current status
rescued: 227113 MB, errsize: 1800 MB, current rate: 0 B/s
ipos: 399975 MB, errors: 5963, average rate: 32716 B/s
opos: 399975 MB, run time: 10.68 m, successful read: 1 s ago
Copying non-tried blocks... Pass 1 (backwards)
I am going to unistall version 1.19 and run 1.17 overnight - at least it will recover more of the disk than 1.19 seems to.
Can you explain this performance?
Would I be better installing the damaged drive in my computer and running Ubuntu live cd as nowto a USB external drive to store the image?
I cannot explain 1.19 being slower than 1.17. I am interested if going back to 1.17 increases the speed again. I suspect that something else is going on, but you should use whatever seems to work best.
As for reading backwards, it will always be slower. Drives have a built in read-ahead feature, but it only works when reading forwards.
It is always recommended to hook the drive strait to the computer instead of through USB. But I don't think a normal Ubuntu live cd comes with ddrescue, and it is a bit difficult to install something onto a cd :) There is systemrescuecd, which looks like it comes with ddrescue 1.18.1 and kernel 3.10. Don’t use ubuntu-rescue-remix, as it is old and has ddrescue 1.14 on it. No matter what live cd you use, you will no longer be able to use my ddrutility since it requires installation.
Do you have the resources to hook up two drives at once in your computer, the current Linux operating system plus the rescue drive? And I am curious what version of Linux you are using. I am most interested in the kernel version which can be found by the command “uname -r”. Newer kernels (somewhere around 3.6 and newer) seem to be much faster at handling errors in my testing. But I don’t know if it makes a difference with a usb connected drive.
I using a laptop (Dell M4300 OS WinXP) for the recovery and running Ubuntu 14.04.1 LTS 32bit from a live cd. By the way I've run ddrescue 1.17 with the 64bit Live CD on a core i7 64 bit laptop and it's not always quicker. I've been downloading and installing ggrescue 1.17 from the respository. I have run ddrtility with the livecd!
uname -r' returned '3.13.0-32-generic'
I can hook another drive up via usb but I don't think you are asking me to do this.
Yesterday I tried removing ddrescue 1.19 but instead removed 1.17 with sudo apt-get remove gddrescue. I left 1.19 running overnight with the follwing results:
Initial status (read from logfile)
rescued: 227113 MB, errsize: 1800 MB, errors: 5963
Current status
rescued: 230597 MB, errsize: 1800 MB, current rate: 0 B/s
ipos: 132190 MB, errors: 5963, average rate: 88895 B/s
opos: 132190 MB, run time: 10.88 h, successful read: 1 s ago
Copying non-tried blocks... Pass 1 (forwards)
No errors and a recovery rate just over half the long term rate with 1.17
I'm definitely going back to 1.17 and will let you know how I get on.
Last edit: simon076 2015-01-18
I'd appreciate your view on the following:
The disk that I am recovering is circa 18 months old. It has been used to store programs and personal files. I have guessed it has approx 150Gb of total information stored. I don't sense there have been lots and lots of files saved then deleted, I think it's more a case of new files being added from time to time.
Will all the information be stored from the start of the disk? If this is the case is it worth taking a copy of the disk image when approx. the first 150GB are recovered and then trying to read the files?
Before reading the files. Do I need to fill the unread portion of the image with 0's? How do I do that please?
Do I try to recover the mtf and should I try and fill in some skipped blocks first? How would i do this?
Since the image is taking so long to create I think it will save 2-3 weeks if not waiting for the blank part of the disc to be copied.
Thanks
I have to ask: how are you running ddrutility using a live cd? I actually have never tried it. It may be something I would like to include in the instructions how to do.
If you are already using a live cd, then you could put the drive in the computer for possibly better results. However, make sure (in the bios) the computer boots to the cd first and does not try to boot from the hard drive! It may or may not read faster, as usb adapters can vary in how they act. But you will have more control, and I can say for sure that the -d option will help with read errors.
As for where the data is stored on the disk, there is no true way to tell without the bitmap file. It may or may not be all at the beginning. The MFT is split in two locations, one towards the beginning and one about 1/3 into the disk. Because the bitmap file is so damaged you should recover the whole disk. However, if you are going to work with a COPY of the image like you say, then you can try anything you want. Just make sure to keep the original image and log files safe and unchanged so you can resume the recovery if needed.
To fill the unrecovered part of the image with zeros you can use ddrescue like this:
ddrescue --fill-mode ?*/- /dev/zero image.dd rescue.log
This will fill all the untried, non-trimmed, non-split, and bad areas with zeros.
You already made the mft domain file. That is what you would use with ddrescue to focus on the MFT. Just don't use the -n option as you want the MFT to be as complete as possible. You should definitely try to get the MFT the best you can. It contains the location of all the files! If the first 1/3 of the MFT is really damaged, then you may have to run something like photorec to scan the disk for files.
That's really helpful.
To install and run ddrescue with the LiveCD I've been using this approach:
From the LiveCd desktop:
Ctrl +Alt + T to access the Terminal
sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) universe"
sudo apt-get update
sudo apt-get install gddrescue
Then to identify the location of the source and destination drives
sudo lsblk -o name,Label,size,fstype,model
then run ddrescue.
If I swap the damaged hard drive into the computer and run LiveCD you are recommending using the -d switch. Is it also worth using ddrescue 1.19 and your passthrough patch? Is running it straightforward? Would this also help recovering the mtf do you think? If you recommend imaging the whole disk I am definitely on for speeding the process up if possible. In 21 days ddrescue has imaged 232GB of 500GB!