ddrutility Wiki
LINUX based utility for use with gnuddrescue to aid with data recovery
Brought to you by:
maximus57
Menu
▾
▴
Home
There is a newer version of this page. You can find it here.
Next: Introduction, Up: (dir)
ddrutility
This manual is for ddrutility (version 1.99-beta2, 16 November 2013).
Next: ddru_findbad, Previous: Top, Up:
Top
1 Introduction
Ddrutility is a linux based set of utilities designed to assist with
data recovery, specifically as a supplement to GNU ddrescue. It contains
the following programs:
ddru_findbad
ddru_ntfsbitmap
ddru_findbad is actually the original ddrutility version 1.6. None of
its functionality has changed, only the name and the documentation. It
is a bash script that will try to find which files are related to bad
sectors in a ddrescue log file. It relies on 3rd party utilities for its
functions. It can be somewhat slow. While I will fix any bugs found in
it, I do not plan on updating or improving its functionality. It is
possible (hopeful) that I may create new utilities to replace it (or
parts of it) in the future. These utilities would not rely on 3rd party
functions, and would be faster. But for now, it is what it is.
ddru_ntfsbitmap is a utility to extract the bitmap file from a NTFS
partition, and then process it and create a domain file to be used with
ddrescue. This would allow for only recovering the used portion of the
partition, and not spending time reading unused and unneeded data.
Next: ddru_ntfsbitmap, Previous:
Introduction, Up: Top
2 ddru_findbad
ddru_findbad is meant to be a compliment to gnuddrescue. It finds which
files are related to the bad sectors. It will also work with a home made
file of a sector list (see LOGFILE below). Currently tested with
ddrescue 1.15.
If you performed a rescue that had some leftover bad sectors, but the
file system seems ok and you are wondering what might still be damaged
by the unrecovered sectors, then this program is for you. The filesystem
of the target must be intact enough for ddru_findbad to work. If you
are unable to mount the target and view its directory, then ddrutility
probably won't work and will just give errors. In this case you may have
to run some sort of repair tools on the target first and hope that the
filesystem can be repaired to a working state. Note that any repair tool
ran on the target could possibly cause ddru_findbad to have inaccurate
results. It is recommended that you always make an extra copy of the
recovered copy and work with that copy.
ddru_findbad is not meant to be run on the drive that actually has the
bad sectors. It is meant to be ran against a recovered copy of the
drive. FAT and HFS+ will fail if ran against the actual drive, as ifind
seems to feel the need to actually read the sectors in question. NTFS
and EXT seem to be processed without trying to read the sectors in
question, however understand that they still will be trying to read the
file table (which could have bad sectors) to get results.
ddru_findbad must be run as root (sudo).
ddru_findbad works on NTFS, EXT2/3/4, FAT16/32, and HFS+ file systems
(note that EXT4 will report as EXT3).
ddru_findbad requires ntfs-3g to be installed for NTFS support, which
seems to be included in current Ubuntu versions. (sudo apt-get install
ntfs-3g)
ddru_findbad requires sleuthkit to be installed for normal operation.
Sleuthkit is not included as standard in current Ubuntu versions. (sudo
apt-get install sleuthkit)
ddru_findbad requires GNU fdisk to be installed for normal operation.
GNU fdisk is not included as standard in current Ubuntu versions. (sudo
apt-get install gnu-fdisk)
Ubuntu Rescue Remix contains the above requirements, so it should run on
the live cd.
ddru_findbad will work on a whole drive recovery, or a single partition
recovery. It will work with a target device or image file.
Results are left in 4 main output files. Results_summary is a condensed
sorted summary file, and is probably the most useful output.
Results_list is the main output file. It contains information about
every sector processed with one line per sector. It is designed to be
easily processed to obtain desired final results (such as the summary
file). Results_info contains information about the partition table and
file system(s) and can be expanded with the moreinfo option.
Results_debug is mostly for troubleshooting purposes and most error
messages are sent to it. If you request help for any problems with
ddru_findbad I may ask for these 4 files.
ddru_findbad makes several temporary files in the directory that it is
run from. It deletes them when finished. Do not mount the target and run
ddru_findbad from within the target, as that would cause writing to the
target and you probably don't want that!
ddru_findbad does not knowingly write or alter any data on the target
drive and does not require the file system of the target to be mounted.
However, it makes calls to different 3rd party tools which SHOULDN'T do
any writing for what they are, but this has not been verified.
Some errors or warnings may appear on screen when running. Don't panic,
some errors can be normal, as long as there is a good output. I have
tried to eliminate all the errors that I have found, but I cannot test
all possibilities. Most error messages are sent to results_debug. If
you question the error, please feel to email or post on the homepage so
that I can determine if it is something I should work on.
The format for running ddru_findbad is:
ddru_findbad target logfile [options]
Where:
target
The drive or partition if using a device (/dev/hda), or the name (and
full path if not in the current directory) of the image file
(/media/mydrive/image.dd). The target should be a copy or image of the
original drive with bad sectors. It should not be the actual drive with
bad sectors! If you should so choose to run it on the actual drive for
some reason, note that FAT and HFS+ will not process with actual bad
sectors. \
logfile
The logfile from ddrescue. It can alternatively be a file containing a
list of sectors, one sector number per line with no other text in the
file. The sector file can also contain sector ranges, with two sector
numbers per line separated by a space, being the first and last sector
of the range. Note that the sectors are in relation to either the whole
disk or a single partition, which depends on what the target is.
ddru_findbad supports the following options:
‘-h’
‘--help’
Print an informative help message describing the options and exit. \
‘-v’
‘--version’
Print the version number and exit. \
‘-o file’
‘--output file’
The base name that will be at the beginning of the multiple results
files. The default is "results". The different output files will have
extensions added to them, for example results_summary, results_list,
etc... \
‘-w seconds’
‘--loopwait seconds’
The number of seconds to wait before doing loop commands. You shouldn't
have to change this. It is only used with image files. The default is 2.
Increase this if you get loop errors such as device busy. \
‘-s bytes’
‘--sectorsize bytes’
The sector size in bytes of the target drive or image. The default is
512. Note that only a sector size of 512 has been tested, as I don't
have any drives of a different sector size to test. \
‘-m’
‘--moreinfo’
Will add what could be considered an insane amount of (mostly) useless
data from fsstat and fls to the results_info file. Note that at this
time fls does not work right on EXT4 (sleuthkit currently only supports
ext2/3). If the file system is large, moreinfo could cause the
results_info file to get very big, and could add a considerable (or
insane) amount of processing time. Only for advanced users, use this if
you understand what output fsstat and fls give. \
‘-e’
‘--extraoutput’
Will produce more types of output files. The processing for this is not
very efficient at this time (it is actually horrible). If there are a
lot of bad sectors, this could take a considerable (or insane) amount of
processing time. If the quick or quickntfs option is enabled then
extraout will process much faster. Currently available with the EXTRAOUT
option: Results_bysector lists the bad sector groups with the files in
each group. \
‘-q’
‘--quick’
Only processes the first and last sector of each group. This allows for
a much faster first run so you can get an idea of what files are
involved. May provide more useful output with the EXTRAOUT option to get
the results_bysector output. If the first and last sector of a group
show the same file then it could be assumed that all the sectors in
between are the same (assume at your own risk). This is most efficient
if there are large sized groups of bad sectors. If most of the bad
sectors are in very small groups or individual sectors then this is not
efficient and will take nearly as long as a normal run. \
‘-Q’
‘--quickntfs’
The same as quick, except it processes NTFS partitions extra special. It
uses the ability of ntfscluster to rapidly check groups of sectors at
once. This can very quickly produce a FULL list of files related to bad
sectors for NTFS partitions, instead of of just checking the first and
last sector of a group. As with quick, it is still most efficient with
large sector groups. Results for NTFS partitions will not end up in the
results_list file, only in results_summary (and in results_bysector
if the extra option is used).
EXAMPLES:
If your ddrescue command was:
ddrescue /dev/sda1 rescued_image rescued_logfile
Then the ddru_findbad command would be:
ddru_findbad rescued_image rescued_logfile
If your ddrescue command was:
ddrescue /dev/sda /dev/sdb rescued_logfile
Then the ddru_findbad command would be:
ddru_findbad /dev/sdb rescued_logfile
NOTE: It is important that the source for ddru_findbad is the same as
the ddrescue outfile. Do not use /dev/sdb in the ddrescue command and
then /dev/sdb1 in the ddru_findbad command. This would cause very
inaccurate results.
It does not matter if the source for ddru_findbad is a disk or image, a
partition or a whole disk, as ddru_findbad will automatically process
it as long as it matches the format used to describe the ddrescue
outfile.
Next: Important Notes, Previous:
ddru_findbad, Up: Top
3 ddru_ntfsbitmap
Ddru_ntfsbitmap is a utility that will extract the bitmap file from an
NTFS partition, and then create a rescue domain file to use with
ddrescue. This will allow recovering only the used portion of an NTFS
partition. Other normal cloning utilities (such as clonezilla, or at
least I think clonezilla uses something similar) use a similar method to
speed up the cloning process.
Ddru_ntfsbitmap requires that a version of ddrescue that includes
ddrescuelog is installed (ddrescue version 1.15 and up). It does not
make any other calls to 3rd party utilities.
The bitmap file is just a map of what clusters are used and free on the
partition. Consider the map as 0's and 1's, the 0's mean unused clusters
and the 1's mean used clusters. If any sectors of the bitmap file are
not readable by ddrescue (or were not tried), then they will be filled
with ones (FF) in the recovered bitmap output file. A section that is
all ones (FF) is considered as used clusters, and therefore the
corresponding data will be copied. Ddru_ntfsbitmap fills all bad (or
untried) bitmap sector(s) in the output file with ones to insure that no
needed data is missed during the recovery process, although it could
cause extra data to be copied (better safe than sorry).
Since the domain logfile that is created relies on the bitmap file from
the NFTS partition, it would be affected by any sort of corruption that
could have happened to the bitmap file caused when the filesystem was
running. So I can offer no guarantee that the results will be accurate.
All I can say is that the idea works, and the results SHOULD be good.
The safest way to make sure of the best possible data recovery would be
to continue on without the domain file after you got as far as you could
with it.
However, in addition to saving time, there COULD be another benefit to
not continuing on without the domain file in certain cases. If you zero
fill all the bad and untried areas of the target, then that could
possibly get rid of "garbage" data in the free space. This could
possibly make file recovery easier for programs that scan the whole disk
looking for files (such as photorec).
Ddru_ntfsbitmap will show the percentage of used vs free space of the
partition after processing the bitmap file, so you can get an idea of
how much time and effort you may be saving. You can also get this number
again later on by running ddrecuelog against the domain_logfile.
ddrescuelog -t domain_logfile
In the results, "rescued" will = used space that you are trying to
recover, and "non-tried" will = free space that you are ignoring. When
your rescue percentage reaches the percentage of used space, then you
should have recovered the needed data, except for any errors of course.
But you must have used the domain_logfile in all of the ddrescue
commands up to this point for this percentage match to be accurate.
After you reach this point and ddrescue exits, and you have done all the
rescuing of the used portion that you can or wish to (retries, no-split,
ect...), you could leave out the domain_logfile from the command and
continue to recover all of the remaining partition/drive if desired.
Ddru_ntfsbitmap uses ddrescue for all the reads and also creates
logfiles, so that you can run it as many times as you wish/need with
different ddrescue options. Because it uses ddrescue for the reads, it
will not try to read the same part of the disk twice, unless any of the
output files are deleted or missing. If it completes without ddrescue
reporting any errors, then running it again it will just read the output
files and not try to read the disk. Please note that this also means
that if you wish to start a new recovery or a recovery of a different
partition of the same drive, you need to make sure that the output files
from a previous project are not in the current directory.
While ddru_ntfsbitmap will only try to read any area once while reading
the bitmap file, the downside is that when you start the actual rescue
attempt, those areas will be read again. But this should be a fairly
small portion of the disk. For instance, at a standard 4k cluster size,
a 500GB drive bitmap file would be approximately 16MB (0.003%).
Ddru_ntfsbitmap will create several output files in the directory where
it is ran. They are required if you need to run ddru_ntfsbitmap again
on the same drive with different ddrescue options to get the best bitmap
file recovery. They can be deleted if you aquired a successful
domain_logfile (but don't delete the domain_logfile until you have
your successful data recovery). They need to be deleted or switched to a
different directory if you are attempting a new rescue of a different
partition or a different disk. The output files are as follows:
__bootsec
This is the recovered partition boot sector (512 bytes). It is needed to
find the location of the MFT, along with the sector and cluster size. \
__bootsec.log
This is the ddrescue logfile for the __bootsec file. \
__mftshort
This is the first 16 entries of the MFT (16384 bytes). It is needed to
find where the bitmap file is located on the disk. \
__mftshort.log
This is the ddrescue logfile for the __mftshort file. \
__bitmapfile
This is the recoved NTFS bitmap file. \
_partX__bitmapfile.log
This is the ddrescue logfile for the NTFS bitmap file. There could be
multiple logfiles, each with its own part# (part0, part1...). Normally
there should only be one logfile, but if for some reason the NTFS bitmap
file is fragmented on the disk, there will be a separate logfile for
every fragment. \
ntfsbitmap_rescue_report.log
This file contains all the results from ddrescuelog –show-status for all
the ddrescue reads used by ddru_ntfsbitmap, and also for the domain
logfile(s).
The format for running ddru_ntfsbitmap is:
ddru_ntfsbitmap source_disk logfile [options]
Where:
source_disk
The drive or partition that you are trying to recover. This needs to be
the EXACT same source as the ddrescue input file that you will be using
for the recovery. \
logfile
The name of the domain logfile that you wish to create to use with
ddrescue.
ddru_ntfsbitmap supports the following options:
‘-h’
‘--help’
Print an informative help message describing the options and exit. \
‘-v’
‘--version’
Print the version number and exit. \
‘-D’
‘--debug’
Turn on debugging and create a debug file. The name of this file is
ntfsbitmap_debug.log. It is mostly for my use in troubleshooting bugs.
I may ask for this file if someone reports a problem that I am unable to
identify from their description. \
‘-V’
‘--verbose’
Show additional information. Right now all this shows is the commands
sent to ddrescue. This could help if you are trying to pass options to
ddrescue and it is not working correctly. \
‘-i bytes’
‘--inputoffset bytes’
Set input offset (partition offset). This is needed if doing a whole
drive recovery, to specify the offset of the NTFS partition. You need to
specify this offset if you will be using an input such as "/dev/sda".
You can get this offset by doing a little math on an "fdisk -lu" command
(the "u" gives the results in sectors, which is needed for this to
work). Let's look at the following results from "fdisk -lu /dev/sda",
where /dev/sda is the source disk you wish to recover.
Disk /dev/sda: 5 GB, 5362882560 bytes
255 heads, 63 sectors/track, 652 cylinders, total 10474380 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 2048 10485759 5245191 7 HPFS/NTFS
Warning: Partition 1 does not end on cylinder boundary.
This tells us that the NTFS partition /dev/sda1 starts at sector 2048.
The units tells the sector size, which is 512 bytes. So multiply the
unit size (512) by the start (2048), which gives us 1048576. This number
is the offset in bytes, which is what you supply to this option. So your
ddru_ntfsbitmap command would look something like this:
ddru_ntfsbitmap /dev/sda -i 1048576 domain_logfile
If you do not supply the correct offset, you will get an error stating
the boot sector ID is not NTFS, and the program will exit. \
‘-o "options"’
‘--options "options"’
Ddrescue options to pass directly to ddrescue. Use quotes around the
whole string of options. For example:
ddru_ntfsbitmap /dev/sda1 domainfile --options "--direct --no-split"
These options will be passed to all calls to ddrescue. Note that you
cannot use the options --input-position, --output-position, or
--size, as these are used by ddru_ntfsbitmap for control. If you try
to pass one of these options, you will likely get an error message from
ddrescue. \
‘-m file’
‘--mftdomain file’
Creates a separate additional domain file for the MFT (master file
table). This mft_domain_file can be used the same way as the examples
below, just insert mft_domain_file in place of the regular
domain_logfile. The purpose of the mft_domain_file is to allow
recovering of the most important part of the filesystem. You can use
this before using the regular domain_logfile to attempt to recover the
MFT first, and then use the regular domain_logfile to then recover the
rest of the data. It could also be useful to see if the filesystem has
errors, and how many, which could help with your decision on how to
proceed.
EXAMPLES:
PARTITION TO IMAGE-
The simplest example is if recovering only the partition to an image
file.
Let's start with the following fdisk -lu result:
Disk /dev/sda: 5 GB, 5362882560 bytes
255 heads, 63 sectors/track, 652 cylinders, total 10474380 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 2048 10485759 5245191 7 HPFS/NTFS
Warning: Partition 1 does not end on cylinder boundary.
The ddru_ntfsbitmap command would be something like:
ddru_ntfsbitmap /dev/sda1 domain_logfile
Your ddrescue command would be something like:
ddrescue -m domain_logfile /dev/sda1 recovered_ntfs_image rescue_logfile
WHOLE DISK TO DISK OR IMAGE SIMPLE-
Let's start with the following fdisk -lu result:
Disk /dev/sda: 5 GB, 5362882560 bytes
255 heads, 63 sectors/track, 652 cylinders, total 10474380 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 2048 10485759 5245191 7 HPFS/NTFS
Warning: Partition 1 does not end on cylinder boundary.
You will need to use the -i (–inputoffset) option, and you will need to
do some math (see the inputoffset option description for more detail).
Multiply the sector size (512) by the partition Start (2048) to get
1048576, which is the input offset for the NTFS partition.
Your ddru_ntfsbitmap command would be something like:
ddru_ntfsbitmap /dev/sda -i 1048576 domain_logfile
Your ddrescue command would be something like this respectively:
ddrescue -f -m domain_logfile /dev/sda /dev/sdb rescue_logfile
Or:
ddrescue -m domain_logfile /dev/sda recovered_disk_image rescue_logfile
If there is more than one NTFS partition on the drive, you would just
repeat the above process for each of them, with the proper input offset
for each. You would use the same rescue_logfile in the ddrescue
command, but you would use a separate domain_logfile for each separate
NTFS partition accordingly.
Important Note: When using the -i (–inputoffset) option, the first
"track" (32256 bytes or 63 * 512 byte sectors) of the disk is
automatically included in the created domain file. This first "track"
contains the MBR, partition table, and possibly more needed boot code.
The reason this is included automatically is so you shouldn't have to
worry about making a rescue copy that won't boot due to missing
important stuff.
WHOLE DISK TO DISK OR IMAGE WITH EXTRA PARTITIONS-
If you are planning on recovering the entire disk structure including
other partitions, it starts to get a bit more tricky. But you may need
to do this with some Windows installations, as there may also be a
smaller active boot partition that contains the boot information for
Windows, and this partition would need to be copied for the OS to boot.
Let's start with the following fdisk -lu result:
Disk /dev/sda: 120 GB, 120031511040 bytes
255 heads, 63 sectors/track, 14593 cylinders, total 234436545 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sda1 63 20466809 10233373 b FAT32
/dev/sda2 * 20466810 143331929 61424527 7 HPFS/NTFS
/dev/sda3 143331930 225247364 40949685 83 Linux
/dev/sda4 225247365 234436544 4586557 f Extended LBA
/dev/sda5 225247428 234436544 4586557 82 Linux swap
Warning: Partition 5 does not end on cylinder boundary.
The first step is to get the NTFS partition (sda2) offset, multiplying
the Start by the sector size (20466810 * 512 = 10479006720). So then
your ddru_ntfsbitmap command would be something like:
ddru_ntfsbitmap /dev/sda -i 10479006720 domain_logfile
Your ddrescue command would be something like this:
ddrescue -f -m domain_logfile /dev/sda /dev/sdb rescue_logfile
Now you want to also recover the FAT32 and Linux partitions. You need to
do some math again. For the FAT32 partition (sda1) multiply the start by
the sector size (63 * 512 = 32256). This is the input position you
provide ddrescue. Next subtract the start from the end, and then
multiply that by the sector size (20466809 - 63 = 20466746, * 512 =
10478973952). This is the size you provide ddrescue.
So now your next ddrescue command would be something like this:
ddrescue -f -i 32256 -s 10478973952 /dev/sda /dev/sdb rescue_logfile
And we do the same for the Linux partition. Input position = (143331930
* 512) = 73385948160, and size = ( (225247364 - 143331930) * 512) =
41940702208.
Ddrescue command:
ddrescue -f -i 73385948160 -s 41940702208 /dev/sda /dev/sdb rescue_logfile
I am not going to get into the Extended partition, except to say that I
am pretty sure that you must copy the first part of it (starting at sda4
in this case). I have probably covered what the average person will ever
do with this already.
Next: Reporting Bugs, Previous:
ddru_ntfsbitmap, Up: Top
4 Important Notes
Here are a few things to consider about data rescue and using this
software.
I would like to point something out that relates to both an "unfinished"
image, and using a utililty as ddru_ntfsbitmap. In any event, anything
less than a 100% recovery will leave portions on the target image/drive
that have not been written to by ddrescue. If copying to a brand new
hard drive, those areas are (hopefully) likely to be zeros. But in any
other case, those areas will contain whatever data was there previously.
For someone that uses their system for data recovery on a regular basis,
and is using image files or reusing hard drives, those areas could
contain data from a previous recovery! This could be a privacy issue in
some cases, but also could cause an issue with running any other sort of
repair/file recovery tools on the recovered image/drive. The unrecovered
parts could contain "garbage" data that could affect accurate file
recovery. In these cases I would recommend using the fill mode of
ddrescue to fill any unfinished/untried areas with zeros. Example
command:
ddrescue --fill-mode=?/*- /dev/zero recovered_image logfile
This would write zeros to any portion of the recovery that was not
successfully read from the source.
Next: Index, Previous: Important Notes,
Up: Top
5 Reporting Bugs
There are likely bugs in the programs of ddrutility. There are almost
definitely errors and omissions in this documentation, and perhaps also
in the individual programs' help and manual files. If you report them,
they will get fixed. If you don't report them, they will just stay the
way they are and will not get fixed.
Report bugs to (maximus57@hotmail.com)
Or better yet, report bugs at the Ddrutility homepage:
http://sourceforge.net/projects/ddrutility/
Please include the version number of ddrutility, and the version of the
individual program. You can find the version by running ddrutility or
the program with the --version option.
Previous: Reporting Bugs, Up: Top
