[sleuthkit-developers] Re: IO Subsystem patch for fstools

[Brian C. <ca...@sl...>]

Wow!  This looks great!

I have been meaning to incorporate the offset option for quite a while, 
but this is much more involved.  I don't have time to look at the code 
in detail right now, but I have some comments from your email and a 
quick skim of the code.

My original plan was to use the '-o' flag to specify the sector offset 
for the file system.  I figured sectors would be easier than bytes 
because mmls and fdisk give you the values in sectors and almost every 
disk uses a 512-byte sector.  This also allows people to use the offset 
value without the '-i' setting.

I like the idea of the '-i' because it is like specifying the image 
type, whereas -f is specifying the file system type. I hadn't thought 
about getting this advanced, but it looks good.

I would actually say that '-i' should only have the type and no other 
options.  If multiple files are needed (splitting and RAID), then they 
should be appended to the end of the command.  For example, to look at 
the file system at offset sector 12345, the following could be used 
(names are made up):

Normal full image:
fls -f linux-ext3 -o 12345 file1.dd
or
fls -f linux-ext3 -i single -o 12345 file1.dd

Split Image:
fls -f linux-ext3 -i split -o 12345 file1.dd file2.dd

LVM RAID Image:
fls -f linux-ext3 -i lvm -o 12345 lvm-config.dat

MS LDM Spanning  Image
fls -f ntfs -i ldm-span -o 12345 ldm-config.dat

It would also be useful if the config file format that you are 
developing for the RAID images could be used for the split images.

To keep the subsystem design similar to what currently exists, have you 
thought about the following:

A new data structure IO_INFO and before fs_open is run, the io_open() 
function is run with either the image lists or the config file etc and 
the offset.  There would probably have to be one for io_open_files(char 
**) and io_open_config(char *).

The IO_INFO structure is filed in with io_open and the needed read 
functions are mapped (like file_walk etc are now in FS_INFO).

The fs_open() function gets the IO_INFO structure passed to it and the 
fs_open() no longer needs to do the open() system call on the images.  
It just checks the magic value and fills in FS_INFO.   Any 
read_random() function in the file system code turns into 
fs_info->io->read_random(...).

This looks great!

brian

On Feb 3, 2004, at 8:47 AM, Michael Cohen wrote:

> Dear List,
>     Please accept this patch to the sleuthkit to implement a pluggable 
> IO
> subsystem for the fstools. (patch against 1.67, fstools directory).
>
>     Background
>     Quite often users are supplied with dd images that do not 
> immediately work
> with sleuthkit. Two notable examples are:
> 	- when a dd image was taken of the hdd - in this case users have to 
> use
> sfdisk to work out the partition offsets and then use dd with 
> appropriate
> skip parameters to extract each partition, before being able to use the
> sleuthkit. This is because the sk expects to have a dd image of a 
> partition
> (i.e. filesystem starts at offset 0 in the image file. This is not 
> always the
> case).
> 	- Sometimes images are split into smaller sizes for example in order 
> to burn
> to cd/dvd etc. This means that images need to be stuck together before
> analysis potentially wasting time and space.
>
>   It would be nice if one could use the images directly - without 
> needing to
> do creative dd manipulations.
>
>    Solution
>    This patch implements a modular io subsystem approach - all 
> filesystem
> operations within the sk are made to use this subsystem, and the user 
> can
> choose the subsystem they want. The subsystem is responsible to 
> seeking into
> the file and extracting data out of the dd image - how that is 
> implemented is
> completely abstracted from the point of view of the fstools.
>
>   The user can choose the subsystem to be used by the -i (io subsystem)
> command line switch. Then a list of arguments can be passed to the 
> subsystem
> to initialise it correctly. Once that is done, the regular sk calls 
> can be
> made (e.g. fs_open etc). The io subsystem will take care of the 
> specifics of
> implementation.
>
>   This patch includes 2 subsystem modules: simple and advanced. The 
> simple
> module is exactly the same as the old sk, while the advanced module 
> allows
> for specifying offsets into the dd file, as well as multiple dd files 
> in
> sequence.
>
>    Example:
> 	As an example the fls and icat tools were modified to support the new 
> sub
> system, more tools will be converted tomorrow once i get some sleep. 
> Example
> of how to seek into a partition within a disk dd:
>
> 	fls -i advanced -o offset=524288 -f linux-ext2 test.dd
>
> 	This selects the advanced io subsystem and passes it the offset option
> specifying 1024 blocks of 512 bytes.
>
> 	Now we can split the dd image across multiple files (maybe using the 
> split
> utility), and still analyse them at once:
>
> 	fls -i advanced -o offset=524288,file=xaa,file=xab,file=xac,file=xad 
> -f
> linux-ext2 xae
>
> 	Note that xae (the last part of the image will be appened to the list 
> of
> parts automatically). Also note that all the options in -o are passed 
> as one
> parameter to the subsystem which then parses them into the relevant
> arguements.
>
>     If the subsystems name is not found, the subsystem will list all 
> known
> subsystems:
>
> 	bash# fls -i help -f linux-ext2 test.dd
>
> Available Subsystems:
>
>         standard - Standard Sleuthkit IO Subsystem
>         advanced - Advanced Sleuthkit IO Subsystem
> fls: Could not set io subsystem help
>
>    To get more help about the options available, try setting an option 
> which
> is not supported:
>
> 	bash# fls -i  advanced -o help -f linux-ext2 test.dd
>
> option help not recognised
>
> Advanced io subsystem options
>
>         offset=bytes            Number of bytes to seek to in the 
> image file.
> Useful if there is some extra data at the start of the dd image (e.g.
> partition table/other partitions
>         file=filename           Filename to use for split files. If 
> your dd
> image is split across many files, specify this parameter in the order
> required as many times as needed for seemless integration
>
> 	Future work:
> 	I am in the process of implementing a raid reassembly functionality. 
> I.e.
> given a raid reconstruction map (a file telling sk the order in which 
> raid
> blocks go together) and a list of dd images of individual drives, the 
> io
> subsystem will transparently reassemble the logical data. I have a 
> working
> prototype so i know its possible. The abstracted io subsystem concept 
> will be
> very handy for that.
> <fstools_diff>