Re: [sleuthkit-users] hashing a file system

[Simson G. <si...@ac...>]

Yes, fiwalk hashes in SleuthKit order. If you want to hash in block order you need to generate the DFXML for the entire drive, sort by the index of the first 

On Sep 4, 2014, at 7:53 PM, Stuart Maclean <st...@ap...> wrote:

> On 09/04/2014 03:46 PM, Simson Garfinkel wrote:
>> Hi Stuart.
>> 
>> You are correct — I put this in numerous presentations but never published it.
>> 
>> The MD5 algorithm won't let you combine a partial hash from the middle of the file with one from the beginning. You need to start at the beginning and hash through the end. (That's one of the many problems with MD5 for forensics, BTW.) So I believe that the only approach is sorting the files by the sector number of the first run, and just leaving it at that.
>> 
>> I saw speedup with both HDs and SSDs, strangely enough, but not as much with SSDs. There may be a prefetch thing going on here.
>> 
>> I think that the Autopsy framework should hash this way, but currently it doesn't. On the other hand, it may be more useful to hash based on the "importance" of the files.
>> 
>> Simson
>> 
>> 
> Hi Simson, currently I have just got as far as noting the 'seek distances' between consecutive runs, across ALL files.  I have yet to actually read the file content.  But I don't think it's that hard.  As you point out, md5 summing must be done with the file content in correct order.  I see an analogy between the 'runs ordered by block address but not necessarily file offset' and the problem the IP layer has in tcp/ip as it tries to reassemble the fragments of a datagram that may arrive in any order. We may have to have some 'pending data' structure for runs whose content has been read but which cannot yet be offered to the md5 hasher due to an as yet unread run being needed first.
> 
> I'll let you know if/when I nail this.  Pehaps Autopsy could benefit?  Is fiwalk doing it the 'regular way' too, i.e reading all file content of each file as the walk proceeds?
> 
> Stuart