Re: [sleuthkit-users] Good vs. Bad Hashes

[Matthias H. <mat...@mh...>]

Brian,

I think we have technical and logical issues here. Logical we need
to maintain a potential huge amount of data and categorize every single
hash entry. Furthermore, we have to decide for each entry if it is a
known-bad or a known-good. I think a useful solution is to maintain a
global database with both freely available hashsums like NSRL,KnownGoods
combined
with selfmade hash set (md5sum/graverobber ...). The interface to
autopsy and sleuthkit should allow to query only certain categories, only
known bads, a certain category as known bad or not(-> e. g. remote
management tools). The biggest problem here is to manage the category
mapping table for all the different tools.

The technical problem is to manage such a huge amount of raw data. With
NSRL alone, we have millions of hash sets. This requires a new query
mechanism. With a RDBMS, we need persistent connections and the
possibility
to bulk query large data sets very fast. With the current sorter|hfind
design, sorter calls hfind one time per hash analyzed. This is definitely
a big bottleneck.

Best regards,

Matthias

--=20
Matthias Hofherr
mail: mat...@mh...
web: http://www.forinsect.de
gpg: http://www.forinsect.de/pubkey.asc

Brian Carrier said:
> Is anyone interested in looking into the best way to manage hashes? The
> definition of "good" versus "bad" is relative to the current
> investigation and I don't know the best way to handle this in The
> Sleuth Kit and Autopsy.  There could be a single database with
> categories of hashes and you choose which are good and which are bad
> for that investigation (similar to the new Forensic Hash Database that
> was announced and NSRL).  Or, you could import tens of hash databases
> and identify them as bad or good (like hashkeeper).
>
> I think hashkeepr is LE-only, so I would rather focus on using NSRL and
> custom hashes made by md5sum.  If anyone is interested in working on a
> workable solution to this, let me know.